AWS Messaging, Streaming, and Workflow Decision Guide

Reading notesDownload original Markdown ↗

With the guarantee boundaries established, the next question is which AWS service owns each part of the flow. Use the comparison matrix to orient yourself, then work through the service families and the concrete fintech decisions. The aim is to justify a composition—such as an event router feeding separate queues—not to memorize a winner. Compute choice comes after message ownership: Lambda, Fargate, and EC2 run consumers whose correctness duties remain explicit.

Evidence notation: C identifies a claim in the claim register, A a dated AWS source, F a foundational source, and CS a finding in the repository case study. The source index supplies the full source details. These labels are lookup aids, not facts to memorize.

This chapter is a requirement-first selection guide, not a product catalog. Start with who owns work, what must be ordered, what must be replayed, how failure is acknowledged, and which business effect must be correct. Then choose one or more services. A router can feed a queue; a retained log can feed a delivery service; an orchestrator can wait on work performed by a container. Composition is normal.

The vocabulary and three guarantee planes come from Event-Driven Foundations. Throughout this chapter, delivery means a named transport handoff, processing means a handler attempt and commit, and business effect means an externally meaningful outcome. No product moves a guarantee between those planes automatically.

Requirement-first decision sequence

Ask these questions in order:

1. Meaning and ownership: Is this one-recipient work, a fact for independent subscribers, a replayable record, table change data, or an explicit business process? Commands and events are not interchangeable.
2. Correctness: Name the authoritative mutation, stable identity, ordering unit, duplicate policy, and any cross-key invariant. Decide where idempotency, version checks, compensation, and reconciliation live.
3. Recovery: State acknowledgement/checkpoint semantics, retry owner, terminal holding path, retention, replay scope/rate, and how recovery is proved.
4. Workload: Give average/peak records per second, bytes, burst duration, fan-out, active keys, skew, latency and lag budgets, and backlog drain target.
5. Isolation: Decide whether consumers, tenants, priorities, and poison items need separate queues, positions, quotas, or compute pools.
6. Integration and operations: Name protocols, targets, runtime duration, state/history needs, deployment skills, observability, and on-call ownership.
7. Unit economics: Compare cost per accepted order, fill, projection update, million updates, or retained audit year. Include transport, compute, storage, logs, transfer, encryption, replay, and engineering effort (C49; A50, retrieved 2026-08-22).

Do not begin with “serverless or Kafka?” A defensible answer sounds like: “The workload needs per-account serialization, competing consumers, a visible backlog, and no historical replay, so I choose a FIFO work queue keyed by accountId; I reject a retained log because independent replay is not required, and I keep the ledger write idempotent because delivery can repeat.”

The shared decision-axis matrix

Read one row as a complete service contract, then compare the columns that your requirement makes decisive. For example, independent historical replay makes consumer position and retention important; one worker owning each item makes acknowledgement and backlog control important. A favorable throughput description cannot compensate for the wrong ownership or recovery model.

“Low/medium/high” below is a relative operating-shape inference, not an AWS service guarantee. Capacity and quotas must be checked for the selected Region, mode, payload, and integration before calculation (C33; A08,A11,A16, retrieved 2026-08-22).

The full axes are distribution model, routing, fan-out, ordering unit, replay/retention, acknowledgement, throughput, latency, consumer isolation, failure recovery, payload, integration reach, operational effort, and cost shape.

Option	Distribution model; routing/fan-out	Ordering unit	Replay/retention; acknowledgement	Throughput/latency and consumer isolation	Payload/integration reach	Operational effort; cost shape
EventBridge event bus	Rule-based router; zero-to-many targets, content matching	No ordering contract; archive replay is not original arrival order	Live routing; optional selected archive/replay to source bus; target delivery success/failure per target	Good for loose integration and cross-account/AWS target reach; isolate slow consumers with a queue target	Structured events and broad AWS/API target integration	Low broker effort; billable events plus archive storage/replay and target costs (C62; A47,A84, retrieved 2026-08-22)
SQS Standard	Competing-consumer work queue; route before enqueueing	Best effort, not ordered	Visibility lease, delete acknowledgement, retention and DLQ/redrive; no independent consumer positions	Elastic buffering and consumer isolation per queue; duplicates and reorder possible	Queue payload and SQS/Lambda/poller integrations	Low broker effort; request/payload-chunk, transfer, encryption, compute, and DLQ/replay costs (C02,C63; A02,A09,A48,A85, retrieved 2026-08-22)
SQS FIFO	Competing-consumer work queue with message groups	Strict sequence within a message group; parallelism across active groups	Same visibility/delete model; bounded send deduplication, retention and DLQ	Inference: hot/blocked groups limit concurrency; good when one key owns serial work	Queue consumers and Lambda event-source mappings	Inference: low broker effort; request shape plus cost of reduced batching/concurrency when groups skew (C03,C19,C63; A02,A10,A11,A48,A85, retrieved 2026-08-22)
SNS Standard	Push publish-subscribe topic; subscription filters and multiple endpoint protocols	No ordering contract; messages can arrive out of order	Protocol-specific delivery retry; possible duplicate delivery; subscription DLQ; no general retained replay	Independent delivery per subscription, but endpoint failure policy varies	Broad push endpoints including SQS/Lambda/HTTP and user-notification channels	Low broker effort; publications, deliveries, payload/data transfer and endpoint costs (C60,C64; A13,A48,A83,A86,A91, retrieved 2026-08-22)
SNS FIFO	FIFO topic fan-out to SQS Standard or FIFO subscribers	Strict order/deduplication requires the SNS FIFO → SQS FIFO path; SQS Standard subscribers have best-effort order and at-least-once delivery	Built-in FIFO archive/replay; the documented exactly-once label requires all listed conditions including no filtering; filtering changes the documented delivery semantic to at-most-once	Inference: use only when compatible subscribers need this path; replay and consumer effects remain idempotent (C20,C69; A12,A92-A94, retrieved 2026-08-22)	No direct email/SMS/mobile/HTTP endpoints; Lambda consumes through subscribed SQS	Inference: low broker effort, but price every subscriber queue/consumer and recovery path (C49,C64; A12,A48,A92-A94, retrieved 2026-08-22)
Kinesis Data Streams	Partitioned retained log; independent consumer applications	Sequence within a shard; partition key selects shard	Consumers checkpoint their own progress; replay within configured retention	Inference: high-rate ingestion; isolation and downstream limits depend on standard versus enhanced fan-out and consumer design (C17; A16,A18, retrieved 2026-08-22)	Producers, Lambda/KCL/Flink and stream ecosystem	Inference: medium effort; on-demand/provisioned ingestion/read, retention, enhanced fan-out, consumers, and recovery capacity (C10,C14,C42,C49; A03,A14-A18,A46, retrieved 2026-08-22)
DynamoDB Streams	Table-specific CDC derived from item changes	Record order for modifications to one item, not table-wide global order	Records retained for 24 hours; consumer checkpoint/retry is separate	Fits projections/outbox relay tied to one table; not general ingestion	Lambda/event-source integrations around DynamoDB change records	Low-to-medium; table writes, optional change images, consumer compute, retries and projection writes (C23; A01,A21,A45, retrieved 2026-08-22)
Step Functions Standard	Inference: explicit per-execution orchestration/process manager	State-machine transition order, not event global order	Durable execution state/history; workflow retry/catch/wait/callback	Inference: long-lived, auditable, lower-volume workflows with per-execution isolation	Broad service integrations, jobs and callback tokens	Inference: low platform effort, medium model/governance effort; priced by state transitions (A25,A26,A49,A87, retrieved 2026-08-22)
Step Functions Express	Inference: high-volume short workflow execution, synchronous or asynchronous	State-machine order within an execution	CloudWatch Logs rather than Standard execution history; no job-run/callback integration	Inference: short, high-volume idempotent processing paths	Broad request/response integrations	Inference: low platform effort; executions, duration and memory shape (A25,A49, retrieved 2026-08-22)
Amazon Data Firehose	Managed buffered delivery to supported destinations	Inference: no general ordering contract on which to build business state (C31; A29, retrieved 2026-08-22)	Inference: destination delivery/retry and optional S3 backup are not consumer-position replay (C31,C66; A29,A88, retrieved 2026-08-22)	Inference: efficient sink delivery and batching, not independent multi-consumer isolation (C66; A29,A88, retrieved 2026-08-22)	S3, analytics/search destinations and supported transforms	Inference: low pipeline effort; bytes ingested plus optional conversion, dynamic partitioning, delivery, transformation and destination costs (C28,C49; A29,A76,A88, retrieved 2026-08-22)
Amazon MSK	Managed Apache Kafka brokers/serverless with topics, partitions and consumer groups	Partition order	Inference: retained log and consumer offsets do not extend Kafka transactions beyond supported Kafka operations (C29; A30, retrieved 2026-08-22; F28)	Inference: strong fit for Kafka clients/connectors/ecosystem and many independent consumers (A30,A90, retrieved 2026-08-22)	Kafka protocol and ecosystem	Inference: medium-to-high effort across topology/version/partitions, upgrades, quotas, storage, security and rebalances; broker/serverless capacity, storage and transfer cost (C49; A30,A77,A90, retrieved 2026-08-22)
Amazon Managed Service for Apache Flink	Stateful stream-processing application over one or more streams	Keyed/event-time application semantics, not source-wide global order	Inference: checkpoints/snapshots plus replayable input bound recovery, and sink compatibility bounds the effect guarantee (C67; A31,A89, retrieved 2026-08-22; F31)	Inference: stateful joins, windows, late data and continuous analytics fit; scale and backpressure are application concerns (C67; A31,A89, retrieved 2026-08-22)	Kinesis/MSK and supported connectors/sinks	Inference: medium-to-high application effort; processing capacity, state/checkpoint/storage, logs and source/sink costs (C49; A31,A80,A89, retrieved 2026-08-22)
Amazon MQ	Managed ActiveMQ Classic/RabbitMQ broker compatibility	Engine/topology/protocol-specific	Broker-specific acknowledgement, durability and DLQ semantics	Inference: migration/interoperability fit; broker capacity/topology is an explicit concern (C55; A61, retrieved 2026-08-22)	JMS, AMQP, STOMP, MQTT or RabbitMQ-compatible workloads as supported	Inference: medium effort across engine/version/topology/broker operations; broker instance, storage and transfer cost (C49; A61,A69, retrieved 2026-08-22)

The matrix deliberately has no “winner.” Selection happens at a workload boundary, and a financial system usually uses several rows.

Guarantee-language traps to reject:

• Inference: SQS FIFO plus Lambda can repeat processing, so the authoritative effect remains idempotent; FIFO is not end-to-end exactly once (C03,C04,C63; A02,A10,A85, retrieved 2026-08-22).
• EventBridge target delivery is separate from Lambda runtime, handler, and business failure; configure and observe each boundary independently (C05-C07,C57; A01,A04,A05,A81, retrieved 2026-08-22).
• DynamoDB Streams provides per-item record order, while consumer retry/commit remains separate and idempotent (C23; A01,A21, retrieved 2026-08-22).
• Inference: Step Functions workflow execution semantics do not make an external side effect or task commit exactly once (C25,C65; A25,A26, retrieved 2026-08-22).
• Inference: Firehose is not a general bus, work queue, or replayable multi-consumer stream (C31,C66; A29,A88, retrieved 2026-08-22).
• Global ordering requires a common serialization scope and bottleneck; no listed partitioned service supplies free global order (C34).

EventBridge, SQS, and SNS

EventBridge: route facts across bounded contexts

Choose EventBridge when producers should publish a structured fact without knowing target addresses, rules need content-based matching, or AWS SaaS/ cross-account integrations matter. Rules evaluate each event and can send one match to multiple targets (C60,C62; A07,A84, retrieved 2026-08-22). Add one SQS queue per stateful/slow consumer when buffering, concurrency control, or failure isolation matters; direct Lambda targets do not create a visible work backlog.

Guarantee boundary: EventBridge target delivery retry applies when the service cannot deliver to a target. A target DLQ holds exhausted target-delivery failures only, and needs the rule-scoped SQS resource policy; a failed DLQ write has its own metric (C05,C57; A05,A81, retrieved 2026-08-22). Target delivery is not Lambda runtime retry or business processing. After Lambda accepts an invocation, runtime, handler, state commit, and reconciliation are separate boundaries. A successful PutEvents response is also only producer-API acknowledgement, not routing or business completion (C08; A06,A82, retrieved 2026-08-22).

An archive can retain selected events and replay them to the source bus, but replay does not preserve original arrival order (C62; A84, retrieved 2026-08-22). Therefore an archive is a recovery input, not a partitioned log, authoritative ledger, or deterministic projector rebuild. Replayed consumers need stable event IDs, version rules, pacing, side-effect controls, and reconciliation.

Poor fit: account-serial commands, a visible competing-consumer backlog, high-rate ordered replay for several independent projections, or a flow that needs one explicit durable process history.

Fintech example: route TradeExecuted to notification, portfolio, and compliance boundaries. Put portfolio and compliance behind separate queues or logs; never treat the notification target as authority.

SQS Standard and FIFO: assign and buffer work

SQS Standard is the default when each message should be owned by one of several workers and reorder/duplicate delivery is acceptable to an idempotent handler. Receiving creates a visibility lease; successful processing is acknowledged by deleting the message. Crash, timeout, or missed deletion can expose it again (C02,C63; A09,A85, retrieved 2026-08-22). Separate queues provide real consumer, priority, and tenant isolation; one shared queue plus autoscaling does not ensure fairness.

Choose FIFO only when a named key needs serial work. MessageGroupId defines the ordering/concurrency unit, while deduplication suppresses qualifying repeated sends only within its documented boundary (C03; A10, retrieved 2026-08-22).

Inference: More active groups allow more parallelism; a hot or poison group creates head-of-line blocking. Global ordering is not provided. You could force global serialization through one group, but that deliberate bottleneck sacrifices parallelism and still does not make consumer effects exactly once (C04,C34,C41; A02,A10, retrieved 2026-08-22).

With Lambda, the event-source mapping polls and invokes synchronously. Batches can repeat, and partial-batch responses prevent already-successful items from being retried unnecessarily (C63; A02, retrieved 2026-08-22). SQS FIFO plus Lambda remains at-least-once at the handler/effect boundary: transact the logical command/event ID with the authoritative mutation, stop a FIFO batch at the first failure when preserving order, and reconcile external outcomes.

Set visibility from worst credible processing time plus margin; extend it only with an owned heartbeat. Configure a source redrive policy and DLQ, but specify alarm, diagnosis, correction, paced redrive, idempotency, order implications, and reconciliation (C43,C45; A43, retrieved 2026-08-22). A poison message moved from a FIFO queue can let later messages advance, which is an explicit business decision—not automatic correctness.

Poor fit: independent consumers needing their own replay positions, long-term history, complex content routing inside one queue, or one message that several owners must each process without a fan-out layer.

Fintech example: serialize PostExecutionToAccount by accountId in a FIFO queue, but make ledger posting idempotent on executionId and validate account sequence. Reject one global group because an active account or poison item would block all accounts.

SNS Standard and FIFO: push fan-out to endpoint subscriptions

Choose SNS when direct topic/subscriber fan-out and endpoint reach are primary: SQS queues, Lambda, HTTP/S, and customer-notification channels can subscribe as supported. Filter policies are attached per subscription; retry behavior depends on the endpoint protocol, and an SQS DLQ attaches to the subscription rather than proving downstream processing (C64; A13,A83,A86, retrieved 2026-08-22). SNS Standard delivery can be duplicated and arrive out of order, so subscriber effects remain idempotent (C64; A91, retrieved 2026-08-22). For durable workers, SNS-to-one-SQS-queue-per-consumer is often more operable than direct invocation because each consumer owns backlog, concurrency, DLQ, and redrive.

SNS FIFO is narrower. It can deliver to SQS Standard or FIFO queues. The documented strict-order/deduplication path is SNS FIFO → SQS FIFO; SQS Standard subscribers instead have best-effort order and at-least-once delivery. AWS uses its exactly-once delivery/processing label only while every listed condition holds: the FIFO queue exists and is writable, its consumer processes and deletes before visibility expiry, no subscription filtering is configured, and no network disruption prevents delivery acknowledgement. With filtering enabled, AWS documents at-most-once delivery because the filter can remove messages (A12,A92,A93, retrieved 2026-08-22).

Inference: These are SNS-to-SQS delivery conditions, not proof that every subscriber database or external effect commits once; consumers remain idempotent (C20,C69; A12,A92,A93, retrieved 2026-08-22).

FIFO topics also have built-in archive/replay: the topic owner configures archive retention, and a subscriber can replay a selected time window to its endpoint (A94, retrieved 2026-08-22).

Inference: Replay is a recovery delivery, not reconciliation; use stable identities, pace downstream work, suppress or idempotently repeat external effects, and prove repaired business state (C69; A94, retrieved 2026-08-22).

Inference: Poor fit: complex event-envelope routing across many AWS/SaaS sources (EventBridge is usually clearer), a general partitioned log with independent consumer positions (Kinesis/MSK), or one work item that exactly one worker should own (SQS directly). SNS FIFO is also a poor fit when direct email/SMS/mobile/HTTP delivery is required: current FIFO topics deliver through SQS Standard or FIFO subscribers rather than customer-managed endpoints (C69; A92, retrieved 2026-08-22).

Fintech example: fan out a non-authoritative StatementReady notification to email and mobile delivery, with subscription-specific failure policy. The client notification is not proof that settlement or statement generation completed.

Decision table

Requirement	Choose	Avoid / why
Complex structured matching, AWS/cross-account integration, loose publishers	EventBridge, often targeting per-consumer SQS	SNS if its simpler topic/filter model is enough; direct Lambda when backlog isolation matters
One owner per unit of buffered work	SQS Standard	SNS/EventBridge alone; they route/fan out rather than expose work ownership
Per-key serial work	SQS FIFO with many well-distributed message groups	One global group; SQS Standard if reorder changes valid state
Direct endpoint fan-out and notification protocols	SNS Standard	EventBridge when endpoint breadth is irrelevant and rule/event integration is the actual need
Ordered topic fan-out to SQS FIFO queues, with optional built-in FIFO archive/replay	Inference: SNS FIFO → SQS FIFO, without filtering when relying on the documented exactly-once label (C69; A92-A94, retrieved 2026-08-22)	Inference: avoid an SQS Standard subscriber when strict order/deduplication is required, and never turn delivery conditions into a one-business-effect claim (C20,C69; A12,A92-A94, retrieved 2026-08-22)

Repository cross-reference: the current bus is useful fan-out, but CS01 through CS05 show producer dual-write and acknowledgement gaps; CS08 shows why target DLQ and Lambda runtime controls must remain separate.

Kinesis Data Streams and DynamoDB Streams

Kinesis Data Streams is general replayable ingestion. Producers choose a partition key; the service maps records to shards, and sequence/order is scoped to a shard (C10,C42; A14, retrieved 2026-08-22). Independent consumers maintain their own progress and can replay within configured retention (C14; A15, retrieved 2026-08-22).

Inference: This is a good fit for high-rate execution/market/audit feeds, multiple projections, and backlog recovery. Consumer isolation, throughput, downstream limits, and cost depend on standard polling versus enhanced fan-out, partition distribution, batching, retention, and consumer compute (C17,C49; A16,A18,A46, retrieved 2026-08-22).

This chapter stops at the selection boundary; partition math, producer partial failures, Lambda checkpointing, resharding, enhanced fan-out, and recovery drills belong in the next Kinesis deep dive. The non-negotiable boundary is that shard order is not global order, ordering does not remove duplicates, and replay can repeat effects. Choose accountId, orderId, or symbol only after deciding which relation must be serial and which skew/parallelism trade-off is acceptable.

DynamoDB Streams is table-specific change data capture. It emits item-level modifications, preserves record order for changes to the same item, and retains records for 24 hours (C23; A21, retrieved 2026-08-22). DynamoDB Streams record order is not consumer-effect semantics: a Lambda/event-source consumer can be retried, so the projection or outbox relay remains idempotent. It is not table-wide global order, general producer ingestion, or a long-retention audit log.

Use DynamoDB Streams when the event should follow a committed table mutation: projection maintenance, cache invalidation, audit export, or an outbox-like relay whose local atomicity is a DynamoDB transaction (C23,C38; A20,A21, retrieved 2026-08-22).

Inference: That DynamoDB transaction does not atomically include publication to EventBridge/SQS or any other external service, and consumption remains retryable. If recovery may exceed 24 hours, retain an independent rebuild source/export or publish controlled records to a longer-retained log (C22,C23; A20,A21, retrieved 2026-08-22).

Requirement	Choose	Poor fit / recovery implication
General high-rate feed with independent consumer positions and replay	Kinesis Data Streams	SQS deletes work; EventBridge archive replay is not a partitioned consumer log
React to a committed DynamoDB item change	DynamoDB Streams	Kinesis if producers are not table mutations or retention/consumer topology must be independent
Deterministic projection order	Partition/version by the actual aggregate and reject stale/gapped versions	Arrival time or a claim of table-wide Stream order
Recovery beyond source retention	Durable authority/export plus controlled republish/rebuild	Assuming Stream retention or a DLQ is the audit history

Repository cross-reference: CS06 needs an inbox identity, and CS07 needs an authoritative sequence/gap policy. Moving the projector from EventBridge to either stream does not cure those application semantics.

Step Functions: explicit workflow orchestration

The preceding services move records or expose changes. Some business processes also need to remember which step is waiting, who owns a timeout, and what happens after an external action becomes irreversible. That process-state requirement leads to orchestration. Keep workflow history separate from the participant's actual database or provider receipt.

Use Step Functions when a business process benefits from an explicit state, deadline, branching, retry, wait, callback, audit, and operator-visible execution identity. This is orchestration/process-manager territory: settlement, withdrawal, KYC review, or a multi-stage corporate-action process. Use choreography when independent reactions are few, local, and do not need a central process history. As dependencies, deadlines, compensations, and questions like “what is waiting?” grow, explicit orchestration becomes easier to reason about.

Property	Standard Workflows	Express Workflows
Execution semantics	Documented exactly-once workflow execution unless explicitly retried	Asynchronous Express is at-least-once; Synchronous Express is at-most-once
Duration	Up to one year	Up to five minutes
History	Durable/auditable execution history; API history retained after completion according to current documented period	CloudWatch Logs is the execution-history route; logging is not enabled merely by choosing Express
Integrations	Request/response, job-run (.sync), and callback task token patterns	Request/response; no job-run or callback pattern
Price shape	State transitions	Executions, duration, and memory
Fit	Long-running, auditable, human/external callbacks, non-idempotent workflow coordination	High-volume, short, idempotent transformations or request flows

These mutable differences, including the one-year/five-minute limits and history model, are current as of retrieval (A25,A49,A87, retrieved 2026-08-22).

Inference: They are selection inputs, not universal capacity claims; verify current quotas and integration support for the target Region.

Inference: retries belong on explicit transient errors with bounded attempts, backoff, jitter, and deadlines. Catch can route to manual review or compensation. A Wait state can model a business deadline without holding compute. Callback tokens let Standard wait for a human, venue, or legacy worker; secure the token, set timeout/heartbeat, and make duplicate/late callbacks safe (C45,C65; A26,A87, retrieved 2026-08-22).

Inference: at the workflow-execution boundary, Standard's documented workflow execution label does not make an external side effect exactly once. A retried payment API, broker submission, email, or ledger task can still repeat or time out ambiguously. Give each task a stable business key, persist request/receipt state, use the provider's idempotency contract where available, and reconcile. A compensation is a new forward action such as release/reversal; it is not an ACID rollback and may be impossible after an irreversible market or external effect (C39,C65; A25,A26,A87, retrieved 2026-08-22; F05,F10).

Failure/recovery: distinguish execution failed/timed out/aborted, task retry exhaustion, callback timeout, and a business state that needs manual repair. Redrive/restart must not blindly repeat already-completed external actions. Keep stable workflow/business IDs and an operator view that joins workflow state to authoritative settlement/ledger state.

Poor fit: a high-rate retained event log, simple independent fan-out with no process state, or a microsecond/millisecond matching loop. Do not turn every service call into a state transition when ordinary code with one transaction is clearer.

Fintech example: a settlement workflow waits for venue confirmation, branches on partial failure, times out/escalates, posts balanced ledger entries idempotently, and compensates only reversible reservations. Ledger and venue records remain authoritative; workflow history explains coordination.

Repository cross-reference: CS04 cannot be repaired merely by drawing those non-atomic writes as states. First define the local transaction/outbox and authoritative invariants; then orchestrate the genuinely cross-boundary process.

Data Firehose, MSK, and Managed Service for Apache Flink

Amazon Data Firehose: managed buffered destination delivery

Inference: choose Firehose when the desired outcome is efficient managed delivery to a supported destination such as S3 or a search/analytics target, with buffering, optional transformation/format conversion, destination retry, and optional backup as supported. Buffer size/interval are hints and destination behavior varies (C66; A88, retrieved 2026-08-22). It is excellent for audit/analytics landing where seconds-scale buffering is acceptable and custom consumer control adds no business value.

Inference: Firehose is not a general event bus, work queue, or replayable multi-consumer stream. Its at-least-once/destination duplicate warning has documented destination exceptions, and none authorizes an end-to-end exactly-once effect (C28,C31,C66; A29, retrieved 2026-08-22). Keep the upstream Kinesis/MSK/S3 or authoritative export if independent replay/rebuild is required.

Poor fit: low-latency commands, per-key state machines, arbitrary subscribers, consumer-controlled checkpoints, or a compliance pipeline whose only recoverable copy would be the in-flight Firehose buffer.

Amazon MSK: choose the Kafka contract and ecosystem deliberately

Choose MSK when Kafka protocol compatibility, existing clients/connectors, partitioned topics, consumer groups, or organizational Kafka expertise is a requirement. Compared with Kinesis, you accept more topology/version/partition/ security and rebalance decisions in exchange for Kafka ecosystem control and portability. MSK manages infrastructure, not topic design, key skew, schema governance, lag budgets, poison handling, or application upgrades.

Kafka transactions can make supported Kafka consume-transform-produce work and offset commits atomic inside Kafka (F28).

Inference: Amazon MSK does not extend that transaction to an arbitrary ledger/database/API. That external-system boundary still requires a supported transactional connector or business idempotency and reconciliation (C29; A30, retrieved 2026-08-22; F28).

Inference: Monitor consumer lag alongside business reconciliation, but remember lag metrics can be absent for some group states/configurations (C68; A90, retrieved 2026-08-22).

Poor fit: a team with no Kafka requirement or operating competence, a simple one-consumer work queue, or a design chosen only because “Kafka scales.”

Managed Service for Apache Flink: stateful event-time computation

Choose Managed Service for Apache Flink when processing depends on state across records: event-time windows and watermarks, keyed aggregates, temporal joins, late-data handling, patterns, or continuous risk/fraud analytics. It is a processor over streams, not a replacement for the source log or authoritative financial store. Event time answers when the fact occurred; processing time answers when the system saw it.

Checkpointing restores application state and positions after failure (A31,A89, retrieved 2026-08-22; F31).

Inference: Exactly-once language stops at the replayable source, checkpointed operator state, and a compatible connector/sink; an arbitrary database or API side effect is outside that boundary. Plan stable operator identities, savepoint-compatible deployments, checkpoint duration/size, state growth, backpressure, restart time, source retention, and catch-up capacity (C67; A31,A89, retrieved 2026-08-22; F31).

Poor fit: stateless routing/filtering, simple delivery to S3, low-latency matching, or a team unwilling to own stateful streaming code and upgrades.

Need	Choose	Reject neighboring option because
Buffered managed landing to S3/search/warehouse target	Firehose	A custom consumer is unjustified unless it needs custom checkpoints, branching, low latency, or recovery behavior
Kafka clients/connectors and consumer-group ecosystem	MSK	Kinesis avoids Kafka operations but does not satisfy a mandatory Kafka contract
Stateful event-time windows/joins/late-data logic	Managed Service for Apache Flink over Kinesis/MSK	Firehose delivers; EventBridge routes; neither owns stateful computation
Durable general replay for several independent consumers	Kinesis or MSK	Firehose is a destination delivery service, not the shared bus/log

Amazon MQ and compute-substrate alternatives

Inference: Amazon MQ is a migration/interoperability choice when an existing ActiveMQ Classic or RabbitMQ application needs supported broker protocols/semantics and rewriting immediately is not justified (C55; A61, retrieved 2026-08-22). Verify engine, version, topology, durability, acknowledgement, failover, client library, and protocol features. Avoid MQ merely because “we need a queue”: SQS/SNS/ EventBridge usually remove broker/version operations when no legacy protocol is required.

Inference: messaging choice and execution substrate are separate. Lambda fits bursty, bounded, event-driven work with managed scaling and short stateless handlers. Choose ECS/Fargate when a long-running container, steady worker pool, custom runtime/process, connection reuse, sidecar, or controlled concurrency is the better unit. Choose EC2 when specialized hardware/networking/kernel/runtime, tightly controlled latency, very sustained utilization, or mature host-level operations justify the additional ownership (C53; A58-A60, retrieved 2026-08-22).

Inference: for a latency-critical matcher, prefer a long-lived, measured single-writer or partitioned engine on containers/instances; use queues/streams around its ingress, durable execution publication, projections, and control plane. Lambda remains a candidate only if explicit latency/throughput/state evidence says it fits—not because it is serverless. Compare Lambda requests/duration/memory with Fargate vCPU/memory/runtime and EC2 capacity/storage/network/idle headroom, using current eu-west-1 inputs rather than a universal price slogan (C49,C53; A44,A58-A60,A67,A68, retrieved 2026-08-22).

Substrate	Choose when	Avoid when	Primary recovery/operations concern
Lambda	Bursty, bounded, stateless handler; managed scaling is valuable	Long-running/stateful process, tight tail latency, specialized runtime/hardware, sustained pool is demonstrably better	Concurrency/downstream protection, retry mode, deployment version, cold/warm behavior, logs
ECS on Fargate	Long-running container and controlled worker pool without host management	Specialized host/hardware or workload makes per-task capacity poor fit	Task replacement, desired count/capacity, connection draining, image/dependency rollout
ECS/other scheduler on EC2	Host/runtime/network control and sustained capacity justify ownership	Small bursty workload or team cannot operate hosts safely	Instance/cluster capacity, patching, failover, placement, headroom, autoscaling
Amazon MQ consumers	Legacy protocol/broker semantics are binding	Native AWS message semantics satisfy the new system	Broker topology/version, connection/failover, storage, consumer acknowledgement and DLQ

Repository cross-reference: CS12 is exactly this separation. CS09 also shows why moving a memory-bound scan from Lambda to a container would change runtime limits but not prove export completeness.

Paired decisions: choose and reject explicitly

Pair	Choose the first when	Choose the second when	Interview rejection sentence
EventBridge vs SNS	Structured event matching, AWS/cross-account integration, event-bus governance, optional source-bus archive/replay	Inference: direct Standard topic/subscriber push and notification protocols, or SNS FIFO → SQS FIFO strict-order fan-out with built-in FIFO archive/replay (C69; A92-A94, retrieved 2026-08-22)	Inference: “I chose EventBridge for rule/event integration; SNS Standard endpoint reach or SNS FIFO queue-path replay was not the requirement” (C69; A92-A94, retrieved 2026-08-22).
EventBridge vs Kinesis	Route current facts to targets; consumer positions and partition order are unnecessary	Retained ordered shards, independent consumer positions, replay and high-rate ingestion are required	“An EventBridge archive is recovery input, not the partitioned log this projector fleet needs.”
Kinesis vs SQS FIFO	Several independent consumers need replay and shard-scoped retained order	Each work item has one owner and per-message-group serialization/visibility/DLQ semantics fit	“This is account work assignment, not a shared historical feed, so FIFO groups match ownership better.”
Kinesis vs MSK	AWS-native managed stream and Kinesis integrations meet the need with less Kafka surface	Kafka protocol, clients/connectors, consumer groups, or portability are requirements worth operating	“Kafka ecosystem compatibility is absent, so MSK's additional operating surface buys us nothing.”
Step Functions vs choreography	Deadlines, waits, callbacks, compensations, audit and operator questions require explicit process state	Few independent reactions can converge through authoritative state/reconciliation without central coordination	“Settlement needs a durable process manager; notification fan-out does not.”
Firehose vs custom consumers	Supported destination delivery, buffering and managed transformation are enough	Custom checkpoint, branching, ordering, validation, low latency, enrichment, or recovery control is essential	“Audit landing is a delivery problem, so custom consumer code would add an unnecessary failure surface.”

These are paired boundary decisions, not mutually exclusive platform choices. A common architecture is Kinesis → Flink projections and Firehose audit delivery, or EventBridge → SQS queues → Lambda/container workers.

Eight fintech flow decisions

Apply the service distinctions to whole obligations now. Start each row at the business authority and follow it through the proposed composition to recovery evidence. The rejected neighbor matters as much as the selected service: it reveals which requirement drove the choice and what change would make you reconsider.

Flow	Requirements first	Recommended composition	Rejected neighbors and correctness boundary	Recovery and evidence
Order commands	One logical owner; idempotent acceptance; pre-trade risk/reservation; per-order/account state rules; bounded client latency	Synchronous API to authoritative command transaction plus outbox; SQS FIFO by account/order only if asynchronous serial admission is allowed	EventBridge/SNS do not own command work; Kinesis is justified only when retained ingress/replay is a stated requirement. Queue acceptance is not order acceptance.	Return scoped status; inbox/idempotency key with mutation; outbox relay; queue age; rejected/duplicate commands; reconcile accepted orders to reservations
Execution events	Immutable venue/matcher fact; independent ledger, portfolio, notification and compliance consumers; replay and per-order/venue sequence	Durable authoritative execution record/outbox to Kinesis (or MSK when Kafka contract exists); isolate derived consumers	EventBridge can route lower-rate facts but lacks partitioned consumer replay; SQS gives one owner. Transport order cannot enforce cross-account cash/securities invariants.	Stable executionId; consumer inbox/version gaps; iterator/consumer lag; replay drill; reconcile venue executions to ledger postings
Account-serial work	One work owner; serialize postings/reservations for accountId; many accounts parallel; poison isolation	SQS FIFO with accountId message group feeding idempotent ledger worker, or a measured partitioned single-writer engine for higher/tighter workloads	One global group destroys parallelism; Kinesis is for retained multi-consumer replay, not automatically work ownership.	Visibility/redrive policy; stop FIFO batch on failure; hot-group age; posting inbox; balance/reconciliation proof
Market data	Very high rate, symbol/key order where needed, many consumers, replay window, skew and late data	Kinesis or MSK partitioned by a measured key; Flink for event-time windows/joins; long-lived compute for latency-critical normalization	SQS FIFO serializes work; EventBridge is routing, Firehose is only a sink, Lambda is not assumed suitable for the tight loop.	Producer partial-failure handling; lag/throttles; hot partition detection; checkpoint health; replay at controlled rate; gap detection against source sequence
Notifications	Derived, multi-channel fan-out; preferences/filters; stated loss/latency contract; no financial authority	SNS Standard for endpoint fan-out, often per-channel SQS queues; EventBridge when domain-event routing is already the integration boundary	Inference: choose SNS FIFO only for the SNS FIFO → SQS FIFO ordered/deduplicated path; filtering changes its documented semantic, while built-in archive/replay can support queue-subscriber recovery. Client receipt is never ledger/settlement evidence (C69; A92-A94, retrieved 2026-08-22).	Subscription delivery failures/DLQ; provider idempotency key/receipt; channel queue age; FIFO replay progress where used; preference/version checks; customer-status reconciliation if contract requires
Long-lived settlement	Hours/days, external callbacks, deadlines, manual review, partial/irreversible actions, auditable process state	Step Functions Standard process manager plus idempotent tasks and authoritative settlement/ledger records; SQS/container workers for external work	Express is time/integration constrained; choreography obscures state when many dependencies/compensations exist. Workflow state is not financial authority.	Timeout/heartbeat/escalation; task request/receipt state; safe workflow redrive; compensation status; reconcile venue/custodian/ledger
Audit delivery	High-volume append-oriented delivery to immutable/object retention, batching acceptable, independently recoverable source	Kinesis/MSK or transactional export as recoverable source → Firehose → S3; catalog/query later	Firehose alone is not the general bus/replay authority; EventBridge archive and CloudWatch logs are not a financial ledger.	Source-to-object counts/hashes/manifests; Firehose freshness/delivery errors; S3 lifecycle/integrity; replay missing ranges; retained-audit-year unit cost
Compliance search	Derived searchable projection; rich queries; freshness SLO; full rebuild; authority remains executions/ledger/audit store	Kinesis/DynamoDB Streams/outbox → idempotent projector → OpenSearch; S3/Athena for durable investigation/rebuild as appropriate	OpenSearch is not authoritative; DynamoDB Streams' 24-hour retention is insufficient as the only rebuild source; Firehose fits bulk delivery but not custom projection state when that is required (C23,C46; A21,A32, retrieved 2026-08-22).	Projection lag/version gaps; indexing failures/quarantine; rebuild from durable authority; compare source and indexed counts; searches show freshness watermark

Authority remains deliberately separate in every row. This chapter does not relabel those projections as CQRS or event sourcing. CQRS is a spectrum and event sourcing is a separate authoritative-state choice (C35); the dedicated later chapter will decide them.

Repository mapping without widening CS01-CS12:

• Order commands expose CS01, CS02, and CS03.
• Execution and projection paths expose CS04, CS05, CS06, and CS07.
• Recovery/export correctness exposes CS08 and CS09.
• Financial authority and compute choice expose CS10, CS11, and CS12.

Failure, recovery, and operational evidence

Failure-boundary table

Boundary	Failure/recovery procedure	Saturation/loss/incorrectness signals
EventBridge producer → bus	Inspect every entry result; retain stable replay identity; validate bus/rules separately; repair/replay and reconcile	Failed entries, rule matches versus target invocations, failed invocations, target DLQ sends and failed DLQ sends
EventBridge/SNS → target	Inference: own target/subscription retry and DLQ permission; quarantine, diagnose, fix, and use paced redrive. For SNS FIFO subscribers, built-in archive/replay can redeliver a selected time window, but effects remain idempotent and reconciled (C69; A94, retrieved 2026-08-22).	Delivery failures by target/protocol, DLQ depth/age, failed DLQ write, FIFO archive/replay progress, downstream acceptance/commit gap
SQS → consumer	Visibility sized/extended deliberately; partial-batch response; source DLQ; order-aware safe redrive	Oldest age, visible/in-flight backlog, receive/delete gap, redelivery/duplicate rate, hot message group, DLQ age
Kinesis/DynamoDB Streams → consumer	Checkpoint only after intended batch semantics; quarantine/skip policy; replay within retention; rebuild/reconcile beyond it	Iterator age, throttling, failed records, per-consumer lag, version gaps, poison retries, backlog drain time
MSK → consumer group	Offset/checkpoint policy; rebalance-safe idempotency; reset/replay with rate and sink controls	Offset/time lag, under-replicated/offline partitions as applicable, rebalance churn, producer/consumer errors, sink discrepancy
Step Functions → task	Bound retry and timeout; persist task key/request/receipt; callback heartbeat; route terminal business state to review	Executions failed/timed out/aborted, task retry count, callback age, throttling, workflow-to-authority mismatch
Flink application → sink	Restore checkpoint/savepoint; keep source retention sufficient; validate connector/sink boundary; reconcile output	Backpressure, checkpoint duration/size/failure, restart/downtime, source lag, state growth, late/dropped records, sink discrepancy
Firehose → destination	Preserve upstream recovery source; inspect destination errors/backup; replay missing interval from source	Delivery freshness, throttling/errors, backup/error objects, source-to-destination count/bytes/manifests

Current AWS metric routes support the categories in the table (A03,A36,A81,A85, A86,A89,A90, retrieved 2026-08-22). Exact metric names and dimensions are mutable; verify them for service mode, endpoint, engine/version, and Region.

Inference: Every dashboard must add business signals: duplicate commands/fills, version gaps, unposted executions, ledger imbalance, stale projections, and unreconciled external receipts. A green transport is not financial correctness (C68; A03,A36,A81,A85,A86,A89,A90, retrieved 2026-08-22).

Cost and operating-shape checklist

There are no numeric prices in this chapter. Later calculations must use dated eu-west-1 inputs. At selection time model:

• published/ingested requests or records, payload billing chunks/bytes, and fan-out multiplication;
• batch size, invocation rate, compute duration/memory/concurrency, and idle long-lived capacity;
• stream/broker partitions or shards/capacity mode, retention/storage, enhanced fan-out, consumer count, and replay/catch-up traffic;
• workflow transitions versus Express executions/duration/memory and log volume;
• Firehose bytes, buffering/transform/conversion/dynamic partitioning, backup, destination and storage requests;
• MSK/MQ/Fargate/EC2 provisioned capacity, storage, transfer, headroom, upgrades, and on-call skill;
• CloudWatch logs/metrics, tracing, KMS calls, cross-AZ/Region/NAT transfer, DLQ retention, and duplicate/reconciliation work.

Price shapes are mutable and must be recalculated from the current official pages (C49; A44-A50,A67-A69,A71,A73-A77,A80, retrieved 2026-08-22). The right conclusion is not “managed/serverless is cheaper,” but “for workload W and recovery target R, option X has unit cost Y and operating risk Z.”

Interview follow-ups

1. Why put SQS behind EventBridge instead of targeting Lambda directly?
2. At which exact boundary can an SQS FIFO/Lambda flow duplicate a ledger write?
3. What is your ordering key, and what happens when one key is hot or poisoned?
4. How does EventBridge target failure differ from Lambda handler failure?
5. When is an EventBridge archive enough, and when is Kinesis/MSK required?
6. Why is DynamoDB Streams suitable for an outbox relay but not a general bus?
7. When would SNS be clearer than EventBridge? When would SNS FIFO be justified?
8. What makes Step Functions Standard preferable to choreography for settlement?
9. Why does “exactly-once workflow execution” not mean one external payment?
10. What boundary does a Flink checkpoint protect, and what sink can escape it?
11. Why is Firehose a good audit-delivery path but a poor command bus?
12. What concrete Kafka requirement justifies MSK over Kinesis?
13. When do ECS/Fargate or EC2 beat Lambda for a consumer or matcher?
14. Which transport and business metrics prove a replay repaired the system?
15. How would you rebuild compliance search after its source Stream retention expired?

Concise senior articulation

“I select from ownership, order, replay, recovery, and workload. EventBridge and SNS route or fan out; SQS assigns buffered work; Kinesis and MSK retain partitioned logs for independent consumers; DynamoDB Streams is short-retained table CDC; Step Functions makes long-lived process state explicit; Firehose delivers buffered data to supported sinks; Flink owns stateful event-time computation; MQ is for broker compatibility. I name every ordering key and retry boundary. FIFO deduplication, Standard workflow execution, Kafka transactions, and Flink checkpoints stop at documented boundaries, so authoritative financial effects still use stable identity, transactions/idempotency, version rules, and reconciliation. Then I compare unit cost and operating burden for the measured workload.”

Sources used in this chapter

Only sources cited above are listed.

Concepts and patterns

• Bounded claim IDs cited above in the claim register.
• Chris Richardson, Saga (F05).
• Garcia-Molina and Salem, Sagas (F10).
• Apache Kafka project, message delivery semantics (F28).
• Apache Flink project, stateful computations, time, and checkpoints (F31).

AWS semantics and operations (retrieved 2026-08-22)

• AWS Lambda: SQS event source (A02), Kinesis event source (A03), event-source mappings (A01), and invocation retries (A04).
• EventBridge: target retry/DLQ (A05), PutEvents (A06), bus concepts (A07), quotas (A08), sending events (A82), DLQ permissions/metrics (A81), patterns, and archive/replay (A84).
• SQS: Standard queues (A09), FIFO terms (A10), message quotas/retention (A11), and visibility/redelivery (A85).
• SNS: FIFO ordering/deduplication (A12), delivery retries (A13), topic/subscriber model (A83), filtering, subscription DLQs (A86), Standard versus FIFO capabilities plus Standard ordering/deduplication (A91), FIFO delivery to SQS Standard/FIFO (A92), FIFO deduplication/filtering (A93), and FIFO archive/replay (A94).
• Kinesis Data Streams: concepts (A14), retention (A15), quotas (A16), PutRecords (A17), and enhanced fan-out (A18).
• DynamoDB: transactions (A20) and Streams (A21).
• Search projection: Amazon OpenSearch Service (A32).
• Step Functions: Standard/Express comparison (A25), error handling (A26), and integration patterns (A87).
• Data Firehose: purpose/destination semantics, duplicate boundary (A29), and buffering (A88).
• Amazon MSK: service guide (A30) and consumer lag (A90).
• Managed Service for Apache Flink: service guide (A31), checkpoint/fault-tolerance scope, and metrics (A89).
• Compute and migration: ECS (A58), Fargate (A59), EC2 (A60), and Amazon MQ (A61).
• Operations and observability: CloudWatch (A36) and AWS Well-Architected Reliability (A43).

AWS price-shape routes (retrieved 2026-08-22; no numeric price copied)

• Lambda (A44), DynamoDB (A45), Kinesis Data Streams (A46), EventBridge (A47), SQS/SNS (A48), Step Functions (A49), and AWS Pricing Calculator (A50).
• Fargate (A67), EC2 (A68), Amazon MQ (A69), Data Firehose (A76), Amazon MSK (A77), and Managed Service for Apache Flink (A80).
• Cross-cutting inputs: CloudWatch (A71), data transfer (A73), KMS (A74), and S3 (A75).

Takeaway and next step

A defensible selection explains ownership, ordering, replay, failure handling, and operating cost together. Before moving on, choose services for account-serial work, execution fan-out, and settlement coordination, and explain a rejected alternative for each. The next stage examines one retained-stream choice in detail: Kinesis partitioning, consumers, and replay.