# Event-driven foundations: choose semantics before services

This chapter assumes AWS vocabulary. Its purpose is to make the service name
the *last* part of an architecture answer. A senior design starts with the
business invariant, accepted latency and loss, work ownership, replay need,
ordering scope, overload policy, and recovery evidence. Only then should it map
those requirements to a concrete service mode.

The controlled definitions live in
[canonical terminology](../research/terminology.md), the high-risk wording in
the [claim register](../research/claim-register.md), and repository observations
in the [evidence map](../research/repository-evidence.md). “Documented” below is
a source-backed semantic. “Inference” is an architecture conclusion from named
premises, never an AWS guarantee.

## The decision sequence

Before saying “EventBridge,” “SQS,” “Kinesis,” or “MSK,” answer these questions
in order:

1. **What fact is authoritative, and what invariant is synchronous?** An order
   acceptance, reservation, execution, settlement obligation, ledger posting,
   portfolio view, and client notification are different facts.
2. **Is the message a command, event, notification, document, or reply?** The
   transport envelope does not decide the payload's business meaning (C37).
3. **Who owns each unit of work?** One worker, every interested consumer, or
   independently positioned consumers that may replay?
4. **What can repeat or disappear?** Name the producer acknowledgement,
   transport delivery, consumer commit, external effect, and reconciliation
   boundary separately.
5. **What order matters?** Name the key, causal/version rule, serialization
   point, and cross-key invariant. “Ordered” alone is incomplete.
6. **How stale can each result become?** Give normal and recovery lag budgets,
   not merely a throughput target.
7. **What happens under overload?** State admission, shedding, isolation,
   fairness, and backlog-drain policies before invoking autoscaling.
8. **How will an old producer, a new consumer, and retained historical data
   coexist?** That is the contract and replay-compatibility question.

That sequence prevents a common interview failure: choosing a familiar managed
service and then retrofitting the correctness story around its marketing label.

## Why create an asynchronous boundary?

Asynchrony moves waiting and failure out of a caller's stack. It does not remove
either. The exchange is usually immediate coupling for delayed state,
additional identities, and an operational recovery loop.

| Reason | What it buys | What it costs / when the benefit fails |
|---|---|---|
| **Temporal decoupling** | Producer and consumer do not need to be available at the same instant; retained work can bridge a short consumer outage. | Completion becomes delayed and ambiguous. The caller needs an acceptance state, status model, deadline, cancellation semantics, and recovery source. Retention is finite. |
| **Load leveling** | A buffer converts a burst into work a bounded consumer fleet can drain. | A queue can hide overload until its oldest item violates the SLO. If sustained arrival rate is at least service rate, backlog has no finite drain time. Storage, duplicate work, and stale outcomes grow. |
| **Independent scaling** | Producers, consumers, and different projections can use distinct capacity and deployment models. | Every scaling dimension can move the bottleneck downstream. More consumer concurrency can throttle a database, exhaust connections, or violate a serialization requirement. |
| **Failure isolation** | A failed optional consumer need not fail the producer or unrelated consumers. | Isolation is bounded by shared capacity, contracts, and authority. Failure often becomes lag, a DLQ item, or divergent state rather than an immediate exception. Operators now own detection and reconciliation. |
| **Latency hiding** | A user can receive `ACCEPTED` after durable intent while slower work continues. | This improves response latency only by changing the product contract. It cannot truthfully return `EXECUTED`, `SETTLED`, or “funds available” before the authoritative transition completes. |
| **Organizational autonomy** | Teams can deploy consumers and read models independently behind stable contracts. | Coupling moves into schemas, event meaning, retention, permissions, SLOs, and incident ownership. Hidden subscribers make change impact and event cascades harder to see (C58/F29,F30). |

### Repository boundary: useful decoupling, missing completion evidence

**Observed:** the order handler writes a `PENDING` order and calls `PutEvents`;
later handlers write the simulated trade, portfolio, and notification. Those
separate calls create temporal decoupling, but the first database write and
publication are not atomic
([CS01](01-repository-case-study.md#cs01--order-state-and-event-publication-are-a-non-atomic-dual-write)).

**Documented:** a DynamoDB transaction can cover supported DynamoDB items in one
account and Region, not EventBridge (C22/A20, retrieved 2026-08-22). A
transactional outbox can commit business state and publication intent locally,
but relay and consumption remain retryable (C38/A20,A21/F04; A20-A21 retrieved
2026-08-22).

**Inference:** asynchrony has converted a thrown downstream error into a
potentially stranded order. The correct response is durable intent, explicit
`ACCEPTED` status, outbox age metrics, idempotent consumers, and reconciliation—not
the claim that the systems are “decoupled.” The same boundary can be a good
design after its delayed-state contract and recovery loop are made explicit.

### When not to add the boundary

Keep work synchronous, or inside one authoritative transaction, when the caller
must know the invariant before proceeding: available-funds reservation, legal
order-state transition, or ledger balance are examples. Asynchrony may publish
the result afterward, but a stale portfolio projection must not approve the
invariant (C46).

Also avoid an asynchronous network boundary when a cohesive module and one
transaction already meet the workload. A queue between every function adds
latency, contracts, retry states, and on-call surfaces without buying a real
independence or overload boundary. For a strict tail-latency matching loop, a
multi-hop general event path is normally a poor fit; use a measured,
capacity-controlled sequencing design and keep asynchronous services around its
control plane
([CS12](01-repository-case-study.md#cs12--the-lambda-simulation-is-not-a-default-latency-critical-matcher)).

## Pattern vocabulary: four different meanings of “event-driven”

First separate payload semantics from interaction style and persistence model.
These patterns can coexist; none implies a particular broker (C58/F07,F29,F30).

### Message, command, event, and notification

- A **message** is the transport envelope. Delivery, acknowledgement,
  visibility, partition, and DLQ are properties of its channel.
- A **command** requests an action and is accepted or rejected: `PlaceOrder`,
  `ReserveFunds`, `CancelOrder`. It has an intended handler or authority.
- An **event** records a past fact: `OrderAccepted`, `FundsReserved`,
  `TradeExecuted`. It cannot be rejected after publication as though the fact
  had not happened; a consumer may reject or quarantine its processing.
- A **notification** announces a fact to a recipient. It may be a pointer or
  carry rich context. It is not intrinsically lossy: durability, acknowledgement,
  replay, and acceptable loss belong to the business contract and transport.

Calling a required action `OrderPlaced` does not turn a command into a fact. If
the sender expects exactly one executor and treats non-execution as failure, say
who owns the `ExecuteOrder` command. Fowler calls the misleading alternative a
“passive-aggressive command” (C58/F29).

### Event notification

The producer announces a past fact and does not prescribe a recipient's
reaction. A minimal `PortfolioChanged {portfolioId, version}` can tell a client
to refresh; a richer notification can avoid that read. The producer remains
coupled to the event's meaning but not to a list of response calls.

**Fit:** optional reactions, cache invalidation hints, client status updates,
and extensibility where zero consumers is valid.

**Costs:** a pointer creates synchronous read-back load and an availability
dependency; rich payloads increase coupling, privacy exposure, and versioning
work. Hidden event cascades make the end-to-end flow difficult to inspect.

**Poor fit:** a reservation or settlement step that must have one accountable
owner and a completion deadline. Use a command or visible process manager;
publish events about accepted/completed facts.

### Event-carried state transfer

The event carries enough producer-owned state for a consumer to maintain the
local subset it needs without a synchronous call back (C58/F29). A portfolio
projector might receive execution ID, account ID, instrument, side, exact
quantity/price, currency/scale, and authoritative aggregate version.

**Fit:** read-heavy projections, independent availability, low-latency local
queries, and several consumers that need the same stable facts.

**Costs:** deliberate data replication, larger messages, privacy/retention
scope, write amplification, stale copies, version/gap handling, bootstrap, and
rebuild procedures. A “current-state” payload also cannot fill a missed
intermediate transition when that transition matters.

**Poor fit:** large or sensitive records where most consumers need one field;
rapidly changing data where copying it costs more than a bounded query; or any
case where a stale copy would authorize a hard financial invariant.

### Event collaboration

Components announce facts and independently react; the overall behavior
emerges from subscriptions and event cascades rather than a central caller
directing every step (C58/F30). It can use notifications, event-carried state,
or both.

**Fit:** adding projections, analytics, compliance search, or optional
notifications without changing the producer; domains that can own local state
and tolerate a declared freshness window.

**Costs:** the workflow graph moves from source code into runtime configuration
and contracts. Cascades can be cyclic or surprising. Partial failure,
compensation, timeout, and completion ownership become harder to see. For a
customer-visible settlement workflow, a process manager may provide a clearer
state machine even when steps still exchange events.

**Poor fit:** a small, fixed sequence whose audit and timeout owners must be
obvious, or a cross-component invariant incorrectly assumed to be ACID.

### Event sourcing

An aggregate's authoritative state changes are stored as an append-oriented
event sequence; current state is derived by replay, often with snapshots
(C36,C58/F07,F29). Event sourcing is a persistence decision. It neither requires
asynchronous processing nor follows from publishing integration events after a
CRUD write.

**Fit:** domains where historical decisions, temporal queries, correction
semantics, and alternative/rebuilt views justify event-store governance—for
example, a carefully modelled order lifecycle or accounting domain.

**Costs:** stable event identity and ordering, optimistic concurrency,
upcasters/schema evolution, snapshot policy, replay-safe consumers, deletion
and privacy handling, projection rebuilds, and operational expertise. Historic
events preserve past meaning; changing code does not rewrite what happened.

**Poor fit:** adopting it only to get an audit log, broker replay, or CQRS. An
append-only technical event archive is not automatically the authoritative
financial ledger (C47).

### The repository through this vocabulary

| Repository path | Observed | Bounded classification / inference |
|---|---|---|
| `OrderPlaced` to trade-executor | One named target is expected to simulate execution. | **Inference:** despite its past-tense name, it behaves command-like because order progress depends on one action. Make command ownership and acceptance explicit; then publish facts. |
| `TradeExecuted` to portfolio-updater | The payload includes trade/order/user, instrument, side, quantity, price, timestamp, and correlation ID. | **Inference:** an integration event with event-carried state sufficient for the current projector to calculate without reading the trade. It is not event sourcing, and it lacks exact money plus an authoritative sequence ([CS07](01-repository-case-study.md#cs07--the-portfolio-has-an-ordering-requirement-but-no-event-sequence), [CS10](01-repository-case-study.md#cs10--binary-floating-point-is-used-for-money)). |
| `TradeExecuted` to notification-sender | A separate consumer turns the fact into an SNS customer message. | Event collaboration plus a client notification. Whether missed delivery is acceptable must be stated; “notification” does not make it lossy. |
| EventBridge archive and DLQs | Transport/recovery paths with bounded behavior; the declared DLQs lack the required send permission (C57/A81, retrieved 2026-08-22). | Neither event sourcing nor a financial ledger. A DLQ is a holding area, not reconciliation ([CS08](01-repository-case-study.md#cs08--the-target-dlqs-lack-write-permission-and-cover-a-separate-boundary)). |
| Mutable portfolio row | Updated from integration events. | A projection, not authoritative holdings. Its optimistic lock is not an inbox or source sequence ([CS06](01-repository-case-study.md#cs06--optimistic-locking-does-not-make-the-portfolio-consumer-idempotent)). |

## Channel semantics: work queue, router, and replayable log

These are semantic centers of gravity, not mutually exclusive product boxes.
Concrete products combine traits; state retention, acknowledgement, ordering,
fan-out, and retry separately (C42/F01,F02).

| Question | Work queue / competing consumers | Publish-subscribe router | Replayable log |
|---|---|---|---|
| Primary purpose | Assign each work item to one worker in a competing group. | Route or replicate a publication to zero, one, or many independent subscriptions/targets. | Retain an ordered sequence so independently positioned consumers can read/re-read within retention. |
| Consumption state | Visibility/lease then acknowledgement/removal or expiry. | Each target/subscription has its own delivery boundary; the router may not retain a consumer position. | Each consumer/group tracks a position/checkpoint; one consumer advancing does not delete the shared record. |
| Parallelism unit | Consumers and, where used, message groups/lanes. | Targets/subscriptions plus the downstream channel chosen for each. | Partitions/shards and consumer-group membership. |
| Ordering | Product/mode-specific; often best effort or per message group. | Do not infer order from fan-out. Add sequence/version checks or an ordered downstream path when required. | Normally per partition/shard, not across the whole stream. Replay preserves only the documented scope and retained data. |
| Replay | Usually redrive of retained/unacknowledged/held work, not arbitrary independent history. | Often needs an archive or downstream durable channel; routing alone is not replay. | Core property within retention, but rebuild time, schema compatibility, and external side effects still constrain it. |
| Natural fit | Document generation, risk calculation jobs, notification delivery work, settlement file ingestion. | Domain integration events, independent notification/projection/compliance reactions. | Execution journals, market-data streams, audit feeds, rebuildable projections. |
| Main failure shape | Poison item, visibility expiry, duplicate work, queue growth, unfair tenant use. | Silent no-match, target-specific delivery failure, divergent consumers, hidden event cascade. | Hot partition, lagging consumer, replay storm, retention overrun, skew, checkpoint error. |

### Bounded AWS examples, not one-to-one mappings

- **Competing work:** SQS Standard is an at-least-once queue and can deliver
  duplicates or out of send order (C02; A09, retrieved 2026-08-22). With Lambda,
  batch processing/acknowledgement failure can cause records to appear again
  (A02, retrieved 2026-08-22). SQS FIFO adds strict order within a
  `MessageGroupId` and bounded
  producer-side deduplication, not one-time database or API effects
  (C03-C04; A10, retrieved 2026-08-22).
- **Routing/fan-out:** EventBridge buses evaluate events against rules and send
  matches to targets; SNS topics deliver publications to subscribers (C60;
  A07,A83, retrieved 2026-08-22). A target may itself be a queue. Neither
  “router” nor “topic” establishes replay, global order, or business completion.
- **Ordered retained consumption:** Kinesis Data Streams contains ordered shard
  sequences and maps partition keys to shards (C10,C42; A14, retrieved
  2026-08-22). Kafka on MSK supplies partitioned-log and consumer-group patterns;
  Kafka transactions remain bounded to supported Kafka operations and do not
  atomically include an arbitrary external database or API (C29/A30/F28; A30
  and F28 retrieved 2026-08-22).

A sound design often composes these: a router gives each consumer its own work
queue for isolation; a retained log feeds several independent projections; a
log consumer creates work-queue items for expensive parallel tasks. This is why
“event bus versus queue versus stream” is not answerable until work ownership,
replay, order, and overload have been stated.

## Guarantees: delivery, processing, and business effect

Use three planes. Never let a guarantee jump from one plane to the next without
an explicit mechanism.

| Plane | The question | Evidence required |
|---|---|---|
| **Delivery** | How often can this named channel attempt to hand this message to this named receiver, and until what deadline/retention? | Producer result, channel identity, delivery acknowledgement/visibility/checkpoint, retry age/attempts, terminal destination. |
| **Processing** | How many handler attempts and commits can occur for one logical identity across timeout, crash, retry, and rebalance? | Stable event/command ID, transaction boundary, checkpoint timing, inbox/deduplication behavior, ambiguous-timeout handling. |
| **Business effect** | Can one logical input create more or fewer externally meaningful outcomes than intended? | Authoritative state transition, idempotency record committed with it where possible, external provider key/receipt, reconciliation and correction evidence. |

### Delivery vocabulary is boundary-scoped

- **At-most-once delivery:** the named boundary does not redeliver after its
  attempt, so loss or non-processing is possible. It says nothing about code
  that independently repeats an effect.
- **At-least-once delivery:** the boundary retries/redelivers until its success
  or terminal condition, so duplicates are possible. It does not promise that
  a valid consumer will eventually succeed before retention expires.
- **Exactly once:** admissible only after naming operation, scope, identity key,
  time window, transaction, and external side effects (C01/F01). Product terms
  such as FIFO send deduplication or workflow execution must be repeated with
  their documented boundary, never promoted to “the system is exactly once.”

The practical default is: **at-least-once delivery plus an idempotent named
business effect**. Business idempotency means the same logical identity produces
one intended transition and a compatible response; a broker ID or time-limited
cache alone is insufficient (C40/F02,F04,F11).

### A crash-window worksheet

For each consumer, enumerate these states:

1. message received, no authoritative commit;
2. authoritative commit succeeded, acknowledgement/checkpoint not recorded;
3. acknowledgement recorded, optional external effect pending;
4. external effect timed out ambiguously;
5. terminal retry or poison quarantine reached.

Then answer: what repeats, what durable identity exists, who retries, how a
duplicate becomes a compatible no-op, and which reconciliation proves the
outcome. If the database mutation and inbox record can be in one local
transaction, use that boundary. If an external venue, email provider, or other
store cannot join it, use its idempotency contract where available and retain a
recoverable state machine; reconcile independently.

### Repository guarantee audit

**Observed:** the order handler stores a cached `201` before publication, and a
same-key retry can return it without evidence of valid routing
([CS02](01-repository-case-study.md#cs02--the-idempotency-cache-can-certify-an-order-with-no-proven-routing)).
Three `PutEvents` callers ignore per-entry results
([CS03](01-repository-case-study.md#cs03--putevents-response-and-configuration-boundaries-are-ignored)).
Trade creation, order update, and publication are separate operations; the
duplicate-trade guard can block repair after a late publication failure
([CS04](01-repository-case-study.md#cs04--trade-creation-order-transition-and-publication-are-separate-operations),
[CS05](01-repository-case-study.md#cs05--publication-failure-is-blocked-by-the-duplicate-trade-guard-on-retry)).

**Documented:** `PutEvents` can return HTTP 200 with failed entries. Even HTTP
200, zero failed entries, and an `EventId` are only producer-API acknowledgement;
AWS documents a nonexistent-bus case in which those signals accompany a dropped
event (C08/A06,A82, retrieved 2026-08-22).

**Inference:** no existing producer, channel, or conditional write establishes
one end-to-end effect. A defendable design transacts state plus outbox, validates
routing configuration independently, retries under stable IDs, transacts
consumer inbox plus mutation, and reconciles orders, executions, ledger postings,
and projections.

## Ordering, concurrency, backpressure, overload, and fairness

### Order is a business relation, not arrival time

Name at least four orders:

1. **source order:** the authority assigns aggregate version 41 after 40;
2. **transport order:** a channel returns records in a documented key/partition
   scope;
3. **processing order:** concurrency and retries determine which handler starts
   or completes first;
4. **commit/effect order:** database transactions and external systems decide
   which state becomes visible first.

Transport order alone does not prove causal business order. A consumer needs an
expected version rule: lower is duplicate/stale, equal/next is applicable, and
higher reveals a gap. Ordering does not remove duplicates, and deduplication
does not establish causal/version order (C41).

Per-key serialization trades throughput for a smaller reasoning boundary. One
partition or writer can create order for an account, order, or instrument;
additional partitions increase parallelism but require coordination or
reconciliation where invariants span keys (C34/F01,F09,F24). Global
serialization is possible only by accepting a common bottleneck or equivalent
coordination—not by declaring that global order is impossible.

### Head-of-line blocking and skew

Strict order means a slow or poison item can stop later work in its lane. If all
orders share one FIFO group or partition, one failure creates global
head-of-line blocking. If groups are too fine, related transitions can race.

A good key preserves the narrowest necessary invariant:

- `orderId` for one order state machine;
- `accountId` where positions/cash within one account require sequence;
- `instrument` or order-book partition for matcher price-time sequencing;
- a workflow ID for settlement steps.

Measure the distribution, not only cardinality. A popular symbol, institutional
account, or noisy tenant can create a hot partition even with millions of keys.
Salting a key increases parallelism but destroys straightforward order, so it
needs an explicit merge/coordination rule.

### Backpressure and lag budgets

Let average arrival rate be `λ`, sustainable service rate be `μ`, and backlog be
`B`. If `μ <= λ`, the backlog cannot drain. When `μ > λ`, the idealized minimum
drain time is:

```text
drain time = B / (μ - λ)
```

Real recovery is slower because retries, skew, downstream throttles, deployment,
and poison records consume capacity. A lag budget therefore includes:

- maximum normal oldest-item age and per-consumer freshness;
- peak/outage duration to absorb;
- spare capacity and maximum recovery time;
- retention margin beyond the recovery time;
- priority/tenant-specific budgets where aggregate lag hides starvation;
- an admission or shedding action before the budget is exhausted.

Backpressure is a design decision: bound concurrency/backlog, defer or shed
lower-value work, protect downstream capacity, and monitor age/lag. Autoscaling
alone can amplify overload or move it to a hard downstream limit (C44/F03,F13,F14).
Retries need a deadline, bounded attempts, exponential backoff, jitter, and one
clear owner where possible; retries at every layer multiply load (C45/F03,F11,F12).

### Fairness is separate from throughput and FIFO

A system can meet aggregate throughput while a quiet retail account waits
behind one institutional tenant or while low-priority reconciliation starves
forever. State the scheduling policy: FIFO, weighted fair share, per-tenant
quota, reserved lanes, strict priority with aging, or isolated capacity. Measure
oldest age and rejection rate *per class*, not just total depth (C61/F03,F13,F14).

Possible controls include:

- per-tenant or per-priority queues/partitions with bounded worker shares;
- reserved concurrency/capacity for critical order and reconciliation paths;
- producer quotas and admission control before shared storage is saturated;
- maximum work age, retry budgets, and poison quarantine;
- paced replay so recovery traffic cannot starve live traffic;
- load shedding for replaceable notifications or stale derived work when the
  business contract permits it—never silently for accepted orders or ledger
  postings.

### Overload review questions

1. Which downstream resource saturates first: compute, database partitions,
   connections, external API quota, or operator capacity?
2. Is the buffer sized for a burst or being used to deny sustained overload?
3. What is the oldest age by key/tenant/priority, and when is new work rejected?
4. Can a poison item block a FIFO lane? How is it quarantined without violating
   order?
5. How long will a replay take, and what live-work capacity remains?
6. Which accepted work may never be shed, and which notification/projection can
   be regenerated under an explicit contract?

In the repository, the portfolio applies arrival order without a source
sequence. **Inference:** retries and concurrency can produce nondeterministic
positions; adding an ordered transport can simplify delivery but does not
replace inbox idempotency, expected versions, gap repair, and reconciliation
([CS06](01-repository-case-study.md#cs06--optimistic-locking-does-not-make-the-portfolio-consumer-idempotent),
[CS07](01-repository-case-study.md#cs07--the-portfolio-has-an-ordering-requirement-but-no-event-sequence)).

## Event contracts and schema evolution

An event contract is not merely a TypeScript interface or JSON schema. It binds
identity, business meaning, units, authority, compatibility, privacy, and
operational behavior. Compatibility is a relationship between writer and reader
versions, including retained records replayed into new code (C59/F01,F25,F26).

### A useful envelope

Keep transport metadata distinct from domain payload. A typical envelope has:

| Field | Purpose / rule |
|---|---|
| `eventId` | Stable identity for this fact across retries and republish. Do not regenerate it during redrive. |
| `eventType` | Stable past-tense semantic name such as `TradeExecuted`; routing name is not a command disguise. |
| `source` | Authority/producer namespace, not merely the current function name. |
| `subject` / aggregate ID | The business entity to which the fact applies. |
| `schemaVersion` | Schema contract version; it does not replace compatibility policy. |
| `aggregateVersion` / sequence | Source-assigned order within the named aggregate/key; not a timestamp substitute. |
| `occurredAt` | When the authoritative fact occurred. Separate from publish/receive/process timestamps. |
| `correlationId` | Groups a customer request or business journey across messages. It is not a deduplication key. |
| `causationId` | Identifies the command/event that directly caused this event; useful for cascade diagnosis. |
| `traceContext` | Technical distributed-trace propagation. Sampling must not erase business correlation. |
| `data` | Versioned domain payload with exact units, currency/scale, nullable/optional meaning, and privacy class. |

CloudEvents standardizes common event metadata; AsyncAPI can describe channels,
messages, bindings, and schemas. Neither standard chooses domain authority,
guarantees semantic compatibility, or makes effects exactly once (C37,C59;
F25,F26).

### Compatibility rules

Evaluate both directions explicitly:

- **Backward compatibility:** a new reader can understand data written by the
  old writer, including retained history.
- **Forward compatibility:** an old reader can tolerate data from a new writer
  during rolling deployment or independent team release.

Practical rules:

1. Adding an optional field can support mixed versions only when absence has a
   defined meaning/default and old consumers tolerate unknown fields.
2. Removing or renaming a required field, changing type, or narrowing allowed
   values is usually breaking for a deployed reader.
3. Reusing a field with a new unit, currency, sign, rounding rule, identity, or
   business meaning is breaking even if the JSON type is unchanged.
4. Do not mutate a past fact into a new interpretation. Introduce a new event
   type/version, dual-publish or translate at a controlled boundary, observe old
   consumer use, then retire according to policy.
5. Validate envelopes and domain invariants at ingress. Quarantine poison data
   with the original bytes, identity, schema version, error, and ownership.
6. Test old-writer/new-reader, new-writer/old-reader, and replay fixtures from
   each retained version. A registry check alone cannot test business meaning.
7. Minimize sensitive event-carried state. Encryption does not remove deletion,
   access, purpose, retention, or non-production replay obligations.

### Consumer-driven risk

Independent consumers are easy to add and easy to forget. Maintain an owner and
compatibility window for each event type, discover actual subscriptions, and
record which fields consumers rely on. Consumer-driven contract tests can catch
known syntactic dependencies, but a producer still owns coherent domain meaning
and cannot outsource it to every downstream implementation.

The repository envelope has a useful `correlationId`, and `TradeExecuted`
contains a stable `tradeId`. **Observed:** it lacks causation ID, schema version,
authoritative aggregate sequence, currency, and scale. **Inference:** these gaps
prevent deterministic projection ordering and safe monetary evolution; adding
fields still requires mixed-version and replay policy. The current TypeScript
interface is evidence of code shape, not proof that deployed producers and
consumers are compatible.

## Fintech pattern decision table

The rows deliberately describe different business cases rather than assigning
one service to one pattern.

| Case | Authority and invariant | Needed interaction/channel semantics | Failure and recovery design | Poor fit / trap |
|---|---|---|---|---|
| **Order acceptance** | Command authority owns request identity, validation, reservation policy, and legal `ACCEPTED` transition. | Synchronous durable acceptance may enqueue/append publication intent. One command owner; later facts can fan out. | Atomically protect command identity, order, and outbox where local; expose status; reconcile accepted orders to executions/cancels. | Returning `EXECUTED` because a message was accepted; styling a required executor command as an unowned event. |
| **Trade execution / fill** | Matcher or venue adapter owns stable execution ID, per-book/order sequence, partial-fill and correction semantics. | Capacity-controlled single writer per ordering scope plus retained journal/integration facts; independent consumers may replay. | Idempotent execution ingestion, conditional order version, outbox, deterministic recovery and venue reconciliation. | General fan-out bus as price-time sequencer; global order claim; retry that generates a new trade ID or price. |
| **Portfolio projection** | Execution/ledger authority, not the portfolio row, decides ownership and cost inputs. | Event-carried state plus independent consumer position/replay; per-account/aggregate expected version. | Transactional inbox + projection mutation, stale/duplicate/gap paths, freshness SLO, rebuild and control totals. | Using stale projection for pre-trade availability; optimistic locking as duplicate defense. |
| **Settlement** | Settlement process and ledger authorities own obligations, deadlines, cash/securities postings, and irreversible external steps. | Commands with accountable owners plus events for facts; often a visible process manager/saga state rather than an invisible cascade. | Durable step state, bounded retries, compensation where meaningful, manual exception queue, statement/ledger reconciliation. | Treating compensation as rollback or assuming pub/sub fan-out proves every party completed. |
| **Client notification** | The underlying order/execution/settlement authority owns the fact; notification state owns its own recipient contract. | Event notification, then per-channel work queues/routing as needed. Loss/retry/deduplication follow channel and product contract. | Stable notification intent, preference/consent, provider receipt where available, bounded retries, suppression, audit and support status. | Making email/SMS proof of execution; assuming notification means best-effort; retrying without recipient-level deduplication. |
| **Compliance/audit** | Ledger/execution/order authorities own facts; audit evidence has governed retention and integrity. | Retained append-oriented records and independently replayable projections/search. | Completeness control totals, immutable/integrity-validated evidence, access/retention policy, replay drills and discrepancy resolution. | Treating an event-bus archive, DLQ, mutable balance, or incomplete daily scan as a financial ledger. |

The repository demonstrates every trap in compressed form: non-atomic order and
trade publication (CS01, CS04-CS05), response-cache idempotency (CS02), ignored
producer results (CS03), unsequenced/non-idempotent projection (CS06-CS07),
unusable DLQ paths (CS08), incomplete snapshot (CS09), floating-point money
(CS10), no apparent ledger/reconciliation authority (CS11), and an unjustified
Lambda matcher (CS12). The full evidence and remedies remain in the
[repository case study](01-repository-case-study.md); later chapters should
reference those IDs rather than restating observations as service guarantees.

## Metrics that reveal the real failure mode

One dashboard should not collapse the planes.

| Plane | Minimum signals | What a healthy aggregate can hide |
|---|---|---|
| Producer | accepted commands, rejected/limited commands, publish API failures, per-entry failures, outbox count/oldest age | An HTTP 200 with failed entries or invalid routing; accepted state with no publication intent. |
| Transport | ingress/egress rate, oldest message/record age, backlog/lag, retries, throttles, no-match/drop, DLQ writes and failed DLQ writes | One hot key, tenant starvation, retention expiry, target-specific failure. |
| Consumer | attempts, success/error/timeout, duration, concurrency, batch reprocessing, poison/schema failures, checkpoint progress | A handler success after a duplicate effect; one partition stopped while others advance. |
| Ordering/contract | duplicate IDs, stale versions, sequence gaps and gap age, unknown versions/types, compatibility/quarantine counts | Silent semantic misinterpretation of unchanged JSON fields. |
| Business correctness | accepted-without-execution age, duplicate fills, execution-without-posting, ledger imbalance, reservation mismatch, settlement breaks, projection discrepancy/freshness | Green availability and empty DLQs alongside incorrect money or holdings (C48). |
| Recovery | replay/redrive rate, estimated drain time, live-versus-recovery capacity, records classified, reconciliation completion and unresolved aging | An empty queue after destructive or incomplete replay. |
| Fairness | wait/oldest age, admission/rejection, throughput and concurrency by tenant/key/priority | Aggregate p50/p95 and total throughput while a quiet class starves (C61). |

Alert on user/business risk, not merely resource use. Queue depth without oldest
age is ambiguous; zero DLQ depth is unsafe if the channel lacks permission to
write it. In this repository, EventBridge's
`InvocationsFailedToBeSentToDLQ` must be alarmed separately because the declared
queues lack required resource policies (C57/A81, retrieved 2026-08-22;
[CS08](01-repository-case-study.md#cs08--the-target-dlqs-lack-write-permission-and-cover-a-separate-boundary)).

## Interview follow-ups

1. What changes if the API must return accepted in 100 ms but reservation must
   be complete before acceptance?
2. Why does reversing “database then publish” to “publish then database” only
   move the dual-write gap?
3. Can an SQS FIFO message group make a portfolio update exactly once? Name the
   send, delivery, consumer transaction, and external-effect boundaries.
4. If two fills for one account arrive out of order, which source sequence lets
   the projector distinguish duplicate, stale, next, and gap?
5. When would event-carried state transfer be better than event notification,
   and what privacy/rebuild costs does it add?
6. Is a Kinesis or Kafka log an event store? Under what authority, retention,
   schema, and replay conditions could it become one?
7. How do you prevent a replay of historic executions from sending customer
   notifications twice?
8. A queue grows for 20 minutes and then input stops. What variables determine
   whether it drains before retention or the business deadline?
9. How can aggregate throughput be healthy while retail orders are unfairly
   delayed?
10. Which schema changes are syntactically compatible but financially breaking?
    Consider cents-to-micros, currency, sign, and partial-fill semantics.
11. When should a settlement flow use choreography, and when should it expose a
    process manager?
12. Which metric proves a trade is financially complete? Why are function
    success, an empty queue, and a sent notification insufficient?

## Concise senior articulation

> I start with authority and invariants, then choose the asynchronous boundary.
> A queue assigns work, a router fans out or filters, and a log retains ordered
> partitions for independent replay, but managed services can combine those
> traits. I name ordering by key and guarantee by boundary: producer
> acknowledgement, transport delivery, consumer commit, and business effect are
> separate. At-least-once delivery normally means stable identities,
> transactional inbox/outbox where possible, version checks, and reconciliation;
> FIFO or broker deduplication does not make external effects exactly once. I
> design overload explicitly with lag budgets, admission control, fairness, and
> replay capacity. Finally I version both syntax and meaning, test mixed
> producers/consumers plus retained-event replay, and keep authoritative
> reservations, executions, and ledger postings separate from stale portfolios,
> notifications, and analytics. Only after those decisions do I choose the AWS
> service and mode.

## Vocabulary contract for later chapters

Later chapters use the following without redefining them:

- command = requested action; event = past fact; message = envelope;
- notification loss/durability = explicit transport and business contract, not
  an intrinsic property of the word;
- event-carried state transfer, event collaboration, and event sourcing =
  separate dimensions;
- queue = competing work, router = independent routing/fan-out, log = retained
  independently positioned replay, subject to concrete service semantics;
- delivery, processing, and business-effect guarantees = separate boundaries;
- order = named key/scope plus source version and failure behavior;
- backpressure = admission/capacity policy; lag budget = normal plus recovery
  freshness; fairness = per-class isolation/scheduling policy;
- replay = controlled reprocessing; reconciliation = independent proof and
  discrepancy resolution;
- authoritative state, ledger, and projection = explicitly scoped roles.

The canonical forms and prohibited shorthand are in
[terminology](../research/terminology.md). Service chapters may add
product-specific terms, but must not broaden these guarantees.

## Sources used in this chapter

Only the sources below support this chapter. Claim wording and retrieval status
are controlled in the [claim register](../research/claim-register.md); full
metadata is in the [source ledger](../research/source-ledger.md).

### Conceptual and pattern sources

- F01 — Kleppmann and Riccomini, [*Designing Data-Intensive Applications*, 2nd
  ed.](https://www.oreilly.com/library/view/designing-data-intensive-applications/9781098119058/),
  “Encoding and Evolution” and “Stream Processing.”
- F02 — Hohpe and Woolf, [*Enterprise Integration
  Patterns*](https://www.enterpriseintegrationpatterns.com/), message channels,
  competing consumers, publish-subscribe, idempotent receiver, and dead letter.
- F03 — Nygard, [*Release It!*, 2nd
  ed.](https://pragprog.com/titles/mnee2/release-it-second-edition/), stability
  antipatterns and patterns.
- F04 — Richardson, [Transactional
  Outbox](https://microservices.io/patterns/data/transactional-outbox.html).
- F08-F09 — Fowler, [Accounting
  Patterns](https://martinfowler.com/eaaDev/AccountingNarrative.html) and [The
  LMAX Architecture](https://martinfowler.com/articles/lmax.html), for scoped
  ledger/authority and single-writer reasoning.
- F07, F29, F30 — Fowler, [Event
  Sourcing](https://martinfowler.com/eaaDev/EventSourcing.html), [What do you mean
  by “Event-Driven”?](https://martinfowler.com/articles/201701-event-driven.html),
  and [Event Collaboration](https://martinfowler.com/eaaDev/EventCollaboration.html).
- F11-F13 — Amazon Builders' Library, [Making retries safe with idempotent
  APIs](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/),
  [Timeouts, retries, and backoff with
  jitter](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/),
  and [Avoiding insurmountable queue
  backlogs](https://aws.amazon.com/builders-library/avoiding-insurmountable-queue-backlogs/).
- F14 — Google, [*Site Reliability
  Engineering*](https://sre.google/sre-book/table-of-contents/), for overload and
  SLO framing.
- F15-F16 — Google, [*The Site Reliability
  Workbook*](https://sre.google/workbook/table-of-contents/) and [Service Level
  Objectives](https://sre.google/sre-book/service-level-objectives/).
- F17-F18 — Harris, [*Trading and
  Exchanges*](https://global.oup.com/academic/product/trading-and-exchanges-9780195144703),
  and FIX Trading Community, [FIXimate FIX.Latest top-level
  specification](https://fiximate.fixtrading.org/en/FIX.Latest/fix.html) (F18,
  retrieved 2026-08-24), for trading-domain authority boundaries.
- F24 — Elhemali et al., [Amazon DynamoDB: A Scalable, Predictably Performant,
  and Fully Managed NoSQL Database
  Service](https://www.usenix.org/conference/atc22/presentation/elhemali), for
  partitioning/concurrency trade-offs.
- F25-F26 — [CloudEvents
  specification](https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md)
  and [AsyncAPI
  specification](https://www.asyncapi.com/docs/reference/specification/latest).
- F28 — Apache Kafka, [message delivery
  semantics](https://kafka.apache.org/documentation/#semantics).

### Current AWS examples (retrieved 2026-08-22)

- A02, A09-A10 — [Lambda with
  SQS](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html), [SQS Standard
  queues](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues.html),
  and [SQS FIFO key
  terms](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-key-terms.html).
- A12, A28 — SNS [FIFO ordering and
  deduplication](https://docs.aws.amazon.com/sns/latest/dg/fifo-message-ordering.html)
  and S3 [event-notification
  considerations](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html),
  used only in the cross-source ordering-versus-deduplication boundary (C41).
- A06-A07, A81-A82 — EventBridge [`PutEvents` API](https://docs.aws.amazon.com/eventbridge/latest/APIReference/API_PutEvents.html),
  [event bus concepts](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus.html),
  [target DLQs](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rule-dlq.html),
  and [sending with `PutEvents`](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html).
- A14 — Kinesis Data Streams [terminology and
  concepts](https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html).
- A20-A21 — DynamoDB [transactions](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/transactions.html)
  and [Streams change data
  capture](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/streamsmain.html).
- A24 — DynamoDB [global-table consistency
  modes](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/V2globaltables_HowItWorks.html),
  used only through the controlled authority/projection claim.
- A26 — Step Functions [error
  handling](https://docs.aws.amazon.com/step-functions/latest/dg/concepts-error-handling.html),
  used only through the bounded retry claim.
- A30 — [Amazon MSK Developer
  Guide](https://docs.aws.amazon.com/msk/latest/developerguide/what-is-msk.html).
- A32-A35 — [OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/what-is.html),
  [DAX](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DAX.html),
  [ElastiCache](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/WhatIs.html),
  and [Aurora replication](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Replication.html),
  used only to bound derived read stores from authoritative financial state.
- A41, A43 — CloudTrail [log-file integrity
  validation](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-log-file-validation-intro.html)
  and the AWS Well-Architected [Reliability
  pillar](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/welcome.html).
- A83 — [What is Amazon
  SNS?](https://docs.aws.amazon.com/sns/latest/dg/welcome.html).

Repository facts come only from the exact local routes in
[the evidence map](../research/repository-evidence.md). No local note or
repository comment is treated as authority for service behavior.