Repository case study: from event demo to financially correct system

Reading notesDownload original Markdown ↗

You have now seen the intended authority and recovery boundaries. Apply them to a compact implementation that deliberately exposes common failure windows: an order API writes state, publishes events, updates a portfolio, and sends notifications. This is the point in the guide to inspect all twelve CS findings in detail. The original filename begins with 01 because of the source package's numbering; here the case study follows the concepts needed to evaluate it. Code paths and line numbers refer to the reviewed reference snapshot.

Evidence notation: C identifies a claim in the claim register, A a dated AWS source, F a foundational source, and CS a finding in the repository case study. The source index supplies the full source details. These labels are lookup aids, not facts to memorize.

This chapter reviews the read-only fintech repository as an interview case study. It is not a claim that the repository is production software. The useful question is: where does each guarantee stop, what state remains after a failure, and what would make recovery provable?

All code paths are relative to the reference repository. “Observed” means the code says it. “Documented” means the statement follows a controlled AWS or pattern claim. “Inference” is an architecture conclusion, never a service guarantee. The full evidence inventory is in the repository evidence map.

Two-minute architecture walk-through

1. POST /orders synchronously invokes order-handler. The handler validates a command, writes a PENDING order, optionally caches a 201 by idempotency key, then calls EventBridge (code/lambdas/order-handler/handler.ts:114-212).
2. EventBridge routes trading.OrderPlaced asynchronously to trade-executor (infra/environments/production/main.tf:345-359). The handler simulates a fill, inserts a trade, updates the order to EXECUTED, then publishes trading.TradeExecuted (code/lambdas/trade-executor/handler.ts:65-148).
3. Two independent EventBridge targets update a mutable portfolio projection and publish an SNS notification (infra/environments/production/main.tf:360-370,565-615).
4. GET /trades reads the user/time GSI, while a daily Lambda scans portfolio rows into S3 JSONL (code/lambdas/trades-reader/handler.ts:45-97; code/lambdas/portfolio-snapshot/handler.ts:64-107).
5. Terraform declares target SQS DLQ ARNs and a bus archive, but neither DLQ has the queue resource policy required for EventBridge sqs:SendMessage. A manually invoked replayer can read only messages that actually reached a queue (infra/modules/eventbridge/main.tf:15-95; infra/environments/production/main.tf:575-605; code/lambdas/dlq-replayer/handler.ts:61-145).

Diagram text source

flowchart LR
    Client[Client] -->|POST /orders| API[API Gateway]
    API --> OH[order-handler]
    OH --> Main[(DynamoDB: orders/trades/portfolios)]
    OH --> Idem[(DynamoDB: idempotency cache)]
    OH -->|PutEvents| EB{EventBridge bus}
    EB -. OrderPlaced .-> TE[trade-executor]
    TE --> Main
    TE -->|PutEvents| EB
    EB -. TradeExecuted .-> PU[portfolio-updater]
    PU --> Main
    EB -. TradeExecuted .-> NS[notification-sender]
    NS --> SNS[SNS]
    Client -->|GET /trades| API
    API --> Reader[trades-reader]
    Reader -->|GSI query| Main
    Timer[Daily schedule] -.-> Snapshot[portfolio-snapshot]
    Snapshot -->|Scan| Main
    Snapshot --> S3[(S3 JSONL)]
    EB -. target-delivery failure .-> DLQGap{{DLQ send denied:<br/>queue resource policy absent}}
    DLQGap -. InvocationsFailedToBeSentToDLQ .-> Metric[CloudWatch metric]
    DLQGap -. after required policy only .-> DLQ[[Declared SQS DLQs]]
    Operator[Operator] --> Replayer[dlq-replayer]
    DLQ -->|Receive/Delete only if populated| Replayer
    Replayer -->|PutEvents| EB

Solid arrows represent synchronous calls made by the current caller. Dashed arrows are scheduled or broker-driven handoffs. A successful arrow proves only its own boundary.

What is authoritative?

The order row is the apparent operational authority for order status, and the trade row is the only persisted execution record. That is an observation about this code, not proof of financial authority. The portfolio row, GSI query, snapshot, and notification are derived views. EventBridge and its archive are transport/recovery mechanisms, not a ledger. The declared SQS DLQs are not even shown writable by EventBridge in the current Terraform (CS08/C57).

Inference: no code here establishes pre-trade cash or securities reservation, clearing/settlement, balanced postings, corrections/reversals, or independent reconciliation. Therefore the mutable portfolio must not decide available balance or ownership. Authoritative financial state and projections must remain distinct (C46-C47; A32-A35,A41; F08,F17,F18).

Guarantee boundaries

Boundary	Success establishes	Boundary ends before
API Gateway / Lambda response	The HTTP API integration returned that response.	Matching, execution, ledgering, portfolio update, and notification (C51/A53).
DynamoDB order write	The order item write succeeded.	A PutEvents producer acknowledgement, valid routing, or downstream progress. A DynamoDB transaction cannot include EventBridge (C22/A20).
PutEvents	HTTP 200, zero failed entries, and EventId are producer-API acknowledgement for the named request.	Valid bus existence/configuration, matching/routing, target delivery, consumer durability, or handler effects. AWS documents that a nonexistent bus can return those response signals and drop the event (C08/A06,A82).
EventBridge target delivery	EventBridge owns delivery attempts at this boundary (C05/A05).	Lambda asynchronous runtime retry/business rollback, or a working target DLQ. The current queues lack the required EventBridge send policy (C57/A81).
Lambda handler invocation	Code ran until it returned or failed.	Atomicity across multiple SDK calls and one-time external effects.
DynamoDB conditional portfolio write	No other writer changed that item version between read and write.	Event novelty and causal order (C40-C41/F11).
Archive or DLQ	The archive is configured for bounded retention; an SQS message is retained only if EventBridge can write it.	A configured DLQ ARN, zero queue depth, diagnosis, safe replay, idempotent repair, or reconciliation. Current DLQ sends can fail because the queue policies are absent (C43,C57/A13,A36,A43,A81).
Business completion	No as-built boundary proves execution plus balanced monetary/securities postings and reconciled projections.	This requires an explicit authoritative design (C46-C48).

Correctness and scalability review

Use the current architecture as a timeline, not as a list of AWS services. The first five findings follow acceptance and publication through separate commits; the next two examine repeated and reordered portfolio effects; the remaining findings cover failed recovery paths, incomplete export, money, domain authority, and compute suitability. For each, compare the observed code with the proposed repair and its remaining limit.

Each issue includes the failure trigger, resulting state, retry behavior, customer/financial effect, detection, recovery, a practical remedy, and what still remains risky.

CS01 — Order state and event publication are a non-atomic dual write

• Observed evidence: order-handler writes the order first (code/lambdas/order-handler/handler.ts:155-165), writes the response cache separately (:167-181), and only then calls EventBridge (:183-208).
• Documented boundary: DynamoDB transactions cover supported DynamoDB items in one account and Region; they do not include EventBridge (C22/A20). A transactional outbox commits business state and an outbox record locally, while publication and consumption remain retryable (C38/A20,A21/F04).
• Trigger: the order write succeeds and the process crashes, the EventBridge call throws, times out ambiguously, or returns a failed entry.
• Resulting state: a durable PENDING order exists without a durable route to execution. The client can receive 500; without a status/recovery process, the order can remain stranded.
• Does retry repair it? Not reliably. No key creates another order; the same key can hit CS02; an ambiguous timeout can also mean the producer API processed the request, so blind republish risks duplicates even though routing and downstream progress remain unknown.
• Impact: a customer sees failure while a command is stored, or later sees an order that never advances. This is both a correctness and support risk.
• Detection: alarm on age/count of PENDING orders, compare accepted orders with outbox/publication acknowledgements, and reconcile order transitions to execution records. The current publication-error log cannot detect an HTTP 200 partial failure.
• Recovery: classify producer acknowledgement separately from validated bus configuration and downstream progress; republish with a stable event ID only through an idempotent consumer, then reconcile the resulting transition.
• Practical remedy: minimally persist publicationStatus and drive a repair worker. Prefer a DynamoDB transaction containing order, idempotency result, and outbox item, then relay from DynamoDB Streams.
• Residual risk: Streams/outbox closes the local database/message gap, not end-to-end exactly-once processing. Relay and consumer retries still demand stable identity, idempotency, alarms, and reconciliation.

CS02 — The idempotency cache can certify an order with no proven routing

• Observed evidence: the handler stores a cached 201 before publication (code/lambdas/order-handler/handler.ts:167-180) and returns the cached value before processing on retry (:229-241). Its comment tells the caller to retry with a new key after publication failure (:11-17), which defeats logical command deduplication.
• Trigger: the cache write succeeds and the producer call throws, has a failed entry, or returns a nominal acknowledgement for a misconfigured or nonexistent bus.
• Resulting state: the original request returns 500, but a same-key retry returns cached 201 even though valid bus routing and downstream progress are unproven. A new-key retry creates a second order.
• Does retry repair it? No. Same-key retry skips publication; new-key retry changes command identity.
• Impact: false success or duplicate orders, with ambiguous customer intent.
• Detection: compare every live idempotency record with its order and durable outbox/publication state; track same customer payload under multiple keys.
• Recovery: do not merely delete the cache and replay. Validate the named bus and rules independently, preserve the original order/event identity, recover through the durable publication path, and reconcile downstream state.
• Practical remedy: atomically transact the idempotency key, canonical request hash, order, and outbox. Store an explicit state such as IN_PROGRESS/ACCEPTED, and make same-key/different-payload reuse a conflict. Business idempotency protects the mutation and response together (C40/F11).
• Residual risk: TTL deletion is asynchronous and is only cleanup (C32/A23). The product still needs a documented key-retention window and behavior for late retries.

CS03 — PutEvents response and configuration boundaries are ignored

• Observed evidence: order-handler (code/lambdas/order-handler/handler.ts:193-205), trade-executor (code/lambdas/trade-executor/handler.ts:134-145), and dlq-replayer (code/lambdas/dlq-replayer/handler.ts:114-133) await the SDK call but discard its response. The replayer then deletes the DLQ message.
• Documented boundary: PutEvents can return HTTP 200 with per-entry failures. Even HTTP 200, FailedEntryCount: 0, and EventId are only producer-API acknowledgement: AWS documents that a nonexistent bus can return those signals while event matching finds no rule and drops the event (C08/A06,A82).
• Trigger: EventBridge reports a failed entry under HTTP 200, or the caller names a nonexistent/misconfigured bus and receives a nominally successful producer response.
• Resulting state: the order path logs success and returns 201, the trade path logs completion without downstream events, or the replayer deletes the only DLQ copy despite having no evidence of valid routing or downstream progress.
• Does retry repair it? No exception is thrown, so these handlers return successfully and their normal retry owner sees nothing to retry. Blind retry against the same bad bus configuration only repeats the drop.
• Impact: silently stranded orders, stale portfolios, missing notifications, and destructive redrive loss.
• Detection: emit FailedEntryCount and per-error-code metrics; validate the configured bus ARN/name and expected rules during deployment and before redrive; reconcile orders/trades/outbox state with downstream effects. An EventId is not the reconciliation signal.
• Recovery: locate the authoritative source record or retained DLQ body, validate/fix the bus and rule configuration, republish under the stable event identity, and verify downstream business state through reconciliation. The current replayer can make recovery impossible after deletion.
• Practical remedy: inspect every response entry and treat a failed entry as failed. A replayer deletes the SQS message only after (1) the entry has no error, (2) bus identity/configuration has been independently validated, and (3) a durable replay audit record preserves the source message/event identity for downstream reconciliation. EventId alone is not proof of bus durability or business progress.
• Residual risk: a client-side timeout remains ambiguous, and validated configuration can drift later. Durable intent, idempotent consumers, deployment checks, and reconciliation remain required.

CS04 — Trade creation, order transition, and publication are separate operations

• Observed evidence: trade-executor conditionally inserts the trade (code/lambdas/trade-executor/handler.ts:83-104), separately updates the order (:106-118), and separately publishes (:120-145). The order update does not require a previous PENDING state.
• Trigger: a crash or rejected operation between any two steps, concurrent order mutation, or a partial PutEvents failure.
• Resulting state: possible states include trade-present/order-pending; trade-present/order-executed/event-missing; or an order transition applied without a validated state-machine predecessor.
• Does retry repair it? Usually not. After the trade exists, the first conditional put fails before the invocation can repair the order or publish.
• Impact: customer-visible status disagrees with execution data; downstream holdings and notifications can be missing; investigation cannot rely on one completion flag.
• Detection: continuously compare accepted orders, trade records, valid order-state transitions, outbox records, ledger postings, and projection checkpoints.
• Recovery: use a repair command that reads the persisted trade and current order state, applies only a legal missing transition, emits the stable event, and records the repair for reconciliation.
• Practical remedy: use TransactWriteItems for the trade record, a conditional order-state transition, and an outbox item. Model partial fills as stable execution IDs rather than assuming one trade per order.
• Residual risk: that local transaction cannot atomically include an external matching engine. Its execution report needs stable identity, idempotent ingestion, and reconciliation.

CS05 — Publication failure is blocked by the duplicate-trade guard on retry

• Observed evidence: each attempt generates a new trade ID and price, but inserts at PK = TRADE#<orderId> with attribute_not_exists(PK) (code/lambdas/trade-executor/handler.ts:79-104). Publication occurs only after that insert and the order update (:120-145).
• Trigger: the trade and order writes succeed, then publication throws; or publication is ambiguous and Lambda retries.
• Resulting state: the trade exists and the order says EXECUTED, but the TradeExecuted event may be absent. The next attempt fails at the duplicate trade put before it reaches publication.
• Does retry repair it? No. The write guard prevents a duplicate trade row but also prevents retry from finishing the workflow.
• Impact: stale holdings and missing client notification despite an executed order. Repeated retries add load and can terminate at the wrong failure story.
• Detection: compare each execution/trade ID with a durable outbox record and each projector's inbox/checkpoint; alert on executed trades without relay progress or reconciled downstream effects beyond the SLO. Validate bus/rule configuration independently of the producer API response.
• Recovery: reconstruct the event from the stored trade under its original stable tradeId, preserve causation/correlation metadata where available, publish through a recovery outbox, and reconcile projections.
• Practical remedy: create trade, transition order, and create the exact outgoing event in one transaction. The relay retries the outbox independently of trade creation.
• Residual risk: repeated relay delivery remains possible. Every projector and notification workflow still needs an idempotent business effect.

CS06 — Optimistic locking does not make the portfolio consumer idempotent

The publication findings concern whether downstream work exists at all. The next two concern what happens after it arrives. A consumer can process the same fill twice or process distinct fills in the wrong business order; those are different failures. Keep event identity separate from the version used to reject a stale concurrent write.

• Observed evidence: TradeExecuted has a tradeId but no consumer stores it (code/shared/types/events.ts:36-47). portfolio-updater reads holdings, applies every event, then performs only a portfolio-version condition (code/lambdas/portfolio-updater/handler.ts:145-186).
• Trigger: EventBridge/Lambda redelivery, operator replay, a response-path failure after DynamoDB committed, or two envelopes for the same trade.
• Resulting state: a duplicate BUY adds shares and cost twice; a duplicate SELL subtracts twice or later fails for insufficient holdings. The item version advances normally, so the lock sees no duplicate.
• Does retry repair it? No. Retries are the trigger. Optimistic locking only resolves concurrent writers that read the same version.
• Impact: materially incorrect positions, cost basis, and downstream P&L.
• Detection: duplicate tradeId count, ledger-to-portfolio reconciliation, projection version/sequence gaps, and negative/implausible position checks.
• Recovery: rebuild the projection from authoritative executions/ledger postings, or apply a reviewed compensating correction. Replaying the same event into the current handler is unsafe.
• Practical remedy: atomically conditionally insert an inbox item keyed by (projector, tradeId) and update the projection. A duplicate becomes a no-op with a compatible result (C40/F11).
• Residual risk: inbox retention must cover all replay windows; schema bugs may require a versioned rebuild rather than ordinary redrive.

CS07 — The portfolio has an ordering requirement but no event sequence

• Observed evidence: TradeExecuted carries a timestamp but no account, order, or execution sequence (code/shared/types/events.ts:36-47). Independent EventBridge targets route the event to the updater (infra/environments/production/main.tf:565-605). The updater uses the arrival-time portfolio version, not a source sequence (code/lambdas/portfolio-updater/handler.ts:145-178).
• Controlled rule: this package makes no ordering claim for an EventBridge bus. A design that requires order must add a sequence/version rule or ordered mechanism (C09/A07/F01). Ordering and deduplication solve different problems (C41).
• Trigger: two trades for the same account arrive late or out of causal order, especially BUY/SELL or cancel/replace/partial-fill sequences.
• Resulting state: a SELL can be rejected before its preceding BUY arrives; weighted cost basis can differ by arrival order; a later old event can mutate an already newer projection.
• Does retry repair it? Not deterministically. Delay may incidentally change arrival order, but there is no gap detector or expected source version.
• Impact: stale or incorrect positions and nondeterministic recovery.
• Detection: per-account expected-versus-observed sequence, gap age, late event count, projector lag, and reconciliation discrepancy.
• Recovery: quarantine a gap, fetch/replay the missing authoritative sequence in order, rebuild that account projection, then reconcile.
• Practical remedy: assign a monotonic version at the authoritative aggregate/single-writer boundary. The projector conditionally accepts the next expected version and treats lower versions as duplicates/stale. An ordered log keyed by account can simplify delivery order but does not remove idempotency.
• Residual risk: one account key can become hot, and invariants spanning accounts/instruments still need coordination and reconciliation (C34).

CS08 — The target DLQs lack write permission and cover a separate boundary

• Observed evidence: the reusable module creates SQS queues and attaches their ARNs to targets (infra/modules/eventbridge/main.tf:29-38,73-95); production repeats that shape for portfolio-updater (infra/environments/production/main.tf:575-605). A repository-wide Terraform search finds no aws_sqs_queue_policy and no sqs:SendMessage grant. The events.amazonaws.com principals that do exist are Lambda invocation permissions (infra/environments/production/main.tf:386-411,608-615). The Lambda module separately has no async failure destination (infra/modules/lambda/main.tf:12-70), while trade-executor comments collapse handler errors into “EventBridge retries (then DLQ)” (code/lambdas/trade-executor/handler.ts:24-26).
• Documented boundary: programmatically configured target DLQs require an SQS resource policy granting events.amazonaws.com sqs:SendMessage, usually constrained by the rule ARN. EventBridge emits InvocationsFailedToBeSentToDLQ when it cannot place an event in the DLQ (C57/A81). Target delivery, DLQ write, Lambda asynchronous invocation, and event-source mappings remain different boundaries (C05-C07/A01,A03-A05).
• Trigger: target delivery fails and EventBridge attempts to send the event to either declared SQS DLQ, or Lambda accepts the invocation but the function later fails.
• Resulting state: for the first path, SQS denies the DLQ write and the event is in neither the target nor the intended DLQ; queue depth can stay zero while InvocationsFailedToBeSentToDLQ rises. For the second path, the target DLQ does not represent the later handler failure and partial business state can remain.
• Does retry repair it? No ordinary retry grants the missing queue policy. AWS documents some target errors, including missing target permission, that go directly to a configured DLQ without target retries; that DLQ attempt still fails here. Lambda runtime retries are separately owned and cannot roll back an already committed database write.
• Impact: terminal delivery events can be lost despite a declared DLQ and a green zero-depth alarm; operators can also confuse Lambda failures with target failures, causing stale financial projections or repeated effects.
• Detection: alarm on EventBridge target invocation failures, InvocationsSentToDLQ, and especially InvocationsFailedToBeSentToDLQ in addition to SQS depth. Dashboard Lambda errors/throttles/runtime destinations and business-state gaps separately. Add deployment assertions that every DLQ has the exact rule-scoped queue policy.
• Recovery: add and test the queue policy, identify which boundary failed, recover the original event from an authoritative outbox/source or the bounded EventBridge archive, replay through an idempotent path, and reconcile business effects. A missing DLQ message cannot be redriven by dlq-replayer.
• Practical remedy: declare an aws_sqs_queue_policy for every module and portfolio DLQ granting principal events.amazonaws.com, action sqs:SendMessage, and an aws:SourceArn condition for the exact rule. Add the failed-DLQ-send alarm and a deployment test. Separately document/configure Lambda async destinations, or select an SQS event-source mapping where durable work-queue semantics are required.
• Residual risk: queue-policy drift or an incorrect source-ARN condition can reopen the loss path; archive retention is bounded; and a working DLQ remains only a holding area. Safe replay still needs ordering, idempotency, rate limiting, durable audit identity, and reconciliation (C43).

CS09 — The daily snapshot is incomplete and memory-bound at scale

• Observed evidence: the Lambda issues one filtered Scan, ignores LastEvaluatedKey, maps all returned rows in memory, joins one JSONL string, and writes one S3 object (code/lambdas/portfolio-snapshot/handler.ts:64-107).
• Trigger: the table exceeds one Scan response page, or the returned portfolio set approaches Lambda memory/time or S3 request-body constraints.
• Resulting state: later pages are silently absent. The filter does not avoid reading non-portfolio items, and records can change while the scan runs, so the output is not a proven point-in-time snapshot.
• Does retry repair it? No; the same code again reads only one page and may observe another moving view.
• Impact: incomplete audit/analytics data, excess read work, timeouts, and an unreliable basis for reconciliation.
• Detection: treat any LastEvaluatedKey as incomplete; compare source and exported counts/control totals; monitor scanned-versus-returned count, bytes, duration, throttles, and object size.
• Recovery: run a complete, bounded export from a known recovery point; generate manifests and control totals; classify and backfill missing partitions.
• Practical remedy: at minimum paginate and stream bounded chunks to separate objects. For scale, use PITR-backed DynamoDB export to S3 for a selected table point, then catalog/query partitions and reconcile counts and monetary totals. Documented: the export is asynchronous, does not consume table RCUs, and has no completion-time SLA (AWS DynamoDB export documentation, retrieved 2026-08-22).
• Residual risk: export freshness needs an SLO, and storage/query work still costs money. A projection export is not an authoritative financial ledger.

CS10 — Binary floating point is used for money

• Observed evidence: prices are JavaScript number values (code/shared/validation/schemas.ts:21-27), execution price uses floating multiplication and cents rounding (code/lambdas/trade-executor/handler.ts:55-63), cost basis/value perform repeated floating arithmetic (code/lambdas/portfolio-updater/handler.ts:64-84,121-135), and snapshots do the same (code/lambdas/portfolio-snapshot/handler.ts:50-61). No currency or scale accompanies the amounts.
• Controlled rule: authoritative monetary amounts use exact decimal or scaled integers with explicit currency/scale, not binary floating point (C47/A41/F08,F17).
• Trigger: decimal prices/fees/taxes or repeated weighted calculations that are not exactly representable in binary, plus inconsistent rounding stages.
• Resulting state: stored and derived totals can drift; cents rounding can discard sub-cent rules without an explicit product policy.
• Does retry repair it? No. Repeating an unspecified rounding policy does not make it financially correct; the random execution calculation can also differ across attempts before the conditional write.
• Impact: incorrect balances, P&L, statements, and reconciliation breaks.
• Detection: exact control-total reconciliation by currency, property tests for rounding, and boundary cases for fees, partial fills, and large values.
• Recovery: recompute projections from exact authoritative postings under a versioned rounding policy; post explicit corrections rather than silently editing historical amounts.
• Practical remedy: represent amount as scaled integer minor/sub-minor units or a decimal type; carry currency, scale, and rounding mode in schemas; keep execution identity and price deterministic.
• Residual risk: exact representation does not choose the business rounding rule. FX, fees, corporate actions, and instrument-specific tick sizes need governed policies and tests.

CS11 — No append-only financial ledger or reconciliation loop is apparent

• Observed evidence: the declared entity types are mutable orders, trades, portfolios, and idempotency records (code/shared/types/dynamodb.ts:16-93); the DynamoDB module declares only the generic table and GSI (infra/modules/dynamodb/main.tf:1-60). A repository-wide search for ledger postings, reconciliation, reservations, settlement, and clearing finds no application implementation. The SELL check occurs only in the asynchronous portfolio projection after the simulated trade (code/lambdas/portfolio-updater/handler.ts:87-100).
• Inference: cash/securities availability is not reserved before acceptance, a mutable portfolio is standing in for ownership, and there is no independent accounting control that can prove completeness and balance.
• Trigger: any BUY/SELL, duplicate or missing fill, partial execution, correction, late event, or cross-system disagreement.
• Resulting state: the system can accept an economically invalid command, report an execution before discovering insufficient holdings, and cannot explain balances through balanced, immutable postings.
• Does retry repair it? No. Retry moves messages; reconciliation compares independent records/control totals and classifies discrepancies.
• Impact: oversell/stale balance exposure, inability to support settlement and audit, and costly manual reconstruction.
• Detection: introduce balanced-ledger invariants, reservation totals, execution-versus-posting counts, cash/securities control totals, unmatched items, aging, and signed-off reconciliation runs. Current transport metrics cannot supply these controls (C48).
• Recovery: freeze affected accounts if necessary, reconstruct from authoritative external executions/statements, post corrections/reversals, rebuild projections, and retain evidence of discrepancy resolution.
• Practical remedy: create explicit order, risk/reservation, execution, clearing/settlement, and cash/securities ledger boundaries. Post stable, append-oriented balanced entries; derive portfolio/P&L; run scheduled and on-demand reconciliation with operator workflow.
• Residual risk: a ledger is not automatically correct. Chart-of-accounts, balancing scope, correction policy, access control, retention, external statements, and operational ownership require domain governance.

CS12 — The Lambda simulation is not a default latency-critical matcher

• Observed evidence: an EventBridge-triggered Lambda computes a random slippage price (code/lambdas/trade-executor/handler.ts:55-81). The reusable Lambda module supplies a general Node.js runtime, configurable timeout/memory, and active tracing (infra/modules/lambda/main.tf:12-31). There is no order book, price-time priority, deterministic sequence, or dedicated matching loop.
• Inference: for strict tail-latency, sustained throughput, deterministic sequencing, or venue fairness requirements, this multi-hop on-demand function is not a justified matching substrate. Lambda is acceptable only if measured requirements and failure semantics support it. Compute choice must be based on workload and unit economics, not “serverless scales/is cheaper” shorthand (C49, C53/A58-A60/F09,F19,F20).
• Trigger: bursty or sustained order flow with a tight p99/p99.9 budget, per-book serialization, market-session load, or recovery backlog.
• Resulting state: unpredictable queue/invocation delay, concurrency races, nondeterministic prices, and no well-defined price-time priority.
• Does retry repair it? No. Retry increases latency and can repeat work; the duplicate row guard does not define matching semantics.
• Impact: missed latency SLOs, unfair or inconsistent execution, and difficult incident recovery.
• Detection: end-to-end accept-to-ack and accept-to-fill percentiles, per-book backlog/sequence gaps, throughput saturation, rejection/load-shed counts, deterministic replay checks, and execution-price controls.
• Recovery: halt or isolate the affected matching partition, replay commands against the authoritative sequenced log under a controlled mode, reconcile resulting executions, and communicate corrections.
• Practical remedy: place the low-latency matching loop on long-lived, capacity-controlled compute with a single writer per order-book partition and an authoritative sequenced journal. Keep serverless APIs, workflows, projections, notifications, reporting, and compliance around it.
• Residual risk: dedicated compute adds capacity planning, failover, deployment, and on-call burden. Partitioning also creates cross-book coordination and hot-market concerns.

Three repair levels

You can now choose a repair scope from concrete evidence. The levels are design alternatives with increasing domain and operating obligations, not a requirement to deploy everything at once. Identify the immediate containment, the local transaction or durable intent that closes a failure window, and the wider financial authorities that a production trading system would still need.

Level 1 — Minimal hardening of the current shape

Keep API Gateway, Lambda, EventBridge, and the existing tables, but remove the most dangerous silent failures:

• inspect every PutEvents entry; independently validate the bus/rules; retain durable replay identity; and reconcile downstream effects before treating a replay as complete;
• persist stable command/event IDs and publication state; run a worker that recovers aged PUBLISH_PENDING records;
• transact trade creation and a conditional legal order transition where both are local DynamoDB items; do not let a duplicate trade guard block completion;
• add a projector inbox keyed by tradeId, and add source versions/gap handling before applying portfolio events;
• add rule-scoped SQS queue policies for every EventBridge target DLQ; alarm InvocationsFailedToBeSentToDLQ; and configure/alarm EventBridge delivery and Lambda runtime failure destinations as separate boundaries;
• use exact monetary representation and explicit currency/scale;
• paginate/chunk the snapshot and verify manifests/control totals;
• introduce daily execution/order/portfolio discrepancy reports even before a full ledger exists.

This reduces silent loss and duplicate effects but retains a state-to-message repair loop and an under-specified financial domain. It is a stabilization step, not the target architecture.

Level 2 — Production serverless with transactional outbox/Streams

Use an access-pattern-first command model and commit local business state plus publication intent atomically:

1. PlaceOrder transacts canonical idempotency record, order, and outbox item. The API returns ACCEPTED, not “executed.”
2. DynamoDB Streams captures the committed outbox/item changes in per-item order and a relay publishes stable integration events. Stream processing can retry, so the relay and consumers remain idempotent (C23, C38/A01,A20,A21/F04).
3. Execution ingestion transacts stable execution record, conditional order transition, and outgoing outbox record. Partial fills use distinct execution IDs and an aggregate version.
4. Each projector transacts its inbox record, expected source version, and projection update. Gaps quarantine the account/aggregate for bounded replay.
5. Export/reconciliation uses PITR-backed DynamoDB export (direct source above) or another bounded snapshot source into partitioned S3, manifests/control totals, Athena/Glue analysis, and signed-off discrepancy repair.

This closes database-to-message atomicity within DynamoDB but not the boundary to an external matching venue, SNS recipient, or other side effect. Streams retain records for a bounded window and do not provide table-wide order (C23/A21); longer replay requires an authoritative journal/archive and tested rebuild process.

Level 3 — Domain-correct trading architecture

Separate the business boundaries rather than treating every component as a Lambda around one mutable table:

• a command gateway provides authentication, validation, idempotent acceptance, and status;
• authoritative risk/reservation services protect available cash and securities before an order can become executable;
• a long-lived matching engine or venue adapter owns deterministic sequencing and execution identity per book/venue;
• order/execution state changes and outbox entries are locally transactional;
• an append-oriented cash and securities ledger posts balanced entries with currency/scale, corrections/reversals, audit history, and access controls;
• portfolio, P&L, search, notification, and analytics stores are explicitly rebuildable projections with freshness SLOs;
• reconciliation compares commands, venue executions, reservations, ledger postings, settlement statements, and projection control totals, then tracks every discrepancy to resolution;
• serverless remains useful for APIs, process management, fan-out, notifications, projections, reporting, compliance, and operator control planes.

This is the strongest correctness model and the highest engineering burden. A ledger and reconciler introduce new authoritative components that themselves need availability, disaster recovery, schema governance, segregation of duties, and operational expertise.

Residual trade-off model

The write counts below are illustrative logical item mutations per accepted order/fill, not DynamoDB billing units. A transaction, item size, GSIs, retries, Streams, backups, and projections change billed work. Migration and operator burden use a 1–5 ordinal: 1 is smallest, 5 is largest.

Dimension	Level 1: harden	Level 2: serverless outbox	Level 3: domain-correct
Command-side logical writes	About 2–3: order, idempotency/publication state, optional repair marker.	3 in one acceptance transaction: order, idempotency, outbox.	At least order, idempotency, reservation, and outbox; exact count follows risk/ledger model.
Fill-side logical writes	About 2–3: trade, order transition, publication state.	3 in one transaction: execution, order transition, outbox; projector adds inbox + view.	Execution/order journal plus at least two balanced ledger legs where double-entry applies, reservation release, and outbox; projections add inbox + view.
Projection lag	Existing asynchronous path; no durable freshness bound until metrics/recovery are added.	Normally Stream/relay lag plus target/consumer lag; measurable and rebuildable, but bounded by 24-hour Stream retention for that source (C23).	Ledger/execution commit stays authoritative; portfolio/search/notification lag is explicit and does not weaken monetary invariants.
Synchronous latency	Small increase for conditional writes/state tracking.	Higher than one write because acceptance uses a DynamoDB transaction; publication leaves the request path.	Highest authoritative work if risk/reservation and ledger must complete synchronously; matcher optimized on dedicated substrate.
Failure recovery	Repair scripts and scans; ambiguity remains.	Outbox backlog, inbox IDs, sequence gaps, replay, and export controls make recovery systematic.	Journal/ledger replay plus external reconciliation gives strongest evidence, but runbooks are more complex.
Duplicate/order defense	Added IDs and versions, but compatibility with current records is awkward.	Transactional inbox plus expected source sequence per projector.	Stable domain identities and single-writer sequences, plus reconciliation across boundaries.
Operator burden (1–5)	2: new alarms and manual repair ownership.	3: relay lag, poison records, rebuilds, and schema evolution.	5: matcher, ledger, settlement controls, reconciliation, and 24/7 domain operations.
Infrastructure/unit cost	Lowest immediate spend; higher expected-loss and support risk remains.	More writes, Stream reads, retained events, export storage/query, and alarms; lower silent-loss risk.	Dedicated compute and authoritative data/control systems add fixed and staff cost; correctness loss exposure is lowest when operated well.
Migration complexity (1–5)	2: additive fields, handlers, metrics, and repair jobs.	4: dual-write/backfill/cutover to outbox and idempotent projectors.	5: domain split, reservation/ledger migration, parallel run, external reconciliation, and controlled cutover.
Main residual risk	Still no full financial authority; repair state can drift.	External matcher/side effects are outside the DynamoDB transaction; replay window and hot keys remain.	Organizational and operational complexity can create new failure modes; correctness depends on governed domain rules and reconciliation.

Use measured workload variables before making a price or substrate choice: accepted orders/second, fills/order, payload bytes, fan-out, item sizes, GSI writes, retries, retention, replay traffic, Lambda duration/memory, matcher utilization, logs/traces, KMS, data transfer, RTO/RPO, and engineering/on-call cost. “Serverless is cheaper” is not an admissible conclusion (C49).

Interview prompts and answer checkpoints

1. Walk from POST /orders to portfolio update and name every commit point. A senior answer distinguishes the HTTP response, each standalone DynamoDB write, PutEvents producer acknowledgement, bus/configuration validation, EventBridge delivery, DLQ write, Lambda runtime, and projection commit.
2. Where is the first dual-write gap? The order commit precedes publication. Ask why reversing the order only moves the gap, and how an outbox changes it.
3. Can PutEvents return 200, zero failures, and EventId, yet drop the event? Yes: a nonexistent bus is the documented case (C08/A06,A82). Ask why the replayer needs independently validated bus configuration, durable replay identity, and reconciliation before deleting its only SQS copy.
4. Why is the idempotency design unsafe? It caches 201 before publication; same-key retry can lie and new-key retry duplicates the command. Ask for a canonical request hash and atomic order/idempotency/outbox transaction.
5. Does attribute_not_exists(PK) make trade execution idempotent? It deduplicates one row, but prevents retry from publishing after a late failure. Business idempotency must protect the whole intended transition.
6. Does portfolio optimistic locking stop duplicate fills? No. It protects concurrent item replacement, not repeated tradeId. Ask for transactional inbox plus update.
7. What ordering does the portfolio require? Per authoritative account or aggregate sequence, with duplicate/stale/gap behavior. Do not claim an EventBridge bus order (C09).
8. Can either declared target DLQ receive anything as built? Not from EventBridge: both lack the queue resource policy granting events.amazonaws.com sqs:SendMessage. The candidate must add the rule-scoped policies, alarm InvocationsFailedToBeSentToDLQ, and separately explain Lambda asynchronous runtime failures (C05-C07,C57/A81).
9. What fails first in the daily snapshot at scale? One Scan page makes the export incomplete; filtering still consumes scanned work; memory/object assembly and moving-state consistency follow. Ask for export manifests and control totals.
10. Why is cents rounding around JavaScript number insufficient? Binary representation and unstated currency/scale/rounding rules remain. Ask how corrections are posted and projections rebuilt.
11. Which record proves a customer's available cash and securities? None in this repository. A portfolio is a projection; require reservations, authoritative ledger postings, and reconciliation.
12. Would you keep matching in Lambda? Only after explicit latency, throughput, ordering, fairness, and recovery requirements justify it. The default low-latency design uses long-lived, partitioned single-writer matching; serverless surrounds it.
13. How do you migrate without losing an order? Expect parallel outbox publication, backfill with stable identities, idempotent dual consumption, control totals, canary accounts, rollback criteria, and signed-off cutover.
14. What does “done” mean after replay? Every input classified, expected authoritative state present, projections caught up, duplicates absent, ledger/control totals balanced, and discrepancies resolved—not merely an empty queue.

Decision summary

If the interview constraint is…	Defensible direction	Do not claim…
Improve the demo quickly	Level 1, with explicit debt and reconciliation alarms.	That PutEvents + DLQ makes the workflow reliable.
Production serverless order workflow	Level 2 outbox/Streams, stable IDs, idempotent sequenced projectors, scalable export.	End-to-end exactly once.
Real trading/financial authority	Level 3 matcher + reservation + ledger + reconciliation, with serverless around the core.	That a portfolio row or event archive is a ledger.
Read-heavy client experience	Rebuildable projections with freshness SLOs and read-your-writes strategy.	That a GSI is strongly current.
Strict matching latency/fairness	Long-lived capacity-controlled single-writer partitions, measured against tail-latency SLOs.	That Lambda is unsuitable or suitable without workload evidence.

Sources used in this chapter

Controlled claim wording and retrieval dates are in the claim register; full bibliographic details are in the source ledger. Direct sources used here are A01, A04-A07, A19-A23, A32-A35, A41, A43-A51, A53, A58-A60, A81-A82 and F01, F04, F08-F09, F11, F17-F20. Repository observations use the exact local routes printed beside each finding. No local note or repository comment is treated as authority for AWS semantics.

Takeaway and next step

The case study turns abstract guarantees into observable crash windows. Rehearse the current request path, distinguish missing publication from duplicate or out-of-order effects, and defend a repair level with residual risk. Keep its logical write counts distinct from billed units. Next, quantify those trade-offs in Cost, performance, and capacity.