# Repository case study: from event demo to financially correct system
This chapter reviews the read-only fintech repository as an interview case
study. It is not a claim that the repository is production software. The useful
question is: **where does each guarantee stop, what state remains after a
failure, and what would make recovery provable?**
All code paths are relative to the reference repository. “Observed” means the
code says it. “Documented” means the statement follows a controlled AWS or
pattern claim. “Inference” is an architecture conclusion, never a service
guarantee. The full evidence inventory is in
`../research/repository-evidence.md`.
## Two-minute architecture walk-through
1. `POST /orders` synchronously invokes order-handler. The handler validates a
command, writes a `PENDING` order, optionally caches a `201` by idempotency
key, then calls EventBridge (`code/lambdas/order-handler/handler.ts:114-212`).
2. EventBridge routes `trading.OrderPlaced` asynchronously to trade-executor
(`infra/environments/production/main.tf:345-359`). The handler simulates a
fill, inserts a trade, updates the order to `EXECUTED`, then publishes
`trading.TradeExecuted` (`code/lambdas/trade-executor/handler.ts:65-148`).
3. Two independent EventBridge targets update a mutable portfolio projection
and publish an SNS notification
(`infra/environments/production/main.tf:360-370,565-615`).
4. `GET /trades` reads the user/time GSI, while a daily Lambda scans portfolio
rows into S3 JSONL (`code/lambdas/trades-reader/handler.ts:45-97`;
`code/lambdas/portfolio-snapshot/handler.ts:64-107`).
5. Terraform declares target SQS DLQ ARNs and a bus archive, but neither DLQ
has the queue resource policy required for EventBridge `sqs:SendMessage`.
A manually invoked replayer can read only messages that actually reached a
queue (`infra/modules/eventbridge/main.tf:15-95`;
`infra/environments/production/main.tf:575-605`;
`code/lambdas/dlq-replayer/handler.ts:61-145`).
```mermaid
flowchart LR
Client[Client] -->|POST /orders| API[API Gateway]
API --> OH[order-handler]
OH --> Main[(DynamoDB: orders/trades/portfolios)]
OH --> Idem[(DynamoDB: idempotency cache)]
OH -->|PutEvents| EB{EventBridge bus}
EB -. OrderPlaced .-> TE[trade-executor]
TE --> Main
TE -->|PutEvents| EB
EB -. TradeExecuted .-> PU[portfolio-updater]
PU --> Main
EB -. TradeExecuted .-> NS[notification-sender]
NS --> SNS[SNS]
Client -->|GET /trades| API
API --> Reader[trades-reader]
Reader -->|GSI query| Main
Timer[Daily schedule] -.-> Snapshot[portfolio-snapshot]
Snapshot -->|Scan| Main
Snapshot --> S3[(S3 JSONL)]
EB -. target-delivery failure .-> DLQGap{{DLQ send denied:
queue resource policy absent}}
DLQGap -. InvocationsFailedToBeSentToDLQ .-> Metric[CloudWatch metric]
DLQGap -. after required policy only .-> DLQ[[Declared SQS DLQs]]
Operator[Operator] --> Replayer[dlq-replayer]
DLQ -->|Receive/Delete only if populated| Replayer
Replayer -->|PutEvents| EB
```
Solid arrows represent synchronous calls made by the current caller. Dashed
arrows are scheduled or broker-driven handoffs. A successful arrow proves only
its own boundary.
## What is authoritative?
The order row is the apparent operational authority for order status, and the
trade row is the only persisted execution record. That is an observation about
this code, not proof of financial authority. The portfolio row, GSI query,
snapshot, and notification are derived views. EventBridge and its archive are
transport/recovery mechanisms, not a ledger. The declared SQS DLQs are not even
shown writable by EventBridge in the current Terraform (CS08/C57).
**Inference:** no code here establishes pre-trade cash or securities
reservation, clearing/settlement, balanced postings, corrections/reversals, or
independent reconciliation. Therefore the mutable portfolio must not decide
available balance or ownership. Authoritative financial state and projections
must remain distinct (C46-C47; A32-A35,A41; F08,F17,F18).
## Guarantee boundaries
| Boundary | Success establishes | Boundary ends before |
|---|---|---|
| API Gateway / Lambda response | The HTTP API integration returned that response. | Matching, execution, ledgering, portfolio update, and notification (C51/A53). |
| DynamoDB order write | The order item write succeeded. | A `PutEvents` producer acknowledgement, valid routing, or downstream progress. A DynamoDB transaction cannot include EventBridge (C22/A20). |
| `PutEvents` | HTTP 200, zero failed entries, and `EventId` are producer-API acknowledgement for the named request. | Valid bus existence/configuration, matching/routing, target delivery, consumer durability, or handler effects. AWS documents that a nonexistent bus can return those response signals and drop the event (C08/A06,A82). |
| EventBridge target delivery | EventBridge owns delivery attempts at this boundary (C05/A05). | Lambda asynchronous runtime retry/business rollback, or a working target DLQ. The current queues lack the required EventBridge send policy (C57/A81). |
| Lambda handler invocation | Code ran until it returned or failed. | Atomicity across multiple SDK calls and one-time external effects. |
| DynamoDB conditional portfolio write | No other writer changed that item version between read and write. | Event novelty and causal order (C40-C41/F11). |
| Archive or DLQ | The archive is configured for bounded retention; an SQS message is retained only if EventBridge can write it. | A configured DLQ ARN, zero queue depth, diagnosis, safe replay, idempotent repair, or reconciliation. Current DLQ sends can fail because the queue policies are absent (C43,C57/A13,A36,A43,A81). |
| Business completion | No as-built boundary proves execution plus balanced monetary/securities postings and reconciled projections. | This requires an explicit authoritative design (C46-C48). |
## Correctness and scalability review
Each issue includes the failure trigger, resulting state, retry behavior,
customer/financial effect, detection, recovery, a practical remedy, and what
still remains risky.
### CS01 — Order state and event publication are a non-atomic dual write
- **Observed evidence:** order-handler writes the order first
(`code/lambdas/order-handler/handler.ts:155-165`), writes the response cache
separately (`:167-181`), and only then calls EventBridge (`:183-208`).
- **Documented boundary:** DynamoDB transactions cover supported DynamoDB items
in one account and Region; they do not include EventBridge (C22/A20). A
transactional outbox commits business state and an outbox record locally,
while publication and consumption remain retryable (C38/A20,A21/F04).
- **Trigger:** the order write succeeds and the process crashes, the EventBridge
call throws, times out ambiguously, or returns a failed entry.
- **Resulting state:** a durable `PENDING` order exists without a durable route
to execution. The client can receive `500`; without a status/recovery process,
the order can remain stranded.
- **Does retry repair it?** Not reliably. No key creates another order; the same
key can hit CS02; an ambiguous timeout can also mean the producer API
processed the request, so blind republish risks duplicates even though
routing and downstream progress remain unknown.
- **Impact:** a customer sees failure while a command is stored, or later sees
an order that never advances. This is both a correctness and support risk.
- **Detection:** alarm on age/count of `PENDING` orders, compare accepted orders
with outbox/publication acknowledgements, and reconcile order transitions to
execution records. The current publication-error log cannot detect an HTTP
200 partial failure.
- **Recovery:** classify producer acknowledgement separately from validated bus
configuration and downstream progress; republish with a stable event ID only
through an idempotent consumer, then reconcile the resulting transition.
- **Practical remedy:** minimally persist `publicationStatus` and drive a
repair worker. Prefer a DynamoDB transaction containing order, idempotency
result, and outbox item, then relay from DynamoDB Streams.
- **Residual risk:** Streams/outbox closes the local database/message gap, not
end-to-end exactly-once processing. Relay and consumer retries still demand
stable identity, idempotency, alarms, and reconciliation.
### CS02 — The idempotency cache can certify an order with no proven routing
- **Observed evidence:** the handler stores a cached `201` before publication
(`code/lambdas/order-handler/handler.ts:167-180`) and returns the cached value
before processing on retry (`:229-241`). Its comment tells the caller to retry
with a new key after publication failure (`:11-17`), which defeats logical
command deduplication.
- **Trigger:** the cache write succeeds and the producer call throws, has a
failed entry, or returns a nominal acknowledgement for a misconfigured or
nonexistent bus.
- **Resulting state:** the original request returns `500`, but a same-key retry
returns cached `201` even though valid bus routing and downstream progress are
unproven. A new-key retry creates a second order.
- **Does retry repair it?** No. Same-key retry skips publication; new-key retry
changes command identity.
- **Impact:** false success or duplicate orders, with ambiguous customer intent.
- **Detection:** compare every live idempotency record with its order and durable
outbox/publication state; track same customer payload under multiple keys.
- **Recovery:** do not merely delete the cache and replay. Validate the named bus
and rules independently, preserve the original order/event identity, recover
through the durable publication path, and reconcile downstream state.
- **Practical remedy:** atomically transact the idempotency key, canonical
request hash, order, and outbox. Store an explicit state such as
`IN_PROGRESS`/`ACCEPTED`, and make same-key/different-payload reuse a conflict.
Business idempotency protects the mutation and response together (C40/F11).
- **Residual risk:** TTL deletion is asynchronous and is only cleanup
(C32/A23). The product still needs a documented key-retention window and
behavior for late retries.
### CS03 — `PutEvents` response and configuration boundaries are ignored
- **Observed evidence:** order-handler (`code/lambdas/order-handler/handler.ts:193-205`),
trade-executor (`code/lambdas/trade-executor/handler.ts:134-145`), and
dlq-replayer (`code/lambdas/dlq-replayer/handler.ts:114-133`) await the SDK
call but discard its response. The replayer then deletes the DLQ message.
- **Documented boundary:** `PutEvents` can return HTTP 200 with per-entry
failures. Even HTTP 200, `FailedEntryCount: 0`, and `EventId` are only
producer-API acknowledgement: AWS documents that a nonexistent bus can return
those signals while event matching finds no rule and drops the event
(C08/A06,A82).
- **Trigger:** EventBridge reports a failed entry under HTTP 200, or the caller
names a nonexistent/misconfigured bus and receives a nominally successful
producer response.
- **Resulting state:** the order path logs success and returns `201`, the trade
path logs completion without downstream events, or the replayer deletes the
only DLQ copy despite having no evidence of valid routing or downstream
progress.
- **Does retry repair it?** No exception is thrown, so these handlers return
successfully and their normal retry owner sees nothing to retry. Blind retry
against the same bad bus configuration only repeats the drop.
- **Impact:** silently stranded orders, stale portfolios, missing notifications,
and destructive redrive loss.
- **Detection:** emit `FailedEntryCount` and per-error-code metrics; validate the
configured bus ARN/name and expected rules during deployment and before
redrive; reconcile orders/trades/outbox state with downstream effects. An
`EventId` is not the reconciliation signal.
- **Recovery:** locate the authoritative source record or retained DLQ body,
validate/fix the bus and rule configuration, republish under the stable event
identity, and verify downstream business state through reconciliation. The
current replayer can make recovery impossible after deletion.
- **Practical remedy:** inspect every response entry and treat a failed entry as
failed. A replayer deletes the SQS message only after (1) the entry has no
error, (2) bus identity/configuration has been independently validated, and
(3) a durable replay audit record preserves the source message/event identity
for downstream reconciliation. `EventId` alone is not proof of bus durability
or business progress.
- **Residual risk:** a client-side timeout remains ambiguous, and validated
configuration can drift later. Durable intent, idempotent consumers,
deployment checks, and reconciliation remain required.
### CS04 — Trade creation, order transition, and publication are separate operations
- **Observed evidence:** trade-executor conditionally inserts the trade
(`code/lambdas/trade-executor/handler.ts:83-104`), separately updates the
order (`:106-118`), and separately publishes (`:120-145`). The order update
does not require a previous `PENDING` state.
- **Trigger:** a crash or rejected operation between any two steps, concurrent
order mutation, or a partial `PutEvents` failure.
- **Resulting state:** possible states include trade-present/order-pending;
trade-present/order-executed/event-missing; or an order transition applied
without a validated state-machine predecessor.
- **Does retry repair it?** Usually not. After the trade exists, the first
conditional put fails before the invocation can repair the order or publish.
- **Impact:** customer-visible status disagrees with execution data; downstream
holdings and notifications can be missing; investigation cannot rely on one
completion flag.
- **Detection:** continuously compare accepted orders, trade records, valid
order-state transitions, outbox records, ledger postings, and projection
checkpoints.
- **Recovery:** use a repair command that reads the persisted trade and current
order state, applies only a legal missing transition, emits the stable event,
and records the repair for reconciliation.
- **Practical remedy:** use `TransactWriteItems` for the trade record, a
conditional order-state transition, and an outbox item. Model partial fills
as stable execution IDs rather than assuming one trade per order.
- **Residual risk:** that local transaction cannot atomically include an
external matching engine. Its execution report needs stable identity,
idempotent ingestion, and reconciliation.
### CS05 — Publication failure is blocked by the duplicate-trade guard on retry
- **Observed evidence:** each attempt generates a new trade ID and price, but
inserts at `PK = TRADE#` with `attribute_not_exists(PK)`
(`code/lambdas/trade-executor/handler.ts:79-104`). Publication occurs only
after that insert and the order update (`:120-145`).
- **Trigger:** the trade and order writes succeed, then publication throws; or
publication is ambiguous and Lambda retries.
- **Resulting state:** the trade exists and the order says `EXECUTED`, but the
`TradeExecuted` event may be absent. The next attempt fails at the duplicate
trade put before it reaches publication.
- **Does retry repair it?** No. The write guard prevents a duplicate trade row
but also prevents retry from finishing the workflow.
- **Impact:** stale holdings and missing client notification despite an executed
order. Repeated retries add load and can terminate at the wrong failure story.
- **Detection:** compare each execution/trade ID with a durable outbox record and
each projector's inbox/checkpoint; alert on executed trades without relay
progress or reconciled downstream effects beyond the SLO. Validate bus/rule
configuration independently of the producer API response.
- **Recovery:** reconstruct the event from the stored trade under its original
stable `tradeId`, preserve causation/correlation metadata where available,
publish through a recovery outbox, and reconcile projections.
- **Practical remedy:** create trade, transition order, and create the exact
outgoing event in one transaction. The relay retries the outbox independently
of trade creation.
- **Residual risk:** repeated relay delivery remains possible. Every projector
and notification workflow still needs an idempotent business effect.
### CS06 — Optimistic locking does not make the portfolio consumer idempotent
- **Observed evidence:** `TradeExecuted` has a `tradeId` but no consumer stores
it (`code/shared/types/events.ts:36-47`). portfolio-updater reads holdings,
applies every event, then performs only a portfolio-version condition
(`code/lambdas/portfolio-updater/handler.ts:145-186`).
- **Trigger:** EventBridge/Lambda redelivery, operator replay, a response-path
failure after DynamoDB committed, or two envelopes for the same trade.
- **Resulting state:** a duplicate BUY adds shares and cost twice; a duplicate
SELL subtracts twice or later fails for insufficient holdings. The item
version advances normally, so the lock sees no duplicate.
- **Does retry repair it?** No. Retries are the trigger. Optimistic locking only
resolves concurrent writers that read the same version.
- **Impact:** materially incorrect positions, cost basis, and downstream P&L.
- **Detection:** duplicate `tradeId` count, ledger-to-portfolio reconciliation,
projection version/sequence gaps, and negative/implausible position checks.
- **Recovery:** rebuild the projection from authoritative executions/ledger
postings, or apply a reviewed compensating correction. Replaying the same
event into the current handler is unsafe.
- **Practical remedy:** atomically conditionally insert an inbox item keyed by
`(projector, tradeId)` and update the projection. A duplicate becomes a no-op
with a compatible result (C40/F11).
- **Residual risk:** inbox retention must cover all replay windows; schema bugs
may require a versioned rebuild rather than ordinary redrive.
### CS07 — The portfolio has an ordering requirement but no event sequence
- **Observed evidence:** `TradeExecuted` carries a timestamp but no account,
order, or execution sequence (`code/shared/types/events.ts:36-47`). Independent
EventBridge targets route the event to the updater
(`infra/environments/production/main.tf:565-605`). The updater uses the
arrival-time portfolio version, not a source sequence
(`code/lambdas/portfolio-updater/handler.ts:145-178`).
- **Controlled rule:** this package makes no ordering claim for an EventBridge
bus. A design that requires order must add a sequence/version rule or ordered
mechanism (C09/A07/F01). Ordering and deduplication solve different problems
(C41).
- **Trigger:** two trades for the same account arrive late or out of causal
order, especially BUY/SELL or cancel/replace/partial-fill sequences.
- **Resulting state:** a SELL can be rejected before its preceding BUY arrives;
weighted cost basis can differ by arrival order; a later old event can mutate
an already newer projection.
- **Does retry repair it?** Not deterministically. Delay may incidentally change
arrival order, but there is no gap detector or expected source version.
- **Impact:** stale or incorrect positions and nondeterministic recovery.
- **Detection:** per-account expected-versus-observed sequence, gap age, late
event count, projector lag, and reconciliation discrepancy.
- **Recovery:** quarantine a gap, fetch/replay the missing authoritative sequence
in order, rebuild that account projection, then reconcile.
- **Practical remedy:** assign a monotonic version at the authoritative
aggregate/single-writer boundary. The projector conditionally accepts the next
expected version and treats lower versions as duplicates/stale. An ordered log
keyed by account can simplify delivery order but does not remove idempotency.
- **Residual risk:** one account key can become hot, and invariants spanning
accounts/instruments still need coordination and reconciliation (C34).
### CS08 — The target DLQs lack write permission and cover a separate boundary
- **Observed evidence:** the reusable module creates SQS queues and attaches
their ARNs to targets (`infra/modules/eventbridge/main.tf:29-38,73-95`);
production repeats that shape for portfolio-updater
(`infra/environments/production/main.tf:575-605`). A repository-wide Terraform
search finds no `aws_sqs_queue_policy` and no `sqs:SendMessage` grant. The
`events.amazonaws.com` principals that do exist are Lambda invocation
permissions (`infra/environments/production/main.tf:386-411,608-615`). The
Lambda module separately has no async failure destination
(`infra/modules/lambda/main.tf:12-70`), while trade-executor comments collapse
handler errors into “EventBridge retries (then DLQ)”
(`code/lambdas/trade-executor/handler.ts:24-26`).
- **Documented boundary:** programmatically configured target DLQs require an
SQS resource policy granting `events.amazonaws.com` `sqs:SendMessage`, usually
constrained by the rule ARN. EventBridge emits
`InvocationsFailedToBeSentToDLQ` when it cannot place an event in the DLQ
(C57/A81). Target delivery, DLQ write, Lambda asynchronous invocation, and
event-source mappings remain different boundaries (C05-C07/A01,A03-A05).
- **Trigger:** target delivery fails and EventBridge attempts to send the event
to either declared SQS DLQ, or Lambda accepts the invocation but the function
later fails.
- **Resulting state:** for the first path, SQS denies the DLQ write and the event
is in neither the target nor the intended DLQ; queue depth can stay zero while
`InvocationsFailedToBeSentToDLQ` rises. For the second path, the target DLQ
does not represent the later handler failure and partial business state can
remain.
- **Does retry repair it?** No ordinary retry grants the missing queue policy.
AWS documents some target errors, including missing target permission, that
go directly to a configured DLQ without target retries; that DLQ attempt still
fails here. Lambda runtime retries are separately owned and cannot roll back
an already committed database write.
- **Impact:** terminal delivery events can be lost despite a declared DLQ and a
green zero-depth alarm; operators can also confuse Lambda failures with target
failures, causing stale financial projections or repeated effects.
- **Detection:** alarm on EventBridge target invocation failures,
`InvocationsSentToDLQ`, and especially `InvocationsFailedToBeSentToDLQ` in
addition to SQS depth. Dashboard Lambda errors/throttles/runtime destinations
and business-state gaps separately. Add deployment assertions that every DLQ
has the exact rule-scoped queue policy.
- **Recovery:** add and test the queue policy, identify which boundary failed,
recover the original event from an authoritative outbox/source or the bounded
EventBridge archive, replay through an idempotent path, and reconcile business
effects. A missing DLQ message cannot be redriven by dlq-replayer.
- **Practical remedy:** declare an `aws_sqs_queue_policy` for every module and
portfolio DLQ granting principal `events.amazonaws.com`, action
`sqs:SendMessage`, and an `aws:SourceArn` condition for the exact rule. Add the
failed-DLQ-send alarm and a deployment test. Separately document/configure
Lambda async destinations, or select an SQS event-source mapping where durable
work-queue semantics are required.
- **Residual risk:** queue-policy drift or an incorrect source-ARN condition can
reopen the loss path; archive retention is bounded; and a working DLQ remains
only a holding area. Safe replay still needs ordering, idempotency, rate
limiting, durable audit identity, and reconciliation (C43).
### CS09 — The daily snapshot is incomplete and memory-bound at scale
- **Observed evidence:** the Lambda issues one filtered `Scan`, ignores
`LastEvaluatedKey`, maps all returned rows in memory, joins one JSONL string,
and writes one S3 object (`code/lambdas/portfolio-snapshot/handler.ts:64-107`).
- **Trigger:** the table exceeds one Scan response page, or the returned
portfolio set approaches Lambda memory/time or S3 request-body constraints.
- **Resulting state:** later pages are silently absent. The filter does not avoid
reading non-portfolio items, and records can change while the scan runs, so
the output is not a proven point-in-time snapshot.
- **Does retry repair it?** No; the same code again reads only one page and may
observe another moving view.
- **Impact:** incomplete audit/analytics data, excess read work, timeouts, and an
unreliable basis for reconciliation.
- **Detection:** treat any `LastEvaluatedKey` as incomplete; compare source and
exported counts/control totals; monitor scanned-versus-returned count, bytes,
duration, throttles, and object size.
- **Recovery:** run a complete, bounded export from a known recovery point;
generate manifests and control totals; classify and backfill missing
partitions.
- **Practical remedy:** at minimum paginate and stream bounded chunks to
separate objects. For scale, use PITR-backed DynamoDB export to S3 for a
selected table point, then catalog/query partitions and reconcile counts and
monetary totals. **Documented:** the export is asynchronous, does not consume
table RCUs, and has no completion-time SLA
([AWS DynamoDB export documentation](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/S3DataExport.HowItWorks.html),
retrieved 2026-08-22).
- **Residual risk:** export freshness needs an SLO, and storage/query work still
costs money. A projection export is not an authoritative financial ledger.
### CS10 — Binary floating point is used for money
- **Observed evidence:** prices are JavaScript `number` values
(`code/shared/validation/schemas.ts:21-27`), execution price uses floating
multiplication and cents rounding (`code/lambdas/trade-executor/handler.ts:55-63`),
cost basis/value perform repeated floating arithmetic
(`code/lambdas/portfolio-updater/handler.ts:64-84,121-135`), and snapshots do
the same (`code/lambdas/portfolio-snapshot/handler.ts:50-61`). No currency or
scale accompanies the amounts.
- **Controlled rule:** authoritative monetary amounts use exact decimal or
scaled integers with explicit currency/scale, not binary floating point
(C47/A41/F08,F17).
- **Trigger:** decimal prices/fees/taxes or repeated weighted calculations that
are not exactly representable in binary, plus inconsistent rounding stages.
- **Resulting state:** stored and derived totals can drift; cents rounding can
discard sub-cent rules without an explicit product policy.
- **Does retry repair it?** No. Repeating an unspecified rounding policy does not
make it financially correct; the random execution calculation can also differ
across attempts before the conditional write.
- **Impact:** incorrect balances, P&L, statements, and reconciliation breaks.
- **Detection:** exact control-total reconciliation by currency, property tests
for rounding, and boundary cases for fees, partial fills, and large values.
- **Recovery:** recompute projections from exact authoritative postings under a
versioned rounding policy; post explicit corrections rather than silently
editing historical amounts.
- **Practical remedy:** represent amount as scaled integer minor/sub-minor units
or a decimal type; carry currency, scale, and rounding mode in schemas; keep
execution identity and price deterministic.
- **Residual risk:** exact representation does not choose the business rounding
rule. FX, fees, corporate actions, and instrument-specific tick sizes need
governed policies and tests.
### CS11 — No append-only financial ledger or reconciliation loop is apparent
- **Observed evidence:** the declared entity types are mutable orders, trades,
portfolios, and idempotency records
(`code/shared/types/dynamodb.ts:16-93`); the DynamoDB module declares only the
generic table and GSI (`infra/modules/dynamodb/main.tf:1-60`). A repository-wide
search for ledger postings, reconciliation, reservations, settlement, and
clearing finds no application implementation. The SELL check occurs only in
the asynchronous portfolio projection after the simulated trade
(`code/lambdas/portfolio-updater/handler.ts:87-100`).
- **Inference:** cash/securities availability is not reserved before acceptance,
a mutable portfolio is standing in for ownership, and there is no independent
accounting control that can prove completeness and balance.
- **Trigger:** any BUY/SELL, duplicate or missing fill, partial execution,
correction, late event, or cross-system disagreement.
- **Resulting state:** the system can accept an economically invalid command,
report an execution before discovering insufficient holdings, and cannot
explain balances through balanced, immutable postings.
- **Does retry repair it?** No. Retry moves messages; reconciliation compares
independent records/control totals and classifies discrepancies.
- **Impact:** oversell/stale balance exposure, inability to support settlement
and audit, and costly manual reconstruction.
- **Detection:** introduce balanced-ledger invariants, reservation totals,
execution-versus-posting counts, cash/securities control totals, unmatched
items, aging, and signed-off reconciliation runs. Current transport metrics
cannot supply these controls (C48).
- **Recovery:** freeze affected accounts if necessary, reconstruct from
authoritative external executions/statements, post corrections/reversals,
rebuild projections, and retain evidence of discrepancy resolution.
- **Practical remedy:** create explicit order, risk/reservation, execution,
clearing/settlement, and cash/securities ledger boundaries. Post stable,
append-oriented balanced entries; derive portfolio/P&L; run scheduled and
on-demand reconciliation with operator workflow.
- **Residual risk:** a ledger is not automatically correct. Chart-of-accounts,
balancing scope, correction policy, access control, retention, external
statements, and operational ownership require domain governance.
### CS12 — The Lambda simulation is not a default latency-critical matcher
- **Observed evidence:** an EventBridge-triggered Lambda computes a random
slippage price (`code/lambdas/trade-executor/handler.ts:55-81`). The reusable
Lambda module supplies a general Node.js runtime, configurable timeout/memory,
and active tracing (`infra/modules/lambda/main.tf:12-31`). There is no order
book, price-time priority, deterministic sequence, or dedicated matching loop.
- **Inference:** for strict tail-latency, sustained throughput, deterministic
sequencing, or venue fairness requirements, this multi-hop on-demand function
is not a justified matching substrate. Lambda is acceptable only if measured
requirements and failure semantics support it. Compute choice must be based
on workload and unit economics, not “serverless scales/is cheaper” shorthand
(C49, C53/A58-A60/F09,F19,F20).
- **Trigger:** bursty or sustained order flow with a tight p99/p99.9 budget,
per-book serialization, market-session load, or recovery backlog.
- **Resulting state:** unpredictable queue/invocation delay, concurrency races,
nondeterministic prices, and no well-defined price-time priority.
- **Does retry repair it?** No. Retry increases latency and can repeat work; the
duplicate row guard does not define matching semantics.
- **Impact:** missed latency SLOs, unfair or inconsistent execution, and
difficult incident recovery.
- **Detection:** end-to-end accept-to-ack and accept-to-fill percentiles,
per-book backlog/sequence gaps, throughput saturation, rejection/load-shed
counts, deterministic replay checks, and execution-price controls.
- **Recovery:** halt or isolate the affected matching partition, replay commands
against the authoritative sequenced log under a controlled mode, reconcile
resulting executions, and communicate corrections.
- **Practical remedy:** place the low-latency matching loop on long-lived,
capacity-controlled compute with a single writer per order-book partition and
an authoritative sequenced journal. Keep serverless APIs, workflows,
projections, notifications, reporting, and compliance around it.
- **Residual risk:** dedicated compute adds capacity planning, failover,
deployment, and on-call burden. Partitioning also creates cross-book
coordination and hot-market concerns.
## Three repair levels
### Level 1 — Minimal hardening of the current shape
Keep API Gateway, Lambda, EventBridge, and the existing tables, but remove the
most dangerous silent failures:
- inspect every `PutEvents` entry; independently validate the bus/rules; retain
durable replay identity; and reconcile downstream effects before treating a
replay as complete;
- persist stable command/event IDs and publication state; run a worker that
recovers aged `PUBLISH_PENDING` records;
- transact trade creation and a conditional legal order transition where both
are local DynamoDB items; do not let a duplicate trade guard block completion;
- add a projector inbox keyed by `tradeId`, and add source versions/gap handling
before applying portfolio events;
- add rule-scoped SQS queue policies for every EventBridge target DLQ; alarm
`InvocationsFailedToBeSentToDLQ`; and configure/alarm EventBridge delivery and
Lambda runtime failure destinations as separate boundaries;
- use exact monetary representation and explicit currency/scale;
- paginate/chunk the snapshot and verify manifests/control totals;
- introduce daily execution/order/portfolio discrepancy reports even before a
full ledger exists.
This reduces silent loss and duplicate effects but retains a state-to-message
repair loop and an under-specified financial domain. It is a stabilization
step, not the target architecture.
### Level 2 — Production serverless with transactional outbox/Streams
Use an access-pattern-first command model and commit local business state plus
publication intent atomically:
1. `PlaceOrder` transacts canonical idempotency record, order, and outbox item.
The API returns `ACCEPTED`, not “executed.”
2. DynamoDB Streams captures the committed outbox/item changes in per-item order
and a relay publishes stable integration events. Stream processing can retry,
so the relay and consumers remain idempotent (C23, C38/A01,A20,A21/F04).
3. Execution ingestion transacts stable execution record, conditional order
transition, and outgoing outbox record. Partial fills use distinct execution
IDs and an aggregate version.
4. Each projector transacts its inbox record, expected source version, and
projection update. Gaps quarantine the account/aggregate for bounded replay.
5. Export/reconciliation uses PITR-backed DynamoDB export (direct source above)
or another bounded snapshot source into partitioned S3, manifests/control
totals, Athena/Glue analysis, and signed-off discrepancy repair.
This closes database-to-message atomicity within DynamoDB but not the boundary
to an external matching venue, SNS recipient, or other side effect. Streams
retain records for a bounded window and do not provide table-wide order
(C23/A21); longer replay requires an authoritative journal/archive and tested
rebuild process.
### Level 3 — Domain-correct trading architecture
Separate the business boundaries rather than treating every component as a
Lambda around one mutable table:
- a command gateway provides authentication, validation, idempotent acceptance,
and status;
- authoritative risk/reservation services protect available cash and securities
before an order can become executable;
- a long-lived matching engine or venue adapter owns deterministic sequencing
and execution identity per book/venue;
- order/execution state changes and outbox entries are locally transactional;
- an append-oriented cash and securities ledger posts balanced entries with
currency/scale, corrections/reversals, audit history, and access controls;
- portfolio, P&L, search, notification, and analytics stores are explicitly
rebuildable projections with freshness SLOs;
- reconciliation compares commands, venue executions, reservations, ledger
postings, settlement statements, and projection control totals, then tracks
every discrepancy to resolution;
- serverless remains useful for APIs, process management, fan-out,
notifications, projections, reporting, compliance, and operator control
planes.
This is the strongest correctness model and the highest engineering burden. A
ledger and reconciler introduce new authoritative components that themselves
need availability, disaster recovery, schema governance, segregation of duties,
and operational expertise.
## Residual trade-off model
The write counts below are **illustrative logical item mutations per accepted
order/fill**, not DynamoDB billing units. A transaction, item size, GSIs,
retries, Streams, backups, and projections change billed work. Migration and
operator burden use a 1–5 ordinal: 1 is smallest, 5 is largest.
| Dimension | Level 1: harden | Level 2: serverless outbox | Level 3: domain-correct |
|---|---|---|---|
| Command-side logical writes | About 2–3: order, idempotency/publication state, optional repair marker. | 3 in one acceptance transaction: order, idempotency, outbox. | At least order, idempotency, reservation, and outbox; exact count follows risk/ledger model. |
| Fill-side logical writes | About 2–3: trade, order transition, publication state. | 3 in one transaction: execution, order transition, outbox; projector adds inbox + view. | Execution/order journal plus at least two balanced ledger legs where double-entry applies, reservation release, and outbox; projections add inbox + view. |
| Projection lag | Existing asynchronous path; no durable freshness bound until metrics/recovery are added. | Normally Stream/relay lag plus target/consumer lag; measurable and rebuildable, but bounded by 24-hour Stream retention for that source (C23). | Ledger/execution commit stays authoritative; portfolio/search/notification lag is explicit and does not weaken monetary invariants. |
| Synchronous latency | Small increase for conditional writes/state tracking. | Higher than one write because acceptance uses a DynamoDB transaction; publication leaves the request path. | Highest authoritative work if risk/reservation and ledger must complete synchronously; matcher optimized on dedicated substrate. |
| Failure recovery | Repair scripts and scans; ambiguity remains. | Outbox backlog, inbox IDs, sequence gaps, replay, and export controls make recovery systematic. | Journal/ledger replay plus external reconciliation gives strongest evidence, but runbooks are more complex. |
| Duplicate/order defense | Added IDs and versions, but compatibility with current records is awkward. | Transactional inbox plus expected source sequence per projector. | Stable domain identities and single-writer sequences, plus reconciliation across boundaries. |
| Operator burden (1–5) | 2: new alarms and manual repair ownership. | 3: relay lag, poison records, rebuilds, and schema evolution. | 5: matcher, ledger, settlement controls, reconciliation, and 24/7 domain operations. |
| Infrastructure/unit cost | Lowest immediate spend; higher expected-loss and support risk remains. | More writes, Stream reads, retained events, export storage/query, and alarms; lower silent-loss risk. | Dedicated compute and authoritative data/control systems add fixed and staff cost; correctness loss exposure is lowest when operated well. |
| Migration complexity (1–5) | 2: additive fields, handlers, metrics, and repair jobs. | 4: dual-write/backfill/cutover to outbox and idempotent projectors. | 5: domain split, reservation/ledger migration, parallel run, external reconciliation, and controlled cutover. |
| Main residual risk | Still no full financial authority; repair state can drift. | External matcher/side effects are outside the DynamoDB transaction; replay window and hot keys remain. | Organizational and operational complexity can create new failure modes; correctness depends on governed domain rules and reconciliation. |
Use measured workload variables before making a price or substrate choice:
accepted orders/second, fills/order, payload bytes, fan-out, item sizes, GSI
writes, retries, retention, replay traffic, Lambda duration/memory, matcher
utilization, logs/traces, KMS, data transfer, RTO/RPO, and engineering/on-call
cost. “Serverless is cheaper” is not an admissible conclusion (C49).
## Interview prompts and answer checkpoints
1. **Walk from `POST /orders` to portfolio update and name every commit point.**
A senior answer distinguishes the HTTP response, each standalone DynamoDB write,
`PutEvents` producer acknowledgement, bus/configuration validation,
EventBridge delivery, DLQ write, Lambda runtime, and projection commit.
2. **Where is the first dual-write gap?** The order commit precedes publication.
Ask why reversing the order only moves the gap, and how an outbox changes it.
3. **Can `PutEvents` return 200, zero failures, and `EventId`, yet drop the
event?** Yes: a nonexistent bus is the documented case (C08/A06,A82). Ask why
the replayer needs independently validated bus configuration, durable replay
identity, and reconciliation before deleting its only SQS copy.
4. **Why is the idempotency design unsafe?** It caches `201` before publication;
same-key retry can lie and new-key retry duplicates the command. Ask for a
canonical request hash and atomic order/idempotency/outbox transaction.
5. **Does `attribute_not_exists(PK)` make trade execution idempotent?** It
deduplicates one row, but prevents retry from publishing after a late
failure. Business idempotency must protect the whole intended transition.
6. **Does portfolio optimistic locking stop duplicate fills?** No. It protects
concurrent item replacement, not repeated `tradeId`. Ask for transactional
inbox plus update.
7. **What ordering does the portfolio require?** Per authoritative account or
aggregate sequence, with duplicate/stale/gap behavior. Do not claim an
EventBridge bus order (C09).
8. **Can either declared target DLQ receive anything as built?** Not from
EventBridge: both lack the queue resource policy granting
`events.amazonaws.com` `sqs:SendMessage`. The candidate must add the
rule-scoped policies, alarm `InvocationsFailedToBeSentToDLQ`, and separately
explain Lambda asynchronous runtime failures (C05-C07,C57/A81).
9. **What fails first in the daily snapshot at scale?** One Scan page makes the
export incomplete; filtering still consumes scanned work; memory/object
assembly and moving-state consistency follow. Ask for export manifests and
control totals.
10. **Why is cents rounding around JavaScript `number` insufficient?** Binary
representation and unstated currency/scale/rounding rules remain. Ask how
corrections are posted and projections rebuilt.
11. **Which record proves a customer's available cash and securities?** None in
this repository. A portfolio is a projection; require reservations,
authoritative ledger postings, and reconciliation.
12. **Would you keep matching in Lambda?** Only after explicit latency,
throughput, ordering, fairness, and recovery requirements justify it. The
default low-latency design uses long-lived, partitioned single-writer
matching; serverless surrounds it.
13. **How do you migrate without losing an order?** Expect parallel outbox
publication, backfill with stable identities, idempotent dual consumption,
control totals, canary accounts, rollback criteria, and signed-off cutover.
14. **What does “done” mean after replay?** Every input classified, expected
authoritative state present, projections caught up, duplicates absent,
ledger/control totals balanced, and discrepancies resolved—not merely an
empty queue.
## Decision summary
| If the interview constraint is… | Defensible direction | Do not claim… |
|---|---|---|
| Improve the demo quickly | Level 1, with explicit debt and reconciliation alarms. | That `PutEvents` + DLQ makes the workflow reliable. |
| Production serverless order workflow | Level 2 outbox/Streams, stable IDs, idempotent sequenced projectors, scalable export. | End-to-end exactly once. |
| Real trading/financial authority | Level 3 matcher + reservation + ledger + reconciliation, with serverless around the core. | That a portfolio row or event archive is a ledger. |
| Read-heavy client experience | Rebuildable projections with freshness SLOs and read-your-writes strategy. | That a GSI is strongly current. |
| Strict matching latency/fairness | Long-lived capacity-controlled single-writer partitions, measured against tail-latency SLOs. | That Lambda is unsuitable or suitable without workload evidence. |
## Sources used in this chapter
Controlled claim wording and retrieval dates are in
`../research/claim-register.md`; full bibliographic details are in
`../research/source-ledger.md`. Direct sources used here are A01, A04-A07,
A19-A23, A32-A35, A41, A43-A51, A53, A58-A60, A81-A82 and F01, F04, F08-F09, F11,
F17-F20. Repository observations use the exact local routes printed beside each
finding. No local note or repository comment is treated as authority for AWS
semantics.