Senior interview drills: reason from boundaries

Reading notesDownload original Markdown ↗

This final stage combines the whole guide. Work through service choice (SC), failure analysis (FA), quantitative exercises (QE), system design (SD), and repository walk-throughs (RW). Answer a prompt aloud or on paper before revealing its reference answer, then use the shared rubric to identify the missing reasoning. The Premises fields are cross-references: AP points to DynamoDB access patterns, FSR/RBK to reliability failures/runbooks, RSP/INV to trading authorities/invariants, and SLO/SEC/REL to operational objectives, security, and releases.

Evidence notation: C identifies a claim in the claim register, A a dated AWS source, F a foundational source, and CS a finding in the repository case study. The source index supplies the full source details. These labels are lookup aids, not facts to memorize.

Use the same loop every time: ask for workload and business constraints; name authoritative state and invariants; choose delivery, order, consistency, and idempotency boundaries; walk the happy path; walk ambiguous failure, retry, replay, and reconciliation; quantify bottleneck, drain, and cost; cover observability, security, release, and DR evidence; then state the trade-off, falsifier, and migration trigger.

Practice in three modes: rapid verbal in 30–120 seconds; deep follow-up in 8–15 minutes; system design in 35–50 minutes. Read only the prompt-side fields, stop, answer aloud, then compare the layered reference answer. Alternate architectures score well when assumptions and proof are sound. Scores are study feedback, not an employer hiring bar.

A projection never becomes authority because it is fresh or strongly read. CQRS is a workload-dependent spectrum and is independent of event sourcing. Money and control totals use integer minor units or exact decimals with currency, scale, and rounding.

Quantitative policy: carry full precision through each formula; show integer counts exactly, show USD/month and ratios to six decimal places, and show time in exact seconds plus a nearest-minute explanation. The validator accepts absolute error no greater than 0.000001 only for recomputed decimal results; integer and ceiling results must match exactly. Every AWS price is the eu-west-1 price retrieved 2026-08-22 from the reviewed Task 10 PRICE rows; planning values are labeled explicitly.

Rapid service-choice drills

SC01 prompt — Domain fan-out versus owned work

• Facts: An accepted order fact feeds risk analytics notification and one executor
• Ask first: Ask peak rate burst payload fan-out latency ownership duplicate tolerance retention replay authority and on-call cost

Reveal SC01 reference answer

SC01 reference answer

• Decision tree: If independent bounded contexts each need the fact use EventBridge routing; if exactly one worker owns buffered work use SQS; SNS plus per-subscriber SQS is valid when push fan-out and subscription/resource-policy controls dominate
• Two minute: I separate fact distribution from work ownership; EventBridge fits content-routed domain facts, while SQS fits competing ownership and buffering, with stable IDs and idempotent effects in either branch
• Deep: Rules isolate consumers but do not establish completion; queues expose backlog and leases but do not create domain authority; choose only after failure ownership and replay source are named
• Guarantee: Producer acknowledgement, target delivery, consumer commit, and business effect remain separate; no cross-service atomic write
• Failure: Outbox intent prevents accepted-without-publication; inspect per-entry results; quarantine and reconcile rather than blind replay
• Cost ops: Requests times fan-out, payload, retry, queue age, DLQ/replay storage, engineering and on-call
• Observability: Outbox age, target failures including failed DLQ sends, oldest work, duplicate effects; least-privilege bus and queue policies
• Alternative: SNS to SQS for simple push subscriptions; Kinesis when retained independently positioned replay is required
• Falsifier: If content routing is unused or backlog ownership and per-message pacing dominate, migrate to queues; if independent retained replay becomes mandatory, use a log
• Followups: Why not direct dual write; who owns redrive; how is one customer effect proved
• Weak: Naming a favorite service; calling bus acceptance completion; retry or DLQ without authority and reconciliation
• Premises: C42 C60 C05 C08 C38 FSR02 SLO01

SC02 prompt — SQS Standard versus FIFO

• Facts: Payment-adjustment work has duplicates and some accounts require serial mutation
• Ask first: Ask ordering unit, group cardinality, throughput skew, retry identity, five-minute dedup boundary, consumer transaction and deadline

Reveal SC02 reference answer

SC02 reference answer

• Decision tree: Use Standard when commutative idempotent work and throughput dominate; FIFO when strict order within a well-distributed MessageGroupId is required; redesign a single hot group rather than claim global order
• Two minute: FIFO orders only within a group and broker dedup is time-bounded; both consumers still need an atomic inbox or effect receipt and reconciliation
• Deep: Standard may reorder and redeliver; FIFO can head-of-line block and dedup does not cover database or provider effects; choose group from the invariant and measure skew
• Guarantee: No global order and no one-time business effect; visibility is a lease and delete is acknowledgement
• Failure: Poison work blocks a FIFO lane; quarantine only with an explicit gap policy, then repair source order and validate account totals
• Cost ops: Batched request count, group distribution, hot lane, retention, retry work and idle consumers
• Observability: Oldest age by group, receive count, dedup conflicts, poison identity; isolate tenants and encrypt/classify payload
• Alternative: Kinesis for retained per-key replay; Standard plus version guards for commutative updates
• Falsifier: If hottest group breaches deadline or ordering is unnecessary, repartition or move; if retained multi-consumer replay appears, use a log
• Followups: Can dedup outlive five minutes; what happens after consumer commit but before delete
• Weak: Saying FIFO is exactly once; one queue means global order; optimistic locking is idempotency
• Premises: C02 C03 C04 C40 C41 C43 C63 FSR05

SC03 prompt — EventBridge routing versus SNS fan-out

• Facts: Several teams subscribe to execution facts across accounts
• Ask first: Ask filtering needs subscriber types push versus queues cross-account policy retry age DLQ ownership archive replay rate and payload

Reveal SC03 reference answer

SC03 reference answer

• Decision tree: Choose EventBridge for rule-based integration routing and bus boundaries; choose SNS for topic push fan-out, often to SQS; either is valid only with explicit resource policies and retry/DLQ ownership
• Two minute: EventBridge expresses content routes; SNS expresses topic subscriptions. Neither proves target effects. Validate destination policies, inspect producer results, retain source identity, and reconcile
• Deep: EventBridge target retry differs from Lambda runtime; a configured DLQ ARN is unusable without queue permission. SNS delivery and subscriber queue behavior are separate boundaries
• Guarantee: PutEvents acknowledgement ends before matching and targets; target DLQ ends before repair; SNS publication ends before endpoint effect
• Failure: Contain failed route or subscriber, retain manifest, repair policy/config, replay original IDs at bounded rate, reconcile every intended consumer
• Cost ops: Published events times matched targets, delivery attempts, archives, queue requests and cross-account operations
• Observability: FailedEntryCount, unmatched intents, target failures, failed DLQ sends, subscriber backlog; SourceArn and SourceAccount conditions
• Alternative: SNS FIFO to SQS FIFO when its documented subscriber/order conditions fit; outbox relay direct to owned queues for command work
• Falsifier: If rules become trivial and subscription push dominates, SNS may simplify; if per-subscriber backlog ownership dominates, attach queues
• Followups: What does HTTP 200 prove; why can DLQ depth be zero during loss
• Weak: Calling archive a ledger; assuming resource policy; checking CloudWatch generically
• Premises: C60 C64 C05 C08 C57 FSR02 FSR03 SEC03

SC04 prompt — Kinesis versus MSK

• Facts: A retained ordered market/execution stream needs multiple replaying consumers
• Ask first: Ask rate bytes key skew retention ecosystem Kafka API/connectors event time operations team regional recovery and total cost

Reveal SC04 reference answer

SC04 reference answer

• Decision tree: Kinesis fits AWS-native managed partitioned logs with shard/on-demand choices; MSK fits a deliberate Kafka protocol, ecosystem, transactions within Kafka, or portability need; either can be wrong for one hot key
• Two minute: Both require partition-key design, idempotent external effects, capacity and replay drills. MSK adds broker/ecosystem control; Kinesis reduces broker operations but has its own shard and consumer modes
• Deep: Order is partition-scoped; Kafka transactions stop before arbitrary external systems; neither is financial authority
• Guarantee: A poison or stalled partition blocks its ordered scope; restore checkpoints and reconcile rather than skip silently
• Failure: Retain original records, detect lag and poison lanes, recover checkpoints, pace replay, suppress external side effects and reconcile
• Cost ops: Peak bytes and records, partitions, retention, consumers/EFO, broker fleet/storage/transfer, staffing and DR
• Observability: Per-partition lag, throttles, ISR/broker health where relevant, poison identity, access and encryption boundaries
• Alternative: SQS for destructively owned work; Data Firehose for buffered landing; Flink for stateful event-time computation on a source
• Falsifier: Kafka ecosystem or existing expertise can select MSK; lower operational appetite and AWS-native integration can select Kinesis; measured hot-key infeasibility can force domain repartition
• Followups: Where do transactions stop; how is reshard/repartition ordering handled
• Weak: Kinesis is serverless Kafka; MSK gives external exactly once; aggregate throughput proves feasibility
• Premises: C10 C29 C75 kinesis_capacity FSR04

SC05 prompt — DynamoDB Streams versus semantic outbox or Kinesis

• Facts: Order state changes must publish stable domain facts and rebuild projections
• Ask first: Ask whether raw item images carry business meaning, transaction boundary, retention, replay horizon, fan-out, sequence and schema ownership

Reveal SC05 reference answer

SC05 reference answer

• Decision tree: Use a semantic outbox item in the same transaction as authority when publication contract matters; Streams can relay that item or expose acceptable CDC; add Kinesis when longer retained independent replay and throughput shape justify it
• Two minute: Raw CDC says an item changed, not why. An outbox fixes identity, type, causation and payload at commit. Kinesis extends replay but does not close the original database-message gap
• Deep: Streams order per item and retain briefly; outbox closes only local atomic intent; relay and consumers remain retryable
• Guarantee: Relay crash or retention expiry can strand or duplicate publication; repair leases and source history before bounded replay
• Failure: Repair outbox leases, never regenerate IDs, recover from authority or archive when retention expires, rebuild side-effect-free and reconcile
• Cost ops: Transactional rounded writes, outbox/GSI amplification, stream consumers, retention/archive and rebuild capacity
• Observability: Outbox oldest age, lease state, stream age, sequence gaps, duplicate effects; minimize sensitive payload and scope stream readers
• Alternative: Relational outbox/CDC when authority is Aurora; direct dual write only as explicitly repaired transitional debt
• Falsifier: If item images are sufficient and no semantic publication exists, raw Streams may fit; if retention/replay exceeds Streams, add a durable log/archive
• Followups: Can a Stream be an event store; what happens after relay publish timeout
• Weak: Calling CDC event sourcing; claiming outbox gives one business effect
• Premises: C23 C38 CS01 AP16 FSR02

SC06 prompt — Step Functions Standard versus Express versus local orchestration

• Facts: Withdrawal needs callbacks timeouts provider calls and possibly human approval
• Ask first: Ask duration callback/human wait invocation mode volume state size external ambiguity audit deadline compensation and cost

Reveal SC06 reference answer

SC06 reference answer

• Decision tree: Standard fits long-lived visible durable state and callbacks; Express fits high-volume short workflows only under its invocation semantics; local orchestration fits one bounded service transaction with no durable cross-boundary wait
• Two minute: Workflow history coordinates steps but is not the ledger. Persist provider idempotency keys and receipts, model UNKNOWN states and deadlines, post balanced ledger entries under ledger authority, and reconcile provider statements
• Deep: Standard workflow execution semantics stop at task/provider effects; Express mode changes retry boundary; compensation is forward action, not rollback
• Guarantee: Provider timeout after effect is ambiguous; workflow retry can duplicate an irreversible transfer without receipt lookup
• Failure: Timeout after provider call is ambiguous; contain account scope, lookup receipt before retry, escalate irreversible uncertainty, reconcile and record human action
• Cost ops: State transitions or duration/memory, callbacks, logs/history retention, provider calls, operator work and break loss
• Observability: Execution age, callback deadline, provider receipt gaps, ledger imbalance, privileged approval audit, secret rotation
• Alternative: A database-backed process manager when custom state/query/control is needed; choreography for few simple reversible reactions
• Falsifier: If all work is short stateless and volume makes Standard expensive, Express may fit; if workflow state must join rich domain queries, local process manager may win
• Followups: What is execution authority; how do you recover a lost callback
• Weak: Step Functions makes payment exactly once; compensation deletes history; Express is cheaper without workload
• Premises: C65 C39 RSP08 FSR07 INV06

SC07 prompt — Data Firehose versus a Kinesis consumer

• Facts: Compliance events need buffered S3 landing and a transformation team proposes custom code
• Ask first: Ask destination latency buffering transform complexity failure ownership source replay record size partitioning and need for per-record decisions

Reveal SC07 reference answer

SC07 reference answer

• Decision tree: Use Data Firehose for managed buffered delivery to supported destinations; use a Kinesis consumer when the team owns stateful validation routing low-latency actions or checkpoints
• Two minute: Firehose reduces delivery operations but destination semantics and duplicate warning vary; a consumer gives control and replay position but adds capacity, code, quarantine and on-call work
• Deep: Firehose is not a work queue or general replay log; successful landing is not ledger completeness
• Guarantee: Destination failure can delay or duplicate delivery; keep source and backup evidence and validate landed manifests
• Failure: Retain source manifest, monitor delivery errors/backups, repair destination/config, reprocess stable identities, validate object and authority control totals
• Cost ops: Ingest and converted bytes, buffering, transforms, destination writes, backup storage versus consumer compute/checkpoints/operations
• Observability: Delivery freshness/errors, backup prefix, object manifests, encryption/key access and data classification
• Alternative: Direct S3 batch/export for scheduled snapshots; Flink for stateful event-time transforms
• Falsifier: If custom state or per-record deadline appears, own a consumer; if processing becomes simple buffered landing, Firehose removes toil
• Followups: Which destinations change duplicate warning; where is replay source
• Weak: Calling Firehose Kinesis; assuming S3 objects prove financial completeness
• Premises: C28 C66 C119 EVD03

SC08 prompt — Lambda consumer versus Flink MSK or another stateful processor

• Facts: A stream computes event-time windows and must replay late data
• Ask first: Ask event-time state size timers lateness checkpoint/source/sink compatibility throughput latency tail replay side effects and team expertise

Reveal SC08 reference answer

SC08 reference answer

• Decision tree: Lambda fits stateless or bounded per-record/batch transforms; Flink fits managed event-time state and checkpoints; MSK plus stream framework fits Kafka ecosystem; long-lived custom workers fit specialized state/control
• Two minute: Select from measured state lifetime, watermark behavior, recovery and sink semantics. Lambda partial batch responses reduce work but do not supply durable event-time state or atomic external effects
• Deep: Checkpoint guarantees stop at compatible source, state and sink; Lambda checkpointing stops before business effect
• Guarantee: Poison data, late events or checkpoint restore can repeat outputs; isolate state and suppress external effects during replay
• Failure: Quarantine poison data with order policy, restore checkpoint/savepoint, replay isolated state, suppress notifications/provider calls and reconcile output
• Cost ops: State bytes and checkpoint storage, compute duty, shuffle, source reads, sink amplification, replay and operator expertise
• Observability: Watermark lag, checkpoint duration/failure, iterator age, late records, output control totals, scoped roles and state encryption
• Alternative: KCL/ECS worker for custom per-shard state; database projection with source versions for simpler aggregates
• Falsifier: If state fits one idempotent transaction and windows disappear, Lambda may simplify; if checkpoints miss SLO or ecosystem burden is excessive, custom workers may win
• Followups: Can Flink make an external API exactly once; how are late corrections represented
• Weak: Defaulting to Lambda because events are present; replaying side effects
• Premises: C11 C12 C53 C67 REL06

SC09 prompt — Lambda versus ECS Fargate or EC2

• Facts: Sustained compute includes a latency-critical matching candidate
• Ask first: Ask measured service time duty peak-to-average tail target cold behavior state connections deterministic sequencing hardware failure recovery deployment and FTE cost

Reveal SC09 reference answer

SC09 reference answer

• Decision tree: Lambda fits bursty stateless edges when measured warm and cold tails pass; ECS/Fargate fits sustained long-lived processes; EC2 fits maximum runtime/hardware/control. A deterministic low-tail matcher defaults to evidenced long-lived compute, not a label
• Two minute: Compute price parity is separate from capacity and operating parity. Model peak headroom, scaling lag, queue growth, HA, state recovery and release fencing before cost
• Deep: No runtime supplies financial correctness; matcher journal, one writer epoch and deterministic replay establish its scoped boundary
• Guarantee: Cold starts, scaling lag, host loss or a second writer can breach tail and sequence invariants; fence and recover from journal
• Failure: Fence writers, preserve journal, fail over one epoch, replay with publication suppressed, compare decision digests, reconcile executions and ledger
• Cost ops: Invocations/GB-s versus task/instance hours, minimum HA fleet, utilization, transfer, observability, deployment and on-call
• Observability: p50/p99/cold tail, queue age, writer epoch conflict, journal flush, decision digest, least-privilege release role
• Alternative: Managed Batch for offline jobs; specialized hardware only with measured need
• Falsifier: If cold tail or sustained fleet economics/SLO fail, leave Lambda; if long-lived fleet idle and complexity dominates while Lambda measurements pass, move edges back
• Followups: Why not Lambda matcher; what does the reviewed no-price-crossing example prove
• Weak: Serverless is cheaper; containers are faster; ignoring utilization and failure recovery
• Premises: C53 C115 CS12 ARCC REL07

SC10 prompt — Authority and read-store choices

• Facts: Orders balances portfolios search and low-latency reads have different access patterns
• Ask first: Ask every access pattern key/range consistency read-your-writes authority invariant volume item size relationships freshness rebuild and operations

Reveal SC10 reference answer

SC10 reference answer

• Decision tree: Start with APs and physical lanes. DynamoDB can own keyed transactional aggregates; Aurora can own relational constraints; cache and search are derived accelerators; choose each path from the invariant, never brand preference
• Two minute: Keep reservation ledger and legal order transition on command authority. Serve portfolio/search/cache projections with version/freshness and overlay durable receipts for read-your-writes
• Deep: GSI/cache/search freshness never transfers authority; transaction boundary is store-specific and cross-service publication needs outbox
• Guarantee: Projection lag, eviction or index gap can serve stale data; fall back or label freshness and rebuild vNext
• Failure: On projection gap serve stale-with-label or authority fallback, rebuild vNext from durable source, catch up, cut over conditionally and reconcile
• Cost ops: Rounded item/read units, GSIs, replicas, cache hit/eviction, search indexing, storage/rebuild and staff
• Observability: Hot physical lanes, throttles, freshness/gaps, cache miss, exact totals; tenant key isolation and evidence access
• Alternative: Aurora for cross-row relational workloads; DynamoDB for bounded key access; no cache when source meets SLO economically
• Falsifier: Measured access-pattern or invariant change falsifies the store; migrate with single authority plus CDC/outbox and reversible reads
• Followups: Can a strong read of portfolio authorize withdrawal; single versus multi-table
• Weak: Schema-first table design; scans as APIs; cache as source of truth
• Premises: AP01 AP07 AP12 C46 C21 C92 REL04

Failure-analysis drills

Service selection established why a component fits. Failure analysis tests whether that choice survives an incomplete or ambiguous result. Draw the commit timeline before proposing recovery, mark what is known versus unknown, and distinguish a safe repeated lookup from repeating a financial effect. Use the observed code evidence where the prompt provides it.

FA01 prompt — Accepted order missing publication

• Observed: Order write and cached 201 precede PutEvents at order-handler lines 155-208 and retry cache return at 229-241
• Trigger: Write succeeds then publish throws times out or is dropped by configuration
• Ask first: Ask command identity timeline exact responses bus/rule version pending age customer retry and outbox evidence

Reveal FA01 reference answer

FA01 reference answer

• Authority: Order row is apparent operational authority; no as-built ledger; publication intent should be durable outbox
• Impact: Unknown acceptance to customer and stranded order or duplicate from a new key
• Severity: High financial correctness and support severity; Critical if invariant or customer loss exposure is active
• Two minute: Freeze duplicate creation, query canonical key, preserve order and request fingerprint, inspect durable intent, then repair publication with original identity and reconcile
• Deep: Prevent with atomic order idempotency result and outbox; detect accepted-without-outbox/receipt; contain account; repair from authority; validate downstream receipt and state
• Retry: Retry cannot create missing durable intent safely; same key can falsely return 201 and a new key changes identity
• Containment: Reject new-key workaround, pause affected relay lane and expose PENDING status
• Repair: Create or repair outbox conditionally from order after investigation, relay original event ID through idempotent consumers
• Reconciliation: Every accepted order maps to outbox, legal transition, execution or cancellation and ledger/reconciliation disposition
• Owner: Order API and messaging owners; RBK01/RBK02
• Signals: Pending age, outbox oldest, publish entries, duplicate keys, downstream receipt, SLO01
• Residual: Ambiguous producer timeout and configuration drift remain; no repair proves an external effect without receipt/reconciliation
• Falsifier: If a proven transactional publication boundary already exists, use it; otherwise escalate when missing-intent count or deadline breaches
• Followups: Would reversing write order help; how do you expose client status
• Weak: Retry; delete cache; trust 201; call EventBridge the authority
• Premises: CS01 CS02 FSR02 RBK01 RBK02 SLO01

FA02 prompt — Partial PutEvents and ambiguous outcome

• Observed: Three handlers discard PutEvents results at exact repository routes in CS03
• Trigger: HTTP 200 has failed entries or client times out after submission
• Ask first: Ask original entry identity and per-entry response bus ARN rule deployment retry owner and downstream receipts

Reveal FA02 reference answer

FA02 reference answer

• Authority: Authoritative order/trade plus durable outbox manifest; EventId is not business authority
• Impact: Some intents absent or duplicated; stale portfolio/notification and destructive redrive risk
• Severity: High; Critical if financial postings or customer decisions depend on missing fact
• Two minute: Stop whole-batch retry and message deletion; preserve manifest; classify explicit failure versus unknown
• Deep: Prevent stable IDs and inspected results; detect control-total gaps; contain bad publisher/config; repair explicit failures and reconcile unknowns; validate each intended effect
• Retry: Retry explicit failed entries only; ambiguous entries may have been accepted, so idempotent consumers and receipt lookup are required
• Containment: Disable destructive replayer, validate named bus/rules and isolate publisher lane
• Repair: Republish original identity only after classification or accept duplicate into atomic inbox; never regenerate fact
• Reconciliation: Attempted equals explicit result plus owned unknown; each intent maps to downstream receipt or reconciled authority
• Owner: Messaging owner plus domain owner; RBK02
• Signals: FailedEntryCount, per-code count, outbox age, rule canary, receipt gaps, SLO01
• Residual: Nominal zero failures can coexist with nonexistent-bus drop; configuration validation is independent
• Falsifier: If producer offers a durable idempotent receipt lookup, use it; breach after deadline opens incident
• Followups: What does EventId prove; how can zero DLQ depth mislead
• Weak: Retry whole batch; delete after HTTP 200; check CloudWatch only
• Premises: CS03 C08 FSR03 RBK02 C57 SLO01

FA03 prompt — Duplicate fill and partial posting

• Observed: Trade insert order update publication are separate at 83-145; portfolio has no inbox and uses floating point
• Trigger: Crash after trade write or duplicate TradeExecuted during partial ledger posting
• Ask first: Ask stable execution ID order version partial-fill semantics posting transaction currency scale provider receipt and replay source

Reveal FA03 reference answer

FA03 reference answer

• Authority: Execution authority owns fill; ledger owns balanced postings/reservations; portfolio is derived
• Impact: Duplicate holdings or one-sided cash/securities postings, incorrect availability and customer harm
• Severity: Critical financial integrity
• Two minute: Fence affected account/order, stop projector/replay and withdrawals, preserve event and posting evidence
• Deep: Prevent deterministic execution ID, conditional order transition, balanced posting transaction and inbox; detect duplicates/imbalance; contain scope; repair append correction; validate reconciliation
• Retry: Retry is safe only when same execution identity returns the prior effect receipt; it cannot blindly repeat external or posting effects
• Containment: Freeze projection publication and financially sensitive actions for affected scope
• Repair: Recover missing side from authority or append reviewed reversal/correction; rebuild projection vNext
• Reconciliation: Exact minor-unit debits equal credits; one execution identity maps to one posting set; order CumQty and ledger reconcile
• Owner: Execution ledger and reconciliation owners; RBK07
• Signals: Duplicate IDs, posting imbalance, version gaps, negative positions, SLO05
• Residual: External venue ambiguity and prior customer action on stale view remain
• Falsifier: If the ledger transaction already atomically covers both sides, focus on duplicate receipt; otherwise incident stays open
• Followups: Why optimistic locking fails; partial fills and corrections
• Weak: Replay fixes it; overwrite balance; portfolio proves ownership
• Premises: CS04 CS05 CS06 CS10 C47 INV06 FSR04 RBK07 SLO05

FA04 prompt — Poison record blocks ordered lane

• Observed: One malformed execution repeatedly fails before checkpoint
• Trigger: A schema-incompatible record reaches the lowest uncommitted sequence and every retry fails deterministically
• Ask first: Ask ordering scope failed sequence batch settings retention age later-record policy schema owner and side effects already made

Reveal FA04 reference answer

FA04 reference answer

• Authority: Retained source plus authoritative execution/ledger; checkpoint is processing position only
• Impact: Freshness breach for one key and possible downstream deadline while other lanes look healthy
• Severity: High by affected financial deadline and scope
• Two minute: Stop unbounded retry, preserve raw bytes and identity, cap concurrency impact, do not skip silently
• Deep: Prevent schema compatibility tests; detect iterator age and poison identity; contain lane; repair decoder/data or explicit compensating sequence; validate gap closure
• Retry: Retry does not fix deterministic poison and consumes retention/headroom; DLQ/quarantine is holding, not terminal repair
• Containment: Quarantine only with explicit ordered-gap policy and park later versions
• Repair: Deploy compatible decoder canary, reinsert or replay original sequence at bounded rate, rebuild if needed
• Reconciliation: Contiguous expected versions, source-to-applied manifest and exact totals reconcile before release
• Owner: Stream consumer and domain owner; RBK04
• Signals: Per-shard age, failed sequence, retry count, retention margin, gap age, SLO03
• Residual: Skipping may violate order; source may expire before fix
• Falsifier: If event is provably irrelevant and invariant-safe, governed skip can close; otherwise rebuild from longer source
• Followups: Bisect versus partial response; why not keep retrying
• Weak: Send to DLQ; replay blindly; aggregate lag only
• Premises: FSR05 RBK04 C11 C12 C43 SLO03

FA05 prompt — Lagging or gapped projection rebuild

• Observed: Portfolio lacks source sequence and daily snapshot is incomplete
• Trigger: Gap alarm or discrepancy appears while live updates continue
• Ask first: Ask authority source completeness retention watermark version side-effect suppression capacity cutover and rollback compatibility

Reveal FA05 reference answer

FA05 reference answer

• Authority: Execution/ledger remains authority; portfolio vCurrent and vNext are projections
• Impact: Stale holdings and unsafe decisions if projection is trusted; rebuild can duplicate notifications
• Severity: High; Critical only if stale view authorizes money/securities
• Two minute: Remove projection from authorization, label freshness, preserve old view and isolate rebuild destination
• Deep: Prevent inbox/version checks; detect gaps and exact discrepancies; contain reads; repair vNext from durable source; validate before conditional alias cutover
• Retry: Retrying later events cannot fill unknown missing versions; replay without idempotency and side-effect suppression can worsen harm
• Containment: Park future versions, cap rebuild below live SLO and disable external outputs
• Repair: Snapshot/manifest then deterministic backfill, catch-up to watermark, compare counts/amounts/versions, canary reads, atomic alias switch
• Reconciliation: No gaps, exact totals and watermark match authority before/after cutover; rollback only while old view compatible
• Owner: Projection owner plus finance reconciliation; RBK05
• Signals: Freshness, gap age, rebuild ETA, source retention, control totals, SLO03
• Residual: Incomplete source or changed semantics may require forward repair rather than rollback
• Falsifier: If source retention cannot finish catch-up, restore archive or rebuild window; abort when live SLO burns
• Followups: Why not rebuild in place; read-your-writes during cutover
• Weak: Replay as repair; strong-read projection; switch on row count alone
• Premises: FSR09 RBK05 REL05 CS06 CS07 CS09 SLO03

FA06 prompt — Region/dependency failure with reconciliation discrepancy

• Observed: Business probes fail, recovery-point age rises, writer epoch conflicts, provider statement differs
• Trigger: Regional outage or dependency ambiguity during failover
• Ask first: Ask RTO RPO authority replication mode provider availability client DNS/keepalive fencing evidence and discrepancy window

Reveal FA06 reference answer

FA06 reference answer

• Authority: One fenced command/ledger writer; provider and ledger have separate authorities joined by reconciliation
• Impact: Unavailable orders or split brain, lost/duplicate external actions and financial break
• Severity: Critical when second writer or unresolved imbalance exists
• Two minute: Stop writes, fence old Region, block stale endpoints, preserve manifests and protect evidence
• Deep: Prevent tested topology and keys; detect business probes/epoch/RPO; contain one writer; repair authority first then outbox/projections; validate provider and ledger totals
• Retry: Failover does not repair ambiguous external calls or propagated corruption; retry only after receipt lookup
• Containment: Route no commands until one epoch and dependency/KMS readiness are proved
• Repair: Promote one writer, replay missing intent under capacity caps, rebuild derived views, reconcile half-open UTC window, fail back separately
• Reconciliation: Measured RTO/RPO, one epoch, stale-client rejection, exact ledger/provider control totals and closed breaks
• Owner: Incident commander, ledger, platform, external ops and compliance; RBK08
• Signals: Business SLOs, epoch conflict, recovery-point age, dependency probes, break aging
• Residual: DNS caches, long connections, provider ambiguity and corruption survive topology move
• Falsifier: If corruption is replicated use clean restore rather than failover; if RPO evidence fails stay read-only
• Followups: RTO versus RPO; how to fence old clients
• Weak: Fail over and retry; multi-Region equals zero loss; encryption proves compliance
• Premises: FSR12 FSR07 RBK08 DR01 DR02 DR03 DR04 REL08 SLO05

Quantitative exercises

Write the inputs and units separately before calculating. Derive aggregate capacity, then test the hottest lane, live-versus-recovery spare, and the complete cost numerator. Keep the original precision until the final presentation. A result such as no safe drain is a valid design finding that needs an operating response, rather than a number to force through a division.

QE01 prompt — Kinesis shards, hot keys, and backlog drain

Given data

peak

8000 records/s

payload

700 B/record

shard_bytes

1000000 B/s/shard

shard_records

1000 records/s/shard

skew

1.25 ratio

util

0.80 ratio

top_share

0.15 ratio

commit

9000 records/s

live

5000 records/s

safety

1000 records/s

backlog

4500000 records

• Questions: Size shards by bytes and records; test the top key; compute spare and drain

Reveal QE01 reference answer

QE01 reference answer

• Formula: shards=ceil(max(peak*payload/shard_bytes,peak/shard_records)*skew/util); spare=commit-live-safety; drain=backlog/spare
• Result: shards=13 shards;hot_key=1200 records/s;spare=3000 records/s;drain=1500 s
• Conclusion: Thirteen aggregate shards fit, but a 1200-record/s single key exceeds the 1000-record/s lane; 4.5M drains in 25 minutes only with protected spare
• Sensitivities: Half payload still record-bound; top_share above 0.125 stays infeasible; commit<=live+safety gives NO_SAFE_DRAIN
• Mistakes: Using MB for records, omitting skew/headroom, dividing by total capacity, or hiding Infinity
• Premises: kinesis_capacity kinesis_recovery kinesis_planned_shards kinesis_drain_seconds C113 C114

QE02 prompt — DynamoDB write amplification and hot lanes

Given data

order

1.2 KiB/item

idem

0.7 KiB/item

outbox

1.6 KiB/item

sentinel

0.2 KiB/item

account_gsi

0.45 KiB/entry

status_gsi

0.35 KiB/entry

ready_gsi

0.45 KiB/entry

txn_multiplier

2 WRU/ordinary-WRU

peak_orders

120 orders/s

order_share

0.50 ratio

• Questions: Round every item; derive base, indexes, aggregate, and hottest physical order lane

Reveal QE02 reference answer

QE02 reference answer

• Formula: base=txn_multipliersum(ceil(each base KiB)); gsi=sum(ceil(each index KiB)); aggregate=(base+gsi)peak_orders; hot=peak_orderssharetxn_multiplier*ceil(order)
• Result: base=12 WRU/order;gsi=3 WRU/order;total_per_order=15 WRU/order;aggregate=1800 WRU/s;hot_order_lane=240 WRU/s
• Conclusion: Billing work is 1800 WRU/s but lanes are separate; aggregate can fit while a physical lane fails, so compare every table/index key lane to its target
• Sensitivities: At order_share 1.0 the lane is 480 WRU/s; a 200-WRU/s lane target is an explicit planning assumption and fails despite aggregate provision; adding a replica changes operation/pricing topology
• Mistakes: Rounding after summing, forgetting transaction doubling, charging GSI transaction multiplier, or treating aggregate as one partition
• Premises: dynamodb_amplification AP06 AP16 ddb_wru_per_order C113

QE03 prompt — Lambda fan-out, replay, and allocated cost

Given data

rate

5000 records/s

seconds

2592000 s/month

batch

100 records/invocation

consumers

3 consumers

retry

0.02 retries/original

replay

0.05 replay-records/original

memory

1 GB

duration

0.08 s/invocation

request_price

0.0000002000 USD/invocation

gb_second_price

0.0000166667 USD/GB-s

shared_engineering

12000 USD/month

shared_oncall

6000 USD/month

shared_compliance

4000 USD/month

• Questions: Using reviewed Task 10 Lambda prices for eu-west-1 retrieved 2026-08-22 and excluding free tier symmetrically, compute replay-inclusive billed invocations for the same three consumers, consumer work, request and duration cost, then marginal and allocated cost

Reveal QE03 reference answer

QE03 reference answer

• Formula: source=rateseconds; processed=source(1+retry+replay); invocations=processed/batchconsumers; work=processedconsumers; request_cost=invocationsrequest_price; gb_seconds=memoryduration; duration_cost=invocationsgb_secondsgb_second_price; marginal=request_cost+duration_cost; allocated=marginal+shared_engineering+shared_oncall+shared_compliance
• Result: source_records=12960000000 records/month;invocations=416016000 invocations/month;consumer_work=41601600000 consumer-records/month;request_cost=83.203200 USD/month;gb_seconds=0.080000 GB-s/invocation;duration_cost=554.689109 USD/month;marginal=637.892309 USD/month;allocated=22637.892309 USD/month
• Conclusion: Replay is explicitly processed through the same three Lambda consumers, so it appears in both billed invocations and work; marginal is this Lambda path while allocated adds the stated shared engineering/on-call/compliance burden
• Sensitivities: Batch 10 multiplies every replay-inclusive invocation and compute charge by ten; double consumers doubles work and marginal cost; poison coupling can reach the whole batch without durable partial checkpoints
• Mistakes: Counting source records as consumer work, omitting replay from billed invocations, applying one invocation to all consumers, hiding regional price inputs, or calling allocated cost marginal
• Premises: batching fanout lambda_request_usd lambda_gb_second_usd engineering_usd oncall_usd compliance_usd C49 C113 A123

QE04 prompt — CQRS fixed costs and break-even sensitivity

Given data

variable_transport

94.392164 USD/month

variable_projection

173.5992 USD/month

fixed_cache

150 USD/month

variable_rebuild

8.67996 USD/month

variable_reconciliation

2.679914 USD/month

fixed_engineering

900 USD/month

fixed_oncall

300 USD/month

fixed_other

1000 USD/month

writes

61560000 writes/month

avoided_read_value

0.000000566 USD/read

• Questions: Using reviewed Task 10 eu-west-1 inputs retrieved 2026-08-22, sum separately labeled fixed and variable burden; derive base, half-volume and double-volume break-even; decide whether ratio alone justifies CQRS

Reveal QE04 reference answer

QE04 reference answer

• Formula: variable=sum(variable components); fixed=sum(fixed components); total=fixed+variable; base=total/(writesavoided_read_value); half=(fixed+0.5variable)/(0.5writesavoided_read_value); double=(fixed+2variable)/(2writes*avoided_read_value)
• Result: variable=279.351238 USD/month;fixed=2350.000000 USD/month;total=2629.351238 USD/month;base=75.462912 reads/write;half=142.908388 reads/write;double=41.740175 reads/write
• Conclusion: The reviewed alternative needs about 75.46 reads/write for base cost parity; authority, 2-second freshness, recovery and operations can still reject it
• Sensitivities: Half volume keeps fixed burden at 2350.000000 and halves variable burden to 139.675619, giving 142.908388 reads/write; double volume keeps fixed burden and doubles variable burden to 558.702476, giving 41.740175 reads/write; zero avoided benefit gives NO_ECONOMIC_WIN before division
• Mistakes: Omitting fixed work, scaling fixed inputs with traffic, using read/write ratio without price units, treating cache hit as invariant proof, or dividing by zero benefit
• Premises: cqrs_components cqrs_break_even cqrs_break_even_reads_per_write C35 C49 C102

Full system-design drills

Build the answer around one coherent obligation from admission to proof of completion. State the workload and authority first, then draw the happy path and a failure path. Use the quantitative work to challenge the design, and finish with the measurement that would cause you to change the architecture.

SD01 prompt — Design a read-heavy brokerage experience

• Ask first: Ask rates/peaks, read paths, p99, freshness/read-your-writes, authority, order unit, replay/retention, RTO/RPO, tenants, cost and team
• Answers: Assume 100 order/s average, 0.25 active duty, 95 percent accepted, 61,560,000 accepted-source writes/month, 50 reads/write, 2-second portfolio freshness, eu-west-1 planning baseline, Task 10 fixed burden 2350.000000 USD/month and variable burden 279.351238 USD/month, avoided read value 0.000000566 USD/read, no projection authorization

Reveal SD01 reference answer

SD01 reference answer

• Scope: Order status, portfolio/search, acceptance receipt and audit; explicit freshness and customer status
• Non goals: No venue matcher, clearing implementation or universal multi-Region guarantee
• Authority: Order/risk/reservation and ledger are authority; projections are rebuildable; exact money and balanced postings
• Boundary: Per-account order/version; transactional command plus outbox; inbox and expected projection version; receipt overlay for read-your-writes
• Architecture: API to command transaction/outbox; relay and routed facts; independent portfolio/search/cache projections; ledger reconciliation and S3 evidence
• Workload: Read amplification can justify denormalized AP07/cache/search, but write amplification, freshness and rebuild remain priced
• Failure: Ambiguous acceptance uses receipt lookup; gaps park and rebuild vNext; replay suppresses side effects; reconciliation closes exact totals
• Capacity: Task 10 gives 61,560,000 accepted-source writes/month from 1002,592,0000.250.95; break-even=(2350.000000+279.351238)/(615600000.000000566)=75.462912 reads/write, so compare the assumed 50 reads/write rather than assume CQRS pays
• Evidence: SLO01/SLO03, outbox age, gaps, exact totals, tenant IAM, PII redaction, canary contracts, REL05 rebuild, tested RTO/RPO
• Two minute: I keep command and ledger authority small, publish durable intent, and buy read models only for named APs with version/freshness and reversible rebuilds
• Deep: Explain AP inventory, physical lanes, overlay receipts, projection manifests, SLOs, security boundary and cost denominator
• Followups: Why cache not authority; how cut over vNext; what if top account is hot; restore after retention; tenant leak response
• Weak: Two databases equals CQRS; eventual consistency everywhere; cache authorizes withdrawals; no reconciliation
• Alternative: Aurora authority if relational invariants dominate; same-table CQRS-lite if projections do not pay
• Falsifier: If freshness, rebuild, or complete-cost evidence fails, serve authority/fewer views; sustained access-pattern change triggers migration
• Rubric: Score authority/AP routes, conditional read-store choice, exact failure proof and quantified falsifier
• Premises: ARCA AP07 C35 AP12 REL05 SLO01 SLO03 SEC02 cqrs_break_even

SD02 prompt — Design write-heavy market and execution ingestion

• Ask first: Ask peak records/bytes, burst/skew, key/order, retention/replay, consumer fan-out, event time, tail, source authority, sink capacity, RTO/RPO and ops
• Answers: Assume 8000 peak records/s, 700 B, 15 percent hot key, three consumers, 24-hour stream retention and longer S3 archive

Reveal SD02 reference answer

SD02 reference answer

• Scope: Durable ingestion, per-symbol/order sequence, independent risk/compliance/projection consumers and paced replay
• Non goals: No cross-symbol total order, retail query UI or claim that broker log is ledger
• Authority: Venue/matcher owns execution sequence; ledger owns postings; stream is retained transport
• Boundary: Partition order only; stable event ID; consumers use inbox/version; external effects outside checkpoint
• Architecture: Fenced ingress writers to Kinesis or MSK, raw archive, Lambda/KCL consumers for simple paths, Flink only for event-time state, outbox into ledger boundaries
• Workload: Minimize synchronous indexes/fan-out; buffer and batch, but top key 1200/s is infeasible against a 1000/s lane until domain repartition
• Failure: Partial producer results classified; poison lane contained; retained or restored replay at protected spare; exact execution/ledger reconciliation
• Capacity: Thirteen aggregate shards by reviewed model, yet hot key is first bottleneck; drain 4.5M in 1500s at 3000/s spare
• Evidence: Iterator age per shard, throttles, gaps, duplicate fills, exact totals, source/sink permissions, immutable release artifacts and DR drills
• Two minute: I size bytes and records with skew, then reject aggregate feasibility when one business key breaches its lane
• Deep: Discuss sequence assignment, repartition migration, producer manifests, checkpoint/external boundary, replay source/sink and cost
• Followups: How preserve order during reshard; what if archive expired; EFO or shared reads; late event correction; no-spare branch
• Weak: Aggregate throughput proves fit; stream is event store/ledger; replay sends notifications again
• Alternative: MSK for Kafka ecosystem/transactions inside Kafka; custom journal for matcher decisions
• Falsifier: If one key cannot partition without breaking invariant, isolate/dedicated writer; if Kafka ecosystem value exceeds ops burden, migrate
• Rubric: Score dimensional capacity, hot-lane recognition, guarantee boundary, replay validation and credible alternative
• Premises: ARCB C10 C29 kinesis_capacity kinesis_recovery FSR04 RBK04

SD03 prompt — Design a mixed exchange with deterministic matching core and serverless edges

• Ask first: Ask matching p99/jitter, order books, writer scope, peak burst, journal durability, risk/ledger boundary, client deadline, DR fencing, release and staff
• Answers: Planning assumptions: 8000 commands/s aggregate peak, hottest book 2000 commands/s, measured service time 0.0002 s/command, 1.5 capacity headroom, symbol-partitioned books, durable acceptance, long-lived single writer per book, matcher fleet 9000 cost-units/month and serverless-edge baseline 3000 cost-units/month

Reveal SD03 reference answer

SD03 reference answer

• Scope: Acceptance, pre-trade reservation, deterministic match/execution, postings, projections, notifications and audit
• Non goals: No Lambda default for matcher, no global order, no instant active-active ledger
• Authority: Command/risk, matcher journal/execution and balanced ledger are distinct authorities; views remain projections
• Boundary: One fenced epoch per book, monotonic order/execution versions, stable IDs, outbox/inbox between authorities
• Architecture: Serverless API/validation and durable command intent; long-lived matcher fleet/journal; transactional posting service; event log/bus to serverless projections and workflow edges
• Workload: Write-heavy core minimizes synchronous work; read-heavy edges denormalize; hot book and journal flush dominate before monthly average
• Failure: Unknown acceptance lookup; matcher crash replays journal with publication suppressed; duplicate fill returns receipt; reconcile executions to postings; fenced regional recovery
• Capacity: A writer provides 1/0.0002=5000 commands/s; aggregate plan is ceil(8000*1.5/5000)=3 book workers at 0.533333 aggregate utilization, hottest-book utilization is 2000/5000=0.400000, and matcher-fleet/edge cost ratio is 9000/3000=3.000000 under these explicit planning inputs
• Evidence: Epoch conflicts, decision digest, journal depth, duplicate fills, imbalance, SLOs, tenant/security evidence, REL07 shadow release and DR01/DR02/DR03 fenced recovery
• Two minute: I put deterministic sequencing on an evidenced long-lived single-writer substrate and keep elastic APIs, workflows and projections around it
• Deep: Expand reservations, cancel/replace/partial fill, outbox boundaries, exact money, canary symbol rollout, recovery proof and unit economics
• Followups: Why not Lambda; failure after match before publish; hot symbol; standby promotion; rollback after new journal format; recompute workers and utilization at 12000 commands/s or 0.0004 s/command
• Weak: Serverless everywhere; EventBridge sequences fills; portfolio is balance; active-active without fencing
• Alternative: Managed low-latency specialized substrate or Kafka-based core only if measurements and deterministic recovery contract fit
• Falsifier: If measured Lambda tail/state and recovery pass it can serve a bounded low-rate case; if single writer saturates partition by invariant or hardware
• Rubric: Score domain authorities, matcher substrate evidence, deterministic recovery, financial reconciliation and migration trigger
• Premises: ARCC CS12 REL07 C53 C115 INV01 INV06 DR01 DR02 DR03

SD04 prompt — Design long-lived settlement or withdrawal with external ambiguity

• Ask first: Ask amount/currency, approval policy, provider idempotency/receipt, callbacks/timeouts, deadlines, compensation limits, ledger/settlement authority, reconciliation, humans, RTO/RPO and security
• Answers: Planning assumptions: 20 withdrawals/s peak, 0.25 monthly peak-duty ratio, 2592000 s/month, 2 s provider service time, 1.5 concurrency headroom, 90 concurrent-call maximum, 10000 obligations due within 600 s, 0.01 breaks/obligation reconciliation share, 12 workflow transitions/withdrawal, 1200-break recovery backlog due within 1800 s, 0.5 breaks/s operator-reconciliation capacity, 0.1 breaks/s live arrival and 0.1 breaks/s protected safety; provider can time out after effect, callback may duplicate, two-person high-value approval, daily statements

Reveal SD04 reference answer

SD04 reference answer

• Scope: Durable request/approval, execution intent, provider interaction, settlement state, balanced postings, exceptions and evidence
• Non goals: No assumption provider supports cancellation, no workflow-as-ledger, no automatic compensation of irreversible transfer
• Authority: Execution request authority, approval authority, provider/settlement receipt and financial ledger are separate; reconciliation joins them
• Boundary: Stable withdrawal and provider keys; legal state transitions; exact postings; callback inbox; UNKNOWN state blocks blind retry
• Architecture: API command transaction/outbox; Step Functions Standard callback/task token or durable process manager; provider adapter receipt lookup; ledger posting commands; exception queue and reconciliation batches
• Workload: Long waits favor durable orchestration; provider calls, transitions, evidence retention, operator hours and expected break loss drive cost
• Failure: Timeout becomes UNKNOWN; contain account, lookup receipt, do not duplicate; callback dedup/version; append correction not overwrite; reconcile statement window and human sign-off
• Capacity: provider_desired=ceil(2021.5)=60 concurrent calls;provider_assigned=min(60,90)=60 concurrent calls;provider_protected=60/(21.5)=20.000000 withdrawals/s;deadline_required=10000/600=16.666667 obligations/s;deadline_reconciliation=16.6666670.01=0.166667 breaks/s;operator_protected=0.5-0.1-0.1=0.300000 breaks/s;deadline_state=SAFE;recovery_required=1200/1800=0.666667 breaks/s;recovery_state=NO_SAFE_DEADLINE;transitions=200.252592000*12=155520000 transitions/month;drain=1200/0.3=4000 s=66.666667 minutes
• Evidence: SLO05/SLO06, receipt gaps, callback duplicates, ledger balance, privileged approvals, secrets/KMS, release compatibility, DR fencing
• Two minute: I model the provider timeout as ambiguous, keep workflow state separate from ledger authority, and require receipt lookup plus reconciliation before retry or closure
• Deep: Walk every state and owner, irreversible boundary, compensation limits, callback security, exact postings, break lifecycle and failover; with the stated 90-call maximum, doubled provider time lowers protected throughput to 15.000000 withdrawals/s versus 16.666667 obligations/s, so the sensitivity is NO_SAFE_DEADLINE
• Followups: Lost callback; provider says sent but ledger absent; human override; secret compromise; Region loss; failback; doubled_service=22=4 s;doubled_desired=ceil(2041.5)=120 concurrent calls;doubled_assigned=min(120,90)=90 concurrent calls;doubled_provider_protected=90/(41.5)=15.000000 withdrawals/s;deadline_required=10000/600=16.666667 obligations/s;sensitivity_state=NO_SAFE_DEADLINE
• Weak: Retry provider; workflow succeeded so money moved; compensation is rollback; CloudTrail alone is audit completeness
• Alternative: Database process manager for richer domain queries; choreography only for few simple reversible reactions
• Falsifier: If callbacks are reliable and flow short Express/local may fit; if Standard history/query limits or cost breach, migrate state while preserving IDs
• Rubric: Score execution/approval/settlement/ledger separation, ambiguity handling, exact reconciliation, security evidence and falsifier
• Premises: C65 C39 RSP08 RSP09 INV06 FSR07 FSR12 SLO05 SLO06 SEC05

Repository walk-through drills

RW01 prompt — Order write plus PutEvents gap

• Evidence: code/lambdas/order-handler/handler.ts:155-165,167-181,183-208; code/lambdas/order-handler/handler.ts:193-205; code/lambdas/trade-executor/handler.ts:134-145; code/lambdas/dlq-replayer/handler.ts:114-133
• Severity: High stranded-order and false-success risk
• Timeline: Order persists, cache persists, publication fails or partially fails, retry returns cached success
• Ask first: Ask exact request identity authority and response timeline bus/rule version and downstream receipts

Reveal RW01 reference answer

RW01 reference answer

• Two minute: The database/event dual write is non-atomic and HTTP acknowledgement is narrower than routing; atomically store order/result/outbox, relay original ID and reconcile
• Deep: Name CS01 and CS03, distinguish explicit failed entry from timeout unknown, keep order authority separate from EventBridge
• Hardening: Inspect every result immediately and add pending-publication repair without changing logical identity
• Repair: Transactional order plus request fingerprint and semantic outbox; idempotent relay/consumer and reconciliation
• Migration: Expand schema first, dual-read status, backfill missing intents under manifest, canary relay, then remove direct publication
• Rollback: Rollback reads/relay while old authority valid; roll forward once new facts use new contract
• Proof: Every accepted order maps to one durable intent and downstream receipt or owned break
• Followups: Why reversing calls fails; how to migrate without duplicates
• Weak: Retry PutEvents; trust EventId; delete cached row
• Falsifier: If a true atomic broker transaction is introduced, outbox choice changes; bus config drift still needs proof
• Premises: CS01 CS03 FSR02 REL04

RW02 prompt — Cached idempotency and retry ambiguity

• Evidence: code/lambdas/order-handler/handler.ts:167-180,229-241
• Severity: High duplicate-order or false-success risk
• Timeline: Original returns failure after cached 201; same key skips publish; new key creates new order
• Ask first: Ask canonical command identity payload fingerprint retention response state lease and repair owner

Reveal RW02 reference answer

RW02 reference answer

• Two minute: Replace response cache with durable IN_PROGRESS/COMPLETED command receipt atomically tied to order/outbox; same-key mismatch conflicts
• Deep: TTL is cleanup, fail-open lookup is unsafe, and retry after ambiguous external effect needs receipt/reconciliation
• Hardening: Stop recommending new keys; preserve fingerprint and return pending lookup token
• Repair: Backfill receipt state from orders/outbox, add lease/takeover and effect receipt, reconcile unknowns
• Migration: Deploy compatible reads, write new receipt transaction, repair old rows, then enforce mismatch and retire cache path
• Rollback: Rollback response presentation only; never delete committed order or regenerate identity
• Proof: One fingerprint maps to one order/version/result; late retry returns compatible outcome
• Followups: TTL expiry; concurrent retries; provider effect boundary
• Weak: Cache equals idempotency; new key solves retry; TTL is deadline
• Falsifier: If product accepts best-effort duplicate creation explicitly severity changes, but financial order semantics usually reject it
• Premises: CS02 C40 AP08 FSR01

RW03 prompt — Trade write publication duplicate and missing paths

• Evidence: code/lambdas/trade-executor/handler.ts:83-118,120-145; code/lambdas/trade-executor/handler.ts:79-104,120-145
• Severity: Critical if execution/posting correctness is affected
• Timeline: Trade commits, order changes, publish fails; retry stops at duplicate insert before publication
• Ask first: Ask deterministic execution identity authority and partial fills legal predecessor outbox state and ledger postings

Reveal RW03 reference answer

RW03 reference answer

• Two minute: Make fill, conditional order transition and semantic outbox one compatible transaction; duplicate returns prior effect receipt and resumes relay
• Deep: A random new trade ID and one-trade-per-order key cannot model partial fills; external matcher boundary still needs idempotent ingestion
• Hardening: Persist repairable publication state and inspect PutEvents entries
• Repair: Migrate deterministic fills, backfill outbox from stored trade, canary relay, reconcile portfolio and ledger
• Migration: Add new schema/index first, dual-read old fills, single authoritative writer, backfill, cut over consumers
• Rollback: Roll back reads before semantic switch; after new executions append corrections and roll forward
• Proof: Every execution ID has legal order version, one posting set, outbox and reconciled projections
• Followups: Failure between trade and order; missing event recovery; partial fill
• Weak: Conditional put makes workflow idempotent; ordinary retry finishes it
• Falsifier: If execution authority is external, local transaction begins at ingestion receipt rather than match decision
• Premises: CS04 CS05 C38 FSR04 AP04

RW04 prompt — Portfolio identity version and money correctness

• Evidence: code/lambdas/portfolio-updater/handler.ts:145-186; code/shared/types/dynamodb.ts:63-79; code/shared/types/events.ts:36-47; code/lambdas/portfolio-updater/handler.ts:145-178; infra/environments/production/main.tf:565-605; code/shared/validation/schemas.ts:21-27; code/lambdas/trade-executor/handler.ts:55-63; code/lambdas/portfolio-updater/handler.ts:64-84,121-135; code/lambdas/portfolio-snapshot/handler.ts:50-61
• Severity: Critical if used for availability; otherwise high projection correctness
• Timeline: Duplicate or late fill applies through version lock; floating arithmetic drifts totals
• Ask first: Ask source sequence event identity currency scale rounding authority and rebuild source

Reveal RW04 reference answer

RW04 reference answer

• Two minute: Treat portfolio as projection; atomically inbox tradeId plus expected version and exact-minor-unit update; park gaps and rebuild
• Deep: Optimistic locking controls concurrent overwrite, not novelty/order. A strong read cannot upgrade provenance. Exact money must cross envelope/storage boundaries
• Hardening: Remove projection from authorization, detect duplicates/gaps and validate serialization
• Repair: Versioned vNext rebuild from authoritative executions/ledger, catch up, compare exact totals and cut over
• Migration: Introduce exact fields and decoder, shadow calculate, backfill vNext, canary reads, then retire float view
• Rollback: Retain old projection alias only if compatible; never roll back authoritative postings
• Proof: One event identity once, contiguous source versions, exact quantities/currency totals equal authority
• Followups: How handle old float history; gap after retention; read-your-writes
• Weak: Version lock deduplicates; timestamp orders events; portfolio is balance
• Falsifier: If operations are mathematically commutative ordering need may narrow, but identity and exact money remain
• Premises: CS06 CS07 CS10 AP07 AP13 REL05

RW05 prompt — Scans indexes access patterns and capacity

• Evidence: code/lambdas/portfolio-snapshot/handler.ts:64-107
• Severity: Medium scalability, high audit risk if called complete
• Timeline: Dataset exceeds one page/memory or hot symbol GSI lane saturates while aggregate looks healthy
• Ask first: Ask AP, result bound, pagination/snapshot semantics, point-in-time requirement, hot share, export deadline and control totals

Reveal RW05 reference answer

RW05 reference answer

• Two minute: Replace request-path scans with AP-specific Query/indexes; use PITR export plus manifest for scalable snapshots and reconcile completeness
• Deep: Alex DeBrie access-pattern-first design requires keys and physical lanes before indexes. LastEvaluatedKey is continuation, not snapshot isolation
• Hardening: Paginate immediately and stop labeling current JSONL complete
• Repair: Create export/reconciliation pipeline, versioned output, checksums/counts/exact totals and bounded catch-up
• Migration: Add cursor-compatible API, run parallel export, compare manifests, switch consumers, retire scan after evidence
• Rollback: Roll back reader to old path only within known bounds; export failures resume from manifest
• Proof: All source segments represented once, exact totals and watermark recorded; hot lanes stay within target
• Followups: Sparse GSI lifecycle; hot symbol; point-in-time versus pagination
• Weak: DynamoDB scan scales; add GSI first; aggregate WCU proves fit
• Falsifier: If access pattern is rare offline, export may beat an index; if low bounded table, pagination may suffice
• Premises: CS09 AP06 C21 C24 C81 C96 C114

RW06 prompt — Missing ledger reconciliation IAM observability deployment and safe sequence

• Evidence: infra/modules/eventbridge/main.tf:29-38,73-95; infra/environments/production/main.tf:575-605; infra/modules/lambda/main.tf:12-70; bounded absence check below; code/shared/types/dynamodb.ts:16-93; infra/modules/dynamodb/main.tf:1-60; repository-wide absence check below; code/lambdas/trade-executor/handler.ts:55-81; infra/modules/lambda/main.tf:12-31
• Severity: Critical architecture gap for production financial authority
• Timeline: A fill occurs, projection changes, DLQ send fails, deployment changes semantics and no control total closes the break
• Ask first: Ask regulated scope ledger owner postings reservation policy IAM principals evidence population SLO release compatibility and RTO/RPO

Reveal RW06 reference answer

RW06 reference answer

• Two minute: Do not infer posture from absence beyond inspected routes; add explicit balanced ledger/reconciliation, fix rule-scoped queue policies, business SLOs and compatible canaries in dependency order
• Deep: Smallest hardening is policy plus failed-DLQ-send alarm and publication manifests; production repair adds ledger/reservation authority, reconciliation and replay-safe releases while matcher substrate follows measurement
• Hardening: Grant EventBridge only scoped SendMessage, alarm failed sends, block portfolio authorization and preserve release artifacts
• Repair: Introduce ledger schema/posting service and reconciliation batches; backfill under manifest; canary views; fence writers; test restore
• Migration: Expand controls and authorities, shadow/control-total compare, single-writer cutover, then contract old paths; never dual-authority casually
• Rollback: Rollback traffic/config while formats compatible; roll forward corrections after financial facts or schemas change
• Proof: Balanced exact postings, one writer, complete manifests, least-privilege negative tests, SLO and DR evidence
• Followups: Why CloudTrail/Object Lock is insufficient; safe order of changes; matcher migration
• Weak: No IAM means insecure; encryption means compliant; deploy everything together
• Falsifier: If an external governed ledger exists, integrate and prove its routes rather than duplicate it; repository alone does not show one
• Premises: CS08 CS11 CS12 SEC03 REL04 REL07 SLO05

Shared self-scoring rubric

Score the explanation you actually gave, then choose a concrete improvement for each weak axis. An alternative service or topology can be sound when its assumptions, boundaries, and recovery proof hold. Use a missed premise to find the relevant chapter; repeating a reference answer without its reasoning does not close the gap.

Score each axis 0–4. Level 4 requires explicit evidence and a falsifier, not extra AWS vocabulary.

Axis	Level 0	Level 1	Level 2	Level 3	Level 4
requirement discovery	Chooses immediately	Asks one generic question	Covers rate or latency but misses business ambiguity	Elicits workload, deadlines, authority and recovery constraints	Prioritizes quantified constraints, exposes conflicting assumptions, cites evidence and states a falsifier
authority and invariant clarity	Calls transport or projection authority	Names data without invariant	Separates some stores	Names command, execution, ledger and projection roles with exact money	Proves invariants at transaction/reconciliation boundaries with evidence and states a falsifier
guarantee-boundary precision	Claims system-wide guarantee	Repeats service label	Names one boundary	Separates producer, transport, consumer and effect with order/idempotency scope	Traces every stopping boundary and ambiguous window to evidence, proof and falsifier
failure/recovery/reconciliation	Says retry, DLQ, replay or fail over	Lists controls without repair	Explains detection and repair partially	Separates prevention, detection, containment, repair and validation	Supplies owner, bounded runbook, exact reconciliation evidence, residual ambiguity and falsifier
quantitative/cost reasoning	No units or arithmetic	Uses averages only	Computes one dimension	Carries units through peak, skew, fan-out, drain and complete cost	Independently checks branches/sensitivities, capacity feasibility, evidence quality and falsifier
observability/security/operations	Says check CloudWatch or encrypt	Lists tools	Names signals or IAM broadly	Ties business SLOs, scoped access, release and DR to owners	Defines evidence populations, negative/security tests, recovery proof, cost and falsifier
communication and trade-off clarity	Unstructured answer	Service trivia	Structured but universal choice	Concise conditional opening plus deeper alternative and migration trigger	Makes assumptions explicit, compares credible alternatives, cites evidence and ends with falsifier

Interviewer calibration: alternate answers can earn full marks when stated assumptions, boundaries, arithmetic, recovery proof and falsifiers are sound. Do not treat the sum as a hiring threshold.

Internal premise routes

Package chapters control all answers: repository evidence, repository case study, foundations, service guide, Kinesis, DynamoDB, CQRS and sagas, reliability, trading architectures, cost and capacity, and operations. Exact claim and formula IDs in each drill are the source route; no external bibliography is duplicated here. The machine-readable contracts and reviewer-facing 30-row route audit are tracked in the Task 12 canonical research artifact.

Takeaway and next step

You have completed the guide's learning sequence when you can explain the service choices, financial authorities, failure windows, and calculations in your own words and defend credible alternatives. Use the quick reference for recall, the source index to verify a claim, and the guide's readiness checklist to choose what needs another pass. The rubric is study feedback, not a prediction of an interview outcome.