# Reliability, correctness, and recovery Reliability is a chain of evidence, not a collection of retry switches. Start with the end-to-end failure surface; then decide which control prevents, detects, contains, or repairs each failure. Availability and empty failure queues do not prove correct orders, positions, or money (C43,C48; F03,F14). ## End-to-end failure surface The authority column names evidence that survives retries and operator action. The ambiguity column states what the customer may safely be told. A response can apply several control classes, but residual ambiguity remains visible. Retries follow C45 and Featonby/Brooker guidance: deadline, bounded attempts, exponential backoff with full jitter, stable identity, and one owner where possible (F03,F11,F12). Backlog and fairness follow C44,C61 and the queue/SRE controls in F13,F14. Stable business identity and business-effect idempotency remain C40; no holding area alone is repair under C43. **Inference:** for this repository, no transport acknowledgement closes the order-to-ledger proof. CS01 through CS12 therefore route to durable authority, containment, and reconciliation; customer state stays pending or unknown until the named proof exists (C48). ```text FAILURE|FSR01|Client or API timeout/disconnect before response|Business-command record and authoritative order lookup by client order ID|Customer sees unknown; acceptance may have committed|Canonical fingerprint; atomic IN_PROGRESS and order/outbox write|Timeout rate; duplicate-key lookup; accepted-without-response count|Do not create a second logical order; return pending with lookup token|Client owns one retry within 8 s deadline; maximum 2 attempts with full jitter; then lookup|Return stored result or reconcile command record with order and outbox|Order API on-call; incident lead if unknown exceeds 2 min|One fingerprint maps to one order ID and durable response; no mismatched payload accepted|prevent detect contain repair|CS01 CS02 FAILURE|FSR02|Producer rejection or ambiguous acknowledgement|Outbox state; per-entry producer result; downstream receipt and business authority|Published false true or unknown; customer order remains accepted-pending|Transactional outbox; stable event ID; validate bus and rules|FailedEntryCount and per-entry errors; outbox age; downstream receipt gaps|Keep outbox pending; circuit-open publisher lane without changing order authority|Retry retryable rejected entries for 2 min and 5 attempts; ambiguous result goes to lookup/reconciliation|Republish original identity only when absence is proven or let idempotent consumer absorb duplicate|Messaging owner; order owner after 2 min|Outbox SENT plus downstream receipt or reconciled authority for every accepted event|prevent detect contain repair|CS01 CS03 CS05 FAILURE|FSR03|Batch API partially accepts records|Per-entry request/result correlation and durable producer manifest|Some records accepted and some rejected; batch-level success is unsafe|Stable entry IDs; retain original manifest; size and quota validation|FailedEntryCount; missing result; accepted versus attempted control total|Freeze blind whole-batch retry; isolate failed or unknown entries|Retry only explicit failures within producer deadline; unknown entries reconcile before retry|Compare manifest IDs with log or downstream inbox and repair missing entries|Producer team; incident if manifest incomplete|Attempted equals explicit success plus explicit failure plus resolved unknown in records|prevent detect contain repair|CS03 FAILURE|FSR04|Delivery is delayed duplicated or out of source order|Stable event ID; source aggregate version; authoritative state|Projection may be stale; business authority unchanged|Inbox idempotency; source version; single writer per invariant scope|Duplicate hit rate; version gaps; oldest age; stale rejection count|Park gaps; reject stale events; isolate replay from live lanes|Transport retry stays bounded; duplicate does not repeat effect; gap waits 60 s then repair|Fetch missing authority range or rebuild projection and reconcile totals|Projection owner; domain owner for source gaps|Each version applied once in order or deterministically superseded; watermark reaches authority|prevent detect contain repair|CS06 CS07 FAILURE|FSR05|Poison or incompatible schema event repeatedly fails|Original payload plus envelope; schema ID/version; validation error; producer identity|One lane or key may be blocked; event effect absent|Compatibility tests; envelope validation; size limits; tolerant readers where semantics permit|Error class and event ID; repeated receive count; iterator age by shard or key|Quarantine exact payload; isolate poison key; never drop silently|Non-retryable validation gets zero ordinary retries; fix producer/consumer then canary replay before retention expires|Transform only through a governed repair job with new lineage; reconcile source and target|Schema owner and consuming domain; page on ordered-lane block|Canary and full replay pass current schema; counts and versions reconcile|prevent detect contain repair|CS08 CS10 FAILURE|FSR06|Handler timeout crash or lost acknowledgement|Inbox state; function logs; checkpoint or queue receipt; protected target transaction|Invocation result unknown; side effect may have committed|Atomic inbox plus state change; timeout shorter than lease; checkpoint after commit|Timeouts; errors; age; redelivery; stale IN_PROGRESS lease|Stop concurrency growth; isolate dependency; preserve record|Invoked-handler retry maximum 3 or event age 15 min; pre-invocation path is separate; then quarantine|Lease takeover after evidence; replay original ID; reconcile protected state before closure|Consumer owner; dependency owner when saturation causes timeout|Inbox COMPLETED and target version or external receipt proves one intended effect|prevent detect contain repair|CS04 CS05 CS06 FAILURE|FSR07|Exchange bank or custodian call times out after possible invocation|Provider request ID; effect receipt; provider status; internal intent and ledger state|Customer sees pending-external; never claim failed or retry safely yet|Provider idempotency key; persist intent before call; callback/status lookup|Timeouts; missing receipt age; provider reconciliation break|Open circuit for new optional work; preserve intent; block conflicting command|Ambiguous outcome gets no blind retry; lookup for 5 min then manual/reconciliation queue|Match provider evidence; post conditional forward completion reversal or correcting entries|Payments or trading operations; compliance escalation by SLA|Provider receipt and internal posting agree under one request ID; break closed with evidence|prevent detect contain repair|CS04 CS11 FAILURE|FSR08|DynamoDB conditional contention throttle or one hot key|Conditional-write result; consumed/throttle metrics; item version and transaction cancellation reason|Command may be rejected busy or pending; no assumed commit|Distribute keys where invariant allows; single writer where it does not; bounded transaction|Throttle and conflict rate by table index tenant and key; latency; queue age|Per-key bulkhead; admission control; reserve authority capacity from projections|Retry retryable throttles within 1 s and 4 attempts full jitter; conflict is lookup/re-evaluate not blind retry|Resolve stale state from authority; repair hot-key model only with ordered versioned migration|Owning bounded context; capacity on-call|Conditional version advances once; invariant and ledger control totals remain valid|prevent detect contain repair|CS02 CS04 CS06 CS12 FAILURE|FSR09|Projection gap lag failed rebuild or stale cutover|Authoritative orders executions ledger plus projection watermark and manifest|Customer view is stale with as-of watermark; cannot authorize correction|Versioned projector; gap buffer; isolated blue-green rebuild; side effects suppressed|Lag age; missing version; source versus projection counts and totals|Keep old projection serving with stale banner; pause cutover; protect authority|Replay at governed rate after dependency-ready gate; stop before live SLO or retention risk|Backfill from durable authority; catch up; validate; conditional alias cutover; reconcile|Projection owner; domain owner approves totals|Manifest complete; no gaps; control totals match; watermark at cutover target|prevent detect contain repair|CS06 CS07 CS09 CS11 FAILURE|FSR10|Notification endpoint or client delivery fails|Notification intent; channel provider receipt; customer inbox state|Business action remains authoritative; customer may not have received notice|Durable intent for required notices; channel preference; client catch-up protocol|Delivery failure; age; provider status; notification projection gap|Isolate channel from command path; degrade to in-app status|Retry only within product expiry and channel budget; do not resend expired one-time codes|Regenerate only from authority and policy; record alternate-channel/manual contact|Notification owner; support or compliance for mandatory notice breach|Required notice has policy-accepted receipt or documented exception; business state unchanged|prevent detect contain repair|CS08 CS11 FAILURE|FSR11|Backlog overload retry storm or dependency saturation|Arrival rate; committed service rate; oldest age; per-class queues and capacity budget|Requests may be rejected busy or delayed with explicit estimate|Admission control; queue cap; bulkheads; reserved and maximum concurrency; fairness lanes|Lambda throttles; queue age; iterator age; rejection and wait by tenant key priority|Shed optional work; cap concurrency and replay; open circuit; reserve command and ledger lanes|One owned retry layer; stop when deadline or age consumes recovery margin|Drain only while mu exceeds lambda plus reserve; reconcile dropped expired or deferred work|Incident commander plus owning service and dependency teams|Age returns below SLO; no starvation; all admitted commands resolved; reconciliation clean|prevent detect contain repair|CS08 CS09 CS12 FAILURE|FSR12|Region loss or unsafe failover|Fenced writer epoch; replicated authority; backups exports logs and external evidence|Commands stop or return unavailable until one writer is proven; projections can be stale|Tested DR tiers; IaC; cross-Region evidence; failover fencing; independent backups|Regional health plus business probes; replication lag; writer-epoch conflict; RTO clock|Stop writes; fence old Region; recover dependencies in authority-first order|Restore service within RTO; recover data only to a point whose age and loss are within RPO; no cross-Region blind replay|Replay outbox and projections; reconcile exchange bank custodian and ledger before unrestricted service|Incident commander; command ledger platform and compliance owners|One active epoch; service restored within RTO; recoverable data loss and recovery-point age within RPO; control totals and game-day evidence complete|prevent detect contain repair|CS01 CS08 CS09 CS11 ``` | ID | Trigger | Authoritative state/evidence | Ambiguity/customer state | Prevention | Detection signal | Containment | Retry/replay decision and budget | Repair/reconciliation | Owner/escalation | Correctness proof | Controls | CS routes | |---|---|---|---|---|---|---|---|---|---|---|---|---| | FSR01 | Client or API timeout/disconnect before response | Business-command record and authoritative order lookup by client order ID | Customer sees unknown; acceptance may have committed | Canonical fingerprint; atomic IN_PROGRESS and order/outbox write | Timeout rate; duplicate-key lookup; accepted-without-response count | Do not create a second logical order; return pending with lookup token | Client owns one retry within 8 s deadline; maximum 2 attempts with full jitter; then lookup | Return stored result or reconcile command record with order and outbox | Order API on-call; incident lead if unknown exceeds 2 min | One fingerprint maps to one order ID and durable response; no mismatched payload accepted | prevent detect contain repair | CS01 CS02 | | FSR02 | Producer rejection or ambiguous acknowledgement | Outbox state; per-entry producer result; downstream receipt and business authority | Published false true or unknown; customer order remains accepted-pending | Transactional outbox; stable event ID; validate bus and rules | FailedEntryCount and per-entry errors; outbox age; downstream receipt gaps | Keep outbox pending; circuit-open publisher lane without changing order authority | Retry retryable rejected entries for 2 min and 5 attempts; ambiguous result goes to lookup/reconciliation | Republish original identity only when absence is proven or let idempotent consumer absorb duplicate | Messaging owner; order owner after 2 min | Outbox SENT plus downstream receipt or reconciled authority for every accepted event | prevent detect contain repair | CS01 CS03 CS05 | | FSR03 | Batch API partially accepts records | Per-entry request/result correlation and durable producer manifest | Some records accepted and some rejected; batch-level success is unsafe | Stable entry IDs; retain original manifest; size and quota validation | FailedEntryCount; missing result; accepted versus attempted control total | Freeze blind whole-batch retry; isolate failed or unknown entries | Retry only explicit failures within producer deadline; unknown entries reconcile before retry | Compare manifest IDs with log or downstream inbox and repair missing entries | Producer team; incident if manifest incomplete | Attempted equals explicit success plus explicit failure plus resolved unknown in records | prevent detect contain repair | CS03 | | FSR04 | Delivery is delayed duplicated or out of source order | Stable event ID; source aggregate version; authoritative state | Projection may be stale; business authority unchanged | Inbox idempotency; source version; single writer per invariant scope | Duplicate hit rate; version gaps; oldest age; stale rejection count | Park gaps; reject stale events; isolate replay from live lanes | Transport retry stays bounded; duplicate does not repeat effect; gap waits 60 s then repair | Fetch missing authority range or rebuild projection and reconcile totals | Projection owner; domain owner for source gaps | Each version applied once in order or deterministically superseded; watermark reaches authority | prevent detect contain repair | CS06 CS07 | | FSR05 | Poison or incompatible schema event repeatedly fails | Original payload plus envelope; schema ID/version; validation error; producer identity | One lane or key may be blocked; event effect absent | Compatibility tests; envelope validation; size limits; tolerant readers where semantics permit | Error class and event ID; repeated receive count; iterator age by shard or key | Quarantine exact payload; isolate poison key; never drop silently | Non-retryable validation gets zero ordinary retries; fix producer/consumer then canary replay before retention expires | Transform only through a governed repair job with new lineage; reconcile source and target | Schema owner and consuming domain; page on ordered-lane block | Canary and full replay pass current schema; counts and versions reconcile | prevent detect contain repair | CS08 CS10 | | FSR06 | Handler timeout crash or lost acknowledgement | Inbox state; function logs; checkpoint or queue receipt; protected target transaction | Invocation result unknown; side effect may have committed | Atomic inbox plus state change; timeout shorter than lease; checkpoint after commit | Timeouts; errors; age; redelivery; stale IN_PROGRESS lease | Stop concurrency growth; isolate dependency; preserve record | Invoked-handler retry maximum 3 or event age 15 min; pre-invocation path is separate; then quarantine | Lease takeover after evidence; replay original ID; reconcile protected state before closure | Consumer owner; dependency owner when saturation causes timeout | Inbox COMPLETED and target version or external receipt proves one intended effect | prevent detect contain repair | CS04 CS05 CS06 | | FSR07 | Exchange bank or custodian call times out after possible invocation | Provider request ID; effect receipt; provider status; internal intent and ledger state | Customer sees pending-external; never claim failed or retry safely yet | Provider idempotency key; persist intent before call; callback/status lookup | Timeouts; missing receipt age; provider reconciliation break | Open circuit for new optional work; preserve intent; block conflicting command | Ambiguous outcome gets no blind retry; lookup for 5 min then manual/reconciliation queue | Match provider evidence; post conditional forward completion reversal or correcting entries | Payments or trading operations; compliance escalation by SLA | Provider receipt and internal posting agree under one request ID; break closed with evidence | prevent detect contain repair | CS04 CS11 | | FSR08 | DynamoDB conditional contention throttle or one hot key | Conditional-write result; consumed/throttle metrics; item version and transaction cancellation reason | Command may be rejected busy or pending; no assumed commit | Distribute keys where invariant allows; single writer where it does not; bounded transaction | Throttle and conflict rate by table index tenant and key; latency; queue age | Per-key bulkhead; admission control; reserve authority capacity from projections | Retry retryable throttles within 1 s and 4 attempts full jitter; conflict is lookup/re-evaluate not blind retry | Resolve stale state from authority; repair hot-key model only with ordered versioned migration | Owning bounded context; capacity on-call | Conditional version advances once; invariant and ledger control totals remain valid | prevent detect contain repair | CS02 CS04 CS06 CS12 | | FSR09 | Projection gap lag failed rebuild or stale cutover | Authoritative orders executions ledger plus projection watermark and manifest | Customer view is stale with as-of watermark; cannot authorize correction | Versioned projector; gap buffer; isolated blue-green rebuild; side effects suppressed | Lag age; missing version; source versus projection counts and totals | Keep old projection serving with stale banner; pause cutover; protect authority | Replay at governed rate after dependency-ready gate; stop before live SLO or retention risk | Backfill from durable authority; catch up; validate; conditional alias cutover; reconcile | Projection owner; domain owner approves totals | Manifest complete; no gaps; control totals match; watermark at cutover target | prevent detect contain repair | CS06 CS07 CS09 CS11 | | FSR10 | Notification endpoint or client delivery fails | Notification intent; channel provider receipt; customer inbox state | Business action remains authoritative; customer may not have received notice | Durable intent for required notices; channel preference; client catch-up protocol | Delivery failure; age; provider status; notification projection gap | Isolate channel from command path; degrade to in-app status | Retry only within product expiry and channel budget; do not resend expired one-time codes | Regenerate only from authority and policy; record alternate-channel/manual contact | Notification owner; support or compliance for mandatory notice breach | Required notice has policy-accepted receipt or documented exception; business state unchanged | prevent detect contain repair | CS08 CS11 | | FSR11 | Backlog overload retry storm or dependency saturation | Arrival rate; committed service rate; oldest age; per-class queues and capacity budget | Requests may be rejected busy or delayed with explicit estimate | Admission control; queue cap; bulkheads; reserved and maximum concurrency; fairness lanes | Lambda throttles; queue age; iterator age; rejection and wait by tenant key priority | Shed optional work; cap concurrency and replay; open circuit; reserve command and ledger lanes | One owned retry layer; stop when deadline or age consumes recovery margin | Drain only while mu exceeds lambda plus reserve; reconcile dropped expired or deferred work | Incident commander plus owning service and dependency teams | Age returns below SLO; no starvation; all admitted commands resolved; reconciliation clean | prevent detect contain repair | CS08 CS09 CS12 | | FSR12 | Region loss or unsafe failover | Fenced writer epoch; replicated authority; backups exports logs and external evidence | Commands stop or return unavailable until one writer is proven; projections can be stale | Tested DR tiers; IaC; cross-Region evidence; failover fencing; independent backups | Regional health plus business probes; replication lag; writer-epoch conflict; RTO clock | Stop writes; fence old Region; recover dependencies in authority-first order | Restore service within RTO; recover data only to a point whose age and loss are within RPO; no cross-Region blind replay | Replay outbox and projections; reconcile exchange bank custodian and ledger before unrestricted service | Incident commander; command ledger platform and compliance owners | One active epoch; service restored within RTO; recoverable data loss and recovery-point age within RPO; control totals and game-day evidence complete | prevent detect contain repair | CS01 CS08 CS09 CS11 | Every repository defect maps explicitly; a row can improve several classes but must still name what remains ambiguous. ```text CSMAP|CS01|FSR01 FSR02 FSR12|Atomic order idempotency and outbox|Outbox age and accepted-without-event manifest|Keep accepted order pending|Replay original event and reconcile order to downstream|An accepted order can remain pending until receipt or reconciliation CSMAP|CS02|FSR01 FSR08|Fingerprint plus durable result in authority|Mismatched fingerprint and stale IN_PROGRESS age|Reject conflict and return pending|Repair command record against order and outbox|Cached response alone remains non-authoritative CSMAP|CS03|FSR02 FSR03|Per-entry producer correlation|FailedEntryCount and unknown result count|Do not retry whole batch|Retry explicit failures and reconcile unknowns|Zero failed entries does not prove routing or effect CSMAP|CS04|FSR06 FSR07 FSR08|Conditional local transaction plus outbox|State transition and publication control totals|Stop conflicting transition|Forward-complete or correct from authority|External execution can remain ambiguous CSMAP|CS05|FSR02 FSR06|Independent publication intent|Trade-with-unsent-outbox age|Keep publication retryable|Publish original identity and reconcile receipt|Duplicate trade guard alone cannot repair publication CSMAP|CS06|FSR04 FSR06 FSR08 FSR09|Inbox plus projection mutation|Duplicate hit and stale version metrics|Reject repeated event effect|Rebuild and reconcile projection|Optimistic version alone is not identity CSMAP|CS07|FSR04 FSR09|Authoritative source sequence|Gap and stale-event counters|Buffer gap and reject stale|Backfill missing versions then cut over|Arrival order remains non-authoritative CSMAP|CS08|FSR05 FSR10 FSR11 FSR12|Correct queue policy and independent retained source|Failed DLQ writes plus queue and source age|Quarantine and preserve payload|Canary redrive then reconcile|Configured DLQ can still be unwritable or incomplete CSMAP|CS09|FSR09 FSR11 FSR12|PITR export or complete paginated manifest|Page count export completion and control totals|Do not publish partial snapshot|Re-export and validate before cutover|Asynchronous export is not a transaction snapshot CSMAP|CS10|FSR05|Exact money schema with currency scale rounding|Schema validation and amount invariant errors|Quarantine unsafe payload|Governed conversion with lineage and balancing proof|Historical floats may require manual evidence CSMAP|CS11|FSR07 FSR09 FSR10 FSR12|Append-only ledger reservations and reconciliation|Ledger imbalance reservation mismatch and external breaks|Freeze affected account or product scope|Reversal or correcting entry from authority only|Missing historical authority can make exact replay impossible CSMAP|CS12|FSR08 FSR11|Measured long-lived or serverless capacity choice|Tail latency saturation age and cost per execution|Admission control and dedicated matcher bulkhead|Scale or migrate with ordered drain and reconciliation|Compute choice does not fix domain correctness ``` | CS | Failure routes | Prevent | Detect | Contain | Repair | Residual ambiguity | |---|---|---|---|---|---|---| | CS01 | FSR01 FSR02 FSR12 | Atomic order idempotency and outbox | Outbox age and accepted-without-event manifest | Keep accepted order pending | Replay original event and reconcile order to downstream | An accepted order can remain pending until receipt or reconciliation | | CS02 | FSR01 FSR08 | Fingerprint plus durable result in authority | Mismatched fingerprint and stale IN_PROGRESS age | Reject conflict and return pending | Repair command record against order and outbox | Cached response alone remains non-authoritative | | CS03 | FSR02 FSR03 | Per-entry producer correlation | FailedEntryCount and unknown result count | Do not retry whole batch | Retry explicit failures and reconcile unknowns | Zero failed entries does not prove routing or effect | | CS04 | FSR06 FSR07 FSR08 | Conditional local transaction plus outbox | State transition and publication control totals | Stop conflicting transition | Forward-complete or correct from authority | External execution can remain ambiguous | | CS05 | FSR02 FSR06 | Independent publication intent | Trade-with-unsent-outbox age | Keep publication retryable | Publish original identity and reconcile receipt | Duplicate trade guard alone cannot repair publication | | CS06 | FSR04 FSR06 FSR08 FSR09 | Inbox plus projection mutation | Duplicate hit and stale version metrics | Reject repeated event effect | Rebuild and reconcile projection | Optimistic version alone is not identity | | CS07 | FSR04 FSR09 | Authoritative source sequence | Gap and stale-event counters | Buffer gap and reject stale | Backfill missing versions then cut over | Arrival order remains non-authoritative | | CS08 | FSR05 FSR10 FSR11 FSR12 | Correct queue policy and independent retained source | Failed DLQ writes plus queue and source age | Quarantine and preserve payload | Canary redrive then reconcile | Configured DLQ can still be unwritable or incomplete | | CS09 | FSR09 FSR11 FSR12 | PITR export or complete paginated manifest | Page count export completion and control totals | Do not publish partial snapshot | Re-export and validate before cutover | Asynchronous export is not a transaction snapshot | | CS10 | FSR05 | Exact money schema with currency scale rounding | Schema validation and amount invariant errors | Quarantine unsafe payload | Governed conversion with lineage and balancing proof | Historical floats may require manual evidence | | CS11 | FSR07 FSR09 FSR10 FSR12 | Append-only ledger reservations and reconciliation | Ledger imbalance reservation mismatch and external breaks | Freeze affected account or product scope | Reversal or correcting entry from authority only | Missing historical authority can make exact replay impossible | | CS12 | FSR08 FSR11 | Measured long-lived or serverless capacity choice | Tail latency saturation age and cost per execution | Admission control and dedicated matcher bulkhead | Scale or migrate with ordered drain and reconciliation | Compute choice does not fix domain correctness | ## Four identities, four stopping boundaries These controls are cumulative, not substitutes (C40,C47,C90; F02,F08,F11). Safety-critical cells use closed `key=value` contracts: semicolons separate fields, `+` means every named item is required together, and the verifier accepts only the row-specific values rendered here. This notation also governs the target-DLQ, DR, and rollback/resume cells below. ```text IDENTITY|ID01|transport dedupe|Producer-to-broker transport acceptance inside one named broker scope|Broker deduplication or message identifier|Transport acceptance and suppression state|Retain for the documented window only; expiry reopens transport delivery and never proves business expiry|Broker-local acceptance and dedupe only; non-atomic with handler database and external effect|Concurrent duplicate sends use broker rules; a late send after expiry is a new transport attempt|Outside the window rely on business command identity and handler inbox; lookup authority before repeating an effect|Duplicate sends suppressed only inside named transport scope; acknowledgement and transport IDs never prove one business effect IDENTITY|ID02|handler inbox idempotency|Consumer and protected effect for one stable business event and effect scope|Stable business event ID plus effect type|PENDING with attemptOwner epoch and leaseUntil; COMPLETED with result or REPAIR_REQUIRED|Retain through source replay dispute and audit horizon; cleanup-only TTL never authorizes a new effect|mode=atomic_single_transaction; writes=inbox+protected_effect; split_commit=forbidden|Conditional create elects one owner epoch; expired leaseUntil permits conditional takeover; fenced completion rejects a stale owner|mode=persisted_intent_before_invocation; resolution=receipt_lookup+reconciliation; terminal=COMPLETED_or_REPAIR_REQUIRED; split_success=not_idempotency_proof|Repeated delivery reads fenced COMPLETED and does not repeat the protected effect IDENTITY|ID03|business-command fingerprint/result|Command and protected order state/outbox for one client command scope|Client command key plus canonical request fingerprint|IN_PROGRESS with attemptOwner epoch leaseUntil; COMPLETED durable response and order ID; REPAIR_REQUIRED on split ambiguity|Retain through client retry dispute and audit horizon; cleanup-only TTL occurs only after protected evidence outlives replay|Atomic command record protected state and outbox in one authoritative transaction when possible|Conditional create elects one owner epoch; same-key races return in-progress or completed; takeover after leaseUntil is conditional and stale completion is fenced|For another store or external effect persist intent before invocation; receipt lookup and reconciliation decide completion or REPAIR_REQUIRED|Same fingerprint returns same durable result; mismatch is rejected; one fenced owner can complete IDENTITY|ID04|ledger posting uniqueness|Posting set and ledger legs for one financial obligation and currency scope|Posting set ID plus account leg and currency|Immutable balanced debit and credit postings with original and correcting lineage|Retain for the ledger and audit horizon and never expires independently of ledger evidence|Atomic balanced posting set uniqueness and expected authority/reservation version in one ledger transaction|Conditional uniqueness rejects concurrent posting sets; ledger writer epoch fences stale Region or worker completion|External settlement remains a persisted intent plus receipt lookup and reconciliation; ambiguity uses correcting entry rather than expiring uniqueness|Posting set balances by currency; duplicate key cannot add a second leg; receipt and reconciliation close external evidence ``` | ID | Control | Scope | Key | State | Retention/expiry | Atomicity | Race/takeover/fencing | Non-atomic fallback | Proof | |---|---|---|---|---|---|---|---|---|---| | ID01 | transport dedupe | Producer-to-broker transport acceptance inside one named broker scope | Broker deduplication or message identifier | Transport acceptance and suppression state | Retain for the documented window only; expiry reopens transport delivery and never proves business expiry | Broker-local acceptance and dedupe only; non-atomic with handler database and external effect | Concurrent duplicate sends use broker rules; a late send after expiry is a new transport attempt | Outside the window rely on business command identity and handler inbox; lookup authority before repeating an effect | Duplicate sends suppressed only inside named transport scope; acknowledgement and transport IDs never prove one business effect | | ID02 | handler inbox idempotency | Consumer and protected effect for one stable business event and effect scope | Stable business event ID plus effect type | PENDING with attemptOwner epoch and leaseUntil; COMPLETED with result or REPAIR_REQUIRED | Retain through source replay dispute and audit horizon; cleanup-only TTL never authorizes a new effect | mode=atomic_single_transaction; writes=inbox+protected_effect; split_commit=forbidden | Conditional create elects one owner epoch; expired leaseUntil permits conditional takeover; fenced completion rejects a stale owner | mode=persisted_intent_before_invocation; resolution=receipt_lookup+reconciliation; terminal=COMPLETED_or_REPAIR_REQUIRED; split_success=not_idempotency_proof | Repeated delivery reads fenced COMPLETED and does not repeat the protected effect | | ID03 | business-command fingerprint/result | Command and protected order state/outbox for one client command scope | Client command key plus canonical request fingerprint | IN_PROGRESS with attemptOwner epoch leaseUntil; COMPLETED durable response and order ID; REPAIR_REQUIRED on split ambiguity | Retain through client retry dispute and audit horizon; cleanup-only TTL occurs only after protected evidence outlives replay | Atomic command record protected state and outbox in one authoritative transaction when possible | Conditional create elects one owner epoch; same-key races return in-progress or completed; takeover after leaseUntil is conditional and stale completion is fenced | For another store or external effect persist intent before invocation; receipt lookup and reconciliation decide completion or REPAIR_REQUIRED | Same fingerprint returns same durable result; mismatch is rejected; one fenced owner can complete | | ID04 | ledger posting uniqueness | Posting set and ledger legs for one financial obligation and currency scope | Posting set ID plus account leg and currency | Immutable balanced debit and credit postings with original and correcting lineage | Retain for the ledger and audit horizon and never expires independently of ledger evidence | Atomic balanced posting set uniqueness and expected authority/reservation version in one ledger transaction | Conditional uniqueness rejects concurrent posting sets; ledger writer epoch fences stale Region or worker completion | External settlement remains a persisted intent plus receipt lookup and reconciliation; ambiguity uses correcting entry rather than expiring uniqueness | Posting set balances by currency; duplicate key cannot add a second leg; receipt and reconciliation close external evidence | ## Retry policy and amplification Timeout ownership starts at the caller's deadline and allocates shorter attempt timeouts to each dependency. Use one owned retry layer where possible. Classify errors before retry: retryable means transient and safe under identity; non-retryable means validation, authorization, invariant, or permanent contract failure; ambiguous means invocation or effect may have happened, so query state before another effect. Every policy names maximum attempts and maximum event age, exponential backoff with full jitter, and a stop point (C45; F03,F11,F12). Task 5's split remains binding: Kinesis/Lambda pre-invocation inability can be bounded by source expiry or maximum event age but not invoked-handler retry count, while invoked-handler errors can additionally use the attempt budget (C76; A98, retrieved 2026-08-22). Step Functions Standard workflow execution is its documented scope; the external effect boundary still needs receipt, provider status lookup, idempotency, compensation where meaningful, and reconciliation (C65; A26, retrieved 2026-08-22). SQS redrive policy is separately bounded by C103; it never expands an attempt budget into permission to repeat an ambiguous effect. **Inference:** repository commands use one caller retry inside an 8 s deadline; publishers own broker retry; consumers own invoked-handler retry. At ambiguity, retry stops for lookup or reconciliation rather than multiplying effects (C45,C65,C76; F11,F12; A98, retrieved 2026-08-22). ```text RETRY|RT01|retryable producer rejection|Outbox relay|2 min publish deadline|2 s attempt timeout|Maximum 5 attempts and outbox age 2 min|Exponential backoff with full jitter|Keep PENDING then page and reconcile|Per-entry success plus downstream receipt RETRY|RT02|non-retryable validation or invariant failure|Domain handler|Immediate classification|No dependency retry|Maximum 1 attempt and zero replay until corrected|None|Quarantine with reason; reject customer command or governed repair|Original payload reason and conditional state unchanged RETRY|RT03|ambiguous external effect|Exchange or bank adapter|5 min lookup deadline|1 s status lookup timeout|Maximum 5 lookups; no blind effect retry|Exponential backoff with full jitter|Manual resolution and reconciliation after lookup budget|Provider receipt agrees with internal intent and ledger RETRY|RT04|pre-invocation Kinesis Lambda inability|Platform event-source owner|Before source retention and business age budget|Invocation platform owns timeout|Maximum event age bounds path; maximum attempts does not|Service retry plus operator containment|Restore capacity or route retained full payload before expiry|Checkpoint advances only after committed handler result RETRY|RT05|invoked-handler timeout crash or Step Functions task failure|Consumer or workflow owner|15 min business deadline|Handler timeout shorter than source lease or state timeout|Maximum 3 handler attempts and maximum event age 15 min|Exponential backoff with full jitter|Quarantine or Catch; lookup external effect before retry|Inbox or effect receipt plus workflow history and reconciliation ``` | ID | Class | Owner | Deadline | Timeout | Attempts/age | Backoff | Stop/action | Proof | |---|---|---|---|---|---|---|---|---| | RT01 | retryable producer rejection | Outbox relay | 2 min publish deadline | 2 s attempt timeout | Maximum 5 attempts and outbox age 2 min | Exponential backoff with full jitter | Keep PENDING then page and reconcile | Per-entry success plus downstream receipt | | RT02 | non-retryable validation or invariant failure | Domain handler | Immediate classification | No dependency retry | Maximum 1 attempt and zero replay until corrected | None | Quarantine with reason; reject customer command or governed repair | Original payload reason and conditional state unchanged | | RT03 | ambiguous external effect | Exchange or bank adapter | 5 min lookup deadline | 1 s status lookup timeout | Maximum 5 lookups; no blind effect retry | Exponential backoff with full jitter | Manual resolution and reconciliation after lookup budget | Provider receipt agrees with internal intent and ledger | | RT04 | pre-invocation Kinesis Lambda inability | Platform event-source owner | Before source retention and business age budget | Invocation platform owns timeout | Maximum event age bounds path; maximum attempts does not | Service retry plus operator containment | Restore capacity or route retained full payload before expiry | Checkpoint advances only after committed handler result | | RT05 | invoked-handler timeout crash or Step Functions task failure | Consumer or workflow owner | 15 min business deadline | Handler timeout shorter than source lease or state timeout | Maximum 3 handler attempts and maximum event age 15 min | Exponential backoff with full jitter | Quarantine or Catch; lookup external effect before retry | Inbox or effect receipt plus workflow history and reconciliation | The shared planning model below is canonical in [`research/cost-model.md`](../research/cost-model.md). Values are illustrative inputs, not quotas or promises. The input domain requires `recovery_capacity_rps > 0 records/s`; zero or negative total capacity is rejected before any capacity-share division, while zero-spare and negative-spare fixtures retain positive total capacity. ```text TASK8_RESULT|nested_retry_worst_case_attempts|27.0 TASK8_RESULT|nested_retry_sensitivity_attempts|64.0 TASK8_RESULT|base_concurrency|24.0 TASK8_RESULT|planned_concurrency|36 TASK8_RESULT|slow_dependency_planned_concurrency|72 TASK8_RESULT|raw_spare_rps|200.0 TASK8_RESULT|effective_redrive_rps|200.0 TASK8_RESULT|recovery_status|POSITIVE_DRAIN TASK8_RESULT|backlog_growth_rps|0 TASK8_RESULT|backlog_drain_seconds|4500.0 TASK8_RESULT|backlog_drain_minutes|75.0 TASK8_RESULT|recovery_action|bounded replay TASK8_RESULT|live_capacity_percent|62.5 TASK8_RESULT|replay_capacity_percent|25.0 TASK8_RESULT|safety_capacity_percent|12.5 ``` | Kind | Name | Value or equation | Unit | |---|---|---:|---| | INPUT | retry_layers | 3 | ratio/count | | INPUT | attempts_per_layer | 3 | ratio/count | | INPUT | sensitivity_attempts_per_layer | 4 | ratio/count | | INPUT | arrival_rps | 200 | records/s | | INPUT | service_time_seconds | 0.12 | s | | INPUT | headroom_ratio | 1.50 | ratio/count | | INPUT | slow_service_multiplier | 2 | ratio/count | | INPUT | backlog_records | 900000 | records | | INPUT | recovery_capacity_rps | 800 | records/s | | INPUT | live_arrival_rps | 500 | records/s | | INPUT | safety_reservation_rps | 100 | records/s | | INPUT | replay_rate_cap_rps | 200 | records/s | | FORMULA | nested_retry | attempts_per_layer ^ retry_layers | attempts | | FORMULA | concurrency | ceil(arrival_rps * service_time_seconds * headroom_ratio) | concurrent requests | | FORMULA | raw_spare | recovery_capacity_rps - live_arrival_rps - safety_reservation_rps | records/s | | FORMULA | effective_replay | max(0, min(replay_rate_cap_rps, raw_spare_rps)) | records/s | | FORMULA | drain_time | if effective_redrive_rps > 0 then backlog_records / effective_redrive_rps else infinite | s | | FORMULA | negative_branch | if raw_spare_rps < 0 then backlog_growth_rps = -raw_spare_rps | records/s | | RESULT | nested_retry_worst_case_attempts | 27 | attempts | | RESULT | nested_retry_sensitivity_attempts | 64 | attempts | | RESULT | base_concurrency | 24 | concurrent requests | | RESULT | planned_concurrency | 36 | concurrent requests | | RESULT | slow_dependency_planned_concurrency | 72 | concurrent requests | | RESULT | raw_spare_rps | 200 | records/s | | RESULT | effective_redrive_rps | 200 | records/s | | RESULT | recovery_status | POSITIVE_DRAIN | state | | RESULT | backlog_growth_rps | 0 | records/s | | RESULT | backlog_drain_seconds | 4500 | s | | RESULT | backlog_drain_minutes | 75 | min | | RESULT | recovery_action | bounded replay | action | | RESULT | live_capacity_percent | 62.50 | % | | RESULT | replay_capacity_percent | 25 | % | | RESULT | safety_capacity_percent | 12.50 | % | Nested retry worst case is `3^3 = 27` attempts; the four-attempt sensitivity is `4^3 = 64`. Concurrency rounds up: `ceil(200 requests/s × 0.12 s × 1.5) = 36 concurrent requests`; a doubled service time needs `ceil(72) = 72`. Here `raw_spare = 800 - 500 - 100 = 200 records/s`, `effective replay = max(0, min(200, 200)) = 200 records/s`, and `900,000 / 200 = 4,500 s = 75 min`. At `raw_spare = 0`, replay is zero and drain time is infinite; at `raw_spare < 0`, replay is zero, backlog grows at `-raw_spare records/s`, and operators reduce admission or restore capacity. Of measured 800 records/s capacity, live, replay, and safety use 62.5%, 25%, and 12.5%. A replay/live-traffic capacity split is an admission decision; if measured commit rate falls, reduce replay before live SLOs fail. ## Overload containment Bulkheads isolate failure domains. Reserved concurrency protects and bounds one function; maximum concurrency caps one SQS event-source mapping. Their sums and downstream capacity must agree (C104; [Lambda concurrency](https://docs.aws.amazon.com/lambda/latest/dg/services-sqs-scaling.html), A117, retrieved 2026-08-22). A circuit breaker stops calls to a failing dependency but does not repair an unknown effect. Load shedding drops or degrades optional work; admission control rejects before accepting an obligation; a queue cap bounds stored debt. Per-tenant, per-key, and priority lanes provide fairness and dependency isolation. Autoscaling may add capacity, but is distinct from these controls (C44,C61; F03,F13,F14). **Inference:** reserve command and ledger capacity first, cap projectors and replay independently, and shed notifications/analytics before financial authority. This allocation is workload policy, not Lambda behavior (C44,C61,C104; F13,F14; A117, retrieved 2026-08-22). Metrics must be actionable: oldest age above 60 s for orders pages the command owner; replay consuming more than 25% of measured commit capacity aborts replay; any tenant above its configured share is throttled while quiet lanes retain a minimum allocation. These thresholds are planning inputs, not AWS guarantees. ## Boundary-specific quarantine and replay There is no generic “the DLQ.” An EventBridge target DLQ is an SQS Standard queue holding failed target deliveries; it is distinct from an EventBridge archive, whose replay returns selected events to the source bus where rules can match again (C57,C62; A81,A84, retrieved 2026-08-22). SNS subscription delivery, SQS source processing, Lambda event-source mapping, Kinesis retention, DynamoDB Streams, and Firehose destination delivery also retain different evidence and have different permissions, age, and ordering behavior (C43,C62-C64,C66,C76; A84,A86,A98, retrieved 2026-08-22). SQS managed redrive moves messages without filtering or modification; operators set a bounded velocity and retention must account for Standard versus FIFO timestamp behavior (C103; A118, retrieved 2026-08-22). SQS visibility and acknowledgement remain lease boundaries under C63, even during recovery. **Inference:** replay starts only after a dependency-readiness gate, dry run, canary, rate limit, and idempotency proof. Completion means post-replay business validation, not an empty holding area (C43,C103; A118, retrieved 2026-08-22). ```text BOUNDARY|RCV01|EventBridge target DLQ using SQS Standard; C57 A81 retrieved 2026-08-22|payload=full_event; failure_metadata=RULE_ARN+TARGET_ARN+ERROR_CODE+ERROR_MESSAGE+RETRY_ATTEMPTS+EXHAUSTED_CONDITION; reconstruction=forbidden|Rule-scoped SQS policy for events.amazonaws.com plus consumer read delete and KMS access|SQS retention exceeds incident diagnosis repair and canary window|source_bus_replay=forbidden; recovery=controlled_direct-target_or_repair_consumer_for_failed_TARGET_ARN; ack_delete=after_committed_target_repair+target_receipt+business_reconciliation|Failed target dependency healthy; direct-target contract and consumer inbox proven|Poll under measured target budget; pause before live target SLO breach|Original event ID failed TARGET_ARN and target-specific inbox or intent receipt|No-effect validation then one event or 1 percent canary against failed target|Target receipt and business reconciliation precede SQS delete acknowledgement|Impossible after DLQ expiry without archive or authoritative regeneration BOUNDARY|RCV02|EventBridge archive replay; C62 A84 retrieved 2026-08-22|Selected full source-bus events retained by archive pattern and time window|Archive replay operator permissions and reviewed optional rule selection|Archive retention exceeds detection fix replay and reconciliation window|Replay goes only to source-bus and may reorder; enumerate every re-matched rule and target; prove idempotency and bound duplicate effects before replay|All selected rules targets consumers and external-effect protections healthy|Narrow UTC window and optional rule set; cancel on duplicate-effect or live SLO budget|Original event ID plus inbox intent and receipt at every re-matched target|Dry run inventory of matched rules and targets then narrow-window canary|DescribeReplay progress plus per-target receipts and business reconciliation|Impossible after archive expiry or when archive pattern omitted the event BOUNDARY|RCV03|SQS source DLQ and managed redrive; C63 C103 A85 A118 retrieved 2026-08-22|Full SQS message body attributes and receive history available at queue boundary|Redrive IAM and KMS access; RedriveAllowPolicy limits source queues|DLQ retention covers original age; Standard expiry uses original enqueue timestamp|Standard order is best effort; FIFO DLQ movement can break source group order|Consumer fix deployed; dependency capacity and queue policy proven|Start 10 messages/s then ramp below reserved replay share|Stable business ID and atomic inbox|Dry-run parser then one message group or 1 percent canary|Source queue age falls; inbox effects and reconciliation totals match|Impossible after message expiry unless durable authority can regenerate BOUNDARY|RCV04|Lambda Kinesis on-failure S3 destination; C76 A98 retrieved 2026-08-22|Complete failed invocation record and payload in S3|Lambda destination write plus operator read KMS and lifecycle permissions|S3 lifecycle exceeds Kinesis expiry and investigation SLA|Custom replay restores source key and version; arrival order is not recovered automatically|Handler fix and all downstream dependencies healthy|Separate replay stream or queue at 25 percent measured capacity|Original event ID source version and inbox|Offline parse and no-effect mode then one shard-key canary|S3 manifest IDs equal completed inbox or explained quarantine IDs|Impossible when only SQS or SNS failure metadata exists and source record expired BOUNDARY|RCV05|Kinesis retained stream replay; C74 C76 A98 retrieved 2026-08-22|Full retained records within configured stream retention|Consumer IAM and checkpoint control isolated from live application|Retention exceeds detection fix drain and margin|Per-shard sequence retained; cross-shard order never inferred|Consumer version compatible and sink has replay lane|Dedicated consumer capped so live traffic and sink stay inside budget|Event ID aggregate version conditional apply|Start from narrow sequence window with side effects disabled|Checkpoint watermark version gaps and control totals reconcile|Impossible after retention expiry without S3 or authoritative backfill BOUNDARY|RCV06|DynamoDB Streams CDC; C84 A102 retrieved 2026-08-22|Configured key and image fields for 24-hour stream record window|Stream read role and consumer checkpoint access|Fixed 24-hour Streams boundary requires independent longer recovery source|Per-item modification order only; transaction records can interleave|Projection fix deployed and target isolated|Backfill from PITR export or authority; catch-up stream rate capped|Semantic outbox ID or item version plus inbox|Blue-green target dry run and sampled item comparison|Manifest totals versions and final watermark match authority|Impossible from Streams alone after 24 hours or when view omitted needed attributes BOUNDARY|RCV07|SNS subscription DLQ; C64 A86 retrieved 2026-08-22|Undelivered subscription message in SQS plus endpoint context|SNS topic to SQS policy and redrive operator permissions|Queue retention covers product notice expiry and investigation|Standard delivery can duplicate or reorder; FIFO scope remains subscription-specific|Endpoint and credentials healthy; notice still meaningful|Channel-specific replayer under provider quota|Notification intent ID and expiry check|Render-only dry run then test recipient canary|Provider receipt or documented policy exception; no business state mutation|Impossible or prohibited after notice expiry or missing consent BOUNDARY|RCV08|Data Firehose S3 backup or failed-delivery objects; C66 A88 retrieved 2026-08-22|Destination-specific buffered records and error output when configured|Firehose delivery role S3 KMS and replay job permissions|S3 lifecycle covers destination repair and audit window|Buffered delivery and retry can duplicate; no business order inferred|Destination schema quota and credentials healthy|New ingestion path isolated and rate-limited below destination headroom|Stable record ID and idempotent destination merge|Validate file schema and sample partition before bounded load|Object manifest row counts destination totals and rejected rows reconcile|Impossible when backup mode omitted the failed payload and source retention expired ``` | ID | Service boundary | Retained | Permissions | Age/retention | Ordering effects | Dependency readiness | Rate limiter | Idempotency | Dry run/canary | Post-replay validation | Replay impossible | |---|---|---|---|---|---|---|---|---|---|---|---| | RCV01 | EventBridge target DLQ using SQS Standard; C57 A81 retrieved 2026-08-22 | payload=full_event; failure_metadata=RULE_ARN+TARGET_ARN+ERROR_CODE+ERROR_MESSAGE+RETRY_ATTEMPTS+EXHAUSTED_CONDITION; reconstruction=forbidden | Rule-scoped SQS policy for events.amazonaws.com plus consumer read delete and KMS access | SQS retention exceeds incident diagnosis repair and canary window | source_bus_replay=forbidden; recovery=controlled_direct-target_or_repair_consumer_for_failed_TARGET_ARN; ack_delete=after_committed_target_repair+target_receipt+business_reconciliation | Failed target dependency healthy; direct-target contract and consumer inbox proven | Poll under measured target budget; pause before live target SLO breach | Original event ID failed TARGET_ARN and target-specific inbox or intent receipt | No-effect validation then one event or 1 percent canary against failed target | Target receipt and business reconciliation precede SQS delete acknowledgement | Impossible after DLQ expiry without archive or authoritative regeneration | | RCV02 | EventBridge archive replay; C62 A84 retrieved 2026-08-22 | Selected full source-bus events retained by archive pattern and time window | Archive replay operator permissions and reviewed optional rule selection | Archive retention exceeds detection fix replay and reconciliation window | Replay goes only to source-bus and may reorder; enumerate every re-matched rule and target; prove idempotency and bound duplicate effects before replay | All selected rules targets consumers and external-effect protections healthy | Narrow UTC window and optional rule set; cancel on duplicate-effect or live SLO budget | Original event ID plus inbox intent and receipt at every re-matched target | Dry run inventory of matched rules and targets then narrow-window canary | DescribeReplay progress plus per-target receipts and business reconciliation | Impossible after archive expiry or when archive pattern omitted the event | | RCV03 | SQS source DLQ and managed redrive; C63 C103 A85 A118 retrieved 2026-08-22 | Full SQS message body attributes and receive history available at queue boundary | Redrive IAM and KMS access; RedriveAllowPolicy limits source queues | DLQ retention covers original age; Standard expiry uses original enqueue timestamp | Standard order is best effort; FIFO DLQ movement can break source group order | Consumer fix deployed; dependency capacity and queue policy proven | Start 10 messages/s then ramp below reserved replay share | Stable business ID and atomic inbox | Dry-run parser then one message group or 1 percent canary | Source queue age falls; inbox effects and reconciliation totals match | Impossible after message expiry unless durable authority can regenerate | | RCV04 | Lambda Kinesis on-failure S3 destination; C76 A98 retrieved 2026-08-22 | Complete failed invocation record and payload in S3 | Lambda destination write plus operator read KMS and lifecycle permissions | S3 lifecycle exceeds Kinesis expiry and investigation SLA | Custom replay restores source key and version; arrival order is not recovered automatically | Handler fix and all downstream dependencies healthy | Separate replay stream or queue at 25 percent measured capacity | Original event ID source version and inbox | Offline parse and no-effect mode then one shard-key canary | S3 manifest IDs equal completed inbox or explained quarantine IDs | Impossible when only SQS or SNS failure metadata exists and source record expired | | RCV05 | Kinesis retained stream replay; C74 C76 A98 retrieved 2026-08-22 | Full retained records within configured stream retention | Consumer IAM and checkpoint control isolated from live application | Retention exceeds detection fix drain and margin | Per-shard sequence retained; cross-shard order never inferred | Consumer version compatible and sink has replay lane | Dedicated consumer capped so live traffic and sink stay inside budget | Event ID aggregate version conditional apply | Start from narrow sequence window with side effects disabled | Checkpoint watermark version gaps and control totals reconcile | Impossible after retention expiry without S3 or authoritative backfill | | RCV06 | DynamoDB Streams CDC; C84 A102 retrieved 2026-08-22 | Configured key and image fields for 24-hour stream record window | Stream read role and consumer checkpoint access | Fixed 24-hour Streams boundary requires independent longer recovery source | Per-item modification order only; transaction records can interleave | Projection fix deployed and target isolated | Backfill from PITR export or authority; catch-up stream rate capped | Semantic outbox ID or item version plus inbox | Blue-green target dry run and sampled item comparison | Manifest totals versions and final watermark match authority | Impossible from Streams alone after 24 hours or when view omitted needed attributes | | RCV07 | SNS subscription DLQ; C64 A86 retrieved 2026-08-22 | Undelivered subscription message in SQS plus endpoint context | SNS topic to SQS policy and redrive operator permissions | Queue retention covers product notice expiry and investigation | Standard delivery can duplicate or reorder; FIFO scope remains subscription-specific | Endpoint and credentials healthy; notice still meaningful | Channel-specific replayer under provider quota | Notification intent ID and expiry check | Render-only dry run then test recipient canary | Provider receipt or documented policy exception; no business state mutation | Impossible or prohibited after notice expiry or missing consent | | RCV08 | Data Firehose S3 backup or failed-delivery objects; C66 A88 retrieved 2026-08-22 | Destination-specific buffered records and error output when configured | Firehose delivery role S3 KMS and replay job permissions | S3 lifecycle covers destination repair and audit window | Buffered delivery and retry can duplicate; no business order inferred | Destination schema quota and credentials healthy | New ingestion path isolated and rate-limited below destination headroom | Stable record ID and idempotent destination merge | Validate file schema and sample partition before bounded load | Object manifest row counts destination totals and rejected rows reconcile | Impossible when backup mode omitted the failed payload and source retention expired | ## Ordering and concurrency repair Order is enforced at the smallest invariant scope, not inferred from arrival. Every mutation carries a source version or sequence, has single-writer ownership for that scope, and uses conditional expected-version apply. A gap buffer has a gap timeout; late, stale, and duplicate policy is deterministic. Backfill runs in an isolated lane and reconciliation proves convergence (C34,C40,C41,C72,C74,C84,C93; F01,F02; A102, retrieved 2026-08-22). **Inference:** account/order aggregates serialize only their own invariants; cross-aggregate projections merge deterministically and reconcile rather than claiming global arrival order (C34,C41,C93; F01). ```text ORDERING|OR01|Order state machine|orderVersion monotonic integer|Order command service owns writes|Condition current version equals n then write n plus 1|Buffer missing n for 60 s then fetch authority|Duplicate returns stored result; stale rejected; concurrent conflict re-reads and re-evaluates|No in-place backfill; replay original IDs in order scope|Accepted order and transition manifest by version|Exactly one valid transition occupies each committed version ORDERING|OR02|Execution and ledger posting set|execution sequence plus postingSetId|Execution authority emits; ledger authority posts|Unique posting legs and expected reservation version in one transaction|Gap blocks affected account only and pages after 30 s|Duplicate is no-op; conflicting fill goes to deterministic break workflow|Replay into isolated ledger validator before any posting|Executions quantities reservations and balanced currency totals|Every execution maps to one balanced posting set or explicit open break ORDERING|OR03|Portfolio projection|authoritative account-instrument version|Projector lane single-writes each projection key|Inbox ID and expected projection version applied atomically|Gap buffer 60 s then source backfill; never skip silently|Late stale duplicate rejected; same-version different fingerprint quarantined|Blue-green rebuild with effects disabled and rate cap|Compare quantity cost basis and watermark to executions and ledger|Serving alias moves only when manifest complete and totals match ORDERING|OR04|Kinesis producer key|domain version plus returned same-client sequence when usable|One producer owner per logical key or fenced epoch|Consumers validate domain version regardless of shard sequence|Ambiguous producer acknowledgement breaks sequence chain and triggers lookup|Duplicate identity absorbed; deterministic version wins only from authority|Replay preserves partition key but uses isolated consumer and source-version checks|Checkpoint sequence version-gap and downstream totals|Per-key history is complete without inferring cross-shard order ``` | ID | Scope | Source version | Single writer | Conditional apply | Gap policy | Late/conflict policy | Replay/backfill | Reconciliation | Proof | |---|---|---|---|---|---|---|---|---|---| | OR01 | Order state machine | orderVersion monotonic integer | Order command service owns writes | Condition current version equals n then write n plus 1 | Buffer missing n for 60 s then fetch authority | Duplicate returns stored result; stale rejected; concurrent conflict re-reads and re-evaluates | No in-place backfill; replay original IDs in order scope | Accepted order and transition manifest by version | Exactly one valid transition occupies each committed version | | OR02 | Execution and ledger posting set | execution sequence plus postingSetId | Execution authority emits; ledger authority posts | Unique posting legs and expected reservation version in one transaction | Gap blocks affected account only and pages after 30 s | Duplicate is no-op; conflicting fill goes to deterministic break workflow | Replay into isolated ledger validator before any posting | Executions quantities reservations and balanced currency totals | Every execution maps to one balanced posting set or explicit open break | | OR03 | Portfolio projection | authoritative account-instrument version | Projector lane single-writes each projection key | Inbox ID and expected projection version applied atomically | Gap buffer 60 s then source backfill; never skip silently | Late stale duplicate rejected; same-version different fingerprint quarantined | Blue-green rebuild with effects disabled and rate cap | Compare quantity cost basis and watermark to executions and ledger | Serving alias moves only when manifest complete and totals match | | OR04 | Kinesis producer key | domain version plus returned same-client sequence when usable | One producer owner per logical key or fenced epoch | Consumers validate domain version regardless of shard sequence | Ambiguous producer acknowledgement breaks sequence chain and triggers lookup | Duplicate identity absorbed; deterministic version wins only from authority | Replay preserves partition key but uses isolated consumer and source-version checks | Checkpoint sequence version-gap and downstream totals | Per-key history is complete without inferring cross-shard order | ## Financial reconciliation control Reconciliation is a scheduled and incident-triggered financial control, not a dashboard afterthought. Every internal manifest uses the mandatory UTC `[startInclusive,endExclusive)` invariant; adjacent windows share the exact endpoint, an event at start is included, and an event at end belongs only to the next window. A run has an immutable manifest: control ID, source versions, timezone, currency/scale, object checksums, record counts, high-water marks, code/config version, and rerun lineage. Provider evidence retains its local business date, timezone, cutoff calendar, and deterministic mapping to the UTC half-open interval. Windows close only when all source watermarks reach or exceed the exclusive end boundary. Control totals retain exact units; counts never substitute for money or quantity (C46-C48,C91; F08,F14-F17). The error-budget and incident practices in F14,F15,F16 govern timeliness, while financial closure still requires the domain evidence below. **Inference:** orders, executions, ledger/reservations, and external statements authorize corrections in their owned domains. Portfolio, search, and notification projections only reveal or receive repair; they never authorize money, quantity, or order corrections (C46,C47,C48,C91; F08,F17). ```text RECON|RC01|Accepted orders|Authoritative order command store|Order IDs and terminal-state version|UTC [startInclusive,endExclusive); adjacent windows share endpoint; start included and end excluded|Count of accepted orders and exact ordered quantity in instrument units|Accepted IDs versus execution pending rejected canceled partition|Maximum orderVersion and ingestion timestamp reaches exclusive window end|Order operations; 2 min intraday accepted-without-route SLA|Repair outbox or conditional order state only from command evidence|Every accepted ID has one classified lifecycle state and manifest checksum RECON|RC02|Executions|Execution venue receipts and internal execution authority|Execution IDs venue IDs order IDs and event sequence|UTC [startInclusive,endExclusive); adjacent windows share endpoint; venue local timestamp maps to UTC before comparison|Fill count and exact fill quantity per instrument; exact price in currency minor units with scale|Internal executions versus accepted orders and venue fills|Venue sequence plus internal execution watermark reaches exclusive window end|Trading operations; 1 min duplicate or missing fill SLA|Import missing receipt or mark break; never invent a fill from portfolio|One-to-one venue receipt lineage and no duplicate execution ID RECON|RC03|Ledger balances and reservations|Append-only balanced ledger and reservation authority|Posting-set IDs account IDs currencies effective times and ledger versions|UTC [startInclusive,endExclusive); adjacent windows share endpoint; effective timestamps use governed UTC|Debit and credit minor units by currency; posting count; balance and reservation minor units|Debits equal credits per currency; executions equal postings; available equals balance minus reservations by policy|Ledger sequence and posting effective-time watermark reaches exclusive window end|Ledger operations; zero imbalance tolerance and 5 min reservation mismatch SLA|Conditional reversal or correcting entries with original-posting lineage; never update history|Balanced totals reservation invariant and independent rerun checksum RECON|RC04|Exchange custodian and bank evidence|Signed or controlled provider reports API receipts and statements|Provider file IDs checksums accounts and statement sequence|UTC [startInclusive,endExclusive); adjacent windows share endpoint; provider local business date timezone cutoff calendar retained and deterministic mapping recorded|Exchange fill count and quantity; custodian position quantity; bank cash minor units by currency|Provider totals versus executions ledger cash and settled positions|Provider statement complete marker and internal watermark reaches mapped exclusive end|Trading payments or settlement operations; product-defined intraday or end-of-day SLA|Forward completion reversal or correcting entry approved under dual control|Provider and internal manifests match or every difference has closed evidence RECON|RC05|Portfolio and search projections|Orders executions and ledger are authority; projection is derived|Projection build ID source manifest alias version and per-key version|UTC [startInclusive,endExclusive); adjacent windows share endpoint; projection as-of maps to source UTC interval|Position quantity and cost minor units; indexed document count; freshness age in seconds|Projection totals and keys versus authoritative source window|Projection watermark reaches source exclusive end and no unresolved gap|Projection owner; 5 min portfolio and 30 min search freshness assumptions|Isolated rebuild and conditional cutover; no authoritative correction from projection|Blue-green validation totals match and rollback target retained RECON|RC06|Notifications|Business authority plus durable notification intent and provider receipt|Intent IDs template version expiry channel and authority version|UTC [startInclusive,endExclusive); adjacent windows share endpoint; provider receipt local time maps to UTC|Intent count delivery receipt count failure count and age in seconds|Required intents versus channel receipts policy exceptions and client inbox|Intent watermark reaches exclusive authority-window end and provider poll complete|Notification owner; mandatory-notice SLA by product and compliance policy|Regenerate only valid intent from authority or record approved exception|Receipt or exception for every required intent; no business mutation ``` | ID | Domain | Authority | Manifest | Window contract | Exact units | Comparison/control totals | Watermark | Break owner/SLA | Repair | Proof | |---|---|---|---|---|---|---|---|---|---|---| | RC01 | Accepted orders | Authoritative order command store | Order IDs and terminal-state version | UTC [startInclusive,endExclusive); adjacent windows share endpoint; start included and end excluded | Count of accepted orders and exact ordered quantity in instrument units | Accepted IDs versus execution pending rejected canceled partition | Maximum orderVersion and ingestion timestamp reaches exclusive window end | Order operations; 2 min intraday accepted-without-route SLA | Repair outbox or conditional order state only from command evidence | Every accepted ID has one classified lifecycle state and manifest checksum | | RC02 | Executions | Execution venue receipts and internal execution authority | Execution IDs venue IDs order IDs and event sequence | UTC [startInclusive,endExclusive); adjacent windows share endpoint; venue local timestamp maps to UTC before comparison | Fill count and exact fill quantity per instrument; exact price in currency minor units with scale | Internal executions versus accepted orders and venue fills | Venue sequence plus internal execution watermark reaches exclusive window end | Trading operations; 1 min duplicate or missing fill SLA | Import missing receipt or mark break; never invent a fill from portfolio | One-to-one venue receipt lineage and no duplicate execution ID | | RC03 | Ledger balances and reservations | Append-only balanced ledger and reservation authority | Posting-set IDs account IDs currencies effective times and ledger versions | UTC [startInclusive,endExclusive); adjacent windows share endpoint; effective timestamps use governed UTC | Debit and credit minor units by currency; posting count; balance and reservation minor units | Debits equal credits per currency; executions equal postings; available equals balance minus reservations by policy | Ledger sequence and posting effective-time watermark reaches exclusive window end | Ledger operations; zero imbalance tolerance and 5 min reservation mismatch SLA | Conditional reversal or correcting entries with original-posting lineage; never update history | Balanced totals reservation invariant and independent rerun checksum | | RC04 | Exchange custodian and bank evidence | Signed or controlled provider reports API receipts and statements | Provider file IDs checksums accounts and statement sequence | UTC [startInclusive,endExclusive); adjacent windows share endpoint; provider local business date timezone cutoff calendar retained and deterministic mapping recorded | Exchange fill count and quantity; custodian position quantity; bank cash minor units by currency | Provider totals versus executions ledger cash and settled positions | Provider statement complete marker and internal watermark reaches mapped exclusive end | Trading payments or settlement operations; product-defined intraday or end-of-day SLA | Forward completion reversal or correcting entry approved under dual control | Provider and internal manifests match or every difference has closed evidence | | RC05 | Portfolio and search projections | Orders executions and ledger are authority; projection is derived | Projection build ID source manifest alias version and per-key version | UTC [startInclusive,endExclusive); adjacent windows share endpoint; projection as-of maps to source UTC interval | Position quantity and cost minor units; indexed document count; freshness age in seconds | Projection totals and keys versus authoritative source window | Projection watermark reaches source exclusive end and no unresolved gap | Projection owner; 5 min portfolio and 30 min search freshness assumptions | Isolated rebuild and conditional cutover; no authoritative correction from projection | Blue-green validation totals match and rollback target retained | | RC06 | Notifications | Business authority plus durable notification intent and provider receipt | Intent IDs template version expiry channel and authority version | UTC [startInclusive,endExclusive); adjacent windows share endpoint; provider receipt local time maps to UTC | Intent count delivery receipt count failure count and age in seconds | Required intents versus channel receipts policy exceptions and client inbox | Intent watermark reaches exclusive authority-window end and provider poll complete | Notification owner; mandatory-notice SLA by product and compliance policy | Regenerate only valid intent from authority or record approved exception | Receipt or exception for every required intent; no business mutation | Breaks use a conditional lifecycle. Only the current owner can advance a break; all evidence is append-only, and closure requires independent proof. ```text BREAK|OPEN|Control ID manifest difference exact units and first-seen timestamp|Reconciliation service immediately; page domain owner by row SLA|Valid difference exists and is not an approved timing item|Customer state pending investigation; affected scope visible|No mutation|Immutable break ID and source checksums BREAK|TRIAGED|Classification timing duplicate missing value or external ambiguity plus hypothesis|Domain operations inside SLA; compliance for regulated scope|Evidence identifies authority and affected accounts orders postings|Customer state pending or restricted; support script attached|Approve containment and repair plan under dual control|Triage decision and approver identities BREAK|CONTAINED|Frozen account product replay lane or outbound action with blast radius|Incident commander and domain owner|No new inconsistent effects enter affected scope|Customer sees explicit delayed restricted or unavailable state|Containment is reversible and does not rewrite authority|Metrics show break count no longer growing BREAK|CORRECTED|Provider receipt restored event or approved reversal/correcting posting lineage|Authority owner plus independent approver within resolution SLA|Conditional expected-version repair succeeds against unchanged premise|Customer sees corrected or processing state with audit reference|Forward completion reversal or correcting entries only|Old and new versions posting IDs and approvals retained BREAK|VERIFIED|Fresh reconciliation run independent query and customer-impact review|Reconciliation owner independent of repairer|Original and adjacent windows match exact-unit control totals|Customer state is consistent across authority and projections|No further mutation unless verification opens a new break|Two independent manifests and zero unexplained differences BREAK|CLOSED|Closure reason root cause linked incident and prevention action|Control owner; compliance sign-off where required|All proof fields complete and SLA disposition recorded|Customer communication delivered or documented unnecessary|Closure is conditional on retained evidence and can reopen|Closed timestamp evidence hashes owner and follow-up ticket ``` | State | Evidence | Owner/SLA | Transition condition | Customer state | Resolution | Proof | |---|---|---|---|---|---|---| | OPEN | Control ID manifest difference exact units and first-seen timestamp | Reconciliation service immediately; page domain owner by row SLA | Valid difference exists and is not an approved timing item | Customer state pending investigation; affected scope visible | No mutation | Immutable break ID and source checksums | | TRIAGED | Classification timing duplicate missing value or external ambiguity plus hypothesis | Domain operations inside SLA; compliance for regulated scope | Evidence identifies authority and affected accounts orders postings | Customer state pending or restricted; support script attached | Approve containment and repair plan under dual control | Triage decision and approver identities | | CONTAINED | Frozen account product replay lane or outbound action with blast radius | Incident commander and domain owner | No new inconsistent effects enter affected scope | Customer sees explicit delayed restricted or unavailable state | Containment is reversible and does not rewrite authority | Metrics show break count no longer growing | | CORRECTED | Provider receipt restored event or approved reversal/correcting posting lineage | Authority owner plus independent approver within resolution SLA | Conditional expected-version repair succeeds against unchanged premise | Customer sees corrected or processing state with audit reference | Forward completion reversal or correcting entries only | Old and new versions posting IDs and approvals retained | | VERIFIED | Fresh reconciliation run independent query and customer-impact review | Reconciliation owner independent of repairer | Original and adjacent windows match exact-unit control totals | Customer state is consistent across authority and projections | No further mutation unless verification opens a new break | Two independent manifests and zero unexplained differences | | CLOSED | Closure reason root cause linked incident and prevention action | Control owner; compliance sign-off where required | All proof fields complete and SLA disposition recorded | Customer communication delivered or documented unnecessary | Closure is conditional on retained evidence and can reopen | Closed timestamp evidence hashes owner and follow-up ticket | ## Disaster recovery and regional loss RTO is maximum acceptable restoration delay; RPO is maximum acceptable data loss measured as time since the last recoverable point. Each row separates the business objective from the selected mechanism and the result measured in a game day; an objective remains unproven until that result exists. Both are set from business impact and tested, not inherited from a service name (C105; [AWS DR objectives](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/disaster-recovery-dr-objectives.html), A119, retrieved 2026-08-22). The values below are business assumptions, not AWS service promises. Recover in dependency order: identity/configuration/KMS and network; fenced command authority; ledger/reservations; outbox/log recovery inputs; external exchange/bank/custodian connectivity; projections/search; notifications. Restore and export boundaries remain those of C88/A106. MREC replication is asynchronous, so neither MREC nor PITR proves a zero RPO. MRSC can provide strongly consistent item operations, but Global Tables MRSC does not support DynamoDB transaction APIs: it is therefore incompatible with Task 6's multi-item transactional ledger-posting design unless that design is replaced by a different synchronously durable journal and its serialization and availability tradeoff is accepted. This design retains that transaction model and declares a nonzero business RPO objective that remains unproven until a game day measures actual replica lag and missing IDs. MREC has no replication- latency SLA. Provider effects remain outside either replica guarantee (C89; A107, retrieved 2026-08-22). Route 53 health checks and ARC recovery controls are pre-provisioned data-plane mechanisms. Failover timing must include DNS TTL, recursive and endpoint caches, existing keepalive connections, SDK connection-pool re-resolution, and client retry identity. A stale client can still reach the old Region, so the old endpoint must return a fenced response rather than accept a command. Failback is a separately approved migration with the same fencing and evidence gates (C106; A120,A121, retrieved 2026-08-22). **Inference:** use a single active command/ledger writer epoch and fail closed until the old writer is fenced. Neither MREC nor MRSC removes external-effect ambiguity; failover finishes only after replay, measured recovery-point age and loss meet the RPO objective, reconciliation closes, and stale-client probes pass, while failback repeats the fence (C89,C105,C106; A107,A119,A120,A121, retrieved 2026-08-22). ```text DR|DR01|Command and order authority|type=RTO; objective=5 min; service=command_and_order_authority_available|type=RPO; objective=1 min; loss_bound=acknowledged_command_recovery_point_age|MREC command replica plus immutable accepted-command receipts and PITR; no zero-RPO claim|Not yet measured; game day records maximum replica lag and missing accepted-command IDs, so the objective remains unproven|PITR and cross-Region replica plus accepted-command manifest; replay only missing IDs within retained evidence|Identity KMS network then command store then API|One writer epoch in conditional command records; fail closed and old endpoint returns fenced response before promotion|Route 53 and ARC data-plane routing; DNS TTL, endpoint caches, keepalive and re-resolution tested; retry identity survives stale routing; A120 A121 retrieved 2026-08-22|Failover uses stale-endpoint probes before writes; failback is separately fenced, canaried, and approved from manifests|MREC conflict is unacceptable for one command key; MRSC transaction limitation and external effects stop the guarantee|Synthetic lookup and accepted-order manifest reconcile one active epoch; game-day measured recovery point age is at or below the RPO objective DR|DR02|Ledger and reservations|type=RTO; objective=5 min; service=ledger_and_reservations_available|type=RPO; objective=1 min; loss_bound=acknowledged_posting_recovery_point_age|MREC posting replica plus immutable posting manifests, PITR exports, and provider statements; no zero-RPO claim|Not yet measured; game day records replica lag, missing posting IDs, debit-credit breaks, and recovery point age, so the objective remains unproven|PITR backups exports and posting manifests survive ledger and audit horizon; never restore balance rows without postings|Command authority then ledger then risk and external adapters|Ledger writer epoch and account expected versions; old endpoint returns fenced response before posting|Route 53 and ARC data-plane routing; DNS TTL, endpoint caches, keepalive and re-resolution tested; retry identity survives stale routing; A120 A121 retrieved 2026-08-22|Failover enables ledger writes only after manifest canary; failback repeats fencing, totals, and provider reconciliation|No concurrent regional ledger writers; MRSC cannot preserve the Task 6 multi-item transaction and providers remain external|Debits equal credits by currency; balances reservations executions and provider evidence reconcile; game-day recovery point age is at or below objective DR|DR03|Outbox and durable logs|type=RTO; objective=15 min; service=outbox_and_durable_log_recovery_available|type=RPO; objective=5 min; loss_bound=outbox_and_log_recovery_point_age|MREC semantic outbox plus retained Kinesis or S3 audit copy and publisher receipts|Not yet measured; game day records oldest missing event, recoverable watermark, replay duplicates, and drain duration, so the objective remains unproven|Retention exceeds detection restore replay and drain; Streams alone is 24 h and archive replay can re-match rules|Authority and ledger before publisher and consumers|Publisher epoch and original event IDs; old Region credentials revoked and writes conditionally fenced|Route 53 and ARC data-plane routing; DNS TTL, endpoint caches, keepalive and re-resolution tested; retry identity survives stale routing; A120 A121 retrieved 2026-08-22|Failover replays only manifest gaps after consumer gate; failback checkpoints replay and revalidates downstream receipts|Replay can duplicate and reorder across sources; retained logs do not make external effects atomic|Accepted authority IDs equal delivered inbox, target receipt, or explicit open break; game-day recovery point age is at or below objective DR|DR04|Portfolio and search projections|type=RTO; objective=24 h; service=validated_projection_rebuild_and_cutover|type=RPO; inherits=DR01+DR02+DR03; independent_authoritative_loss_budget=none; consequence=rebuild_to_recovered_authority_watermarks|Blue-green rebuild from orders executions ledger and versioned S3 manifests|Not yet measured; game day records rebuild duration, source watermark, missing versions, and alias-cutover time, so the RTO objective remains unproven|Durable authority and versioned manifests cover the full rebuild and catch-up boundary; live Streams only catch up and never authorize correction|All authorities and outbox before projector then search alias|Projection build token and conditional alias switch; command writer fencing remains upstream|Route 53 and ARC data-plane routing; DNS TTL, endpoint caches, keepalive and re-resolution tested; retry identity preserves commands; A120 A121 retrieved 2026-08-22|Failover serves stale as-of view until validated cutover; failback rebuilds separately and conditionally switches alias|Derived stores have no independent authoritative loss budget; they inherit recovered authority points and can remain unavailable or stale|Manifest counts quantities exact values and watermark match recovered authority before either cutover DR|DR05|Notifications|type=RTO; objective=24 h; service=required_notice_delivery_recovery|type=RPO; objective=5 min; loss_bound=durable_notification_intent_recovery_point_age; consequence=regenerate_only_policy_valid_notices_from_recovered_authority|Regenerate valid notification intents from recovered authority and retained durable intent store|Not yet measured; game day records oldest missing durable intent, expired count, duplicate receipts, and delivery completion, so both objectives remain unproven|Intent retention covers the RPO plus restore and investigation margin but remains bounded by consent template version expiry and legal policy; expired one-time notices are not replayed|Authority and projections before channel replay|Channel sender epoch and intent uniqueness; old sender is revoked before new sender starts|Route 53 and ARC data-plane routing; DNS TTL, endpoint caches, keepalive and re-resolution tested; retry identity protects command lookup; A120 A121 retrieved 2026-08-22|Failover canaries valid notices under provider quota; failback fences sender, resumes manifest checkpoint, and validates receipts|Provider delivery remains external; notices missing beyond the intent RPO must be regenerated from recovered authority when policy-valid or recorded as breaks|Receipt or approved exception for each required notice; durable-intent recovery point age is at or below objective; no duplicate business action ``` | ID | Tier | Illustrative objective RTO | Illustrative objective RPO | Selected mechanism | Measured result | Backup/retention/replay boundary | Dependency order | Single-writer fencing | Routing/client behavior | Failover/failback | Conflict/external-effect limit | Proof | |---|---|---|---|---|---|---|---|---|---|---|---|---| | DR01 | Command and order authority | type=RTO; objective=5 min; service=command_and_order_authority_available | type=RPO; objective=1 min; loss_bound=acknowledged_command_recovery_point_age | MREC command replica plus immutable accepted-command receipts and PITR; no zero-RPO claim | Not yet measured; game day records maximum replica lag and missing accepted-command IDs, so the objective remains unproven | PITR and cross-Region replica plus accepted-command manifest; replay only missing IDs within retained evidence | Identity KMS network then command store then API | One writer epoch in conditional command records; fail closed and old endpoint returns fenced response before promotion | Route 53 and ARC data-plane routing; DNS TTL, endpoint caches, keepalive and re-resolution tested; retry identity survives stale routing; A120 A121 retrieved 2026-08-22 | Failover uses stale-endpoint probes before writes; failback is separately fenced, canaried, and approved from manifests | MREC conflict is unacceptable for one command key; MRSC transaction limitation and external effects stop the guarantee | Synthetic lookup and accepted-order manifest reconcile one active epoch; game-day measured recovery point age is at or below the RPO objective | | DR02 | Ledger and reservations | type=RTO; objective=5 min; service=ledger_and_reservations_available | type=RPO; objective=1 min; loss_bound=acknowledged_posting_recovery_point_age | MREC posting replica plus immutable posting manifests, PITR exports, and provider statements; no zero-RPO claim | Not yet measured; game day records replica lag, missing posting IDs, debit-credit breaks, and recovery point age, so the objective remains unproven | PITR backups exports and posting manifests survive ledger and audit horizon; never restore balance rows without postings | Command authority then ledger then risk and external adapters | Ledger writer epoch and account expected versions; old endpoint returns fenced response before posting | Route 53 and ARC data-plane routing; DNS TTL, endpoint caches, keepalive and re-resolution tested; retry identity survives stale routing; A120 A121 retrieved 2026-08-22 | Failover enables ledger writes only after manifest canary; failback repeats fencing, totals, and provider reconciliation | No concurrent regional ledger writers; MRSC cannot preserve the Task 6 multi-item transaction and providers remain external | Debits equal credits by currency; balances reservations executions and provider evidence reconcile; game-day recovery point age is at or below objective | | DR03 | Outbox and durable logs | type=RTO; objective=15 min; service=outbox_and_durable_log_recovery_available | type=RPO; objective=5 min; loss_bound=outbox_and_log_recovery_point_age | MREC semantic outbox plus retained Kinesis or S3 audit copy and publisher receipts | Not yet measured; game day records oldest missing event, recoverable watermark, replay duplicates, and drain duration, so the objective remains unproven | Retention exceeds detection restore replay and drain; Streams alone is 24 h and archive replay can re-match rules | Authority and ledger before publisher and consumers | Publisher epoch and original event IDs; old Region credentials revoked and writes conditionally fenced | Route 53 and ARC data-plane routing; DNS TTL, endpoint caches, keepalive and re-resolution tested; retry identity survives stale routing; A120 A121 retrieved 2026-08-22 | Failover replays only manifest gaps after consumer gate; failback checkpoints replay and revalidates downstream receipts | Replay can duplicate and reorder across sources; retained logs do not make external effects atomic | Accepted authority IDs equal delivered inbox, target receipt, or explicit open break; game-day recovery point age is at or below objective | | DR04 | Portfolio and search projections | type=RTO; objective=24 h; service=validated_projection_rebuild_and_cutover | type=RPO; inherits=DR01+DR02+DR03; independent_authoritative_loss_budget=none; consequence=rebuild_to_recovered_authority_watermarks | Blue-green rebuild from orders executions ledger and versioned S3 manifests | Not yet measured; game day records rebuild duration, source watermark, missing versions, and alias-cutover time, so the RTO objective remains unproven | Durable authority and versioned manifests cover the full rebuild and catch-up boundary; live Streams only catch up and never authorize correction | All authorities and outbox before projector then search alias | Projection build token and conditional alias switch; command writer fencing remains upstream | Route 53 and ARC data-plane routing; DNS TTL, endpoint caches, keepalive and re-resolution tested; retry identity preserves commands; A120 A121 retrieved 2026-08-22 | Failover serves stale as-of view until validated cutover; failback rebuilds separately and conditionally switches alias | Derived stores have no independent authoritative loss budget; they inherit recovered authority points and can remain unavailable or stale | Manifest counts quantities exact values and watermark match recovered authority before either cutover | | DR05 | Notifications | type=RTO; objective=24 h; service=required_notice_delivery_recovery | type=RPO; objective=5 min; loss_bound=durable_notification_intent_recovery_point_age; consequence=regenerate_only_policy_valid_notices_from_recovered_authority | Regenerate valid notification intents from recovered authority and retained durable intent store | Not yet measured; game day records oldest missing durable intent, expired count, duplicate receipts, and delivery completion, so both objectives remain unproven | Intent retention covers the RPO plus restore and investigation margin but remains bounded by consent template version expiry and legal policy; expired one-time notices are not replayed | Authority and projections before channel replay | Channel sender epoch and intent uniqueness; old sender is revoked before new sender starts | Route 53 and ARC data-plane routing; DNS TTL, endpoint caches, keepalive and re-resolution tested; retry identity protects command lookup; A120 A121 retrieved 2026-08-22 | Failover canaries valid notices under provider quota; failback fences sender, resumes manifest checkpoint, and validates receipts | Provider delivery remains external; notices missing beyond the intent RPO must be regenerated from recovered authority when policy-valid or recorded as breaks | Receipt or approved exception for each required notice; durable-intent recovery point age is at or below objective; no duplicate business action | Quarterly game days restore backups to a new environment, promote a fenced writer, replay outbox/logs at the planned split, reconcile every exact-unit control, and verify customer states. At least annually, exercise full Region failover and failback. Abort on a second writer, growing ledger break, missing encryption/config dependency, or live-traffic SLO breach. Retain timelines, commands, manifests, approvals, and proof against the RTO/RPO clock (F14,F15; A106,A119-A121, retrieved 2026-08-22). ## Operator runbooks Thresholds below are illustrative planning inputs. Replace them only through an owned SLO/capacity review. Each runbook embodies detect, contain, bounded action, and repair; no runbook declares success from queue depth alone (C43,C45,C48,C68; F03,F13,F14). ```text RUNBOOK|RBK01|Ambiguous order response|Client timeout plus command IN_PROGRESS older than 2 s or accepted-without-response count above zero|Stop duplicate logical command; return pending and lookup token|Query fingerprint order ID order version and outbox; preserve request correlation|One client retry inside 8 s then lookup every 1 s for 5 s|restore=retry_policy_version; checkpoint=command_manifest+lookup_cursor; release=one_fingerprint_one_order_one_result_proven; irreversible=escalate_customer_or_ledger_mutation_to_order_incident_lead|Any mismatched fingerprint or two order IDs|Order API on-call; incident lead if unresolved 2 min|Pending not failed and not safely resubmittable|Reconcile command order outbox and return durable result|One fingerprint one order ID one final customer result and no open break RUNBOOK|RBK02|Producer partial or unknown result|FailedEntryCount above zero missing response entry or outbox age above 30 s|Stop whole-batch retry; persist attempted manifest and isolate unknowns|Inspect each entry event ID error bus rule receipt and downstream inbox|Retry explicit failures maximum 5 over 2 min at publisher cap; lookup unknowns|restore=publisher_cap+retry_policy_version; checkpoint=attempted_manifest+per_entry_result_cursor; release=all_unknown_entries_resolved; irreversible=escalate_republish_or_external_effect_to_messaging_and_order_owners|Unknown count grows or downstream age exceeds 60 s|Messaging owner; order owner for accepted commands|Accepted-pending until downstream proof|Compare attempted explicit success explicit failure and resolved unknown; replay original IDs|Manifest balances and downstream receipt or open break exists for every entry RUNBOOK|RBK03|Kinesis or queue backlog|Iterator or oldest age above 60 s for 5 min or modeled retention margin below 2 h|Pause replay; shed optional work; cap concurrency to protect ledger and live traffic|Measure lambda mu skew throttles poison keys dependency latency and per-class wait|Replay at maximum 25 percent measured commit capacity; target 200 records/s in example|restore=concurrency+admission_policy_version; checkpoint=replay_manifest+last_committed_source_position; release=live_age_below_30s_for_15m_and_net_drain_positive; irreversible=escalate_expiry_or_dropped_obligation_to_incident_commander|Live p99 breaches SLO net drain non-positive or any reconciliation break grows|Incident commander consumer owner and dependency owner|Orders show delayed with watermark; authority remains available if safe|Drain under B divided by net rate; validate checkpoints gaps inbox and control totals|Age below 30 s for 15 min no starvation and all admitted work reconciled RUNBOOK|RBK04|Poison item|Same event fails twice or non-retryable schema validation occurs once|Quarantine full payload; isolate key or group; preserve ordered lane evidence|Record schema version producer hash error source sequence and downstream compatibility|Zero ordinary retries for non-retryable; offline dry run then one-key or 1 percent canary|restore=consumer_version+event_source_configuration; checkpoint=quarantine_manifest+last_safe_sequence; release=canary_effect_and_gap_reconciliation_pass; irreversible=escalate_transform_or_side_effect_to_schema_and_domain_owners|Canary error any unintended side effect or live age growth|Schema and consumer owners; domain owner for correction approval|Affected item pending; unrelated keys continue where order permits|Governed transform with new lineage or consumer fix; replay original identity and reconcile|Canary and batch pass; gap closes; counts versions and effects match RUNBOOK|RBK05|Projection gap|Expected version n but receives above n or watermark lag exceeds 5 min|Park gap; keep old view with as-of time; block cutover and suppress side effects|Query authority versions manifest checkpoint inbox and target build ID|Fetch narrow missing range; otherwise blue-green backfill capped at 25 percent|restore=old_alias+projector_version; checkpoint=old_and_new_build_manifests+replay_watermark; release=no_gaps_and_control_totals_match_authority; irreversible=escalate_index_deletion_or_authority_correction_to_projection_and_domain_owners|Authority mismatch unexplained duplicate version or live SLO impact|Projection owner; domain owner approves source totals|Stale view clearly labeled; never used for authorization|Catch up isolated target compare quantities values counts then conditional alias switch|No gaps; watermark equals target; two manifests match and rollback target retained RUNBOOK|RBK06|External exchange or bank timeout|Call timed out and no receipt after 1 s or callback age exceeds product threshold|Do not repeat effect; open circuit for optional requests; preserve intent and block conflict|Use provider request ID status API callback log statement and internal ledger intent|Five status lookups with full jitter over 5 min; no blind effect retry|restore=circuit+timeout_policy_version_after_provider_ready; checkpoint=intent_manifest+last_provider_receipt_cursor; release=provider_receipt_intent_and_ledger_agree; irreversible=escalate_provider_cancel_reversal_or_posting_to_operations_and_dual_control|Provider reports conflicting states or lookup budget expires|Trading or payments operations; compliance by product SLA|Pending-external and not failed; support cannot resubmit|Match provider evidence then forward-complete reverse or correct under dual control|Provider receipt intent and ledger agree; customer state updated; break independently closed RUNBOOK|RBK07|Ledger or reconciliation break|Any debit-credit imbalance duplicate fill missing posting or reservation mismatch; zero tolerance|Freeze affected account or product writes; preserve statements and manifests|Recompute exact units by currency and instrument; trace orders executions postings reservations provider evidence|No automated retry of money mutation; bounded evidence queries then conditional approved correction|restore=query+capacity_policy_only; checkpoint=original_and_adjacent_manifests+repair_version; release=independent_reconciliation_verified; irreversible=escalate_reversal_or_correcting_entry_to_dual_control|Break scope expands authority is uncertain or evidence checksum differs|Ledger operations incident commander compliance and independent approver|Restricted or pending with explicit support state|Use reversal or correcting entries with lineage; rerun original and adjacent windows|Balanced totals and reservations; provider match; two-person VERIFIED then CLOSED RUNBOOK|RBK08|Region failover|Regional business probe fails 2 min and incident commander declares disaster|Stop writes; revoke old writer credentials or epoch; do not shift command traffic yet|Verify replication lag backups KMS config network external connectivity accepted-command manifest; inspect Route 53 and ARC data-plane state DNS TTL endpoint caches keepalive re-resolution retry identity and fenced old-Region responses; A120 A121 retrieved 2026-08-22|Recover authority-first; canary reads then writes; replay within 25 percent capacity split|restore=last_fenced_routing+capacity_configuration; checkpoint=failover_manifest+writer_epoch+replay_watermark; release=stale_client_probes_and_financial_control_totals_pass; irreversible=escalate_promotion_data_correction_or_failback_to_incident_commander|Second writer detected ledger break grows RPO evidence missing or stale endpoint accepts a write|Incident commander platform command ledger external and compliance owners|Unavailable for commands until fenced; projections may show stale as-of state|Promote one epoch restore missing data replay outbox rebuild projections reconcile externals; separately fence and approve failback|One writer; measured restoration duration and recovery-point age/loss meet RTO/RPO objectives; DNS and keepalive stale-client probes pass; control totals clean; failback game day approved ``` | ID | Scenario | Entry signal/threshold | Immediate safety action | Diagnosis/evidence | Bounded action/rate | Rollback/resume | Abort condition | Owner/escalation | Customer state | Reconciliation/proof | Exit criteria | |---|---|---|---|---|---|---|---|---|---|---|---| | RBK01 | Ambiguous order response | Client timeout plus command IN_PROGRESS older than 2 s or accepted-without-response count above zero | Stop duplicate logical command; return pending and lookup token | Query fingerprint order ID order version and outbox; preserve request correlation | One client retry inside 8 s then lookup every 1 s for 5 s | restore=retry_policy_version; checkpoint=command_manifest+lookup_cursor; release=one_fingerprint_one_order_one_result_proven; irreversible=escalate_customer_or_ledger_mutation_to_order_incident_lead | Any mismatched fingerprint or two order IDs | Order API on-call; incident lead if unresolved 2 min | Pending not failed and not safely resubmittable | Reconcile command order outbox and return durable result | One fingerprint one order ID one final customer result and no open break | | RBK02 | Producer partial or unknown result | FailedEntryCount above zero missing response entry or outbox age above 30 s | Stop whole-batch retry; persist attempted manifest and isolate unknowns | Inspect each entry event ID error bus rule receipt and downstream inbox | Retry explicit failures maximum 5 over 2 min at publisher cap; lookup unknowns | restore=publisher_cap+retry_policy_version; checkpoint=attempted_manifest+per_entry_result_cursor; release=all_unknown_entries_resolved; irreversible=escalate_republish_or_external_effect_to_messaging_and_order_owners | Unknown count grows or downstream age exceeds 60 s | Messaging owner; order owner for accepted commands | Accepted-pending until downstream proof | Compare attempted explicit success explicit failure and resolved unknown; replay original IDs | Manifest balances and downstream receipt or open break exists for every entry | | RBK03 | Kinesis or queue backlog | Iterator or oldest age above 60 s for 5 min or modeled retention margin below 2 h | Pause replay; shed optional work; cap concurrency to protect ledger and live traffic | Measure lambda mu skew throttles poison keys dependency latency and per-class wait | Replay at maximum 25 percent measured commit capacity; target 200 records/s in example | restore=concurrency+admission_policy_version; checkpoint=replay_manifest+last_committed_source_position; release=live_age_below_30s_for_15m_and_net_drain_positive; irreversible=escalate_expiry_or_dropped_obligation_to_incident_commander | Live p99 breaches SLO net drain non-positive or any reconciliation break grows | Incident commander consumer owner and dependency owner | Orders show delayed with watermark; authority remains available if safe | Drain under B divided by net rate; validate checkpoints gaps inbox and control totals | Age below 30 s for 15 min no starvation and all admitted work reconciled | | RBK04 | Poison item | Same event fails twice or non-retryable schema validation occurs once | Quarantine full payload; isolate key or group; preserve ordered lane evidence | Record schema version producer hash error source sequence and downstream compatibility | Zero ordinary retries for non-retryable; offline dry run then one-key or 1 percent canary | restore=consumer_version+event_source_configuration; checkpoint=quarantine_manifest+last_safe_sequence; release=canary_effect_and_gap_reconciliation_pass; irreversible=escalate_transform_or_side_effect_to_schema_and_domain_owners | Canary error any unintended side effect or live age growth | Schema and consumer owners; domain owner for correction approval | Affected item pending; unrelated keys continue where order permits | Governed transform with new lineage or consumer fix; replay original identity and reconcile | Canary and batch pass; gap closes; counts versions and effects match | | RBK05 | Projection gap | Expected version n but receives above n or watermark lag exceeds 5 min | Park gap; keep old view with as-of time; block cutover and suppress side effects | Query authority versions manifest checkpoint inbox and target build ID | Fetch narrow missing range; otherwise blue-green backfill capped at 25 percent | restore=old_alias+projector_version; checkpoint=old_and_new_build_manifests+replay_watermark; release=no_gaps_and_control_totals_match_authority; irreversible=escalate_index_deletion_or_authority_correction_to_projection_and_domain_owners | Authority mismatch unexplained duplicate version or live SLO impact | Projection owner; domain owner approves source totals | Stale view clearly labeled; never used for authorization | Catch up isolated target compare quantities values counts then conditional alias switch | No gaps; watermark equals target; two manifests match and rollback target retained | | RBK06 | External exchange or bank timeout | Call timed out and no receipt after 1 s or callback age exceeds product threshold | Do not repeat effect; open circuit for optional requests; preserve intent and block conflict | Use provider request ID status API callback log statement and internal ledger intent | Five status lookups with full jitter over 5 min; no blind effect retry | restore=circuit+timeout_policy_version_after_provider_ready; checkpoint=intent_manifest+last_provider_receipt_cursor; release=provider_receipt_intent_and_ledger_agree; irreversible=escalate_provider_cancel_reversal_or_posting_to_operations_and_dual_control | Provider reports conflicting states or lookup budget expires | Trading or payments operations; compliance by product SLA | Pending-external and not failed; support cannot resubmit | Match provider evidence then forward-complete reverse or correct under dual control | Provider receipt intent and ledger agree; customer state updated; break independently closed | | RBK07 | Ledger or reconciliation break | Any debit-credit imbalance duplicate fill missing posting or reservation mismatch; zero tolerance | Freeze affected account or product writes; preserve statements and manifests | Recompute exact units by currency and instrument; trace orders executions postings reservations provider evidence | No automated retry of money mutation; bounded evidence queries then conditional approved correction | restore=query+capacity_policy_only; checkpoint=original_and_adjacent_manifests+repair_version; release=independent_reconciliation_verified; irreversible=escalate_reversal_or_correcting_entry_to_dual_control | Break scope expands authority is uncertain or evidence checksum differs | Ledger operations incident commander compliance and independent approver | Restricted or pending with explicit support state | Use reversal or correcting entries with lineage; rerun original and adjacent windows | Balanced totals and reservations; provider match; two-person VERIFIED then CLOSED | | RBK08 | Region failover | Regional business probe fails 2 min and incident commander declares disaster | Stop writes; revoke old writer credentials or epoch; do not shift command traffic yet | Verify replication lag backups KMS config network external connectivity accepted-command manifest; inspect Route 53 and ARC data-plane state DNS TTL endpoint caches keepalive re-resolution retry identity and fenced old-Region responses; A120 A121 retrieved 2026-08-22 | Recover authority-first; canary reads then writes; replay within 25 percent capacity split | restore=last_fenced_routing+capacity_configuration; checkpoint=failover_manifest+writer_epoch+replay_watermark; release=stale_client_probes_and_financial_control_totals_pass; irreversible=escalate_promotion_data_correction_or_failback_to_incident_commander | Second writer detected ledger break grows RPO evidence missing or stale endpoint accepts a write | Incident commander platform command ledger external and compliance owners | Unavailable for commands until fenced; projections may show stale as-of state | Promote one epoch restore missing data replay outbox rebuild projections reconcile externals; separately fence and approve failback | One writer; measured restoration duration and recovery-point age/loss meet RTO/RPO objectives; DNS and keepalive stale-client probes pass; control totals clean; failback game day approved | ## Senior interview contract Each major topic must be defended as an operating decision, not a vocabulary definition. The source routes below deliberately reuse C43-C48,C61 and the named conceptual anchors F01-F03,F08,F11-F14. ```text TOPIC|TP01|Failure surface|Expose ambiguity before choosing mechanisms|Fits multi-boundary financial workflows; poor fit only for a truly atomic local operation|Proof stops at each producer transport consumer store and external-effect boundary|Order acceptance through execution ledger projection and notification|Prevent detect contain repair each row; unresolved state remains explicit|Variable evidence storage and checks; fixed failure taxonomy ownership and drills|Unknown age per boundary accepted-without-effect and reconciliation breaks|I trace authority ambiguity budget owner and proof before naming AWS controls|Ambiguous order publication from CS01-CS03|A green dashboard or DLQ checkbox proves recovery|C43 C45 C48 F03 F11 F12 F14 TOPIC|TP02|Idempotency layers|Prevent retry from multiplying one logical effect|Fits repeatable commands and consumers; poor fit as a substitute for domain uniqueness|Transport handler command and ledger identities stop at different transactions|Client order ID event ID command fingerprint and posting set|Lookup mismatches expired leases and ambiguous effects require repair|Variable idempotency reads writes retention; fixed key governance and support policy|Duplicate-hit mismatch stale-IN_PROGRESS and repeated-posting counters|One ID never proves all layers; I name key state transaction expiry and effect|Order retry plus projector duplicate plus ledger posting|FIFO or acknowledgement means one business effect|C40 C47 C90 F02 F08 F11 TOPIC|TP03|Retry and timeout|Recover transient faults without storms or duplicate effects|Fits classified transient failures; poor fit for validation conflicts and ambiguous money movement|Deadline timeout owner maximum attempts and age stop at lookup or reconciliation|Producer rejection handler timeout and provider status lookup|Full jitter one owned layer and explicit pre-invocation versus invoked-handler paths|Variable extra attempts latency and capacity; fixed policy testing and incident ownership|Attempts by class deadline exhaustion oldest age and ambiguous queue|I retry only safe transient work; ambiguity changes the operation to lookup|Exchange timeout after possible fill|Retry until success or retry at every layer|C45 C65 C76 F03 F11 F12 A26 A98 TOPIC|TP04|Overload containment|Protect authority and fairness when demand exceeds safe commit rate|Fits burst dependency slowdown and replay; poor fit as a correction for wrong data|Bulkhead cap breaker shedding admission queue and fairness each stop at load not correctness|Live commands share DynamoDB and Lambda with projection and replay|Shed optional work cap replay reserve authority and reconcile expired obligations|Variable rejected deferred and idle reserve capacity; fixed SLO policy and capacity tests|Lambda throttle oldest age wait and rejection by tenant key priority|Autoscaling may add supply; admission and bounded debt decide safety|Reserve ledger lane while throttling search rebuild|Conflating autoscaling with backpressure or circuit breaking with correctness|C44 C61 C104 F03 F13 F14 A117 TOPIC|TP05|Quarantine and replay|Preserve failed evidence and reprocess only after readiness|Fits retained replayable inputs; poor fit after payload expiry or irreversible effect|Each EventBridge target DLQ archive SQS Lambda Kinesis Streams SNS and Firehose boundary has distinct payload route and order|Poison projection event and failed notification|Permissions retention canary rate idempotency delete acknowledgement and post-validation govern replay|Variable retained bytes and replay work; fixed tooling permissions and game days|Failure destination success age replay rate and business control totals|A target DLQ holds one target failure; archive replay can re-match source-bus rules; neither alone proves repair|Kinesis failed invocation retained in S3 and replayed through isolated lane|All failures go to the DLQ then redrive at full speed|C43 C57 C62 C63 C64 C66 C76 C103 A81 A84 A86 A98 A118 TOPIC|TP06|Ordering repair|Preserve per-invariant causality under duplicates concurrency and backfill|Fits versioned aggregates; poor fit for fabricated global arrival order|Source version single writer expected-version apply and gap policy stop at named scope|Order aggregate ledger posting set and portfolio key|Buffer gaps reject stale isolate replay and reconcile deterministic merge|Variable serialization wait and buffers; fixed sequence ownership conflict tooling|Gap age stale duplicate conflict rate and watermark|I pay serialization only where an invariant needs it and reconcile wider views|Account-instrument portfolio versions from execution authority|FIFO removes idempotency or optimistic lock deduplicates events|C34 C40 C41 C72 C74 C84 C93 F01 F02 TOPIC|TP07|Financial reconciliation|Detect and repair silent divergence using independent authoritative evidence|Fits money quantity and regulated notices; poor fit when a projection is treated as authority|Manifest window watermark exact units dual control and proof of closure|Orders executions postings reservations provider statements and projections|Break lifecycle contains then forward-completes reverses or corrects conditionally|Variable comparison queries evidence retention and manual cases; fixed control ownership and SLA|Imbalance missing posting duplicate fill reservation mismatch and break age|Availability is not correctness; every accepted obligation must reconcile to authority|Bank minor-unit statement versus cash ledger and reservations|Derived portfolio authorizes a money correction|C46 C47 C48 C91 F08 F14 F15 F16 F17 TOPIC|TP08|Disaster recovery|Restore one correct system rather than two conflicting writers|Fits Region loss and corruption recovery; poor fit as an untested multi-Region badge|Business RTO/RPO objectives selected mechanisms measured restoration duration and recovery-point age/loss backup replay fence routing external effects and reconciliation stop at declared tier|Command ledger outbox projections and notifications recover in dependency order|Fence old writer route through Route 53 and ARC data plane probe stale clients restore canary replay reconcile failover and separately failback|Variable replica backup replay and idle capacity; fixed IaC runbooks and game days|Restoration clock recovery-point age/loss replication lag DNS TTL caches connections writer epoch and control totals|Multi-Region shortens some recovery paths but never proves zero loss or globally atomic effects|Ledger Region failover reconciled with exchange custodian and bank|Treating MREC or PITR objectives as measured lossless recovery or assuming Route 53 instantly removes old clients|C88 C89 C105 C106 A106 A107 A119 A120 A121 F14 F15 ``` | ID | Topic | Why | Fit/poor fit | Guarantee/stopping boundary | Workload | Failure/recovery | Variable/fixed burden | Actionable metrics | Concise defense | Fintech example | Anti-pattern | Sources | |---|---|---|---|---|---|---|---|---|---|---|---|---| | TP01 | Failure surface | Expose ambiguity before choosing mechanisms | Fits multi-boundary financial workflows; poor fit only for a truly atomic local operation | Proof stops at each producer transport consumer store and external-effect boundary | Order acceptance through execution ledger projection and notification | Prevent detect contain repair each row; unresolved state remains explicit | Variable evidence storage and checks; fixed failure taxonomy ownership and drills | Unknown age per boundary accepted-without-effect and reconciliation breaks | I trace authority ambiguity budget owner and proof before naming AWS controls | Ambiguous order publication from CS01-CS03 | A green dashboard or DLQ checkbox proves recovery | C43 C45 C48 F03 F11 F12 F14 | | TP02 | Idempotency layers | Prevent retry from multiplying one logical effect | Fits repeatable commands and consumers; poor fit as a substitute for domain uniqueness | Transport handler command and ledger identities stop at different transactions | Client order ID event ID command fingerprint and posting set | Lookup mismatches expired leases and ambiguous effects require repair | Variable idempotency reads writes retention; fixed key governance and support policy | Duplicate-hit mismatch stale-IN_PROGRESS and repeated-posting counters | One ID never proves all layers; I name key state transaction expiry and effect | Order retry plus projector duplicate plus ledger posting | FIFO or acknowledgement means one business effect | C40 C47 C90 F02 F08 F11 | | TP03 | Retry and timeout | Recover transient faults without storms or duplicate effects | Fits classified transient failures; poor fit for validation conflicts and ambiguous money movement | Deadline timeout owner maximum attempts and age stop at lookup or reconciliation | Producer rejection handler timeout and provider status lookup | Full jitter one owned layer and explicit pre-invocation versus invoked-handler paths | Variable extra attempts latency and capacity; fixed policy testing and incident ownership | Attempts by class deadline exhaustion oldest age and ambiguous queue | I retry only safe transient work; ambiguity changes the operation to lookup | Exchange timeout after possible fill | Retry until success or retry at every layer | C45 C65 C76 F03 F11 F12 A26 A98 | | TP04 | Overload containment | Protect authority and fairness when demand exceeds safe commit rate | Fits burst dependency slowdown and replay; poor fit as a correction for wrong data | Bulkhead cap breaker shedding admission queue and fairness each stop at load not correctness | Live commands share DynamoDB and Lambda with projection and replay | Shed optional work cap replay reserve authority and reconcile expired obligations | Variable rejected deferred and idle reserve capacity; fixed SLO policy and capacity tests | Lambda throttle oldest age wait and rejection by tenant key priority | Autoscaling may add supply; admission and bounded debt decide safety | Reserve ledger lane while throttling search rebuild | Conflating autoscaling with backpressure or circuit breaking with correctness | C44 C61 C104 F03 F13 F14 A117 | | TP05 | Quarantine and replay | Preserve failed evidence and reprocess only after readiness | Fits retained replayable inputs; poor fit after payload expiry or irreversible effect | Each EventBridge target DLQ archive SQS Lambda Kinesis Streams SNS and Firehose boundary has distinct payload route and order | Poison projection event and failed notification | Permissions retention canary rate idempotency delete acknowledgement and post-validation govern replay | Variable retained bytes and replay work; fixed tooling permissions and game days | Failure destination success age replay rate and business control totals | A target DLQ holds one target failure; archive replay can re-match source-bus rules; neither alone proves repair | Kinesis failed invocation retained in S3 and replayed through isolated lane | All failures go to the DLQ then redrive at full speed | C43 C57 C62 C63 C64 C66 C76 C103 A81 A84 A86 A98 A118 | | TP06 | Ordering repair | Preserve per-invariant causality under duplicates concurrency and backfill | Fits versioned aggregates; poor fit for fabricated global arrival order | Source version single writer expected-version apply and gap policy stop at named scope | Order aggregate ledger posting set and portfolio key | Buffer gaps reject stale isolate replay and reconcile deterministic merge | Variable serialization wait and buffers; fixed sequence ownership conflict tooling | Gap age stale duplicate conflict rate and watermark | I pay serialization only where an invariant needs it and reconcile wider views | Account-instrument portfolio versions from execution authority | FIFO removes idempotency or optimistic lock deduplicates events | C34 C40 C41 C72 C74 C84 C93 F01 F02 | | TP07 | Financial reconciliation | Detect and repair silent divergence using independent authoritative evidence | Fits money quantity and regulated notices; poor fit when a projection is treated as authority | Manifest window watermark exact units dual control and proof of closure | Orders executions postings reservations provider statements and projections | Break lifecycle contains then forward-completes reverses or corrects conditionally | Variable comparison queries evidence retention and manual cases; fixed control ownership and SLA | Imbalance missing posting duplicate fill reservation mismatch and break age | Availability is not correctness; every accepted obligation must reconcile to authority | Bank minor-unit statement versus cash ledger and reservations | Derived portfolio authorizes a money correction | C46 C47 C48 C91 F08 F14 F15 F16 F17 | | TP08 | Disaster recovery | Restore one correct system rather than two conflicting writers | Fits Region loss and corruption recovery; poor fit as an untested multi-Region badge | Business RTO/RPO objectives selected mechanisms measured restoration duration and recovery-point age/loss backup replay fence routing external effects and reconciliation stop at declared tier | Command ledger outbox projections and notifications recover in dependency order | Fence old writer route through Route 53 and ARC data plane probe stale clients restore canary replay reconcile failover and separately failback | Variable replica backup replay and idle capacity; fixed IaC runbooks and game days | Restoration clock recovery-point age/loss replication lag DNS TTL caches connections writer epoch and control totals | Multi-Region shortens some recovery paths but never proves zero loss or globally atomic effects | Ledger Region failover reconciled with exchange custodian and bank | Treating MREC or PITR objectives as measured lossless recovery or assuming Route 53 instantly removes old clients | C88 C89 C105 C106 A106 A107 A119 A120 A121 F14 F15 | ## Sources used in this chapter Stable concepts: [Kleppmann and Riccomini](https://www.oreilly.com/library/view/designing-data-intensive-applications/9781098119058/) (F01), [Enterprise Integration Patterns](https://www.enterpriseintegrationpatterns.com/) (F02), [Nygard, *Release It!*](https://pragprog.com/titles/mnee2/release-it-second-edition/) (F03), [Fowler accounting patterns](https://martinfowler.com/eaaDev/AccountingNarrative.html) (F08), [Featonby on idempotent APIs](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) (F11), [Brooker on timeouts/retries/jitter](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/) (F12), [Yanacek on queue backlogs](https://aws.amazon.com/builders-library/avoiding-insurmountable-queue-backlogs/) (F13), [Google SRE](https://sre.google/sre-book/table-of-contents/) (F14), [SRE Workbook](https://sre.google/workbook/table-of-contents/) (F15), [SLO chapter](https://sre.google/sre-book/service-level-objectives/) (F16), and the [FIXimate FIX.Latest top-level specification](https://fiximate.fixtrading.org/en/FIX.Latest/fix.html) (F18, retrieved 2026-08-24) with Harris's trading-domain anchor (F17). Current AWS behavior, all retrieved 2026-08-22: [Lambda with SQS](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html) (A02), [Lambda with Kinesis](https://docs.aws.amazon.com/lambda/latest/dg/with-kinesis.html) (A03), [Step Functions error handling](https://docs.aws.amazon.com/step-functions/latest/dg/concepts-error-handling.html) (A26), [CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) (A36), [EventBridge target DLQs](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rule-dlq.html) (A81), [EventBridge archive/replay](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-archive.html) (A84), [SQS visibility](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html) (A85), [SNS subscription DLQs](https://docs.aws.amazon.com/sns/latest/dg/sns-dead-letter-queues.html) (A86), [Lambda/Kinesis parameters and failures](https://docs.aws.amazon.com/lambda/latest/dg/services-kinesis-parameters.html) (A98), [DynamoDB backup/export](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Backup-and-Restore.html) (A106), [DynamoDB Global Tables](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-global-table-design.html) (A107), [Lambda SQS scaling](https://docs.aws.amazon.com/lambda/latest/dg/services-sqs-scaling.html) (A117), [SQS redrive](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-dead-letter-queue-redrive.html) (A118), [AWS Well-Architected DR](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/plan-for-disaster-recovery-dr.html) (A119), [Route 53 DNS failover best practices](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/best-practices-dns.html) (A120), and [ARC routing-control best practices](https://docs.aws.amazon.com/r53recovery/latest/dg/route53-arc-best-practices.regional.html) (A121).