# Kinesis Data Streams deep dive: order, lag, and recovery by design

Amazon Kinesis Data Streams is a retained partitioned log. It is valuable when
several independently positioned consumers need high-rate ingestion, bounded
replay, and an ordering scope. It is not a work queue, a global sequencer, a
matching engine, an accounting ledger, or an end-to-end exactly-once system.

This chapter uses the guarantee planes from
[Event-Driven Foundations](02-event-driven-foundations.md) and the selection
axes from the [AWS decision guide](03-aws-messaging-decision-guide.md). AWS
semantics, quotas, modes, metric names, and prices are mutable; every such use
below includes `retrieved 2026-08-22`. `Inference:` labels an architectural
deduction rather than a vendor guarantee.

## Mental model: a retained partitioned log

```text
producer record
  {partitionKey, data, eventId, aggregateVersion}
                  |
                  v  hash(partitionKey)
stream -> shard A: 101, 104, 109 ... -> consumer checkpoint A
       -> shard B: 102, 103, 108 ... -> consumer checkpoint B
       -> shard C: 105, 106, 107 ... -> consumer checkpoint C
                  ^ retained records remain until retention expiry
```

| Concept | Working definition | Boundary that matters |
|---|---|---|
| **Stream** | The named retained collection of shards. Independent consumer applications can maintain independent progress. | It is a transport log, not automatically authoritative domain history or a ledger. |
| **Record** | Data plus a required partition key and service metadata such as sequence number and approximate arrival time. | The default maximum is 1 MiB; a supported stream can be configured up to 10 MiB only for intermittent large records. Sustained shard baselines remain unchanged (C71; A16,A95, retrieved 2026-08-22). |
| **Partition key** | A producer-chosen Unicode string hashed into a shard's hash-key range. Equal keys map to the same current shard. | It defines routing and often an intended ordering scope; it does not create an aggregate version or cross-key invariant (C10; A14,A96, retrieved 2026-08-22). |
| **Shard** | A hash-range and the unit of provisioned write/read capacity and consumer parallelism. | Records are sequenced within a shard. There is no free global order across shards (C10,C34; A14, retrieved 2026-08-22). |
| **Sequence number** | Kinesis-assigned record identifier/order within the stream/shard behavior. | **Inference:** it is not a portable business version and cannot be used as a stream-wide query index. Put retries can create a second record with another sequence number (C72; A96, retrieved 2026-08-22). |
| **Retention** | Time for which records remain readable: 24 hours by default and configurable up to 8,760 hours/365 days. | Retention is an opportunity to replay, not evidence that detection plus deployment plus drain will finish in time (C14; A15, retrieved 2026-08-22). |
| **Consumer** | Lambda mapping, KCL application, Flink application, Firehose, or custom reader that reads records and owns downstream effects. | Each consumer's position, retry, state, and sink contract are separate. One consumer succeeding says nothing about another. |
| **Checkpoint** | A durable consumer position after records the application considers processed. | **Inference:** a checkpoint is not a transaction over DynamoDB, an HTTP API, or a customer notification. Crash after an effect but before checkpoint can repeat the effect (C74; A97, retrieved 2026-08-22). |
| **Replay** | Starting a consumer/version at `TRIM_HORIZON`, `AT_TIMESTAMP`, or another retained position and reprocessing a bounded range. | Replay repeats delivery; it needs a target version, rate, idempotency/side-effect policy, completion watermark, and reconciliation. |

### Ordering has four layers

1. The domain authority assigns `aggregateVersion = 42` after `41`.
2. Kinesis routes a partition key to a shard and assigns sequence numbers.
3. The consumer scheduler chooses batch/invocation concurrency.
4. The database or external system decides commit/effect order.

**Inference:** only the first layer tells a projector whether `42` is next, stale, duplicate,
or a gap. Kinesis shard order can transport that relation efficiently, but it
does not invent it. Ordering also does not remove duplicate delivery (C41).

**Inference:** an execution projector should store stable `executionId` and the
expected authoritative sequence in the same transaction as its state mutation.
Kinesis sequence numbers remain transport/checkpoint metadata; reconciliation
compares venue executions, ledger postings, and projections.

### Replay is not redrive and not reconciliation

A replay plan states:

- source range by stream/shard and timestamp or checkpoint;
- immutable event identity and schema/upcaster version;
- isolated target or projection version;
- live-versus-replay capacity reservation and maximum rate;
- suppression or idempotent repetition of email, venue, payment, and other
  non-rebuild effects;
- completion watermark, source/output counts, version gaps, exact control
  totals, and discrepancy sign-off.

Empty iterator lag means a consumer caught up to its position. It does not prove
one fill, balanced cash/securities postings, or correct holdings.

## Partition keys: put the invariant before the hash

The key is a correctness-and-capacity decision. Start with the relation that
must be serialized, then measure its distribution. High cardinality alone does
not prevent a popular symbol, institutional account, or large tenant from
dominating one shard.

| Candidate key | Preserved relation | Parallelism | Skew/failure risk | Defensible use |
|---|---|---|---|---|
| `accountId` | Account-scoped execution/projection order | Across accounts | One institutional account can be hot; cross-account transfer still needs coordination | Account portfolio/ledger projection when the authority emits an account version |
| `symbol` / book ID | Instrument/book sequence | Across instruments | Hot symbols at open/news; all clients for one symbol share a lane | Market-data normalization or journal distribution; not a substitute for the matching core |
| `tenantId` | Tenant-level order/isolation | Across tenants | Large tenant monopolizes a shard; quiet tenants sharing it can suffer | Tenant export/compliance lanes only when tenant order is required |
| `orderId` / aggregate ID | One order's state transitions | Across orders | Excellent distribution but cannot serialize account cash/securities invariants | Order lifecycle projection with authoritative order version |
| `accountId#symbol` | Position order for one account/instrument | Across positions | More parallelism, but cash or whole-account risk spans keys | Derived position updates when broader invariants stay authoritative elsewhere |
| salted `symbol#bucket` | No simple whole-symbol order; requires merge sequence | High | Merge/gap logic and late data become application responsibilities | Replaceable analytics where source sequence enables deterministic merge |

### The irreducible trade-off

**Inference:** a single hot key cannot be spread by adding shards because one partition key
maps to one current shard. Provisioned selective splitting can isolate a hot
hash range from neighbors, but it cannot split that key. On-demand also retains
the single-key shard limit (C75; A95,A97, retrieved 2026-08-22).

**Inference:** if one symbol must sustain more than a shard while preserving
price-time order, Kinesis is not the sequencing mechanism for that hot book.
Use a capacity-controlled single-writer matching/journal partition, then publish
ordered results to Kinesis for independent downstream consumers. If the data is
only analytics, salt the key and merge by a source sequence.

### Detect skew before throttling becomes aggregate lag

**Inference:** track producer-side records/bytes by business key and returned `ShardId`, then
compare hottest shard with mean/p95. Enable enhanced shard-level CloudWatch
metrics during diagnosis; stream averages can hide one hot shard (C77;
A99, retrieved 2026-08-22).

```text
k_skew = hottest_shard_rate / average_shard_rate
key_share = key_rate / total_rate
```

Alarm on `WriteProvisionedThroughputExceeded`, but diagnose it with per-shard
incoming bytes/records, failed `PutRecords` entry shard IDs, key distribution,
and application admission metrics. Remedies differ:

- aggregate shortage: add provisioned shards or prepare on-demand warm capacity;
- collision of many busy keys: selectively split/rebalance hash ranges;
- one irreducible key: change serialization scope/substrate or admit/shed work;
- producer retry storm: bound retry budget and add full jitter;
- downstream backlog: increase *measured commit* capacity without moving the
  bottleneck to a database or external service.

## Producer design: acknowledgement is not one effect

### `PutRecord` versus `PutRecords`

| Concern | `PutRecord` | `PutRecords` |
|---|---|---|
| Unit | One Kinesis record | Up to 500 entries, within current request-size constraints |
| Result | Shard ID and sequence number or request error | HTTP success can contain entry successes and failures; response entries correspond to request order |
| Order | **Inference:** only one client submitting same-key records serially can chain order: feed record n-1's successful returned sequence into record n's `SequenceNumberForOrdering`; an ambiguous response breaks this chain (C72; A96, retrieved 2026-08-22) | Later entries are attempted after an earlier failure; request order is not guaranteed |
| Efficiency | More calls and simpler per-record acknowledgement | Better request amortization; application must correlate and retry only failures |
| Use | When same-client/same-key put order is load-bearing and latency/call cost is acceptable | Default high-throughput path when domain versions make consumer order explicit |

**Inference:** these API semantics require entry-level acknowledgement handling
(C16,C72; A17,A96, retrieved 2026-08-22).

`PutRecords` safe loop:

1. assign `eventId`, authority, aggregate/version, schema, occurred time, and
   partition key before the first attempt;
2. retain the original request-entry index and identity;
3. inspect `FailedRecordCount` and every result entry;
4. remove acknowledged successes from the retry set;
5. retry retriable failures with exponential backoff, full jitter, a deadline,
   and admission control; do not regenerate event identity;
6. durably record/quarantine terminal producer failures and reconcile accepted
   domain facts to published facts.

**Inference:** a timeout can be ambiguous, so retry may append a second Kinesis
record and cannot safely supply the missing successful sequence to the next
same-key put; stop/reconcile the chain from the authoritative domain version.
The consumer must make the *business effect* idempotent on stable
`eventId`/`executionId`; Kinesis sequence number is not that idempotency key
(C72; A17,A96, retrieved 2026-08-22).

### Batching versus aggregation

- **Collection/batching** sends multiple Kinesis records in one `PutRecords`
  request.
- **KPL aggregation** packs several user records into one Kinesis record, which
  can improve tiny-record throughput and provisioned PUT-unit packing.
- **Inference:** KPL buffering and asynchronous completion mean packing efficiency adds
  producer latency and memory/flush behavior (C73; A96, retrieved 2026-08-22).

**Inference:** enable aggregation only after every Lambda/KCL/Flink/Firehose or
custom consumer proves compatible deaggregation, record identity, metrics, and
replay behavior. With Lambda `ParallelizationFactor`, non-EFO aggregated inner
events need matching partition keys for order; with EFO, Lambda can drop inner
events whose partition keys do not correspond, without processing or failure
destination (C73; A03,A96,A98, retrieved 2026-08-22). Treat this as a release-blocking
contract test, not a tuning footnote.

### Record size and payload pointers

The safe sentence is: **default maximum 1 MiB; configurable up to 10 MiB for
intermittent large records in supported Regions; sustained per-shard limits do
not increase** (C71; A16,A95, retrieved 2026-08-22). Lambda's synchronous event
payload is 6 MiB including base64 and metadata, so stream acceptance does not
prove that Lambda can invoke the handler (C71,C80; A95, retrieved 2026-08-22).

| Full-path component | Current large-record compatibility |
|---|---|
| Stream | Default maximum remains 1 MiB; opt in per supported stream up to 10 MiB for intermittent records. Sustained shard baselines do not increase (C71; A95, retrieved 2026-08-22). |
| KCL | KCL 2.x supports large records after the stream maximum is updated (C80; A95, retrieved 2026-08-22). |
| KPL | KPL 1.0.5 and later supports large records after the stream maximum is updated (C80; A95, retrieved 2026-08-22). |
| Lambda ESM | The 6 MiB limit includes base64 expansion and ESM metadata. An oversized record goes through the configured on-failure destination; configure and test that failure path before enabling larger records (C80; A95, retrieved 2026-08-22). |
| Data Firehose | S3 delivery supports large records, including format conversion, but dynamic partitioning is not supported. HTTP, Snowflake, and Redshift destinations are not supported for large records; Firehose-to-Lambda is not recommended for this path (C80; A95, retrieved 2026-08-22). |
| Flink Kinesis connector | The source supports records up to 10 MiB; do not use the sink for records above 1 MiB (C80; A95, retrieved 2026-08-22). |

**Inference:** require full-path compatibility across producer, KCL/KPL version,
Lambda envelope and failure destination, Firehose destination/features, Flink
source/sink, archive, and replay tooling before raising the stream maximum. If
one load-bearing path is incompatible, keep 1 MiB and use a payload pointer
(C80; A95, retrieved 2026-08-22).

**Inference:** for sustained large payloads, put immutable encrypted content in
S3 and stream a versioned pointer, checksum, size, and business identity. Then
operate S3 retention/access and missing-object reconciliation; a pointer does
not make two resources atomic.

### Explicit hash keys: exceptional, not routine

`ExplicitHashKey` overrides the partition-key hash for routing while the record
still carries its partition key (A96, retrieved 2026-08-22).

**Inference:** if that mapping changes, the
same logical partition key can reach different shards and forfeit the normal
same-key order assumption. Permit it only behind one authoritative stable
mapping and an ordered, version-checked cutover that proves the old lane is
drained before the new lane starts; otherwise reject it where logical-key order
is load-bearing (C79; A96, retrieved 2026-08-22). Ordinary producers should not
pin shard IDs or embed topology: resharding changes ranges, independent
producers can disagree, and a manual map can create skew.

## Consumer boundaries: Lambda, KCL, enhanced fan-out, and Flink

| Consumer | Owns for you | You still own | Choose when | Poor fit |
|---|---|---|---|---|
| Lambda event-source mapping | Poll/push connection, batching, shard progress, retries, partial/bisect options, invocation scaling | Idempotent effects, versions/gaps, poison policy, destination permissions, downstream capacity, replay/reconciliation | Stateless bounded per-record/batch work with native scaling | Long-lived state, tight deterministic latency, large payloads, or sink transaction that cannot tolerate repeats |
| KCL application | Shard leases/load balancing, worker failure, checkpoints, reshard discovery, parent-before-child handling; at-least-once delivery | Worker runtime/scale, checkpoint timing, state/sink transaction, deployment, poison/replay | Long-lived custom consumer with connection/state control | Team does not need that runtime/control or cannot operate it |
| Enhanced fan-out | Registered consumer with dedicated up-to-2 MB/s/shard push pipe | Same processing/checkpoint/effect duties plus registration lifecycle and price | Multiple consumers contend, low propagation delay or catch-up isolation matters | One light consumer with ample shared read capacity |
| Managed Flink | Managed Flink runtime for keyed/event-time state and checkpoints | Operator/state design, watermarks, late data, savepoints, source/sink guarantee, cost and recovery | Stateful windows, joins, temporal risk/fraud analytics | Stateless routing, simple projection write, low-latency matching |

**Inference:** KCL's documented behavior makes it the managed-library boundary for shard balancing, checkpointing,
at-least-once delivery, and parent-before-child processing across resharding
(C74; A97, retrieved 2026-08-22). Enhanced fan-out provides each registered
consumer up to 2 MB/s/shard dedicated read throughput; current registration
limits are 20 for Provisioned/On-demand Standard and 50 for On-demand Advantage
where supported (C17,C70; A16,A18,A95, retrieved 2026-08-22).

**Inference:** EFO isolates Kinesis read throughput, not Lambda concurrency,
DynamoDB capacity, vendor quotas, or a hot producer key. Pay for it when measured
contention/latency/recovery justifies it, and reserve downstream catch-up
capacity.

**Inference:** Managed Flink checkpoints can restore compatible source position and operator
state, but an arbitrary REST API
or database side effect is outside that checkpoint transaction. Use a compatible
sink/transactional design or stable identity plus idempotency and reconciliation
(C67; A31,A89, retrieved 2026-08-22).

## Lambda failure semantics and poison records

Lambda invokes synchronously from its event-source mapping and processes
records at least once. By default it checkpoints a Kinesis batch only after
complete success; a failed batch is retried and can block later work on the
affected shard (C07,C11; A03, retrieved 2026-08-22).

**Inference:** current failure ownership must be split before selecting retry
controls. **Before invocation**, throttling or another system inability to
invoke the function is retried until source retention expires or the record
exceeds `MaximumRecordAgeInSeconds`; `MaximumRetryAttempts` does not bound this
path and no handler attempt occurred. **During invocation**, a handler error is
retried until source expiry, maximum record age, or `MaximumRetryAttempts` is
reached. Bisection is only for function errors, and splitting a batch does not
consume retry quota (C76; A98, retrieved 2026-08-22).

### Control-by-control

| Control | Current behavior | Design consequence |
|---|---|---|
| Batch size/window | Batch default 100, maximum 10,000; Kinesis window defaults 0 and is configurable 0–300 seconds; invocation stops at batch/window/6 MB payload | Larger batches amortize calls but add latency, retry blast radius, memory, and downstream burst (A03,A98, retrieved 2026-08-22). |
| `ParallelizationFactor` | 1–10 concurrent batches per shard; Lambda preserves partition-key order | **Inference:** capacity can exceed shard count, but same-key serialization remains and aggregation rules are strict (C73,C76; A03,A98, retrieved 2026-08-22). |
| Partial response | With `ReportBatchItemFailures`, the lowest returned failed sequence becomes the checkpoint and Lambda retries from it | **Inference:** successful later records can repeat; this reduces work, not duplicates (C12,C76; A98, retrieved 2026-08-22). |
| Bisect | For a function error, split a failed batch; with partial reporting, bisect at the returned sequence; splits do not consume retry quota | **Inference:** this finds a handler-poison region but does not apply to a pre-invocation failure and is not quarantine or transaction (C13,C76; A03,A98, retrieved 2026-08-22). |
| Maximum record age | `MaximumRecordAgeInSeconds` is `-1` (default) or 60–604,800; source retention still applies | **Inference:** age bounds both pre-invocation and invoked-error retry, so it must fit the business deadline while preserving repair time and retained payload (C76; A98, retrieved 2026-08-22). |
| Maximum retries | `MaximumRetryAttempts` is `-1` (default) through 10,000 and applies to handler/function errors, not pre-invocation inability | **Inference:** a finite quota bounds handler poison only; pre-invocation failure still needs age/retention alarms and repair (C76; A98, retrieved 2026-08-22). |
| On-failure destination | SQS/SNS receive failure metadata; S3 receives metadata plus the complete invocation record | **Inference:** S3 is normally the safer replay source for discarded Kinesis payloads; permissions, encryption, lifecycle, alarms, and object validation remain required (C76; A98, retrieved 2026-08-22). |

### Two runbooks, then a poison-record policy

1. **Inference:** **Unable to invoke:** alert on rising iterator age plus
   `FailedInvokeEventCount`/throttles and corroborate mapping state,
   permissions, concurrency, network/configuration, and whether
   `InvokedEventCount` advances. Restore invocation before age/retention expiry;
   do not call these handler attempts or expect `MaximumRetryAttempts` to stop
   them (C76,C77; A98, retrieved 2026-08-22).
2. **Inference:** **Invoked handler poison/failed effect:** use function errors, partial-failure
   identities, repeated event IDs, iterator age, and `DroppedEventCount`; repair
   code/data or quarantine according to the business order policy. Separately
   alarm `DestinationDeliveryFailures` and reconcile it against expected
   `OnFailureDestinationDeliveredEventCount` so destination failure is not
   confused with handler failure (C76,C77; A98, retrieved 2026-08-22).

Handler-poison workflow:

1. Validate envelope/schema and stable identity before side effects.
2. Classify transient dependency failure separately from deterministic poison.
3. Retry transient failures within a time/attempt budget with idempotent effects.
4. Use partial response and bisection to minimize repeated good work.
5. Send exhausted/aged batches to S3 so the original invocation payload is
   recoverable; alarm destination delivery failure separately.
6. Record the failed shard, sequence range, event IDs, schema, code version,
   exception class, attempt/age, and owner in an operator case.
7. Fix or deploy a versioned projector, replay to an isolated target at a paced
   rate, and reconcile every event/effect before closing.

**Inference:** skipping a poison record restores liveness by weakening strict
lane order. The business policy must decide whether later records may proceed,
whether the account/book is quarantined, or whether the projection is rebuilt.
A DLQ/destination is a holding mechanism, not that decision (C43,C76;
A98, retrieved 2026-08-22).

### Why Kinesis plus Lambda is not end-to-end exactly once

The event-source mapping is at least once and successful records can repeat
during failed-batch/partial-checkpoint recovery (C11-C12; A03,A98, retrieved
2026-08-22). A handler can commit DynamoDB and crash before the mapping advances;
an external API can time out after accepting the request; replay intentionally
redelivers history.

**Inference:** Kinesis/Lambda cannot promise one end-to-end business effect.
Transact an inbox identity with the protected DynamoDB mutation where possible,
use an external provider idempotency key/receipt when available, validate
aggregate versions, and reconcile authoritative outcomes (C40-C41; F11).

## Capacity modes, resharding, and backlog recovery

### Current mode boundaries

| Mode | Current behavior | Best fit | Primary risk/cost shape |
|---|---|---|---|
| Provisioned | You choose shards; capacity sums shard limits; selective split/merge and shard count changes are your responsibility | Predictable traffic, controlled headroom, selective hot-range work | Shard hours + 25 KiB PUT units + optional EFO/retention; operational scaling |
| On-demand Standard | Starts with documented 4 MB/s write and 8 MB/s read capacity; auto-manages shards; rapid growth and skew can throttle | Unknown/variable stream traffic without capacity planning | Per-stream hour + 1 KiB-rounded Data-In + Data-Out + EFO/retention |
| On-demand Advantage | Account-level regional mode, warm throughput, up to 50 EFO consumers where supported, different discounted data price shape | Aggregate on-demand estate that earns the commitment and warm/EFO features | At least 25 MiB/s ingest and 25 MiB/s retrieval usage/shortfall across on-demand streams |

The mode facts and `eu-west-1` support were retrieved 2026-08-22 (C70; A16,A95,
retrieved 2026-08-22). On-demand can see read/write exceptions for roughly 15
minutes if traffic exceeds twice the prior peak, and one partition key remains
bounded by one shard. Advantage warm throughput is the pre-event control; retry
with backoff/jitter and producer admission is still required (A95, retrieved
2026-08-22).

### Provisioned sizing with skew and headroom

```text
byte_shards   = ceil(peak_bytes_per_second / 1,000,000)
record_shards = ceil(peak_records_per_second / 1,000)
planned       = ceil(max(unrounded byte_shards, unrounded record_shards)
                     * skew_factor / target_utilization)
```

For `8,000 records/s`, `700 B`, measured `1.25` hottest/average skew, and
`80%` target utilization:

```text
bytes baseline   = ceil(5.6 MB/s / 1 MB/s) = 6
records baseline = ceil(8,000 / 1,000) = 8
planned shards   = ceil(8 * 1.25 / 0.80) = 13
```

Current per-shard values are mutable and were retrieved 2026-08-22 (C15; A16).
The complete unit model and sensitivity calculations are in
[the cost model](../research/cost-model.md).

### Reshard without losing parent history

**Inference:** provisioned resharding routes new writes to child shards while records already
in parent shards remain readable until retention expiry. Reading child shards
before draining/checkpointing parents can violate per-key order; KCL handles the
parent-before-child relationship (C74-C75; A97, retrieved 2026-08-22).

Runbook:

1. classify aggregate shortage, collision, or one-key overload;
2. calculate target capacity and downstream/replay headroom;
3. split the hot hash range or update shard count;
4. verify stream active, producer throttle falling, and child discovery;
5. confirm every consumer drains/checkpoints parents before children;
6. validate per-key versions/gaps and business control totals;
7. merge only after a sustained cold period and another capacity/recovery check.

### Backlog growth and drain

```text
growth = max(0, arrival_rate - commit_rate)
backlog = initial_backlog + growth * duration
drain = backlog / (recovery_commit_rate - live_arrival_rate)
```

If recovery rate is not greater than live arrival, drain time is infinite. A
15-minute outage at `5,000 records/s` creates `4.5M` records or `3.15 GB` raw at
700 B. At `9,000 commits/s` while live remains `5,000/s`, ideal drain is
`1,125 s = 18.75 min`; recovery finishes 33.75 minutes after the outage began,
before detection/deployment/retry margin. If commit falls to `6,000/s`, drain is
75 minutes. Do not report the first figure without that sensitivity.

**Inference:** provision live and replay lanes so a backfill cannot starve the
market-session feed. Bound replay concurrency/rate at both the consumer and
downstream store, monitor age per shard/key/tenant, and stop before retention or
financial freshness budgets are threatened (C44,C61).

## Observability and game-day drills

### Signals by owned boundary

| Boundary | Minimum signals | Interpretation/trap |
|---|---|---|
| Producer | `PutRecord.Success`/`PutRecords.Success`, `FailedRecordCount`, error code, retry age/attempt, `IncomingBytes/Records`, `WriteProvisionedThroughputExceeded` | HTTP success is not all-entry success; retries without stable IDs can append duplicates |
| Partition/skew | optional shard-level incoming bytes/records and write throttles; key/share and returned shard ID | Stream average can hide one hot shard; cardinality is not distribution |
| Standard reader | `GetRecords.Bytes/Records`, `ReadProvisionedThroughputExceeded`, `GetRecords.IteratorAgeMilliseconds` | Age rising with no read throttles points to compute/sink/poison; age alone does not locate business gaps |
| EFO | `SubscribeToShardEvent.*`, registered consumer state/count and per-consumer lag/throughput | Registration/delivery health does not prove consumer commit |
| Lambda mapping | `IteratorAge`, function `Errors`, `Throttles`, duration/concurrency, opt-in `PolledEventCount`, `FailedInvokeEventCount`, `DroppedEventCount`, `OnFailureDestinationDeliveredEventCount`, `DestinationDeliveryFailures` | A destination success means payload/metadata was retained, not repaired |
| Business | duplicate IDs, stale/gap sequence, unposted execution, ledger imbalance, projection freshness, unreconciled discrepancy age | This is the only plane that detects correct-looking transport with wrong money/holdings |

**Inference:** metric names/enablement are current mutable inputs to the operational design (C77; A98,A99, retrieved
2026-08-22). Alarm `Maximum` iterator age against a consumer-specific lag and
retention budget; use optional shard metrics for diagnosis, then control their
CloudWatch cost.

### Game-day drills

1. **Hot key:** drive one symbol above its lane capacity. Expect shard-specific
   throttles, identify the key, exercise admission/scope/substrate decision, and
   prove unrelated keys' lag stays within budget.
2. **Consumer outage/backlog:** pause one consumer for 15 minutes. Predict
   backlog/drain first; resume with paced capacity, protect live traffic, and
   compare predicted to actual iterator age.
3. **Poison record:** inject a valid envelope with deterministic handler failure.
   Verify partial/bisect behavior, bounded attempts/age, full payload in S3,
   destination alarms, controlled replay, and business reconciliation.
4. **Partial producer failure:** fault/throttle selected `PutRecords` entries.
   Verify only failed identities retry, successful entries do not get blindly
   republished, and consumer inboxes suppress ambiguous duplicates.
5. **Reshard:** split a hot range while traffic continues. Verify parent drain
   before child processing, KCL/Lambda shard discovery, versions with no gaps,
   and planned recovery capacity.
6. **Inference:** **Aggregation compatibility:** publish mixed inner partition keys in a test
   stream with and without EFO. The release gate must detect the documented
   Lambda drop/order hazard; never run this destructive experiment on business
   traffic (C73; A03, retrieved 2026-08-22).
7. **Retention edge:** restore a consumer near retention expiry. Verify alarms
   fire early enough, the S3/authoritative backfill route works, and every
   missing interval is classified.

Success criteria are version/count/control-total reconciliation, not merely
zero throttles or an empty destination.

## Fintech applications and hard non-fits

### Market data

Partition a normalized feed by the ordering scope the consumer actually needs:
symbol/book for per-instrument sequence, or a source channel if upstream
sequence is channel-scoped. Detect gaps against the venue sequence, isolate hot
symbols, and use Flink only when event-time windows/joins/state justify it.

**Inference:** Kinesis is good downstream distribution and bounded replay, but
not the low-latency price-time matching loop. A hot book needs a measured
single-writer execution substrate; Kinesis carries resulting market/execution
facts to projections.

### Executions

The matcher or venue adapter assigns stable `executionId`, order/account/book
versions, exact price/quantity/currency/scale, and correction semantics before
publication. Ledger, portfolio, notification, and compliance consumers maintain
separate positions and idempotent effects.

Repository mapping: replacing EventBridge with Kinesis would give the portfolio
a retained ordered transport only after choosing a key. It does not repair the
missing inbox or source version in
[CS06](01-repository-case-study.md#cs06--optimistic-locking-does-not-make-the-portfolio-consumer-idempotent)
and [CS07](01-repository-case-study.md#cs07--the-portfolio-has-an-ordering-requirement-but-no-event-sequence),
nor the trade/outbox gap in
[CS04](01-repository-case-study.md#cs04--trade-creation-order-transition-and-publication-are-separate-operations).

### Risk and compliance fan-out

Use independent consumers because real-time risk, surveillance, search, and
portfolio projections have different state, lag, deployment, and replay needs.
EFO is justified by measured shared-read contention or latency, not the number
of team names. Stateful event-time risk belongs in Flink only when windows,
watermarks, joins, and late-data policy are explicit.

Hard pre-trade risk and reservations stay on the authoritative command path. A
lagging Kinesis projection cannot authorize cash, securities, or exposure that
must be current (C46).

### Audit archival and backfill

**Inference:** Kinesis can feed Amazon Data Firehose to S3 for buffered landing; source-to-object
manifests, counts/hashes, encryption, retention, and object access govern audit
evidence. Firehose is not the shared replay authority (C31,C66; A29,A88,
retrieved 2026-08-22). Keep an authoritative execution/ledger source and use S3
for recovery beyond stream retention.

Backfill into an isolated projection version or dedicated stream, preserve
original identities/occurred times/source versions, pace against downstream
capacity, suppress notifications/external actions, and cut over after
reconciliation. Do not pour years of S3 data into the live stream without a lag
and cost budget.

### Explicit hard non-fits

- **Matching engine:** Kinesis does not implement an order book, price-time
  priority, deterministic decision loop, or venue fairness. See
  [CS12](01-repository-case-study.md#cs12--the-lambda-simulation-is-not-a-default-latency-critical-matcher).
- **Global sequencer:** shards provide parallel order scopes. Global
  serialization is possible only through a common bottleneck/coordinator, not a
  free Kinesis setting (C34).
- **Authoritative ledger:** retention, sequence numbers, and replay do not create
  balanced postings, reversals, access controls, or reconciliation (C47).
- **One-owner work queue:** if each item belongs to one worker and visibility,
  acknowledgement, and DLQ/redrive are the desired semantics, SQS is clearer.

## Cost and neighboring alternatives

Current `eu-west-1` cost math and machine-readable price inputs are in
[the cost model](../research/cost-model.md). Keep formulas portable and inputs
dated (C49,C78; A46,A99, retrieved 2026-08-22).

For the printed example (`5,000 records/s`, 700 B, two full consumers, one-day
retention), on-demand Standard rounds each record to 1 KiB for Data-In and
models about `$1.9k/month`; 13 provisioned shards with two EFO consumers model
about `$0.94k/month`. Both exclude consumer compute/sinks, storage, logs,
transfer, KMS, replay, support, and engineering. Report those as rough scenario
results, never a universal break-even.

| Need/cost driver | Kinesis | SQS | EventBridge | MSK | Data Firehose |
|---|---|---|---|---|---|
| Independent positions/replay | Native within retention | Work removed after acknowledgement; redrive, not arbitrary consumer history | Optional archive replays to source bus, not partitioned positions/order | Native topic partitions/consumer groups | No general consumer positions; destination delivery |
| Ordering | Shard/key scope | FIFO message-group scope when selected | No package ordering claim | Kafka partition scope | Do not build business order on it |
| Fan-out cost | Standard read contention or EFO per consumer; on-demand data-out multiplies | One queue per independent owner plus fan-out layer | Event multiplied by matching targets and downstream | Broker/serverless capacity, storage, network and operations | Bytes/transforms/conversion/dynamic partition/destination |
| Capacity burden | Mode, key skew, shards/warm throughput, retention, recovery | Queue age, request chunks, consumers/downstream | Rule/target quotas, retries/archive, target isolation | Brokers/partitions/storage/rebalances/upgrades or serverless capacity | Buffer/destination limits and upstream replay source |
| Choose | High-rate retained AWS-native log, Lambda/KCL/Flink integrations | One-owner buffered work, Standard or FIFO semantics | Structured content routing/AWS integration | Kafka protocol/ecosystem/portability justifies operations | Managed buffered delivery to S3/search/warehouse |
| Reject | One hot globally ordered matcher lane or no replay need | Several consumers need independent history | Deterministic partition replay/high-rate stream | No Kafka requirement/operating competence | Command bus, shared log, custom stateful processing |

Use workload variables, not blanket service prices: records/s, payload
distribution, peak duration, key skew, consumers, retention, replay, batch/
aggregation, Lambda/Flink work, sinks, logs/metrics, transfer, KMS, RTO/RPO,
correctness loss, and on-call cost.

## Myth and correction table

Local Kinesis/Kafka notes and the reference application's Kinesis documents are
cross-references only. The controlled corrections below supersede them.

| Local shorthand/myth | Correction | Why it matters |
|---|---|---|
| “Each shard is 1 MB/s write, so N shards always handle N MB/s.” | Provisioned capacity is constrained by both bytes and records, key skew, retry headroom, and downstream drain. Large-record burst does not change sustained baselines (C15,C71; A16,A95, retrieved 2026-08-22). | Prevents hot-key loss/throttling hidden by aggregate math. |
| “Maximum record size is 10 MiB.” | Default is 1 MiB; supported streams may opt up to 10 MiB for intermittent records, and consumer limits may be lower (C71; A95, retrieved 2026-08-22). | Avoids deploying payloads Lambda/Firehose/Flink paths cannot process safely. |
| “Same partition key means one-at-a-time portfolio writes; conflicts disappear.” | Same key routes to one shard, but Lambda can use parallelization while preserving key order; retries duplicate attempts, and consumer code/state can still race with other writers. Keep inbox, version, and conditional mutation. | Transport order is not database atomicity or novelty. |
| “`PutRecords` is a batch atomic write in request order.” | **Inference:** entries can partially succeed and later entries continue; retry only failures. Use `PutRecord` plus `SequenceNumberForOrdering` only when same-client/same-key put order truly requires it (C72; A17,A96, retrieved 2026-08-22). | Prevents silent loss and blind duplicate republish. |
| “On-demand absorbs any instant spike.” | It can throttle for roughly 15 minutes beyond twice prior peak; one hot key retains shard limits. Advantage warm throughput is a separate account-level mode/commitment (C70; A95, retrieved 2026-08-22). | Makes market-open/flash-event preparation explicit. |
| “Enhanced fan-out gives every consumer exactly one copy.” | **Inference:** it gives dedicated read throughput, not one processing effect. Lambda/KCL processing is still at least once and effects can repeat (C17,C74,C76; A18,A97,A98, retrieved 2026-08-22). | Throughput isolation is not processing exactly once. |
| “Partial batch response means successful records never repeat.” | **Inference:** Lambda retries from the lowest returned failed sequence; later successes can repeat (C12,C76; A98, retrieved 2026-08-22). | Requires idempotent sinks and conservative checkpoint design. |
| “Bisect sends the bad item to the DLQ.” | **Inference:** bisect only splits failed batches. Terminal destination requires age/retry policy and destination; S3 retains full invocation while SQS/SNS retain metadata (C13,C76; A98, retrieved 2026-08-22). | Makes payload recovery and destination permissions testable. |
| “Kinesis/Lambda is end-to-end exactly once.” | Event-source processing is at least once; crash windows, partial failures, external calls, and replay repeat effects. Use stable identity, transactional inbox/outbox where possible, version checks, and reconciliation. | Prevents double fills/postings/notifications. |
| “Kinesis Data Firehose / Kinesis Data Analytics.” | Current names are **Amazon Data Firehose** and **Amazon Managed Service for Apache Flink** (C30; A29,A31, retrieved 2026-08-22). | Avoids stale interview vocabulary and product conflation. |
| “Kinesis versus MSK is a fixed throughput/latency/dollar cutoff.” | Choose from required protocol/ecosystem, workload/key distribution, retention, consumer topology, recovery, operations, and dated regional unit economics. | Removes stale dollar and absolute-latency folklore (X04-X06). |

## Decision table

| Requirement | Choice | Non-negotiable controls | Do not use when |
|---|---|---|---|
| Bursty AWS-native retained feed, unknown capacity | On-demand Standard; Advantage only after aggregate commitment/warm/EFO analysis | stable IDs, key-skew telemetry, retry/jitter, retention/drain budget | one key exceeds a shard or instant spike is assumed free of throttling |
| Predictable feed with measured keys | Provisioned Kinesis | byte+record+skew sizing, headroom, reshard runbook, parent/child validation | team will not operate capacity and recovery |
| Stateless Lambda projections | Lambda mapping | inbox/effect idempotency, version/gap policy, bounded poison handling, S3 failure payload, iterator-age SLO | long-lived state/tight deterministic loop/oversized payload |
| Custom long-lived consumers | KCL, EFO when isolation earns cost | checkpoint after intended effect, lease/state store operations, reshard/replay drills | Lambda already meets semantics and operations |
| Stateful event-time risk | Managed Flink over Kinesis | watermarks/late data, checkpoint/sink boundary, state growth, savepoint/restart, reconciliation | simple stateless routing or hard pre-trade synchronous invariant |
| Managed S3 audit landing | Firehose fed from a recoverable source | manifests/control totals, error backup, S3 governance, upstream replay | Firehose buffer would be the only copy |
| One-owner account-serial work | SQS FIFO keyed by account, or measured single-writer engine | idempotent effect, group skew/poison/redrive policy | several independent consumers need retained history |
| Low-latency matcher/global order | long-lived capacity-controlled sequencer/matching core | deterministic journal, failover/replay, fairness and reconciliation | do not use Kinesis/Lambda as the matching algorithm |

### Two-minute senior answer

> I use Kinesis when I need a retained partitioned log with independent consumer
> positions. I first choose the ordering key from the invariant, then measure
> skew; one hot key still has one-shard capacity, so adding shards does not solve
> every overload. Producers give every fact stable identity and version, inspect
> every `PutRecords` entry, and retry only failures with jitter. Consumers are
> at least once: Lambda gives simple batching/retry controls, KCL gives explicit
> long-lived checkpoint workers, EFO isolates read throughput, and Flink is for
> stateful event-time logic. None makes an external ledger write exactly once.
> I size bytes and records with skew/headroom, calculate backlog drain against
> live arrival, keep retention margin, and operate throttles, iterator age,
> shard skew, poison destinations, version gaps, and reconciliation together.
> Kinesis distributes execution/market facts; it is not the matcher, global
> sequencer, or financial ledger.

## Interview follow-ups

1. Why is `accountId` safer for one portfolio relation but worse for an
   institutional hot account than `orderId`?
2. What happens when one symbol alone exceeds one shard and strict book order is
   required?
3. Compare `PutRecord`, `PutRecords`, collection, and aggregation. Which can
   reorder or partially fail?
4. A producer times out after a put. Which identity makes retry safe at the
   ledger projector?
5. Why can `ReportBatchItemFailures` still repeat a successful record?
6. How do partial response and bisection interact, and when is S3 a better
   failure destination than SQS/SNS?
7. What does `ParallelizationFactor = 5` preserve, and what aggregation
   partition-key hazard must be tested?
8. When does EFO solve the bottleneck, and when does it only move it to
   DynamoDB or an API?
9. How does KCL preserve per-key order across a split, and what must a custom
   consumer implement?
10. On-demand traffic jumps to three times its prior peak at market open. What
    do you expect, and how does Advantage warm throughput change preparation?
11. Calculate shards for peak records, bytes, `k_skew`, and target utilization;
    which constraint binds?
12. If live arrival is 5k/s and recovery commit is 5k/s, when does a 4.5M
    record backlog drain?
13. Which signals distinguish producer throttling, one hot shard, Lambda poison
    retry, EFO subscription failure, and a slow sink?
14. How do you replay six months of executions without sending duplicate client
    notifications?
15. Why are Kinesis stream retention and an S3 audit landing not automatically
    a regulated accounting ledger?
16. Give a workload where SQS FIFO, EventBridge, MSK, or Firehose is more
    defensible than Kinesis.

## Sources used in this chapter

Claim wording and freshness are controlled in the
[claim register](../research/claim-register.md); full metadata is in the
[source ledger](../research/source-ledger.md). No local note or repository
document is authority for AWS behavior.

### Concepts and recovery

- F01 — Kleppmann and Riccomini, *Designing Data-Intensive Applications*, 2nd
  ed., stream processing and partitioned logs.
- F11-F13 — Amazon Builders' Library on idempotent APIs, retries/backoff/jitter,
  and insurmountable backlogs.
- F31 — Apache Flink stateful processing and checkpoints.

### Current AWS semantics (retrieved 2026-08-22)

- A03, A98 — Lambda [Kinesis integration](https://docs.aws.amazon.com/lambda/latest/dg/with-kinesis.html),
  [mapping parameters](https://docs.aws.amazon.com/lambda/latest/dg/services-kinesis-parameters.html),
  [partial batch responses](https://docs.aws.amazon.com/lambda/latest/dg/services-kinesis-batchfailurereporting.html),
  and [on-failure destinations](https://docs.aws.amazon.com/lambda/latest/dg/kinesis-on-failure-destination.html).
- A14-A18, A95-A97 — Kinesis [concepts](https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html),
  [retention](https://docs.aws.amazon.com/streams/latest/dev/kinesis-extended-retention.html),
  [quotas](https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html),
  [`PutRecords`](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecords.html),
  [EFO](https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html),
  [capacity modes](https://docs.aws.amazon.com/streams/latest/dev/how-do-i-size-a-stream.html),
  [large records](https://docs.aws.amazon.com/streams/latest/dev/large-records.html),
  [`PutRecord`](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html),
  [producer SDK behavior](https://docs.aws.amazon.com/streams/latest/dev/developing-producers-with-sdk.html),
  [KPL](https://docs.aws.amazon.com/streams/latest/dev/developing-producers-with-kpl.html),
  [KCL](https://docs.aws.amazon.com/streams/latest/dev/kcl.html), and
  [resharding](https://docs.aws.amazon.com/streams/latest/dev/kinesis-using-sdk-java-after-resharding.html).
- A31,A89 — Managed Service for Apache Flink [purpose](https://docs.aws.amazon.com/managed-flink/latest/java/what-is.html)
  and [checkpoint/fault tolerance](https://docs.aws.amazon.com/managed-flink/latest/java/how-fault.html).
- A98-A99 — [Kinesis CloudWatch metrics](https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-cloudwatch.html)
  and [Lambda/event-source metrics](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html).
- A29,A88 — Data Firehose [purpose/destination behavior](https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html)
  and [buffering](https://docs.aws.amazon.com/firehose/latest/dev/create-configure-backup.html).

### Pricing (retrieved 2026-08-22)

- A46 — [Kinesis Data Streams pricing and billing-unit definitions](https://aws.amazon.com/kinesis/data-streams/pricing/).
- A99 — AWS Price List [`eu-west-1` dated Kinesis offer snapshot](https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonKinesis/20260813193954/eu-west-1/index.json),
  publication date 2026-08-13.