Kinesis Data Streams deep dive: order, lag, and recovery by design

Reading notesDownload original Markdown ↗

The messaging guide introduced Kinesis as a retained stream for independent consumers. Here, follow a record from a business partition key through a producer acknowledgement, shard, consumer checkpoint, and downstream effect. The same path explains both correctness and capacity: a hot key can limit throughput, while a successful checkpoint can still be insufficient evidence of an external result. KCL is the Kinesis Client Library; KPL is the Kinesis Producer Library; EFO means enhanced fan-out.

Evidence notation: C identifies a claim in the claim register, A a dated AWS source, F a foundational source, and CS a finding in the repository case study. The source index supplies the full source details. These labels are lookup aids, not facts to memorize.

Amazon Kinesis Data Streams is a retained partitioned log. It is valuable when several independently positioned consumers need high-rate ingestion, bounded replay, and an ordering scope. It is not a work queue, a global sequencer, a matching engine, an accounting ledger, or an end-to-end exactly-once system.

This chapter uses the guarantee planes from Event-Driven Foundations and the selection axes from the AWS decision guide. AWS semantics, quotas, modes, metric names, and prices are mutable; every such use below includes retrieved 2026-08-22. Inference: labels an architectural deduction rather than a vendor guarantee.

Mental model: a retained partitioned log

producer record
  {partitionKey, data, eventId, aggregateVersion}
                  |
                  v  hash(partitionKey)
stream -> shard A: 101, 104, 109 ... -> consumer checkpoint A
       -> shard B: 102, 103, 108 ... -> consumer checkpoint B
       -> shard C: 105, 106, 107 ... -> consumer checkpoint C
                  ^ retained records remain until retention expiry

Concept	Working definition	Boundary that matters
Stream	The named retained collection of shards. Independent consumer applications can maintain independent progress.	It is a transport log, not automatically authoritative domain history or a ledger.
Record	Data plus a required partition key and service metadata such as sequence number and approximate arrival time.	The default maximum is 1 MiB; a supported stream can be configured up to 10 MiB only for intermittent large records. Sustained shard baselines remain unchanged (C71; A16,A95, retrieved 2026-08-22).
Partition key	A producer-chosen Unicode string hashed into a shard's hash-key range. Equal keys map to the same current shard.	It defines routing and often an intended ordering scope; it does not create an aggregate version or cross-key invariant (C10; A14,A96, retrieved 2026-08-22).
Shard	A hash-range and the unit of provisioned write/read capacity and consumer parallelism.	Records are sequenced within a shard. There is no free global order across shards (C10,C34; A14, retrieved 2026-08-22).
Sequence number	Kinesis-assigned record identifier/order within the stream/shard behavior.	Inference: it is not a portable business version and cannot be used as a stream-wide query index. Put retries can create a second record with another sequence number (C72; A96, retrieved 2026-08-22).
Retention	Time for which records remain readable: 24 hours by default and configurable up to 8,760 hours/365 days.	Retention is an opportunity to replay, not evidence that detection plus deployment plus drain will finish in time (C14; A15, retrieved 2026-08-22).
Consumer	Lambda mapping, KCL application, Flink application, Firehose, or custom reader that reads records and owns downstream effects.	Each consumer's position, retry, state, and sink contract are separate. One consumer succeeding says nothing about another.
Checkpoint	A durable consumer position after records the application considers processed.	Inference: a checkpoint is not a transaction over DynamoDB, an HTTP API, or a customer notification. Crash after an effect but before checkpoint can repeat the effect (C74; A97, retrieved 2026-08-22).
Replay	Starting a consumer/version at TRIM_HORIZON, AT_TIMESTAMP, or another retained position and reprocessing a bounded range.	Replay repeats delivery; it needs a target version, rate, idempotency/side-effect policy, completion watermark, and reconciliation.

Ordering has four layers

1. The domain authority assigns aggregateVersion = 42 after 41.
2. Kinesis routes a partition key to a shard and assigns sequence numbers.
3. The consumer scheduler chooses batch/invocation concurrency.
4. The database or external system decides commit/effect order.

Inference: only the first layer tells a projector whether 42 is next, stale, duplicate, or a gap. Kinesis shard order can transport that relation efficiently, but it does not invent it. Ordering also does not remove duplicate delivery (C41).

Inference: an execution projector should store stable executionId and the expected authoritative sequence in the same transaction as its state mutation. Kinesis sequence numbers remain transport/checkpoint metadata; reconciliation compares venue executions, ledger postings, and projections.

Replay is not redrive and not reconciliation

A replay plan states:

• source range by stream/shard and timestamp or checkpoint;
• immutable event identity and schema/upcaster version;
• isolated target or projection version;
• live-versus-replay capacity reservation and maximum rate;
• suppression or idempotent repetition of email, venue, payment, and other non-rebuild effects;
• completion watermark, source/output counts, version gaps, exact control totals, and discrepancy sign-off.

Empty iterator lag means a consumer caught up to its position. It does not prove one fill, balanced cash/securities postings, or correct holdings.

Partition keys: put the invariant before the hash

The key is a correctness-and-capacity decision. Start with the relation that must be serialized, then measure its distribution. High cardinality alone does not prevent a popular symbol, institutional account, or large tenant from dominating one shard.

Candidate key	Preserved relation	Parallelism	Skew/failure risk	Defensible use
accountId	Account-scoped execution/projection order	Across accounts	One institutional account can be hot; cross-account transfer still needs coordination	Account portfolio/ledger projection when the authority emits an account version
symbol / book ID	Instrument/book sequence	Across instruments	Hot symbols at open/news; all clients for one symbol share a lane	Market-data normalization or journal distribution; not a substitute for the matching core
tenantId	Tenant-level order/isolation	Across tenants	Large tenant monopolizes a shard; quiet tenants sharing it can suffer	Tenant export/compliance lanes only when tenant order is required
orderId / aggregate ID	One order's state transitions	Across orders	Excellent distribution but cannot serialize account cash/securities invariants	Order lifecycle projection with authoritative order version
accountId#symbol	Position order for one account/instrument	Across positions	More parallelism, but cash or whole-account risk spans keys	Derived position updates when broader invariants stay authoritative elsewhere
salted symbol#bucket	No simple whole-symbol order; requires merge sequence	High	Merge/gap logic and late data become application responsibilities	Replaceable analytics where source sequence enables deterministic merge

The irreducible trade-off

Inference: a single hot key cannot be spread by adding shards because one partition key maps to one current shard. Provisioned selective splitting can isolate a hot hash range from neighbors, but it cannot split that key. On-demand also retains the single-key shard limit (C75; A95,A97, retrieved 2026-08-22).

Inference: if one symbol must sustain more than a shard while preserving price-time order, Kinesis is not the sequencing mechanism for that hot book. Use a capacity-controlled single-writer matching/journal partition, then publish ordered results to Kinesis for independent downstream consumers. If the data is only analytics, salt the key and merge by a source sequence.

Detect skew before throttling becomes aggregate lag

Inference: track producer-side records/bytes by business key and returned ShardId, then compare hottest shard with mean/p95. Enable enhanced shard-level CloudWatch metrics during diagnosis; stream averages can hide one hot shard (C77; A99, retrieved 2026-08-22).

k_skew = hottest_shard_rate / average_shard_rate
key_share = key_rate / total_rate

Alarm on WriteProvisionedThroughputExceeded, but diagnose it with per-shard incoming bytes/records, failed PutRecords entry shard IDs, key distribution, and application admission metrics. Remedies differ:

• aggregate shortage: add provisioned shards or prepare on-demand warm capacity;
• collision of many busy keys: selectively split/rebalance hash ranges;
• one irreducible key: change serialization scope/substrate or admit/shed work;
• producer retry storm: bound retry budget and add full jitter;
• downstream backlog: increase measured commit capacity without moving the bottleneck to a database or external service.

Producer design: acknowledgement is not one effect

Once the partition key reflects the required order, publication must preserve the identity and version assigned by the business authority. The next producer choices trade request efficiency against ordering and retry complexity. Keep an original request manifest so that a partial success becomes a bounded repair, rather than a reason to resend unrelated successful entries.

PutRecord versus PutRecords

Concern	PutRecord	PutRecords
Unit	One Kinesis record	Up to 500 entries, within current request-size constraints
Result	Shard ID and sequence number or request error	HTTP success can contain entry successes and failures; response entries correspond to request order
Order	Inference: only one client submitting same-key records serially can chain order: feed record n-1's successful returned sequence into record n's SequenceNumberForOrdering; an ambiguous response breaks this chain (C72; A96, retrieved 2026-08-22)	Later entries are attempted after an earlier failure; request order is not guaranteed
Efficiency	More calls and simpler per-record acknowledgement	Better request amortization; application must correlate and retry only failures
Use	When same-client/same-key put order is load-bearing and latency/call cost is acceptable	Default high-throughput path when domain versions make consumer order explicit

Inference: these API semantics require entry-level acknowledgement handling (C16,C72; A17,A96, retrieved 2026-08-22).

PutRecords safe loop:

1. assign eventId, authority, aggregate/version, schema, occurred time, and partition key before the first attempt;
2. retain the original request-entry index and identity;
3. inspect FailedRecordCount and every result entry;
4. remove acknowledged successes from the retry set;
5. retry retriable failures with exponential backoff, full jitter, a deadline, and admission control; do not regenerate event identity;
6. durably record/quarantine terminal producer failures and reconcile accepted domain facts to published facts.

Inference: a timeout can be ambiguous, so retry may append a second Kinesis record and cannot safely supply the missing successful sequence to the next same-key put; stop/reconcile the chain from the authoritative domain version. The consumer must make the business effect idempotent on stable eventId/executionId; Kinesis sequence number is not that idempotency key (C72; A17,A96, retrieved 2026-08-22).

Batching versus aggregation

• Collection/batching sends multiple Kinesis records in one PutRecords request.
• KPL aggregation packs several user records into one Kinesis record, which can improve tiny-record throughput and provisioned PUT-unit packing.
• Inference: KPL buffering and asynchronous completion mean packing efficiency adds producer latency and memory/flush behavior (C73; A96, retrieved 2026-08-22).

Inference: enable aggregation only after every Lambda/KCL/Flink/Firehose or custom consumer proves compatible deaggregation, record identity, metrics, and replay behavior. With Lambda ParallelizationFactor, non-EFO aggregated inner events need matching partition keys for order; with EFO, Lambda can drop inner events whose partition keys do not correspond, without processing or failure destination (C73; A03,A96,A98, retrieved 2026-08-22). Treat this as a release-blocking contract test, not a tuning footnote.

Record size and payload pointers

The safe sentence is: default maximum 1 MiB; configurable up to 10 MiB for intermittent large records in supported Regions; sustained per-shard limits do not increase (C71; A16,A95, retrieved 2026-08-22). Lambda's synchronous event payload is 6 MiB including base64 and metadata, so stream acceptance does not prove that Lambda can invoke the handler (C71,C80; A95, retrieved 2026-08-22).

Full-path component	Current large-record compatibility
Stream	Default maximum remains 1 MiB; opt in per supported stream up to 10 MiB for intermittent records. Sustained shard baselines do not increase (C71; A95, retrieved 2026-08-22).
KCL	KCL 2.x supports large records after the stream maximum is updated (C80; A95, retrieved 2026-08-22).
KPL	KPL 1.0.5 and later supports large records after the stream maximum is updated (C80; A95, retrieved 2026-08-22).
Lambda ESM	The 6 MiB limit includes base64 expansion and ESM metadata. An oversized record goes through the configured on-failure destination; configure and test that failure path before enabling larger records (C80; A95, retrieved 2026-08-22).
Data Firehose	S3 delivery supports large records, including format conversion, but dynamic partitioning is not supported. HTTP, Snowflake, and Redshift destinations are not supported for large records; Firehose-to-Lambda is not recommended for this path (C80; A95, retrieved 2026-08-22).
Flink Kinesis connector	The source supports records up to 10 MiB; do not use the sink for records above 1 MiB (C80; A95, retrieved 2026-08-22).

Inference: require full-path compatibility across producer, KCL/KPL version, Lambda envelope and failure destination, Firehose destination/features, Flink source/sink, archive, and replay tooling before raising the stream maximum. If one load-bearing path is incompatible, keep 1 MiB and use a payload pointer (C80; A95, retrieved 2026-08-22).

Inference: for sustained large payloads, put immutable encrypted content in S3 and stream a versioned pointer, checksum, size, and business identity. Then operate S3 retention/access and missing-object reconciliation; a pointer does not make two resources atomic.

Explicit hash keys: exceptional, not routine

ExplicitHashKey overrides the partition-key hash for routing while the record still carries its partition key (A96, retrieved 2026-08-22).

Inference: if that mapping changes, the same logical partition key can reach different shards and forfeit the normal same-key order assumption. Permit it only behind one authoritative stable mapping and an ordered, version-checked cutover that proves the old lane is drained before the new lane starts; otherwise reject it where logical-key order is load-bearing (C79; A96, retrieved 2026-08-22). Ordinary producers should not pin shard IDs or embed topology: resharding changes ranges, independent producers can disagree, and a manual map can create skew.

Consumer boundaries: Lambda, KCL, enhanced fan-out, and Flink

Consumer	Owns for you	You still own	Choose when	Poor fit
Lambda event-source mapping	Poll/push connection, batching, shard progress, retries, partial/bisect options, invocation scaling	Idempotent effects, versions/gaps, poison policy, destination permissions, downstream capacity, replay/reconciliation	Stateless bounded per-record/batch work with native scaling	Long-lived state, tight deterministic latency, large payloads, or sink transaction that cannot tolerate repeats
KCL application	Shard leases/load balancing, worker failure, checkpoints, reshard discovery, parent-before-child handling; at-least-once delivery	Worker runtime/scale, checkpoint timing, state/sink transaction, deployment, poison/replay	Long-lived custom consumer with connection/state control	Team does not need that runtime/control or cannot operate it
Enhanced fan-out	Registered consumer with dedicated up-to-2 MB/s/shard push pipe	Same processing/checkpoint/effect duties plus registration lifecycle and price	Multiple consumers contend, low propagation delay or catch-up isolation matters	One light consumer with ample shared read capacity
Managed Flink	Managed Flink runtime for keyed/event-time state and checkpoints	Operator/state design, watermarks, late data, savepoints, source/sink guarantee, cost and recovery	Stateful windows, joins, temporal risk/fraud analytics	Stateless routing, simple projection write, low-latency matching

Inference: KCL's documented behavior makes it the managed-library boundary for shard balancing, checkpointing, at-least-once delivery, and parent-before-child processing across resharding (C74; A97, retrieved 2026-08-22). Enhanced fan-out provides each registered consumer up to 2 MB/s/shard dedicated read throughput; current registration limits are 20 for Provisioned/On-demand Standard and 50 for On-demand Advantage where supported (C17,C70; A16,A18,A95, retrieved 2026-08-22).

Inference: EFO isolates Kinesis read throughput, not Lambda concurrency, DynamoDB capacity, vendor quotas, or a hot producer key. Pay for it when measured contention/latency/recovery justifies it, and reserve downstream catch-up capacity.

Inference: Managed Flink checkpoints can restore compatible source position and operator state, but an arbitrary REST API or database side effect is outside that checkpoint transaction. Use a compatible sink/transactional design or stable identity plus idempotency and reconciliation (C67; A31,A89, retrieved 2026-08-22).

Lambda failure semantics and poison records

Consumer selection determines who owns checkpoints, but it does not settle what happens when progress stops. Separate a platform that cannot invoke the function from a handler that ran and failed: the retry controls apply differently. Then decide what evidence survives terminal failure and whether later records in the affected ordering scope may proceed.

Lambda invokes synchronously from its event-source mapping and processes records at least once. By default it checkpoints a Kinesis batch only after complete success; a failed batch is retried and can block later work on the affected shard (C07,C11; A03, retrieved 2026-08-22).

Inference: current failure ownership must be split before selecting retry controls. Before invocation, throttling or another system inability to invoke the function is retried until source retention expires or the record exceeds MaximumRecordAgeInSeconds; MaximumRetryAttempts does not bound this path and no handler attempt occurred. During invocation, a handler error is retried until source expiry, maximum record age, or MaximumRetryAttempts is reached. Bisection is only for function errors, and splitting a batch does not consume retry quota (C76; A98, retrieved 2026-08-22).

Control-by-control

Control	Current behavior	Design consequence
Batch size/window	Batch default 100, maximum 10,000; Kinesis window defaults 0 and is configurable 0–300 seconds; invocation stops at batch/window/6 MB payload	Larger batches amortize calls but add latency, retry blast radius, memory, and downstream burst (A03,A98, retrieved 2026-08-22).
ParallelizationFactor	1–10 concurrent batches per shard; Lambda preserves partition-key order	Inference: capacity can exceed shard count, but same-key serialization remains and aggregation rules are strict (C73,C76; A03,A98, retrieved 2026-08-22).
Partial response	With ReportBatchItemFailures, the lowest returned failed sequence becomes the checkpoint and Lambda retries from it	Inference: successful later records can repeat; this reduces work, not duplicates (C12,C76; A98, retrieved 2026-08-22).
Bisect	For a function error, split a failed batch; with partial reporting, bisect at the returned sequence; splits do not consume retry quota	Inference: this finds a handler-poison region but does not apply to a pre-invocation failure and is not quarantine or transaction (C13,C76; A03,A98, retrieved 2026-08-22).
Maximum record age	MaximumRecordAgeInSeconds is -1 (default) or 60–604,800; source retention still applies	Inference: age bounds both pre-invocation and invoked-error retry, so it must fit the business deadline while preserving repair time and retained payload (C76; A98, retrieved 2026-08-22).
Maximum retries	MaximumRetryAttempts is -1 (default) through 10,000 and applies to handler/function errors, not pre-invocation inability	Inference: a finite quota bounds handler poison only; pre-invocation failure still needs age/retention alarms and repair (C76; A98, retrieved 2026-08-22).
On-failure destination	SQS/SNS receive failure metadata; S3 receives metadata plus the complete invocation record	Inference: S3 is normally the safer replay source for discarded Kinesis payloads; permissions, encryption, lifecycle, alarms, and object validation remain required (C76; A98, retrieved 2026-08-22).

Two runbooks, then a poison-record policy

1. Inference: Unable to invoke: alert on rising iterator age plus FailedInvokeEventCount/throttles and corroborate mapping state, permissions, concurrency, network/configuration, and whether InvokedEventCount advances. Restore invocation before age/retention expiry; do not call these handler attempts or expect MaximumRetryAttempts to stop them (C76,C77; A98, retrieved 2026-08-22).
2. Inference: Invoked handler poison/failed effect: use function errors, partial-failure identities, repeated event IDs, iterator age, and DroppedEventCount; repair code/data or quarantine according to the business order policy. Separately alarm DestinationDeliveryFailures and reconcile it against expected OnFailureDestinationDeliveredEventCount so destination failure is not confused with handler failure (C76,C77; A98, retrieved 2026-08-22).

Handler-poison workflow:

1. Validate envelope/schema and stable identity before side effects.
2. Classify transient dependency failure separately from deterministic poison.
3. Retry transient failures within a time/attempt budget with idempotent effects.
4. Use partial response and bisection to minimize repeated good work.
5. Send exhausted/aged batches to S3 so the original invocation payload is recoverable; alarm destination delivery failure separately.
6. Record the failed shard, sequence range, event IDs, schema, code version, exception class, attempt/age, and owner in an operator case.
7. Fix or deploy a versioned projector, replay to an isolated target at a paced rate, and reconcile every event/effect before closing.

Inference: skipping a poison record restores liveness by weakening strict lane order. The business policy must decide whether later records may proceed, whether the account/book is quarantined, or whether the projection is rebuilt. A DLQ/destination is a holding mechanism, not that decision (C43,C76; A98, retrieved 2026-08-22).

Why Kinesis plus Lambda is not end-to-end exactly once

The event-source mapping is at least once and successful records can repeat during failed-batch/partial-checkpoint recovery (C11-C12; A03,A98, retrieved 2026-08-22). A handler can commit DynamoDB and crash before the mapping advances; an external API can time out after accepting the request; replay intentionally redelivers history.

Inference: Kinesis/Lambda cannot promise one end-to-end business effect. Transact an inbox identity with the protected DynamoDB mutation where possible, use an external provider idempotency key/receipt when available, validate aggregate versions, and reconcile authoritative outcomes (C40-C41; F11).

Capacity modes, resharding, and backlog recovery

Now connect those failure paths to capacity. Size for both bytes and records, then check skew and downstream commit rate. A shard count that accepts the live feed is only the first calculation; recovery also needs spare capacity and enough retained history to cover detection, repair, and catch-up.

Current mode boundaries

Mode	Current behavior	Best fit	Primary risk/cost shape
Provisioned	You choose shards; capacity sums shard limits; selective split/merge and shard count changes are your responsibility	Predictable traffic, controlled headroom, selective hot-range work	Shard hours + 25 KiB PUT units + optional EFO/retention; operational scaling
On-demand Standard	Starts with documented 4 MB/s write and 8 MB/s read capacity; auto-manages shards; rapid growth and skew can throttle	Unknown/variable stream traffic without capacity planning	Per-stream hour + 1 KiB-rounded Data-In + Data-Out + EFO/retention
On-demand Advantage	Account-level regional mode, warm throughput, up to 50 EFO consumers where supported, different discounted data price shape	Aggregate on-demand estate that earns the commitment and warm/EFO features	At least 25 MiB/s ingest and 25 MiB/s retrieval usage/shortfall across on-demand streams

The mode facts and eu-west-1 support were retrieved 2026-08-22 (C70; A16,A95, retrieved 2026-08-22). On-demand can see read/write exceptions for roughly 15 minutes if traffic exceeds twice the prior peak, and one partition key remains bounded by one shard. Advantage warm throughput is the pre-event control; retry with backoff/jitter and producer admission is still required (A95, retrieved 2026-08-22).

Provisioned sizing with skew and headroom

byte_shards   = ceil(peak_bytes_per_second / 1,000,000)
record_shards = ceil(peak_records_per_second / 1,000)
planned       = ceil(max(unrounded byte_shards, unrounded record_shards)
                     * skew_factor / target_utilization)

For 8,000 records/s, 700 B, measured 1.25 hottest/average skew, and 80% target utilization:

bytes baseline   = ceil(5.6 MB/s / 1 MB/s) = 6
records baseline = ceil(8,000 / 1,000) = 8
planned shards   = ceil(8 * 1.25 / 0.80) = 13

Current per-shard values are mutable and were retrieved 2026-08-22 (C15; A16). The complete unit model and sensitivity calculations are in the cost model.

Reshard without losing parent history

Inference: provisioned resharding routes new writes to child shards while records already in parent shards remain readable until retention expiry. Reading child shards before draining/checkpointing parents can violate per-key order; KCL handles the parent-before-child relationship (C74-C75; A97, retrieved 2026-08-22).

Runbook:

1. classify aggregate shortage, collision, or one-key overload;
2. calculate target capacity and downstream/replay headroom;
3. split the hot hash range or update shard count;
4. verify stream active, producer throttle falling, and child discovery;
5. confirm every consumer drains/checkpoints parents before children;
6. validate per-key versions/gaps and business control totals;
7. merge only after a sustained cold period and another capacity/recovery check.

Backlog growth and drain

growth = max(0, arrival_rate - commit_rate)
backlog = initial_backlog + growth * duration
drain = backlog / (recovery_commit_rate - live_arrival_rate)

If recovery rate is not greater than live arrival, drain time is infinite. A 15-minute outage at 5,000 records/s creates 4.5M records or 3.15 GB raw at 700 B. At 9,000 commits/s while live remains 5,000/s, ideal drain is 1,125 s = 18.75 min; recovery finishes 33.75 minutes after the outage began, before detection/deployment/retry margin. If commit falls to 6,000/s, drain is 75 minutes. Do not report the first figure without that sensitivity.

Inference: provision live and replay lanes so a backfill cannot starve the market-session feed. Bound replay concurrency/rate at both the consumer and downstream store, monitor age per shard/key/tenant, and stop before retention or financial freshness budgets are threatened (C44,C61).

Observability and game-day drills

Signals by owned boundary

Boundary	Minimum signals	Interpretation/trap
Producer	PutRecord.Success/PutRecords.Success, FailedRecordCount, error code, retry age/attempt, IncomingBytes/Records, WriteProvisionedThroughputExceeded	HTTP success is not all-entry success; retries without stable IDs can append duplicates
Partition/skew	optional shard-level incoming bytes/records and write throttles; key/share and returned shard ID	Stream average can hide one hot shard; cardinality is not distribution
Standard reader	GetRecords.Bytes/Records, ReadProvisionedThroughputExceeded, GetRecords.IteratorAgeMilliseconds	Age rising with no read throttles points to compute/sink/poison; age alone does not locate business gaps
EFO	SubscribeToShardEvent.*, registered consumer state/count and per-consumer lag/throughput	Registration/delivery health does not prove consumer commit
Lambda mapping	IteratorAge, function Errors, Throttles, duration/concurrency, opt-in PolledEventCount, FailedInvokeEventCount, DroppedEventCount, OnFailureDestinationDeliveredEventCount, DestinationDeliveryFailures	A destination success means payload/metadata was retained, not repaired
Business	duplicate IDs, stale/gap sequence, unposted execution, ledger imbalance, projection freshness, unreconciled discrepancy age	This is the only plane that detects correct-looking transport with wrong money/holdings

Inference: metric names/enablement are current mutable inputs to the operational design (C77; A98,A99, retrieved 2026-08-22). Alarm Maximum iterator age against a consumer-specific lag and retention budget; use optional shard metrics for diagnosis, then control their CloudWatch cost.

Game-day drills

1. Hot key: drive one symbol above its lane capacity. Expect shard-specific throttles, identify the key, exercise admission/scope/substrate decision, and prove unrelated keys' lag stays within budget.
2. Consumer outage/backlog: pause one consumer for 15 minutes. Predict backlog/drain first; resume with paced capacity, protect live traffic, and compare predicted to actual iterator age.
3. Poison record: inject a valid envelope with deterministic handler failure. Verify partial/bisect behavior, bounded attempts/age, full payload in S3, destination alarms, controlled replay, and business reconciliation.
4. Partial producer failure: fault/throttle selected PutRecords entries. Verify only failed identities retry, successful entries do not get blindly republished, and consumer inboxes suppress ambiguous duplicates.
5. Reshard: split a hot range while traffic continues. Verify parent drain before child processing, KCL/Lambda shard discovery, versions with no gaps, and planned recovery capacity.
6. Inference: Aggregation compatibility: publish mixed inner partition keys in a test stream with and without EFO. The release gate must detect the documented Lambda drop/order hazard; never run this destructive experiment on business traffic (C73; A03, retrieved 2026-08-22).
7. Retention edge: restore a consumer near retention expiry. Verify alarms fire early enough, the S3/authoritative backfill route works, and every missing interval is classified.

Success criteria are version/count/control-total reconciliation, not merely zero throttles or an empty destination.

Fintech applications and hard non-fits

Market data

Partition a normalized feed by the ordering scope the consumer actually needs: symbol/book for per-instrument sequence, or a source channel if upstream sequence is channel-scoped. Detect gaps against the venue sequence, isolate hot symbols, and use Flink only when event-time windows/joins/state justify it.

Inference: Kinesis is good downstream distribution and bounded replay, but not the low-latency price-time matching loop. A hot book needs a measured single-writer execution substrate; Kinesis carries resulting market/execution facts to projections.

Executions

The matcher or venue adapter assigns stable executionId, order/account/book versions, exact price/quantity/currency/scale, and correction semantics before publication. Ledger, portfolio, notification, and compliance consumers maintain separate positions and idempotent effects.

Repository mapping: replacing EventBridge with Kinesis would give the portfolio a retained ordered transport only after choosing a key. It does not repair the missing inbox or source version in CS06 and CS07, nor the trade/outbox gap in CS04.

Risk and compliance fan-out

Use independent consumers because real-time risk, surveillance, search, and portfolio projections have different state, lag, deployment, and replay needs. EFO is justified by measured shared-read contention or latency, not the number of team names. Stateful event-time risk belongs in Flink only when windows, watermarks, joins, and late-data policy are explicit.

Hard pre-trade risk and reservations stay on the authoritative command path. A lagging Kinesis projection cannot authorize cash, securities, or exposure that must be current (C46).

Audit archival and backfill

Inference: Kinesis can feed Amazon Data Firehose to S3 for buffered landing; source-to-object manifests, counts/hashes, encryption, retention, and object access govern audit evidence. Firehose is not the shared replay authority (C31,C66; A29,A88, retrieved 2026-08-22). Keep an authoritative execution/ledger source and use S3 for recovery beyond stream retention.

Backfill into an isolated projection version or dedicated stream, preserve original identities/occurred times/source versions, pace against downstream capacity, suppress notifications/external actions, and cut over after reconciliation. Do not pour years of S3 data into the live stream without a lag and cost budget.

Explicit hard non-fits

• Matching engine: Kinesis does not implement an order book, price-time priority, deterministic decision loop, or venue fairness. See CS12.
• Global sequencer: shards provide parallel order scopes. Global serialization is possible only through a common bottleneck/coordinator, not a free Kinesis setting (C34).
• Authoritative ledger: retention, sequence numbers, and replay do not create balanced postings, reversals, access controls, or reconciliation (C47).
• One-owner work queue: if each item belongs to one worker and visibility, acknowledgement, and DLQ/redrive are the desired semantics, SQS is clearer.

Cost and neighboring alternatives

Current eu-west-1 cost math and machine-readable price inputs are in the cost model. Keep formulas portable and inputs dated (C49,C78; A46,A99, retrieved 2026-08-22).

For the printed example (5,000 records/s, 700 B, two full consumers, one-day retention), on-demand Standard rounds each record to 1 KiB for Data-In and models about $1.9k/month; 13 provisioned shards with two EFO consumers model about $0.94k/month. Both exclude consumer compute/sinks, storage, logs, transfer, KMS, replay, support, and engineering. Report those as rough scenario results, never a universal break-even.

Need/cost driver	Kinesis	SQS	EventBridge	MSK	Data Firehose
Independent positions/replay	Native within retention	Work removed after acknowledgement; redrive, not arbitrary consumer history	Optional archive replays to source bus, not partitioned positions/order	Native topic partitions/consumer groups	No general consumer positions; destination delivery
Ordering	Shard/key scope	FIFO message-group scope when selected	No package ordering claim	Kafka partition scope	Do not build business order on it
Fan-out cost	Standard read contention or EFO per consumer; on-demand data-out multiplies	One queue per independent owner plus fan-out layer	Event multiplied by matching targets and downstream	Broker/serverless capacity, storage, network and operations	Bytes/transforms/conversion/dynamic partition/destination
Capacity burden	Mode, key skew, shards/warm throughput, retention, recovery	Queue age, request chunks, consumers/downstream	Rule/target quotas, retries/archive, target isolation	Brokers/partitions/storage/rebalances/upgrades or serverless capacity	Buffer/destination limits and upstream replay source
Choose	High-rate retained AWS-native log, Lambda/KCL/Flink integrations	One-owner buffered work, Standard or FIFO semantics	Structured content routing/AWS integration	Kafka protocol/ecosystem/portability justifies operations	Managed buffered delivery to S3/search/warehouse
Reject	One hot globally ordered matcher lane or no replay need	Several consumers need independent history	Deterministic partition replay/high-rate stream	No Kafka requirement/operating competence	Command bus, shared log, custom stateful processing

Use workload variables, not blanket service prices: records/s, payload distribution, peak duration, key skew, consumers, retention, replay, batch/ aggregation, Lambda/Flink work, sinks, logs/metrics, transfer, KMS, RTO/RPO, correctness loss, and on-call cost.

Myth and correction table

Local Kinesis/Kafka notes and the reference application's Kinesis documents are cross-references only. The controlled corrections below supersede them.

Local shorthand/myth	Correction	Why it matters
“Each shard is 1 MB/s write, so N shards always handle N MB/s.”	Provisioned capacity is constrained by both bytes and records, key skew, retry headroom, and downstream drain. Large-record burst does not change sustained baselines (C15,C71; A16,A95, retrieved 2026-08-22).	Prevents hot-key loss/throttling hidden by aggregate math.
“Maximum record size is 10 MiB.”	Default is 1 MiB; supported streams may opt up to 10 MiB for intermittent records, and consumer limits may be lower (C71; A95, retrieved 2026-08-22).	Avoids deploying payloads Lambda/Firehose/Flink paths cannot process safely.
“Same partition key means one-at-a-time portfolio writes; conflicts disappear.”	Same key routes to one shard, but Lambda can use parallelization while preserving key order; retries duplicate attempts, and consumer code/state can still race with other writers. Keep inbox, version, and conditional mutation.	Transport order is not database atomicity or novelty.
“PutRecords is a batch atomic write in request order.”	Inference: entries can partially succeed and later entries continue; retry only failures. Use PutRecord plus SequenceNumberForOrdering only when same-client/same-key put order truly requires it (C72; A17,A96, retrieved 2026-08-22).	Prevents silent loss and blind duplicate republish.
“On-demand absorbs any instant spike.”	It can throttle for roughly 15 minutes beyond twice prior peak; one hot key retains shard limits. Advantage warm throughput is a separate account-level mode/commitment (C70; A95, retrieved 2026-08-22).	Makes market-open/flash-event preparation explicit.
“Enhanced fan-out gives every consumer exactly one copy.”	Inference: it gives dedicated read throughput, not one processing effect. Lambda/KCL processing is still at least once and effects can repeat (C17,C74,C76; A18,A97,A98, retrieved 2026-08-22).	Throughput isolation is not processing exactly once.
“Partial batch response means successful records never repeat.”	Inference: Lambda retries from the lowest returned failed sequence; later successes can repeat (C12,C76; A98, retrieved 2026-08-22).	Requires idempotent sinks and conservative checkpoint design.
“Bisect sends the bad item to the DLQ.”	Inference: bisect only splits failed batches. Terminal destination requires age/retry policy and destination; S3 retains full invocation while SQS/SNS retain metadata (C13,C76; A98, retrieved 2026-08-22).	Makes payload recovery and destination permissions testable.
“Kinesis/Lambda is end-to-end exactly once.”	Event-source processing is at least once; crash windows, partial failures, external calls, and replay repeat effects. Use stable identity, transactional inbox/outbox where possible, version checks, and reconciliation.	Prevents double fills/postings/notifications.
“Kinesis Data Firehose / Kinesis Data Analytics.”	Current names are Amazon Data Firehose and Amazon Managed Service for Apache Flink (C30; A29,A31, retrieved 2026-08-22).	Avoids stale interview vocabulary and product conflation.
“Kinesis versus MSK is a fixed throughput/latency/dollar cutoff.”	Choose from required protocol/ecosystem, workload/key distribution, retention, consumer topology, recovery, operations, and dated regional unit economics.	Removes stale dollar and absolute-latency folklore (X04-X06).

Decision table

Requirement	Choice	Non-negotiable controls	Do not use when
Bursty AWS-native retained feed, unknown capacity	On-demand Standard; Advantage only after aggregate commitment/warm/EFO analysis	stable IDs, key-skew telemetry, retry/jitter, retention/drain budget	one key exceeds a shard or instant spike is assumed free of throttling
Predictable feed with measured keys	Provisioned Kinesis	byte+record+skew sizing, headroom, reshard runbook, parent/child validation	team will not operate capacity and recovery
Stateless Lambda projections	Lambda mapping	inbox/effect idempotency, version/gap policy, bounded poison handling, S3 failure payload, iterator-age SLO	long-lived state/tight deterministic loop/oversized payload
Custom long-lived consumers	KCL, EFO when isolation earns cost	checkpoint after intended effect, lease/state store operations, reshard/replay drills	Lambda already meets semantics and operations
Stateful event-time risk	Managed Flink over Kinesis	watermarks/late data, checkpoint/sink boundary, state growth, savepoint/restart, reconciliation	simple stateless routing or hard pre-trade synchronous invariant
Managed S3 audit landing	Firehose fed from a recoverable source	manifests/control totals, error backup, S3 governance, upstream replay	Firehose buffer would be the only copy
One-owner account-serial work	SQS FIFO keyed by account, or measured single-writer engine	idempotent effect, group skew/poison/redrive policy	several independent consumers need retained history
Low-latency matcher/global order	long-lived capacity-controlled sequencer/matching core	deterministic journal, failover/replay, fairness and reconciliation	do not use Kinesis/Lambda as the matching algorithm

Two-minute senior answer

I use Kinesis when I need a retained partitioned log with independent consumer positions. I first choose the ordering key from the invariant, then measure skew; one hot key still has one-shard capacity, so adding shards does not solve every overload. Producers give every fact stable identity and version, inspect every PutRecords entry, and retry only failures with jitter. Consumers are at least once: Lambda gives simple batching/retry controls, KCL gives explicit long-lived checkpoint workers, EFO isolates read throughput, and Flink is for stateful event-time logic. None makes an external ledger write exactly once. I size bytes and records with skew/headroom, calculate backlog drain against live arrival, keep retention margin, and operate throttles, iterator age, shard skew, poison destinations, version gaps, and reconciliation together. Kinesis distributes execution/market facts; it is not the matcher, global sequencer, or financial ledger.

Interview follow-ups

1. Why is accountId safer for one portfolio relation but worse for an institutional hot account than orderId?
2. What happens when one symbol alone exceeds one shard and strict book order is required?
3. Compare PutRecord, PutRecords, collection, and aggregation. Which can reorder or partially fail?
4. A producer times out after a put. Which identity makes retry safe at the ledger projector?
5. Why can ReportBatchItemFailures still repeat a successful record?
6. How do partial response and bisection interact, and when is S3 a better failure destination than SQS/SNS?
7. What does ParallelizationFactor = 5 preserve, and what aggregation partition-key hazard must be tested?
8. When does EFO solve the bottleneck, and when does it only move it to DynamoDB or an API?
9. How does KCL preserve per-key order across a split, and what must a custom consumer implement?
10. On-demand traffic jumps to three times its prior peak at market open. What do you expect, and how does Advantage warm throughput change preparation?
11. Calculate shards for peak records, bytes, k_skew, and target utilization; which constraint binds?
12. If live arrival is 5k/s and recovery commit is 5k/s, when does a 4.5M record backlog drain?
13. Which signals distinguish producer throttling, one hot shard, Lambda poison retry, EFO subscription failure, and a slow sink?
14. How do you replay six months of executions without sending duplicate client notifications?
15. Why are Kinesis stream retention and an S3 audit landing not automatically a regulated accounting ledger?
16. Give a workload where SQS FIFO, EventBridge, MSK, or Firehose is more defensible than Kinesis.

Sources used in this chapter

Claim wording and freshness are controlled in the claim register; full metadata is in the source ledger. No local note or repository document is authority for AWS behavior.

Concepts and recovery

• F01 — Kleppmann and Riccomini, Designing Data-Intensive Applications, 2nd ed., stream processing and partitioned logs.
• F11-F13 — Amazon Builders' Library on idempotent APIs, retries/backoff/jitter, and insurmountable backlogs.
• F31 — Apache Flink stateful processing and checkpoints.

Current AWS semantics (retrieved 2026-08-22)

• A03, A98 — Lambda Kinesis integration, mapping parameters, partial batch responses, and on-failure destinations.
• A14-A18, A95-A97 — Kinesis concepts, retention, quotas, PutRecords, EFO, capacity modes, large records, PutRecord, producer SDK behavior, KPL, KCL, and resharding.
• A31,A89 — Managed Service for Apache Flink purpose and checkpoint/fault tolerance.
• A98-A99 — Kinesis CloudWatch metrics and Lambda/event-source metrics.
• A29,A88 — Data Firehose purpose/destination behavior and buffering.

Pricing (retrieved 2026-08-22)

• A46 — Kinesis Data Streams pricing and billing-unit definitions.
• A99 — AWS Price List eu-west-1 dated Kinesis offer snapshot, publication date 2026-08-13.

Takeaway and next step

Kinesis is useful when its partitioned retention and consumer positions match the workload. The complete argument also needs stable producer identities, version-aware consumers, a hot-key check, and a recoverable failure destination. Recompute the shard and drain examples, including the case with no spare capacity. Next, examine the store receiving those effects in DynamoDB: access patterns and authority.