The DynamoDB Hot Partition That Only Showed Up on Black Friday

We spent six weeks load-testing the checkout service before Black Friday. Synthetic traffic at 3x forecast peak, chaos drills, the whole thing. Then at 19:04 on the day itself, one DynamoDB table started returning ProvisionedThroughputExceededException while sitting at 22% of its provisioned capacity. This is the story of that outage, why our tests missed it, and the partition key redesign that fixed it.

What broke, in one paragraph

We run an order-tracking table on DynamoDB. The partition key was merchantId, the sort key was orderId#timestamp. Roughly 40,000 merchants, average traffic well-distributed across them. Under normal load and even under our load tests, throughput was smooth. On Black Friday, one merchant — a large fashion retailer running a flash sale at the top of the hour — pushed something like 35% of all writes to the table for a 90-second window. That's a classic hot partition. DynamoDB throttled writes to that partition while the rest of the table sat idle. Checkout failures cascaded, retries piled on, and we spent 40 minutes bleeding orders before we routed that merchant's writes to a separate table as a stopgap.

Why our load tests didn't catch it

This is the part that stings. Our load generator modelled traffic as a weighted random draw across merchants, using historical share-of-traffic as weights. That produces a smooth distribution. Real Black Friday traffic is not smooth — it's spiky, and the spikes are correlated with individual merchants' marketing calendars. A single retailer sending a push notification at 19:00 sharp will concentrate writes on one partition key in a way that a Poisson-ish load generator will never reproduce.

The lesson: if your load test smooths traffic over your partition key, you are testing a scenario that cannot fail the way production does.

We now generate load with a mixture model — 70% of RPS drawn from the smooth distribution, 30% drawn from a small number of "burst merchants" that fire correlated spikes. That single change would have surfaced the hot partition in staging.

Finding the hot key with Contributor Insights

At 19:11 we knew we were throttled. We did not yet know why, because table-level metrics in CloudWatch looked fine. Consumed write capacity was well below provisioned. The tell was WriteThrottleEvents climbing while ConsumedWriteCapacityUnits did not.

DynamoDB Contributor Insights is the tool that pays for itself the first time you have this incident. If you enable it on a table, it publishes the top-N most active partition keys to CloudWatch. It costs a bit, but it turns a five-hour investigation into a five-minute one.

aws dynamodb update-contributor-insights \
  --table-name orders \
  --contributor-insights-action ENABLE

With it on, the top-partition-key graph showed a single merchantId receiving roughly 12,000 write requests per second while the next-busiest merchant was around 400. That was our smoking gun.

The math of why this throttles

A DynamoDB partition can handle up to 1,000 WCU per second (roughly 1,000 writes/sec for items under 1KB). If a single partition key receives 12,000 writes/sec, it doesn't matter that the table is provisioned for 50,000 WCU — that key lives on one partition, and that partition tops out at 1,000. Adaptive capacity helps, but it takes minutes to kick in, and a 90-second spike is over before the rebalancing finishes. You get throttled, hard, while your dashboards insist you have headroom.

The fix: write sharding

There are a few standard patterns for this. We chose write sharding, because we needed the fix live before the following weekend.

Instead of merchantId as the partition key, we use merchantId#shard, where shard is an integer 0–N. For most merchants, N=1 (a single logical shard, unchanged behaviour). For the top 200 merchants by traffic, N is computed from their p99 write rate, rounded up to the nearest power of two — typically 8, 16, or 32.

Writes pick a shard at random:

function partitionKey(merchantId: string, shardCount: number): string {
  const shard = Math.floor(Math.random() * shardCount);
  return `${merchantId}#${shard}`;
}

Reads are the tricky part. A query for "all orders for merchant X" now has to fan out across all shards for that merchant and merge results. We store the shard count for each merchant in a small config table cached in-process with a 60-second TTL.

async function queryMerchantOrders(merchantId: string) {
  const shardCount = await getShardCount(merchantId);
  const queries = Array.from({ length: shardCount }, (_, i) =>
    ddb.query({
      TableName: 'orders',
      KeyConditionExpression: 'pk = :pk',
      ExpressionAttributeValues: { ':pk': `${merchantId}#${i}` },
    })
  );
  const results = await Promise.all(queries);
  return results.flatMap(r => r.Items ?? []).sort(byTimestampDesc);
}

The tradeoffs are real:

• Read cost goes up proportionally to shard count. For a merchant with 16 shards, every query is 16 queries. In practice we cache aggressively and most reads never hit the table.
• Sort order across shards has to be reconstructed client-side. Fine for us; painful if you're paginating.
• Shard count changes need a migration or a dual-read window. We built a background job that re-shards a merchant's history when we bump their shard count.

What we changed operationally

The code fix is only half the story. The other half is making sure this class of problem is visible before it takes production down.

Alarms that actually fire

We added a CloudWatch alarm on WriteThrottleEvents > 0 for any period longer than 60 seconds. Sounds obvious. It wasn't alarmed before, because our alerting was built around capacity utilisation, and throttling on a hot partition doesn't move the capacity graph.

We also alarm on the Contributor Insights metric for top-partition-key traffic exceeding 60% of a single partition's ceiling. That gives us a warning before the throttling starts.

Runbook, not heroics

During the incident, we didn't have a documented path for "single merchant is hot." We invented one under pressure and it worked, but it was luck. The runbook now says: if a single partition key exceeds 70% of partition ceiling for more than 2 minutes, we have a pre-baked Terraform change that routes that merchant to an isolated table. It's one PR, one apply, ten minutes end-to-end.

Load tests that spike

We rebuilt the load generator to mix smooth traffic with correlated bursts. Every release now runs against a scenario labelled flash-sale-single-merchant that pushes 40% of RPS through one partition key for 2 minutes. If write throttling appears, the build fails.

When not to shard

Write sharding is not free and it's not always right. If your access pattern is dominated by point reads on a well-distributed key, don't bother. If you can restructure the schema so the natural partition key is higher-cardinality — for example, orderId instead of merchantId when you rarely query by merchant — that's cleaner. And if your workload really is uniformly hot, DynamoDB is possibly the wrong store; a log-structured system like Kinesis or a partitioned Kafka topic in front of a batch writer will behave better.

We considered moving to Aurora for this table. The migration cost, and the loss of DynamoDB's operational simplicity, outweighed the benefit for us. Your calculation may differ.

Where we'd start

If you run a DynamoDB table under bursty, tenant-driven traffic and you have not looked at Contributor Insights this quarter, turn it on today. Watch the top-key graph for a week. If any single key is consistently in the top 3, model what happens when its traffic doubles. Then decide whether to shard, restructure, or isolate — before a marketing team you've never met picks 19:00 on your busiest day to send a push notification.

If you want help stress-testing a data layer against realistic spike patterns, that's the kind of work we do in our reliability and cloud engineering practice.