The DynamoDB On-Demand Bill That Ate Our Margin: A Capacity Mode War Story

We flipped a DynamoDB table to on-demand the week before a product launch because someone in the room said the words "peace of mind." Three weeks later, that one table was costing us roughly 6x what it did on provisioned capacity, and the traffic hadn't even doubled. This is the story of how we got there, what the math actually looks like, and the rule we now use to decide capacity mode.

The setup: why we switched to on-demand

The table in question was the write-heavy core of an event ingestion pipeline. Every user action produced 2 – 5 writes across a main table and two GSIs. Pre-launch, it ran on provisioned capacity with autoscaling between 200 and 1,200 WCU, and it sat around 40 – 60% utilization most of the day.

We were nervous about a marketing push that we couldn't size. Product was promising a 3 – 5x traffic bump. The autoscaling target was 70%, and DynamoDB autoscaling is famously lazy on the way up — it uses CloudWatch alarms that take minutes to trigger, and the scale-up steps are conservative. We'd been throttled before during a smaller spike, and nobody wanted to eat that during a launch keynote.

So we made the call: switch to on-demand, ride out the launch, review in a month.

The mental model we had (and why it was wrong)

The pitch for on-demand is simple: pay per request, no capacity planning, instant scaling. The mental model most engineers carry is "it's like Lambda for tables." That's technically true. What it hides is the unit price.

At the time of writing, in us-east-1, standard DynamoDB pricing looks roughly like this:

• Provisioned: ~$0.00065 per WCU-hour, ~$0.00013 per RCU-hour
• On-demand: ~$1.25 per million write request units, ~$0.25 per million read request units

Do the arithmetic. One WCU sustained for an hour handles 3,600 writes and costs $0.00065. The same 3,600 writes on-demand cost $0.0045. That's roughly 6.9x more expensive per write if you can keep a provisioned WCU busy the whole hour.

That's the number that ate us.

What the bill actually looked like

Here's the shape of what happened, pulled from Cost Explorer with the numbers rounded and anonymized:

Week	Mode	Avg writes/sec	Table + GSI cost
-2	Provisioned + autoscale	~480	~$310
-1	Provisioned + autoscale	~520	~$340
0 (launch)	On-demand	~890	~$1,150
+1	On-demand	~950	~$1,780
+2	On-demand	~910	~$1,690

Traffic went up ~1.8x. Cost went up ~5.5x. The launch itself was fine — no throttling, no incidents — but the ongoing steady-state was brutal because our workload was, in retrospect, extremely predictable.

Why the ratio was so bad for us

On-demand is a great deal when:

• Traffic is spiky and unpredictable
• Peak-to-average ratio is high (say, 10x or more)
• The table sits mostly idle
• You genuinely cannot forecast within an order of magnitude

On-demand is a bad deal when:

• Traffic is sustained and roughly flat
• You have a reasonable floor and ceiling
• You're willing to run autoscaling with a sane buffer
• Your peak-to-average is under ~4x

Ours was case two, pretending to be case one because we were scared of the launch curve. The event pipeline had a strong daily rhythm — 3x peak-to-trough — but no dramatic bursts. Every hour, we were paying the on-demand premium to solve a problem we didn't actually have after the first 48 hours.

The heuristic we now use

After the rollback, we wrote this down and pinned it in the platform channel. The rough break-even point:

If your table would sit above ~15 – 18% of its provisioned peak on average, provisioned + autoscaling is cheaper. Below that, on-demand starts to win.

The math: on-demand costs ~6.9x per write vs a fully-utilized WCU. If you provision for peak and only use 1/6.9 of it on average (~14.5%), you break even. Add autoscaling and you can push effective utilization to 40 – 60%, which is where provisioned really shines.

Here's the quick check we now run before choosing a mode, using the AWS CLI against CloudWatch:

# Get consumed WCU for the last 14 days, 1-hour granularity
aws cloudwatch get-metric-statistics \
  --namespace AWS/DynamoDB \
  --metric-name ConsumedWriteCapacityUnits \
  --dimensions Name=TableName,Value=my-table \
  --start-time $(date -u -d '14 days ago' +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 3600 \
  --statistics Maximum Average \
  --query 'Datapoints[*].[Timestamp,Average,Maximum]' \
  --output text | sort

Divide the average by the maximum across a representative window. If that ratio is above ~0.2, provisioned with autoscaling is almost certainly cheaper. If it's under ~0.1, on-demand probably wins. In the middle, model both.

A back-of-envelope calculator

We use this in code review when someone proposes an on-demand table:

def monthly_cost(avg_wps, peak_wps, mode, hours=730):
    # Rough us-east-1 standard pricing, writes only.
    if mode == "on_demand":
        writes_per_month = avg_wps * 3600 * hours
        return (writes_per_month / 1_000_000) * 1.25
    elif mode == "provisioned":
        # Provision for peak with a 30% headroom, assume autoscaling
        # keeps us at ~60% of the ceiling on average.
        ceiling_wcu = peak_wps * 1.3
        avg_wcu = ceiling_wcu * 0.6
        return avg_wcu * 0.00065 * hours

print(monthly_cost(500, 1500, "on_demand"))     # ~$1,643
print(monthly_cost(500, 1500, "provisioned"))   # ~$555

These are order-of-magnitude estimates, not quotes. Real bills include GSIs, storage, backups, streams, and replicated writes for global tables. But the shape is right, and it's usually enough to end the debate in the PR.

What we did instead for the next launch

Six months later we had another launch, bigger this time. We didn't reach for on-demand. Instead:

1. Pre-warmed provisioned capacity. We manually scaled the table to ~1.5x our forecast peak an hour before the announcement, then let autoscaling take over afterward. DynamoDB provisioned capacity changes are near-instant on the way up; the delay is only on scale-down.
2. Tightened the autoscaling target from 70% to 60%. More headroom for burst, small cost increase.
3. Enabled the built-in burst capacity awareness. DynamoDB gives you up to 5 minutes of unused capacity as burst credit per partition. We stopped treating it as free insurance and started monitoring ConsumedWriteCapacityUnits at 1-minute granularity so we'd see partition-level hot spots before throttling started.
4. Set a hard billing alarm at 1.5x forecast so a runaway loop couldn't quietly burn a weekend.

The launch went fine. Cost was ~$420 for launch week versus ~$1,780 the previous time. Same reliability outcome.

When on-demand is still the right answer

We kept on-demand for two tables:

• A rarely-used admin audit log — writes maybe 20 times an hour, but occasionally 500 in a burst during a bulk import. Peak-to-average is roughly 100x. On-demand costs pennies.
• A short-lived table backing a webhook receiver for a third-party integration that fires unpredictably. We genuinely don't know the shape, and the volume is low enough that even a bad month is under $50.

Both match the profile: low absolute volume, high variance, no meaningful floor. That's the on-demand niche. It's real, it's just narrower than the marketing suggests.

Where we'd start

If you're staring at a DynamoDB bill that's grown faster than your traffic, do three things this week:

1. Pull 14 days of ConsumedWriteCapacityUnits and ConsumedReadCapacityUnits per table at 1-hour granularity. Compute the average-to-peak ratio.
2. For any table above 15% average utilization on on-demand, model the switch to provisioned with autoscaling using real numbers, not vendor calculators.
3. For any table on provisioned that's throttling, don't jump to on-demand — check whether it's a hot partition first. On-demand doesn't fix hot partitions, it just makes them more expensive.

On-demand isn't a mistake. Treating it as the default because capacity planning feels tedious is. Ten minutes with CloudWatch and a spreadsheet will pay for itself many times over — and if you'd rather have someone else do that analysis, our team does this kind of cloud cost and reliability work regularly.