The CloudWatch Logs Bill That Doubled Overnight: Tracing a Rogue Lambda Retry Storm

A client pinged us on a Monday morning: their AWS bill was tracking 2.1x the previous week, and the entire delta was CloudWatch. No new features had shipped. No traffic spike. Just a quietly broken Lambda chewing through log ingestion while everyone slept.

This is the postmortem — the actual queries we ran, the misconfiguration that caused it, and the three guardrails we now put on every Lambda we deploy.

The symptom: $400 to $900 in 48 hours

The account was a mid-sized SaaS backend. Roughly 30 Lambdas, an Aurora Postgres cluster, SQS between most services, and a smattering of Step Functions. CloudWatch Logs usually sat around $12–$15/day in ingestion plus storage. On the Saturday it jumped to $180. Sunday was $220. Nobody noticed because nobody looks at Cost Explorer on weekends.

The billing breakdown told us where to look but not why:

• AmazonCloudWatch – DataProcessing-Bytes was 94% of the delta
• Log storage was flat (we had 7-day retention on most groups)
• No new log groups had been created

So: existing log groups, suddenly ingesting a lot more data. That narrows it enormously.

Finding the noisy log group

CloudWatch doesn't give you per-log-group ingestion in the console in any useful way. You have to query it. The fastest path is the IncomingBytes metric in the AWS/Logs namespace, which is published per LogGroupName dimension.

We ran this via the AWS CLI to rank log groups by ingestion over the previous 24 hours:

aws cloudwatch get-metric-data \
  --start-time $(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --metric-data-queries file://queries.json \
  --query 'MetricDataResults[*].[Label,Values[0]]' \
  --output table

With queries.json iterating over every log group's IncomingBytes sum. One group dominated: /aws/lambda/invoice-webhook-processor, with roughly 180 GB ingested in 24 hours. The baseline for that function was about 400 MB/day. That's a 450x jump.

The root cause: a retry storm masquerading as normal traffic

invoice-webhook-processor consumed an SQS queue of webhook deliveries from a payment provider. A recent change — merged Friday afternoon, naturally — had updated the handler to validate a new field on the incoming payload. The validation threw a TypeError when the field was missing, which it was on about 12% of messages (older webhook versions still in flight).

Three things combined to turn a bug into a bill:

1. No DLQ configured on the Lambda event source mapping. SQS kept redelivering the failed messages until maxReceiveCount was hit. The queue's redrive policy pointed at a DLQ that had been deleted months earlier during a cleanup. Messages just recycled.
2. batchSize: 1 and maximumRetryAttempts not set at the function level. Each failure was a full invocation with a full stack trace logged.
3. The stack trace was huge. The validation library serialized the entire incoming payload into the error message. Each failed invocation produced about 8 KB of logs. Multiply by millions of retries.

The math checks out. ~12% failure rate × ~400 msg/min baseline × redelivery loop every 30 seconds × 8 KB per log entry × 48 hours ≈ the 180 GB we saw.

Why nothing alerted

There were alarms. They just measured the wrong thing.

• A CloudWatch alarm on Lambda Errors existed but was set to > 50 in 5 minutes. It was permanently in ALARM state before the incident too, because the function had always had a steady trickle of errors. Alert fatigue had muted it in Slack weeks earlier.
• SQS ApproximateNumberOfMessagesVisible was alarmed at 10,000. The queue stayed around 2,000–3,000 because messages were being consumed (and failing) fast enough to not pile up.
• No alarm on IncomingBytes for any log group. Nobody we've worked with has one by default. This is the gap.

The fix, in order of what we did

1. Stop the bleeding

We disabled the event source mapping. Queue backed up, but ingestion dropped to zero within two minutes. Then:

aws lambda update-event-source-mapping \
  --uuid <mapping-uuid> \
  --enabled false

2. Add a real DLQ and sane retry limits

We recreated the DLQ, pointed the source queue's redrive policy at it with maxReceiveCount: 3, and set maximumRetryAttempts on the function itself as a belt-and-braces measure. In Terraform:

resource "aws_sqs_queue" "webhook_dlq" {
  name                      = "invoice-webhook-dlq"
  message_retention_seconds = 1209600
}

resource "aws_sqs_queue" "webhook" {
  name = "invoice-webhook"
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.webhook_dlq.arn
    maxReceiveCount     = 3
  })
}

3. Fix the log volume per failure

The handler was logging the full payload on every error. We changed the error path to log a hash of the payload plus the first failing field, and dropped the full payload to a sampled 1-in-100 debug log. This alone would have cut ingestion by about 95% even if the retry storm had continued.

try {
  await processInvoice(payload);
} catch (err) {
  logger.error({
    msg: 'invoice processing failed',
    payloadHash: sha256(JSON.stringify(payload)).slice(0, 12),
    errorType: err.constructor.name,
    errorMessage: err.message,
    // full payload only on 1% of errors
    ...(Math.random() < 0.01 ? { payload } : {}),
  }, 'invoice.failed');
}

4. Alarm on ingestion, not just errors

The guardrail we now ship on every account is a CloudWatch alarm on IncomingBytes per log group, with a threshold set to roughly 5x the 30-day p95. It's noisy at first but it catches exactly this failure mode.

resource "aws_cloudwatch_metric_alarm" "log_ingestion_spike" {
  for_each            = toset(var.lambda_names)
  alarm_name          = "log-ingestion-spike-${each.key}"
  metric_name         = "IncomingBytes"
  namespace           = "AWS/Logs"
  statistic           = "Sum"
  period              = 300
  evaluation_periods  = 2
  threshold           = var.ingestion_threshold_bytes[each.key]
  comparison_operator = "GreaterThanThreshold"
  dimensions = {
    LogGroupName = "/aws/lambda/${each.key}"
  }
  alarm_actions = [var.ops_sns_topic_arn]
}

Lessons we'd generalize

Error logs are an unbounded cost vector

If a single failure costs 8 KB of logs and you have no cap on retries, your log bill is bounded by whatever a broken deploy can produce in the time it takes a human to notice. That window is longer than most teams admit — ours was ~40 hours. Treat log volume per error path the way you treat memory allocation in a hot loop.

Errors alarms rot

Any alarm that's been in ALARM state for more than a few hours is dead to your team. If your baseline error rate is non-zero, alarm on rate-of-change or on a dependent metric (ingestion, DLQ depth, downstream latency) instead of a raw count.

DLQs need to be in IaC with explicit references

The deleted DLQ was deleted because it wasn't in Terraform — it had been created in the console years ago. Any resource that silently absorbs failures (DLQs, backup buckets, replica queues) should be in the same module as the thing it protects, so you can't delete one without the other screaming at plan time.

CloudWatch is a fine default, but know its pricing shape

Ingestion in us-east-1 is roughly $0.50/GB at the time of writing. That sounds cheap until a loop multiplies it. If a log group routinely ingests more than ~50 GB/month, we usually route it to S3 via a subscription filter and keep CloudWatch for the last 24 hours of hot queries. The economics flip hard around that volume.

Where we'd start

If you've never looked at per-log-group ingestion on your AWS account, do it today. The CLI snippet above runs in under a minute. Rank your log groups by IncomingBytes sum over the last 7 days. The top three are almost always more expensive than you think, and one of them is probably logging more than it should on an error path.

After that: audit every Lambda-to-SQS mapping for a configured DLQ, a sane maxReceiveCount, and a maximumRetryAttempts on the function. If you want help wiring the alarms and IaC guardrails across an existing account, that's the kind of work our DevOps and cloud team does regularly — usually as a one-week audit before anything else.

The bug was a missing field validator. The bill was a missing DLQ. Those are two different problems, and only one of them had an owner.