CloudWatch Logs to Grafana Loki: A Migration That Actually Paid Off

CloudWatch Logs is fine until it isn't. Ours crossed the "isn't" line somewhere around 3 TB/month of ingest, at which point the bill started outpacing our RDS spend. This is the story of moving off it — not because CloudWatch is bad, but because the economics stopped working — and the parts of the migration that we'd do differently if we ran it again.

Why we stopped tolerating CloudWatch Logs

CloudWatch Logs charges you three times: ingestion, storage, and Logs Insights scans. The ingestion line item (roughly $0.50 per GB in us-east-1, more in other regions) is what most teams notice first. What killed us was the third one: engineers running broad Logs Insights queries during incidents. A single 7-day scan across a chatty Lambda log group could ring up double-digit dollars, and during a bad week we saw four-figure query spend from panicked debugging.

We had three concrete pains:

• Query cost anxiety. Engineers hesitated before running queries. That's a cultural failure caused by a pricing model.
• No good cross-service view. Correlating ECS, Lambda, and API Gateway logs required either exporting to S3 and running Athena, or juggling multiple Insights tabs.
• Retention was all-or-nothing per log group. Fine in theory, painful in practice when you have hundreds of groups created by Serverless Framework and CDK defaults.

We evaluated OpenSearch Service (too expensive at our volume, and we've been bitten by cluster ops before), Datadog Logs (excellent product, painful pricing above ~1 TB), and Grafana Loki. Loki won on cost per GB and on the fact that we already ran Grafana for metrics.

The target architecture

We deployed Loki in simple scalable mode on EKS — three write pods, three read pods, one backend pod — with S3 as the object store and a small DynamoDB table for the ring. Promtail was our first shipper choice, but we ended up on Fluent Bit for most workloads because it handles ECS and Lambda more gracefully.

Here's the rough shape:

App containers ──► Fluent Bit (sidecar or DaemonSet) ──► Loki write ──► S3
                                                             │
CloudWatch Logs (Lambda) ──► Firehose ──► Lambda transform ──┘
                                                             
Grafana ──► Loki read ──► S3

Lambda was the awkward one. You can't run a sidecar in Lambda, and the CloudWatch Logs subscription filter to Firehose route is the least-bad option. We wrote a small transform Lambda that reshapes CloudWatch's envelope into Loki's push API format.

Fluent Bit config that actually worked

Our first Fluent Bit config lost logs during pod restarts because we didn't persist the tail position. This is the version we settled on:

[SERVICE]
    Flush         5
    Log_Level     warn
    storage.path  /var/log/flb-storage/
    storage.sync  normal
    storage.checksum off

[INPUT]
    Name              tail
    Path              /var/log/containers/*.log
    Parser            cri
    Tag               kube.*
    Refresh_Interval  5
    Mem_Buf_Limit     50MB
    storage.type      filesystem
    DB                /var/log/flb-storage/tail.db

[FILTER]
    Name                kubernetes
    Match               kube.*
    Merge_Log           On
    Keep_Log            Off
    K8S-Logging.Parser  On

[OUTPUT]
    Name        loki
    Match       *
    Host        loki-write.observability.svc
    Port        3100
    Labels      job=fluentbit, cluster=prod, namespace=$kubernetes['namespace_name'], app=$kubernetes['labels']['app']
    Line_Format json
    Auto_Kubernetes_Labels off

Two things worth calling out. First, Auto_Kubernetes_Labels off — turning this on cardinality-bombs Loki within a day because every pod name becomes a label. Second, the filesystem-backed storage means a Fluent Bit crash doesn't cost you the buffer.

The label discipline talk nobody wants to have

Loki's whole cost model rests on low-cardinality labels. If you treat labels like Elasticsearch fields, you'll build a system that's slower and more expensive than what you left. We learned this by shipping request_id as a label for exactly one afternoon before ingestion latency spiked to 30 seconds.

Rules we now enforce in code review:

• Labels: cluster, namespace, app, env, level. That's basically it.
• Anything high-cardinality (user IDs, request IDs, trace IDs) goes in the log line as structured JSON, queried via | json and |= filters.
• No labels derived from log content unless there are fewer than ~10 possible values.

This is the single biggest source of Loki disasters we've seen at other teams. If your ops person says "let's add pod_name as a label," push back.

What the numbers actually looked like

Rough monthly figures, rounded, for our workload (~3 TB ingest, ~30-day retention, moderate query volume):

Line item	CloudWatch	Loki on EKS
Ingest	~$1,500	$0 (compute-bounded)
Storage	~$100	~$70 (S3)
Query	~$400 (variable, scary)	$0
Compute	—	~$350 (EKS nodes)
Egress / misc	~$50	~$40
Total	~$2,050	~$460

Call it a 75–80% reduction, with the caveat that you're now running software instead of consuming a service. Our SRE spends maybe 2–3 hours a month on Loki upkeep in steady state, and had a rough first two weeks. If your team can't absorb that, the math changes.

The incidents we caused ourselves

The retention cliff

Our Terraform for Loki set retention_period: 720h and compactor.retention_enabled: true. What we missed: the compactor runs on a schedule, and if it falls behind (ours did after a node reschedule), it silently keeps old chunks around. We noticed when S3 costs crept up 40% over three weeks. Fix was to alert on compactor lag directly:

- alert: LokiCompactorBehind
  expr: time() - loki_compactor_apply_retention_last_successful_run_timestamp_seconds > 86400
  for: 1h
  labels:
    severity: warning

The query that ate the cluster

A well-meaning engineer wrote a Grafana dashboard with a panel like {app="api"} |= "error" over 30 days, refreshing every 30 seconds. Loki doesn't cost you per query, but it does cost you memory on the querier pods. The cluster OOMed at 2 a.m. We added max_query_length: 168h and max_query_parallelism: 32 to the limits config and moved long-range queries to a separate read path.

Lambda logs and the Firehose backpressure trap

Firehose buffers before invoking the transform Lambda. If your transform Lambda errors, Firehose retries for up to 24 hours and then dumps to an S3 error bucket. We didn't notice a schema change in a Lambda's log format for three days because the transform silently failed on a subset. Now we alarm on DeliveryToHttpEndpoint.Records divergence between input and output, and we ship the transform Lambda's own logs to Loki via a separate, simpler path so a failure there doesn't hide itself.

When we'd tell you not to do this

Be honest about your situation before you migrate:

• Under ~500 GB/month of ingest? Stay on CloudWatch. The bill isn't worth the operational surface area.
• No existing Kubernetes or Grafana practice? The learning curve will eat the savings for at least a year.
• Compliance requirements that mandate managed logging? Some auditors are easier to satisfy with a managed service. Get this in writing before you start.
• Small team without on-call rotation? Loki is fine software, but it is software you now own. It will page you.

We'd also flag that Loki's query language, LogQL, is a real skill investment. It's close enough to PromQL to be familiar, but the log-specific parts (| json, | logfmt, | pattern) take practice. Budget a week of team ramp-up.

Where we'd start if we did it again

Run both systems in parallel for at least 30 days. We rushed the cutover on one service and immediately regretted it when a debugging session needed logs that were only in the new system, which we didn't yet trust. Dual-shipping cost us maybe 15% extra during migration and saved our credibility.

Start with one noisy, non-critical service. Something with high volume and low sensitivity — a background worker, a webhook receiver. Get the label model, the retention, and the alerting right there. Then move the tier-1 services once you've been paged by Loki at least twice and know how it fails.

If you're weighing this migration for your own stack and want a second set of eyes, our DevOps and cloud team has done this pattern a few times now — happy to compare notes. And if you're earlier in the observability journey, the blog archive has more war stories in this vein.