All articles
DevOps & CloudNovember 11, 2026 6 min read

CloudWatch to Grafana Cloud: What Our Logging Bill Actually Did

We moved 2.4 TB/month of application logs off CloudWatch Logs onto Grafana Cloud. The bill dropped, but not by as much as the slide decks promised, and two things broke in ways we didn't expect.

Someone on the finance side flagged that CloudWatch Logs was quietly the third-largest line item on our AWS bill. Not compute, not egress — logs. We spent a quarter moving the bulk of our application logs to Grafana Cloud Loki, and the result was useful but messier than the vendor comparison tables suggest.

This is what actually happened, what the real numbers looked like, and where we'd tell you to stay put.

Why CloudWatch Logs got expensive in the first place

CloudWatch Logs pricing has three parts that stack up fast: ingestion (around $0.50/GB in us-east-1 at the time of writing), storage (around $0.03/GB-month), and — the one people forget — Logs Insights scans, charged per GB scanned per query. If you have a dashboard that runs a Logs Insights query every 60 seconds across a 7-day window, you are paying for that window on every refresh.

Our breakdown before the migration, roughly:

  • Ingestion: ~$1,200/month at 2.4 TB
  • Storage: ~$650/month (we kept 90 days hot)
  • Logs Insights scans: ~$900/month, driven mostly by three dashboards nobody loved but everyone opened during incidents

Call it $2,750/month for logs alone, not counting metrics or X-Ray. The Insights number was the surprise. A single engineer with a bad fields @timestamp, @message | filter habit on a 30-day window can quietly burn $50 in an afternoon.

What we considered before touching anything

Before picking a new vendor we actually tried to lower the CloudWatch bill in place. Two things helped before we committed to a migration:

  1. Log group retention audits. About 40% of our log groups had no retention set, which defaults to "never expire". Setting sane retentions dropped storage cost immediately.
  2. Subscription filter to drop noise at ingest. A Lambda in front of a few chatty services stripped health-check lines before they hit CloudWatch.

That got us from roughly 3.1 TB/month to 2.4 TB. Useful, but not transformative, and it didn't touch the Insights scan cost.

Why Grafana Cloud, and why not Datadog or self-hosted Loki

We looked at four options: stay on CloudWatch, Datadog Logs, Grafana Cloud (managed Loki), and self-hosted Loki on EKS.

  • Datadog has the nicest UX of the four. Pricing starts reasonable and becomes aggressive once you add indexed log retention and Live Tail seats. For our volume, the quoted annual contract was more than we were paying CloudWatch.
  • Self-hosted Loki is cheap in software license terms ($0) and expensive in engineer-hours. We've run it before. S3 as the backend works well; the operational tax of ingesters, compactors, and the query frontend during traffic spikes does not.
  • Grafana Cloud sits in the middle: managed Loki, pay for ingested GB and active series, with a free tier that's generous enough to validate the pipe before committing.

We picked Grafana Cloud with a one-year commit at a middle tier. No discount theater — we negotiated once, got a reasonable number, and moved on.

The actual migration shape

The pipeline we ended up with:

App containers (ECS Fargate)
  └─> stdout
       └─> FireLens (Fluent Bit sidecar)
            ├─> CloudWatch Logs  (critical groups only, 14-day retention)
            └─> Grafana Cloud Loki (everything, via HTTP push)

We did not cut CloudWatch cold. For the first eight weeks we dual-wrote. That cost us money but it caught two issues that would have been nasty in a one-way migration.

The Fluent Bit config that actually shipped

Trimmed, but close to what runs in production:

[OUTPUT]
    Name            loki
    Match           app.*
    Host            logs-prod-us-central1.grafana.net
    Port            443
    TLS             On
    HTTP_User       ${LOKI_USER}
    HTTP_Passwd     ${LOKI_API_KEY}
    Labels          job=fluentbit, env=prod, service=$kubernetes['labels']['app']
    Label_Keys      $container_name,$trace_id
    Line_Format     json
    Auto_Kubernetes_Labels off

[OUTPUT]
    Name            cloudwatch_logs
    Match           app.critical.*
    region          us-east-1
    log_group_name  /ecs/critical
    log_stream_prefix from-fluentbit-
    auto_create_group true

Two things worth calling out. First, Label_Keys is where people blow up their Loki bill. Every unique label value creates a new stream, and high-cardinality labels (like trace_id or user_id) will murder your active series count. We learned this the expensive way — more below. Second, keeping a narrow CloudWatch output for critical services means AWS-native tooling (EventBridge rules, metric filters, GuardDuty) still works where it needs to.

What the numbers actually looked like after

After the dual-write period ended and we tuned labels properly, our monthly logging spend settled at roughly:

  • Grafana Cloud: ~$1,100/month for ~2.3 TB ingest and the retention tier we picked
  • CloudWatch Logs (reduced scope): ~$280/month
  • Total: ~$1,380/month

So we went from roughly $2,750 to roughly $1,380. A ~50% reduction, not the 80% the migration doc promised. The gap between promise and reality came from two places: the critical CloudWatch footprint we kept on purpose, and the engineering time we had to spend on label hygiene we hadn't budgeted for.

If you amortise the migration work over a year, the payback was around five months. Reasonable, but not life-changing for a mid-sized team.

Where we actually got value beyond cost

The surprise win wasn't cost. LogQL plus Grafana dashboards made ad-hoc log exploration faster than Logs Insights, mostly because iterating on a query doesn't cost money per attempt. Engineers stopped flinching when they opened the logs tool. That cultural shift mattered more than the dollars.

Two things that broke

1. Active series explosion from a well-meaning label

About three weeks in, our Grafana Cloud usage dashboard showed active series climbing linearly. A service team had added request_id as a Loki label, thinking it would help with searching. Each request_id is unique, so every request was creating a new stream. We were on track to hit 2 million active series before the end of the month, well above our plan.

The fix: request_id belongs in the log line, not as a label. You search it with a LogQL filter (|= "request_id=abc123"), not as a stream selector. We added a CI check that lints Fluent Bit configs for suspicious high-cardinality label keys.

2. CloudWatch metric filters we forgot about

Three alarms were driven by CloudWatch metric filters sitting on log groups we stopped writing to. The alarms didn't fire for a week because the metric just... stopped producing data points. CloudWatch treats missing data as "not breaching" by default. We only noticed because an unrelated runbook pointed at one of the dashboards.

If you move logs, inventory every metric filter and alarm tied to those log groups first. Either recreate the signal in Grafana (with an alert on a LogQL query) or keep enough of the log flowing to CloudWatch to feed the filter.

When we'd tell you to stay on CloudWatch

This migration made sense for us. It would not make sense for every team. Stay on CloudWatch if:

  • Your log volume is under ~500 GB/month. The CloudWatch bill is annoying but not worth a migration quarter.
  • Your team lives inside the AWS console and your on-call runbooks are built around Logs Insights syntax.
  • You rely heavily on CloudWatch-native integrations: metric filters, Contributor Insights, subscription filters to Kinesis, EventBridge patterns on log events. Rebuilding those elsewhere is real work.
  • You don't already run Grafana for metrics. Adopting Grafana purely for logs means a second tool for your team to learn.

The break-even in our experience sits somewhere around 1 TB/month of ingest plus heavy Logs Insights usage. Below that, the juice is rarely worth the squeeze.

Where we'd start

If you're staring at a CloudWatch bill that's bigger than it should be, do these in order before you shop for a new vendor:

  1. Run a retention audit this week. Set 30 or 90 days on everything, 7 days on dev.
  2. Find your top five Logs Insights queries by scan volume. Cache them as dashboards with narrower time windows, or kill them.
  3. Drop known-noisy lines (health checks, readiness probes) at the source with a subscription filter or Fluent Bit filter.
  4. Only after those three, price out Grafana Cloud or Datadog with your real post-cleanup volume — not your current bloated number.

If you'd like help sizing a migration like this or auditing an observability stack before you commit, that's the kind of work our DevOps and cloud team does day to day.

#AWS#Observability#Grafana#Cost Optimization#DevOps

Want a team like ours?

72Technologies builds production software for the kind of teams who actually read this blog.

Start a project