On Grafana alerting, PromQL, and the difference between a system that is monitored and a system that can actually speak.

A while back I picked up a task that sounded boring on the ticket: fill the observability gaps.

There was already an internal alerting platform. Built, deployed, service teams onboarded, dashboards everywhere. On paper we were covered. In practice, people were still opening incidents by hand, someone in a Slack channel typing "is checkout slow for anyone else?" twenty minutes before anything fired. And in other corners of the estate there was nothing at all. Just logs. Or just events. Or a service that had been running in production for two years emitting absolutely nothing except stdout.

That's the job most of us actually get. Not a greenfield platform. A half-instrumented estate where the tools are fine and the rules are wrong.

Here's the thing I keep coming back to: your system is observed exactly as well as your alert configs are written. Not as well as your stack is built. You can run Prometheus, Thanos, Mimir, OTel, Alloy, the full pipeline, and still be blind, because blindness and health look identical on a dashboard nobody is staring at.

These are the six things that cost me the most to learn. Written for you so they cost you less.

01 Start with the manual incidents, not the shiny service

When you're handed "fill the gaps," the instinct is to open a blank rule file and start writing good alerts. Don't. You already have a dataset telling you exactly where the holes are, and it's free.

Pull every manually-raised incident from the last three to six months and sort them into four buckets:

• An alert existed and fired late. Your for: duration or threshold is wrong. Tuning job.

• An alert existed, fired on time, went nowhere. Routing problem. Labels, notification policy, a Slack channel nobody is in. The query was never the issue.

• An alert existed, fired on time, got ignored. This one hurts, because it means the alert is buried in noise. Go fix noise before you write anything new.

• No alert existed. The actual gap.

The distribution across those four buckets is your roadmap, and it's almost never what you assumed. On my run, bucket four was the smallest. Most of the pain was buckets two and three, alerts that technically worked and practically didn't.

The second thing to do before writing anything: inventory what signal actually exists per service. Metrics? Logs only? Events only? Nothing? Because you cannot alert on what isn't emitted, and the honest first ticket for half those services was "add a counter," not "add an alert." That's not a detour. That is the gap.

Worth the effort

Every manual incident is a free bug report on your alerting, written by someone who was there. It's the highest-signal input you will ever get, and most teams throw it away by closing the ticket and moving on.

02 Alerts overfit and underfit, exactly like models do

This framing helped me more than any monitoring blog post, so I'll hand it over.

Underfit alerts are the ones everyone complains about. Threshold too loose, window too short, scope too wide. CPU > 90% for 1m on every pod in the fleet. It fires constantly, it's right roughly never, and after three weeks your on-call has a Slack filter that hides it. That alert is now worse than having no alert, because the org believes it's covered.

Overfit alerts are the quiet ones and almost nobody talks about them. Someone had a bad Tuesday, did a postmortem, and wrote a rule so precisely shaped around that one incident that it will never fire again:

# tuned to a single incident that will never repeat in this exact shape
avg by (pod) (rate(container_cpu_usage_seconds_total{
    namespace="payments",
    pod=~"checkout-v2-.*",
    node="node-17"
}[3m])) > 0.937

It looks fantastic in the repo. It is dead weight. Worse, it produces false confidence "we have an alert for that" and nobody checks that the alert is structurally incapable of firing.

The tell is easy to check. Look at your firing history and find every rule with zero firings in six months. Each one is either overfit, broken, or genuinely guarding something you never breach. It is almost never the third.

The middle ground is boring and correct: alert on a symptom your user actually feels, scoped to a level you can actually act on. Not CPU. Not one pod. Error ratio per service, latency per endpoint, queue not draining, data gone stale.

Careful here

Adding labels to a noisy query feels like tuning. Sometimes it's just overfitting with extra steps. Ask yourself honestly: am I narrowing the scope because this alert genuinely only matters in this scope, or because I want the pages to stop? Those are very different commits with very similar diffs.

03 Noise isn't too many alerts. It's the same truth told too many times

This distinction took me embarrassingly long. Two hundred pods failing the same way is not two hundred problems. It's one problem, reported two hundred times. Fixing noise is mostly about collapsing repetition, and PromQL gives you the tools if you're deliberate about it.

Aggregate to the level you'd actually act on

# noisy: one series per pod. 200 pods = 200 notifications
rate(http_requests_total{status=~"5.."}[5m]) > 0

# what you actually want to know
sum by (cluster, service) (rate(http_requests_total{status=~"5.."}[5m]))
  /
sum by (cluster, service) (rate(http_requests_total[5m]))
  > 0.05

Aggregating up throws away the pod label, and that's fine. The pod label belongs in the dashboard you open after you're paged, not in the page itself.

Use group_left() to carry context without fanning out

The usual objection to aggregating is "but then I lose the team and ownership labels I need for routing." You don't. Join them back from a metadata metric:

(
  sum by (cluster, service) (rate(http_requests_total{status=~"5.."}[5m]))
    /
  sum by (cluster, service) (rate(http_requests_total[5m]))
    > 0.05
)
* on (cluster, service) group_left(team, tier, runbook)
  service_metadata{env="prod"}

One series per service, carrying team, tier and runbook so your Grafana notification policy can route it properly. This single pattern removed more noise for me than any threshold change, because it let me aggregate aggressively without losing the routing I needed.

Match for: to how long the system actually takes to heal itself

Here's the one that got me, and it's the reason for the card at the top of this page.

An AC power reclaim in the datacenter. Power gets pulled and restored on a rack, nodes go down, nodes come back, retries fire, things settle. Normal, planned, understood by everyone who has been there longer than me. Total time to settle: ten to fifteen minutes.

My rule had for: 5m.

So every reclaim produced a burst of pages that resolved themselves, then fired again as the next batch of nodes cycled, then resolved, then fired. Four events that mattered turned into somewhere north of twenty notifications, in the middle of the night, all of them technically correct and all of them useless.

The fix is not clever. If the system genuinely recovers on its own in fifteen minutes, then five minutes is not an incident, it's a stage of normal operation, and your rule is reporting on the weather.

- alert: NodeUnreachableAfterPowerEvent
  expr: up{job="node-exporter"} == 0
  for: 20m
  keep_firing_for: 10m
  labels:
    severity: critical
    team: infra
  annotations:
    summary: "{{ $labels.instance }} unreachable for 20m"
    description: "Past the normal reclaim window. Check power and BMC before assuming host failure. {{ $value }} instances affected."
    runbook_url: "https://runbooks.internal/node-down"

keep_firing_for is the underrated half of that. It holds the alert open for a grace period after the condition clears, which kills the fire/resolve/fire/resolve flapping that makes an incident channel unreadable.

This is where you get the heat

If you're junior and you're about to widen a page window from 5m to 20m, do not do it alone. Get your senior, your manager, or the service owner to say yes, in writing, in a thread you can find later.

Because the query is the easy part. The hard part is the day something doesn't self-heal in twenty minutes, and the first question in the postmortem is "who widened this?" You want that thread. I promise you want that thread. Nobody blames you for asking; everybody remembers when you didn't.

Then let Grafana collapse the rest

Query-level work gets you most of the way. Notification policies do the last mile. Group by the labels that define the incident, not the ones that define the instance:

• group_by: [alertname, cluster, service] : fifty firing series arrive as one notification.

• group_wait : hold briefly so related alerts from the same event land together instead of trickling in.

• repeat_interval :if you set this to five minutes you have built a very expensive way to teach people to mute you.

• Mute timings for known windows. Nightly batch spikes consumer lag every night at 01:00? That's a schedule, not an incident. Mute it and alert on the batch failing instead.

04 Writing from scratch: watch the system before you threshold it

For the corners with nothing at all, the temptation is to copy a starter rule pack and move on. Resist for about two weeks. That's roughly what it costs to write alerts that survive contact with production.

The order that worked for me:

Find the symptom, not the cause. Ask the service team one question: "what does a user notice when this is broken?" The answers are things like payments fail, the queue stops draining, the nightly reconciliation didn't run, the dashboard is showing yesterday's data. None of those are CPU. Cause-based alerts multiply endlessly because there are infinite ways to break; symptom-based alerts stay small because there are only a few ways to be broken.

Get the metric emitted if it doesn't exist. Often the whole ticket. A counter and a last-success timestamp will carry you further than any amount of log parsing.

Then sit in Grafana Explore for a while. Thirty days back. What does a normal Tuesday look like? What does deploy day look like? What does month-end look like? What does that 03:00 spike turn out to be? Pick your threshold from your data, not from a blog post that confidently said five percent.

Prefer thresholds that age well. Static numbers rot. disk > 85% is meaningless on a 20 TB volume and screaming on a 50 GB one. Alert on trajectory instead:

# will we run out in the next 4 hours, and are we already low?
predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 4*3600) < 0
and
node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}
  / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"} < 0.15

That second clause matters. predict_linear alone will page you because someone wrote a large temp file for ten minutes.

The alert that never fires because your exporter died

This is the one I'd tattoo on people. A rule evaluating over a metric that has stopped arriving does not fire. It evaluates to nothing and sits there, silently, permanently green.

Green because healthy and green because blind look exactly the same in Grafana. There is no visual difference. You will not notice.

# is the thing that watches the thing still alive?
absent(up{job="payments-exporter"} == 1)

# did the nightly job actually succeed, or did it just stop reporting?
time() - batch_job_last_success_timestamp_seconds{job="nightly-recon"} > 26*3600
or
absent_over_time(batch_job_last_success_timestamp_seconds{job="nightly-recon"}[26h])

Every critical path needs one of these. It's the cheapest alert you will ever write and it is the one that catches the failure mode nobody plans for.

05 A dashboard panel is not an alert rule, and copy-pasting one into the other will hurt you

Everyone who moves from Grafana dashboards into Grafana alerting does this once. Here's your once, for free.

Template variables don't exist in alert rules. Your panel query is full of $namespace and $__rate_interval. Those are dashboard-time substitutions. An alert rule has no dashboard and no time picker.

# in a panel: fine
sum(rate(http_request_duration_seconds_count{namespace="$namespace"}[$__rate_interval]))

# in an alert rule: pin everything explicitly
sum by (namespace) (rate(http_request_duration_seconds_count[5m]))

Panels are range queries; rules are instant queries on a schedule. A panel shows you a spike because you're looking at a window. A rule evaluating every minute with for: 5m may never see that spike at all. If the panel and the alert disagree, the panel is usually not the one that's wrong, but it's also not the one that pages anybody.

Choose your rule type on purpose

• Grafana-managed rules give you the expression pipeline (query → reduce → threshold), multi-datasource conditions, and screenshots in notifications. Great for teams who live in the UI.

• Data-source-managed rules (the Prometheus or Mimir ruler) live as YAML in the same repo as your recording rules. Reviewable, diffable, and this is the part people forget they keep evaluating when Grafana is down.

Think carefully about where the rules that watch your monitoring stack should live. Putting the alert for "Grafana is unhealthy" inside Grafana is a joke that stops being funny at 02:00.

No Data and Error handling is not a footnote

It's the setting at the bottom of the form that people scroll past, and the default is to treat it as alerting. Which means a scrape gap, a datasource timeout, or a Mimir hiccup becomes a page and worse, it becomes a page on every rule at once, which is how a five-minute blip turns into an incident channel with four hundred messages in it.

Decide per rule:

• Payment path stops reporting? No Data should page. Silence there is the emergency.

• A batch job that only runs 09:00–18:00? No Data overnight is normal. Set it to OK or you've just built a nightly alarm clock.

• Error state on a flaky remote datasource? Usually OK, with a separate dedicated alert on the datasource itself.

Precompute anything expensive

If your alert query takes eight seconds to evaluate on a quiet afternoon, it will time out under load, which is precisely when you need it to work. Push the heavy aggregation into a recording rule and let the alert read a cheap series:

# recording rule: evaluated once, reused by every alert and panel
- record: slo:http_error_ratio:rate5m
  expr: sum by (cluster, service) (rate(http_requests_total{status=~"5.."}[5m]))
        / sum by (cluster, service) (rate(http_requests_total[5m]))

# fast-burn alert: budget being consumed 14.4x too quickly,
# confirmed over both a short and a long window so one blip can't trigger it
- alert: CheckoutErrorBudgetFastBurn
  expr: slo:http_error_ratio:rate5m{service="checkout"} > 14.4 * 0.001
        and slo:http_error_ratio:rate1h{service="checkout"} > 14.4 * 0.001
  for: 2m

The two-window trick is the same noise-reduction idea from section three, applied to SLOs: a short window makes it fast, a long window makes it real, and requiring both means a thirty-second blip doesn't wake anyone.

06 An alert with no owner and no next step is just a notification

You can write the best PromQL of your career and still ship something worthless. The query is maybe sixty percent of an alert. Here's the rest.

Labels are routing, and routing is the whole point. team, service, env, severity. Get one wrong and your perfect rule fires into a Slack channel that was archived last quarter. That's not a monitoring failure anyone will catch, it looks exactly like the alert working. Test the route, don't assume it.

Annotations are what the tired person reads. At 2 a.m. nobody parses a PromQL expression. Write the summary as if you're waking up a colleague who has never seen this service:

• summary : what broke and where, in one line, with the labels interpolated.

• description : the actual numbers. {{ $value }}, {{ $labels.instance }}. "Error ratio 12%" beats "error ratio high" every single time.

• runbook_url : even if the runbook is three bullets in a wiki. Three bullets beats opening Grafana and guessing.

Severity has to mean something. If everything is critical, nothing is. My working rule: critical means a human gets woken up and there is something that human can do right now. If you can't name the action, it isn't critical, it's a warning, and warnings can wait for business hours.

Review on a schedule, and delete things. Once a month, three lists: what fired the most, what fired and got closed with no action taken, and what never fired at all. The middle list is your noise. The last list is your overfitting. Deleting a dead alert is real engineering work, even though it feels like you're removing coverage. You're removing the illusion of coverage, which is more dangerous than the gap.

The social part nobody puts in the runbook

When you change someone else's alert, tell them. You are modifying the thing that wakes them up at night. Even a one-line message in their channel "widening the node-down window to 20m because of DC reclaims, shout if that's wrong for you" buys you enormous goodwill and occasionally saves you from a mistake, because they know something about their service that you don't.

What this is actually for

By the end of that quarter the numbers were fine. Noise down, coverage up, manual incidents way down. But the thing that stuck with me wasn't a metric.

You can build the most beautiful observability platform in the world. Prometheus scraping cleanly, Thanos and Mimir holding years of history, OTel traces threaded end to end, Alloy collecting everything, pipelines tuned, a model summarizing your incidents. Genuinely impressive work, and I'd enjoy building every piece of it.

And if, at 02:14, checkout is failing and nobody gets told none of it counted. Not one line.

Everything else in observability is how you investigate a problem. Alerting is how you find out there is one. It's the only part of the stack that reaches outward and taps a human on the shoulder, and it's the only part anyone outside your team will ever experience.

So write the boring rules first. Get the windows right, get the routing right, get the runbook links in. Then go build the rest of the platform, knowing your system can speak when it needs to.

If you take one thing: go find every rule that hasn't fired in six months, and check whether it can. That afternoon will teach you more about your alerting than this whole article.