Alerting on errors

Alerting on errors is the practice of automatically notifying a human when something goes wrong in production - before a customer reports it. At the basic level (L1), this means hooking your application's error output to a notification system: PagerDuty, OpsGenie, or simply a Slack channel that receives error messages. When an unhandled exception is thrown, an HTTP 500 is returned, or an error log line is written, a human gets pinged. The human then logs in, investigates, and decides what to do.

This is an enormous step up from pure reactive operations - where the first signal of a production problem is a customer complaint - but it remains fundamentally human-driven. The alert is a page to a person, not a trigger for automated investigation. The on-call engineer receives "Error: NullPointerException in PaymentService" at 2am and must manually trace what caused it, assess the impact, and determine the fix. The alerting system provides the notification; everything after that is manual.

At L1, alerts are typically coarse-grained: any error triggers a page, any 5xx rate above a threshold fires an alarm. There is little sophistication around alert severity, routing, or suppression. The same alert fires for a transient database connection blip and for a total payment system outage. This leads to the classic "alert fatigue" failure mode: teams that page too aggressively train their engineers to ignore pages, which defeats the purpose of alerting entirely.

The critical gap at L1 alerting is the absence of context in the alert. The notification tells you something broke; it does not tell you what changed, which users were affected, whether it is getting worse or better, or what the likely cause is. All of that investigation happens manually after the page fires. For AI-assisted operations, this gap is the primary bottleneck: agents can be paged just like humans, but without structured context attached to the alert, they have as little to work with as a human receiving a terse error message.

Even basic error alerting is a significant maturity step with real operational benefits:

• Shifts from reactive to proactive - the team learns about production issues from the monitoring system rather than from customer complaints, reducing the time-to-detection from hours to minutes
• Creates an incident record - alert history provides a log of production events that informs post-mortems and helps identify recurring patterns
• Establishes the on-call culture - defining who receives alerts, how they escalate, and what the response SLA is builds the operational discipline that advanced observability practices require
• Baseline for improvement - you cannot tune alerts, reduce noise, or add context to them until you have alerts running; basic alerting is the prerequisite for every more sophisticated alerting practice
• Enables agent notification - even crude error alerts can be routed to an agent as the starting signal for investigation, provided the agent has access to supporting context elsewhere

1. Integrate an error tracking tool - Sentry is the standard choice for application-level error tracking. It captures unhandled exceptions with full stack traces, groups similar errors, and provides alert routing. Add the Sentry SDK to your application in an afternoon. For infrastructure-level alerting, connect your metrics to PagerDuty or OpsGenie.
2. Define alert severity tiers - Not every error should page an engineer at 2am. Define three tiers: P1 (customer-impacting, page immediately), P2 (degraded functionality, notify during business hours), P3 (noise/investigation needed, log and review weekly). Apply these tiers to every alert before enabling it.
3. Set thresholds, not just boolean triggers - Alerting on every single 5xx response will produce noise. Alert on 5xx rate exceeding 1% of requests over a 5-minute window, or error count exceeding 10 per minute. Rate-based thresholds filter out transient blips while catching sustained problems.
4. Configure alert routing - Alerts should go to the team responsible for the affected service, not a single shared inbox. Use PagerDuty or OpsGenie routing rules to match alert source to on-call team. This prevents alert fatigue from irrelevant pages and ensures the right person investigates.
5. Add runbooks to every alert - Every alert should link to a runbook: a document that says "when this alert fires, check these things in this order." The runbook does not need to be comprehensive on day one - even a paragraph helps. This runbook becomes the context that agents will later use for automated investigation.
6. Review alert noise weekly - For the first month, review which alerts fired and whether the page was actionable. Alerts that fire frequently but never require action should be tuned or removed. Alert fatigue is a failure mode that undermines your entire observability investment.

Alerting on everything, paging on everything. The fastest way to destroy an on-call culture is to page engineers for every non-critical error. When pages fire 20 times a week for issues that resolve themselves, engineers stop treating pages as urgent. Distinguish between "this needs immediate human attention" and "this is worth knowing about" before routing any alert to PagerDuty.

No runbooks attached to alerts. An alert that says "PaymentService error rate high" with no additional context forces the on-call engineer to start from scratch every time. Even a minimal runbook - what to check, what the likely causes are, what the last time this happened looked like - dramatically reduces mean time to resolution and is the prerequisite for any agent-assisted investigation.

Not setting up alert suppression or deduplication. A single cascading failure can trigger dozens of alerts from different systems. Without deduplication, the on-call engineer receives 40 pages in 5 minutes - an unhelpful flood rather than actionable signal. Configure alert grouping in PagerDuty/OpsGenie and deduplication in your alerting rules before going live.

Forgetting infrastructure alerts alongside application alerts. Application error tracking (Sentry) captures code-level exceptions but misses disk full, memory pressure, network partition, and database connection pool exhaustion. These infrastructure-level alerts are often the root cause of application errors. A complete alerting setup covers both layers.

No post-mortem process connected to alerts. Alerts that fire and are resolved without a follow-up record create no organizational learning. When the same alert fires three times in a month, you should know that - and have documented what was done each time. Connect your alerting tool to your incident management process (even a simple Slack channel or Jira ticket) so every significant alert creates a record.