A Month of On-Call Taught Me More Than Two Years of Sprints

The on-call schedule was introduced at our company when we moved to continuous deployment and couldn't guarantee that any given release would behave the same in production as it had in staging. Someone needed to be reachable when things went sideways. The rotation was one week per engineer, roughly every ten weeks. I had been on the team for two years before my first rotation came up.

I had been writing production software for two years without anyone ever seriously asking me to think about what happened to it after it shipped. Sprint reviews, code reviews, QA cycles, staging deployments, none of these had given me a concrete sense of what my services looked like at 2am on a Tuesday when traffic patterns shifted in ways nobody had anticipated and something started failing silently.

My first week on call changed that. Not because it was a dramatic week, it wasn't. But because the perspective shift was immediate and permanent.

The First Page

PagerDuty fired at 02:47am on the Thursday of my first rotation. The alert title was: p95 latency on order-service exceeds threshold (2000ms). I was awake within about ten seconds and staring at my laptop screen within two minutes.

I had written tests for the order service. I had deployed it. I had watched it work in staging. What I had never done was look at it from the perspective of someone trying to understand why it was misbehaving at 2:47am with no prior context and no one else online.

The first thing I discovered: I didn't know where the relevant logs were. The service was deployed on Kubernetes and the logs were in Elasticsearch, but I didn't know the index name or the filter syntax. I spent six minutes finding the right logs, which is six minutes during which whatever was affecting users was continuing to affect them.

The second thing: the logs showed that requests to one specific endpoint were timing out, but they didn't show why. The log messages said "request timed out" with a duration. They did not say what internal step had timed out, which downstream service it was waiting for, or whether the timeout was new (suggesting a recent change had caused it) or ongoing (suggesting a pre-existing condition we simply hadn't noticed).

The third thing: there was no runbook for this service. A runbook is a document that says, in plain language: here's how this service works, here's how to read its metrics, here's the list of things to check when it starts misbehaving, here's how to roll back, here's who to call if you can't fix it yourself. Every mature on-call setup has runbooks. We didn't have any, because nobody had needed them before.

I eventually traced the latency to a connection pool exhaustion in a downstream database, too many concurrent requests were holding connections open past their expected duration because of a query that had started running slowly after a schema change three days earlier. The fix was increasing the connection pool timeout so that requests failed fast rather than queuing indefinitely, buying time for the team to address the underlying slow query the next morning. From alert to resolution was 51 minutes. A runbook and clearer log messages could probably have cut that to 15.

What I Learned About Logging

Before on-call, I wrote logs when I thought something interesting might happen. After on-call, I write logs as if someone I don't know will need to diagnose a problem at 3am with no other context available.

That's a different mental model. It changes what you log, how you structure messages, and what you include in the context. A log message that says "payment failed" is not useful at 2am. A log message that says "payment failed: PaymentProcessor returned TIMEOUT after 5023ms; order_id=ORD-8812377, customer_id=CUS-442011, payment_method=CARD, idempotency_key=KEY-8812377-1" gives you something to work with.

Structured logging, where each log entry is a JSON object rather than a free-form string, sounds like over-engineering until the first time you need to aggregate all payment failures across ten service instances and correlate them with latency data from the same time period. Free-form strings require regex parsing that fails when the format changes slightly. Structured fields are queryable directly.

Trace IDs, a unique identifier attached to every request that flows through every service that handles it, sound theoretical until the first time you're looking at an error in service C and need to find the originating request in service A that caused it. Without a trace ID, you're correlating by timestamp and approximate request counts, which is unreliable under concurrent load. With a trace ID, you can follow a single request across a dozen service hops in seconds.

These are not abstract best practices. They are things that directly determine how long you spend awake at 3am when something breaks.

The Difference Between Cause and Trigger

One of the most useful distinctions I learned from the postmortem process is the difference between the trigger of an incident and its cause.

The trigger is the thing that happened immediately before the incident started: a deployment, a configuration change, a traffic spike, a third party service going offline. The trigger is what you fix to restore service. The cause is the underlying condition that made the system vulnerable to that trigger, the missing index that made a query slow under load, the absence of a circuit breaker that allowed a slow dependency to exhaust connection pools, the lack of graceful degradation that turned a partial failure into a complete one.

Postmortems that focus on the trigger miss the point. "We had a traffic spike and the service couldn't handle it" is a trigger-level analysis. "We had a traffic spike and the service couldn't handle it because the connection pool was sized for normal traffic patterns and had no mechanism to fail fast when exhausted" is a cause-level analysis. The difference matters because fixing only the trigger means the same incident will happen again the next time the same conditions occur. Fixing the cause means it won't.

Blameless postmortems, a practice where the goal is to understand what happened and improve the system, not to identify and punish whoever made the mistake, make cause analysis possible because they create the psychological safety to say "here's what I misunderstood about how this system would behave" without fear of career consequences. Teams that skip the blameless part get trigger-level analyses and repeated incidents. This isn't a moral argument. It's a practical one.

What Changed in How I Write Code

After my first rotation, I started asking a question I hadn't asked before when I finished implementing a feature: if this breaks at 3am and I'm the one on call, what do I need to know to fix it in under fifteen minutes?

That question led to better health check endpoints. It led to metrics being exported for every external call, not just the ones I thought were likely to be slow. It led to timeout handling being explicit in every place my service called something else, rather than relying on default timeout behavior that I hadn't investigated. It led to circuit breakers on integrations with third party services that had historically been unreliable. It led to runbooks, not elaborate ones, just enough to tell a future on-call engineer where the logs are, what the most common failure patterns look like, and how to roll back.

None of this made the code more complex to write. Most of it made it simpler, because explicit failure handling is easier to reason about than implicit. What it required was a shift in when I considered a feature "done", not when it worked correctly under normal conditions, but when it would fail gracefully and diagnoseably under abnormal ones.

The Thing About Being On Call

Being paged at 2:47am is not inherently valuable. Being paged at 2:47am and then improving the system so the next on-call engineer has a better experience, shorter diagnosis time, clearer failure signals, runbooks that actually help, is how on-call rotations make software better over time rather than just redistributing the pain.

The best on-call cultures I've observed treat every incident as an investment opportunity. Something failed. Someone woke up. Now you know something you didn't know before about how the system behaves under that condition. You can either respond by fixing the immediate problem and moving on, or you can fix the immediate problem and then use what you learned to make the system more resilient, more observable, or easier to operate. The second option costs more upfront. Over a year of rotations, it compounds into a system that wakes people up less often and for shorter periods when it does.

Two years after my first rotation, our p95 alert threshold was the same. Our mean time to resolution had dropped by about 60%. Most of that improvement came from runbooks, structured logging, and circuit breakers added by engineers who had recently been on call and knew exactly what information they needed and hadn't been able to find.

I've worked with engineers who avoid on-call rotations when they can. I understand the instinct. The hours are inconvenient and the stress is real. What I would tell someone starting in this career is that the fastest way to understand what your software actually does in production, not what you think it does, not what it does in staging, but what it does when real users are hitting it under real conditions, is to be the person who gets called when it stops doing what it's supposed to.

The education is expensive. It is also thorough.