I Deployed to Production on My Second Day. Here's What Happened.

It was my second day. I had been given access to the deployment pipeline because the senior engineer said it would be useful to see how it works. I was told to run a test deployment on the feature branch to check the build steps. Nothing consequential. Just observe the process.

The pipeline had two environments listed in a dropdown: staging and production. They were the same visual weight, the same font size, the same distance apart. I selected what I thought was staging. The labels were ambiguous enough that I was genuinely uncertain which I had clicked even as I clicked it. I was wrong.

The Moment You Know

There is a specific silence that falls over an engineering floor when something breaks in production. It is not dramatic. It is the silence of people stopping what they are doing and paying attention to something they weren't paying attention to a moment ago. Two people make eye contact across the room. Someone starts typing faster. Someone else picks up their phone and then puts it down again.

That silence fell about four minutes after I clicked deploy. The monitoring dashboard started showing elevated error rates. The senior engineer looked at the logs, looked at the deployment history, looked at me with an expression that was less angry than I expected and more tired than I was prepared for.

"Did you just deploy to prod?"

I had. And in that moment I understood the difference between knowing that production is important and understanding what it means for something to be live.

What Actually Broke

The build I deployed was a work-in-progress. It was missing an environment variable that the new feature used to resolve a downstream service endpoint. In staging, that variable existed. In production, it didn't, because nobody had added it yet, the feature wasn't supposed to be anywhere near production.

With the variable missing, every request that touched the new code path was returning a 500 error. We had roughly 2,000 active users at the time. Not all of them were hitting the affected endpoint, but enough were that the error rate spike was immediately visible on the dashboard.

The fix took eleven minutes. Roll back the deployment to the previous build, wait for the rollback to propagate, watch the error rate fall back to baseline, verify nothing else had broken in the process, write a brief incident note. From a systems perspective it was a small incident, under fifteen minutes of degraded service, no data loss, no financial transactions affected, no customer data compromised.

From the perspective of being the person who caused it while still learning where the bathrooms were, it felt considerably larger. The walk from my desk to the senior engineer's felt longer than it was.

What My Manager Said

I expected some version of anger, or at least the pointed disappointment that is somehow worse. What I got was a ten-minute conversation that I have thought about many times since.

My manager's position was this: the deployment pipeline should not have been accessible to someone on their second day without a supervised walkthrough. The environment dropdown should have been visually distinct, production should look different from staging, not just have a different label in the same font. The pipeline should have required a confirmation step for production deployments. None of those safeguards existed, and their absence was a design failure that predated me by months.

"This is our fault for giving you access without a walkthrough," she said. "Write up what happened and what confused you. That's your action item."

I wrote a one-page document that evening. It described the sequence of events, what I had understood the task to be, what the UI showed me, what I had expected to happen, and what actually happened. I included a screenshot of the dropdown as it appeared to me at the time.

Two weeks later, the deployment UI had been redesigned. Production was now visually marked with a red indicator. Deploying to production required typing the environment name into a confirmation field, a pattern borrowed from how database deletion prompts work. Three weeks after that, a different engineer told me they had caught themselves mid-deployment to production because of the confirmation step. They hadn't realised they had selected the wrong environment until the prompt asked them to confirm it.

The mistake cost the team eleven minutes of incident response. The write-up cost me an evening. The UI change probably saved a more serious incident later.

What Breaks Production (Usually)

In the years since, I have been involved in a number of production incidents, some caused by me, some by others, some by the interaction between two systems that each worked fine in isolation. The ones caused by deliberate, carefully planned changes are genuinely rare. Most production breaks come from a predictable set of patterns.

Missing environment variables. The code works in staging because staging has the variable configured. Production doesn't, because whoever added it to staging forgot to add it to the production configuration, or because the deployment checklist that was supposed to cover this wasn't followed, or because there wasn't a checklist. Every deploy pipeline should validate required environment variables before marking a deployment as successful. This is not difficult to implement and it catches this failure mode entirely.

Schema migrations running against live traffic. Adding a nullable column to a large table is usually fine. Renaming a column while the old code is still running against it is not, the old code looks for the old name and finds nothing. Removing a column before every service that reads it has been updated to stop reading it is not. The order of operations in database migrations matters enormously, and the correct sequence, deploy the new code that handles both old and new schema, then migrate, then remove the compatibility layer, is not intuitive until you have been burned by skipping it.

Cached assumptions becoming invalid. A service reads a piece of configuration or a data shape and caches it to avoid repeated lookups. The underlying data changes. The cache doesn't know. Requests continue using the stale cached version until the cache expires or is explicitly cleared. This is particularly insidious because the service appears to be working, it's returning responses, just wrong ones. Cache invalidation being hard is a cliché because it is consistently, actually hard.

Human interface errors. Like mine. Someone selects the wrong environment, runs a script against the wrong cluster, passes the wrong argument to a command that does something irreversible. These failures are not caused by incompetence, they are caused by tooling that makes the mistake easy to make. The solution is not to hire more careful people. It is to design tooling that makes the consequential action harder to do accidentally than the safe one.

What to Do When You Break Production

If you are reading this in the middle of a production incident you caused, here is the sequence that matters:

Say so immediately. Do not wait until you are certain. Do not try to fix it quietly before anyone notices. The moment you suspect you have caused a problem in a live environment, tell someone, your team lead, the on-call engineer, whoever is the right person to know. Every minute of silence is a minute the team could be helping, and silence during an incident is often interpreted as concealment even when it isn't.

Rollback before you investigate. Unless the rollback itself carries significant risk, a database migration that cannot be reversed, for example, or a data transformation that has already processed records that the old code can't handle, the default response to a deployment-caused incident is to undo the deployment. Get the system back to the last known good state first. Understand what went wrong second. The temptation to fix forward rather than roll back is understandable, but it often makes incidents longer.

Write down what happened and when. Even during the incident, keep a running note: what you did, at what time, what the system did in response, what you did next. "14:32, deployed build 447 to production. 14:36, error rate increase observed on dashboard. 14:39, identified missing environment variable as likely cause. 14:43, initiated rollback to build 446. 14:47, error rate returned to baseline." This is the incident timeline. You will need it for the post-mortem, the incident report, and your own memory of events when you are asked to reconstruct them a week later.

Apologise once, then focus. Acknowledge what happened. Don't spend the next twenty minutes apologising while the incident is still active. The team needs you thinking clearly about the problem, not processing guilt about having caused it. Save the retrospective conversation for after the system is stable. The best thing you can do for everyone, including yourself, is to be useful.

The Long View

Every experienced engineer I have worked with has a production incident story. Usually more than one. The ones who are most trusted in a crisis are not the ones who have never broken anything. They are the ones who, when something breaks, stay calm, communicate clearly, work methodically toward resolution, document what happened, and make sure the conditions that caused the failure become harder to reproduce.

Breaking production on my second day was not a good start. But the write-up I produced that evening, the document that led to a confirmation step being added to the deployment pipeline, which caught at least one serious mistake before it happened, is probably the most directly useful thing I did in my first month.

The incident lasted eleven minutes. The change it triggered outlasted my time at the company.

Sometimes the mistakes are the work. Not because mistakes are good, but because a mistake handled well, documented honestly, and used to improve the system is worth considerably more than a near-miss that nobody told anyone about.