The Day I Crashed Production (And Didn't Get Fired)

It's high time I shared a story about my worst day at CloudCore, back when I was a junior dev. I was working on our database migration tool, which Noah was supposed to hand off to me. I made a huge mistake and pushed an untested script directly to production at 4 PM on a Friday. I remember watching our main dashboard turn completely red, as if the entire system were melting down.

I immediately panicked. In my previous company, there was a terrible blame culture where managers would dismiss your explanations and shoot down any defense. I expected our engineering manager, Rachel, to step in and fire me on the spot. I thought she would second-guess my ability to write even basic code and belittle my skills in front of everyone.

Instead of shouting, Rachel did something unexpected. She came over to my desk, sat down, and spoke in a very calm, non-judgmental voice. She didn't take sides or look for someone to blame. She simply asked, 'What do we need to do to get the database back online?' Her absolute humility in that stressful moment completely shocked me.

Noah quickly stepped in to help me rollback the deployment. Suppose we had panicked and tried to hide the mistake; the outage would have lasted hours instead of fifteen minutes. Because Rachel didn't make me feel terrible, I was able to own up to my mistake immediately. I didn't feel the need to stay silent out of a fear of failure.

During the post-mortem meeting, I was terrified that there would be a personality clash or that our ideas would clash. I would rather Rachel had just given me a written warning in private than make me explain my failure to the whole team. But she had designed the meeting to focus entirely on building a learning culture.

She refused to let anyone point fingers. She wanted to normalize making mistakes so we could learn from them. It was the first time I felt safe enough to let my guard down at work. Rachel validated our concerns about the deployment process and made sure every single engineer was able to feel heard without facing interpersonal risk.

That single day changed how I work. Today, I oversee my own team of developers, and I always try to emulate Rachel's leadership style. I encourage candor and dissent, and I never dismiss anyone's ideas. We don't tiptoe around problems anymore because we know that admitting ignorance is the first step toward finding a real solution.

I am sharing this now because I want to congratulate the person who taught me what psychological safety really means. Rachel, the incredible manager who protected me and helped me grow after my biggest failure, has just been promoted to our new CTO. We are extremely lucky to have her leading us.

Read again and choose the right answer

What caused the database outage at CloudCore?

• Noah handed off a corrupted migration script to the author.
• The author deployed an untested script directly to production.
• A major server hardware failure occurred at 4 PM on a Friday.

How did Rachel react immediately after the incident occurred?

• She stepped in to write the rollback script herself.
• She asked a calm question focused on solving the problem.
• She scheduled a meeting to second-guess the author's coding skills.

Why did the author prefer a private warning over the post-mortem meeting?

• They wanted to hide their mistake from Noah.
• They were afraid of having a personality clash with Rachel.
• They dreaded explaining their failure to the entire team.

What is the main reason the author is sharing this story now?

• To complain about the old blame culture at their former company.
• To congratulate Rachel on her promotion to Chief Technology Officer.
• To explain how to rollback a database migration script safely.

Sources: photo by Serkan Dinç on Pexels