An SRE Carol

We’re probably all familiar with Dickens’ story of Scrooge and the Three Ghosts of Christmas, written all the way back in 1843. What we may not know is that ghosts providing visions and teaching lessons is still common practice today! Let’s look into the carol of an ambitious, but unreliable, tech CEO.‍

The site was down; there was no doubt whatsoever about that. And while my engineers ate free pizza and pretended to fix it, I laid back in my bed, wondering how I’d been cursed with such an incompetent team. It wasn’t so difficult, was it? The site was up, so the site should stay up. As I contemplated what sort of disciplinary methods I’d need going forward, a dreadful rattling filled the room. In front of me loomed an apparition: it looked human, but spectrally clear. It must have long since separated from its earthly form, but the face immediately called a name to my mind.“

Jacob, is that you? My old cofounder Jacob?” I stammered. “But that’s impossible – you left the Valley years ago!”

“Seven years ago!” the ghost howled. “Seven years ago, on this day, you drove me out of our venture! Seven years of wandering the desert, alone and unfunded!”

“But Jacob, that’s business. Surely you understand!” I shouted. “When something goes wrong, someone must take the blame!”

“Be at ease. I visit you tonight not for revenge, but in warning. Unless you change your ways, you are fated to be as destitute as me!” Jacob’s ghost exclaimed.

“Change my ways? What do you mean?”

“Tonight, you will be haunted by three ghosts. Each will show you a vision of –” the ghost continued, but I waved him away.

“No, no, I get it. I’ve seen this movie before. Let’s get on with it. Bring in the first ghost!”

The ghost of Jacob sighed and floated away. In his place, another ghost formed up from the floorboard, youthful and vibrant, dazzling eyes full of hope: I knew they must be an intern.

“I am the Ghost of Incidents Past,” they announced.

My bedroom began to transform: rows of desks materialized, accompanied by chairs, computers, monitors, keyboards, mugs of coffee, and bottles of water. Then some figures emerged slowly, first just as still shades, then they moved, talked, typed: even people had appeared. And not just any people – my engineers! My bed disappeared, and I plopped onto the floor of my own company headquarters. I looked across the room towards the glass-walled office I called my own. I saw myself at my desk-- this was from earlier today!

“Hey,” I said to the ghost. “Isn’t this part usually from my childhood or something? You can hardly call this the past!”

The ghost shrugged. “I don’t work on visual-cortex graphics design yet. I’m more like a tour guide. They’re saying next quarter I could start on some–”

I cut the ghost off as I checked the clock on the wall. It said 4:47pm. Any second now…

“It’s down!” I heard myself scream. All the engineers froze in their seats. I watched myself storm into the room.  “Whose fault is this!?” The responses began immediately.

“He pushed code with a bug to the load balancer!”

“Well, she didn’t review it properly!”

“I was busy speeding up your login code!”

“Maybe the login code wouldn’t be so bad if we had some competent database people!”

“Hey, the database people are busy enough dealing with your terrible load balancer!” And on and on.

I watched myself silently fume amid the cacophony of blame. My past self had sensed only anonymous noise, but as I stood among them, I saw their faces and remembered their names.

“But who is responsible for this, right now? Who made the site go down?!” I heard my past self shout.

Bob, an engineer, raised a shaky hand. “I was the one who wrote the code for the faulty load balancer,” he said. “So, I suppose I’m to blame.” His head down, he followed me into my office, where I closed the blinds.

“And what happened to him then?” the ghost asked me.

“I fired him,” I said. “That’s the only way. If he’s the problem, then he’s got to go. How else will people learn? The site has to be up 100% of the time! That’s how it has to be… right?”

The site has to be up 100% of the time! That’s how it has to be… right?

I turned to the ghost, only to find it had disappeared and left another in its place. This one had a weary smile, carried a chipped yet steaming mug, and wore a hoodie picked up at some long-forgotten convention.

“I am the Ghost of Incidents Present,” they proclaimed. “However, this won’t be a present for you.”

We were still in the office, but it was dark outside and the team had been whittled down to a tiny crew of survivors. Some dozed at their desks, others paced.

“I don’t suppose I can tell these slackers to get back to work, can I?” I asked the ghost.

“Of course not,” the ghost said, with a hearty laugh. “What sort of supernatural vision would that be?”

I sighed. I thought they would be goofing off by now, texting and tossing a stress ball around. But I thought that as I laid in my own bed relaxing. And here they were, at midnight, at their desks working hard. Had I really gotten this so wrong?

“Hey, you should look at this,” I heard Jane, another engineer, say. I wandered over. She was leaning across the table, trying to get the attention of another engineer named Mike. He seemed to be typing while asleep.

“Huh? What?” he replied groggily. “Look at what?”

“The issue with the load balancer… I think it’s a lot more widespread than we thought. Even if we get the site back online, it’ll only be a matter of…”

He shot up. “Hey!” he said. “Don’t tell me that! I don’t want to know that!”

“It’s really bad, though,” Jane continued. “This library we’ve been using has some serious flaws…”

“Look, I’m not listening, I can’t hear you,” Mike said, covering his ears. “This conversation just isn’t happening. Keep this to yourself, okay? You think the boss won’t blame you if you raise this? You saw what happened with Bob this afternoon, right?”

“Actually, I don’t really know what happened,” she said. “Nobody was allowed to say anything.”

“Well, I don’t know either,” Mike replied. “But I’m sure it wasn’t good. That could be you.”

“We’ll just keep making the same mistakes,” she sighed. “We haven’t learned anything.”

“I’ve learned that I’m gonna keep my head down and just get the site back up,” he said.

“And then what?”

“And then I’ll look for a new job.”

We’ll just keep making the same mistakes.

I slumped away from the two engineers. “Oh Ghost, take me away from here,” I pleaded. “I understand it now, the cruelty I have wrought upon these poor folks! I’ll be more open! I’ll hear people out! We’ll track incidents! We’ll do postmortems! Just spare me from the third ghost! It’s always so scary!”

“Sorry, it’s part of the job spec,” the ghost said, donning a ghastly robe over their hoodie and unfolding a scythe from a messenger bag. “Oh, right, I’m actually acting part-time as the third ghost right now. Big opportunity! It’s just temporary right now, but I think I’m really nailing it--Wait, dang, I’m not supposed to talk.”

Settling into character, the ghost slowly unfurled a finger back across the office. All the furniture was gone, the people presumably long gone too. Only a few old convention fliers drifted about in the air conditioning like tumbleweeds. I could see myself pacing in my office before a whiteboard, talking frantically on the phone. As I approached, I saw the names of VCs listed on the board, most of them crossed out.

“I’d call it more of a refinement of our team, not a ‘mass exodus,’ as everyone seems to be saying... No, really, I mean, the genius technology is still here! We just need a little bit of cash to get everything back online, and then…” I watched myself set down the phone and glumly cross another name off the list.

I crumbled to the floor in misery. To see myself in such a pathetic state! I knew my errors now, but terror still grasped me – how could I make these changes? We had no infrastructure for the tracking and communication. Everything would have to be built from the ground up, wouldn’t it?

But would it really help? Even if we figured out these logistics, we would still yell at each other, still resent each other, still blame each other. Every incident was sure to still cause a storm. We didn’t just need a software solution, but another way of thinking. I knew it must exist, but how would I learn?

I grabbed at the robes of the ghost, begging them to tell me the answer, the solution that would spare me from this fate. In my hands the robes became my bedsheets and I was back in my bedroom. It was morning. My head spun with visions of the night before. Was it real? I couldn’t say. But I’d learned my lesson.

“Siri,” I shouted across the room. “What’s happening today?”

“Today you have scheduled: Blameless demo at 3pm,” my phone responded.

Blameless? Of course! It wasn’t too late after all! I sprung from my bed and was out the door before I’d even changed from my pajamas. I called Bob in the car to ask if he’d forgive my outburst yesterday and return to work. It wasn’t his fault after all, it was the system’s! With a box of the finest donuts money could buy, I strode through the office doors. A dozen haggard faces turned to me as I walked in. The bitterness and fear I saw in them hurt me, but I would soon make things right.

“My beloved team, lend me your ears,” I said, taking my spot at the head of the room. “For too long we’ve lived under the tyranny of fear and blame. When something goes wrong, I yell, and we all point fingers, and someone is walked out the door. And then what happens? Something goes wrong again! But I know we can move away from this. Together, we can learn, we can change, we can be blameless.”

Together, we can learn, we can change, we can be blameless.

Indeed, our CEO quickly righted his ways and avoided the grim future he’d foreseen. With new policies of transparency and cooperation, his employees enjoyed a bountiful uptime of many nines.If you’d like to see how Blameless can help you avoid terrifying supernatural visions of your own failure, schedule a demo before it’s too late!