The blameless Resources

Assemble the right team for incident management fast with the new bidirectional integration of Blameless and OpsGenie. Blameless's Aaron Lober, Paul Chu, and Nicolas Philip show you how to seamlessly connect your alerting and service registry to your incident response processes.

Product reliability requires investment from all areas of the business. Technology leaders must effectively communicate the implications of service reliability to the rest of the organization. As a leader, how do you prove that a more reliable product is critical to success? Experts from BetterCloud, Machinify and Blameless come together to discuss how to talk to your executive leadership team about reliability in this webinar.

In this episode, Lauren Caliolio, Director of Reliability Engineering at Nike, and Cat Sewel, Director of Engineering at Nubank, join Matt Davis and Kurt Andersen from Blameless to give the proper severity to the topic of incident severity. Incident management always begins with prioritizing and triaging based on the incident’s impact. Judging that impact isn’t trivial! Join our panel as they break down the nuances and details of incident severity.

Join Thai Wood and Matt Davis as they discuss building an effective incident on-call program by looking at all angles from the socio-technical complexities to training, policies, and learning from incidents.

When an incident is resolved, the learning has just begun. Blameless Incident Retrospectives help you improve your system based on each and every incident. Our seamless logging process makes it easy to gather information, and integration with ticketing services help you track follow-up tasks that emerge. Learn how in this video.

One of the biggest potential interruptions to incident management is needing to communicate with management, customers, and other stakeholders. Blameless CommsFlow makes it easy, with predetermined messages sent to specific groups at customizable incident triggers, like incident status changes. See how it all works in this video.

Don't want to jump between apps to manage your incident as it develops? Blameless keeps your engineers focused by allowing you to manage everything from Slack. See how the Blameless SlackBot brings responders through all the tasks for their roles in this video.

The Blameless SlackBot allows you to manage an incident without ever leaving Slack. See how to manage the integration and start new incidents in this video.

When something goes wrong, it can be tempting to gather as many people as you can to fix it. Each person can contribute tremendous value through diverse viewpoints, but too many people can overcrowd your response, leading to miscommunication, redundant work, and much more.

Have you ever wondered if there was a better way to respond to incidents? When you are in the midst of an incident, does the process help you and your teammates or is it more of a burden? There have been a variety of approaches to organizing people and teams over the 30+ years of online services. Each of them have benefits and drawbacks. This talk will dive into a representative set of these approaches to examine them and help the audience to have a wider context by which they can evaluate their own arrangements for incident response. The talk will also look at incident response from a more abstract, task/intent-focused perspective to give a framework against which processes can be examined and adjusted to be more enabling, less burdensome. (And no, this is not a lite beer commercial ;-))

Blameless is streamlining your incident management runbook with this 60-minute workshop.

Blameless is supercharging your incident response with a modern, integrated framework for incident management.

In this episode, Joanna Mazgaj, Director, Production Support, and Laura Nolan, SRE at Flatiron, join Matt Davis and Kurt Andersen from the Blameless team to detect the problems of problem detection! Knowing what's going wrong isn't always easy. Learn how to get ahead by building collective intelligence, stopping things from slipping, and more!

In this video, Paul Chu, Head of Customer Success at Blameless, talks about building a Reliability Insights report that tracks the progress of completing follow-up actions from incident-related tickets. Follow along as he walks you through how to set this up in Blameless.

In this video, Matthew Dodge, Customer Success Manager at Blameless, walks you through all of the out-of-the-box Reliability Insights reports you get in Blameless automatically. He'll explain where the data comes from, why it's helpful in understanding reliability, and how you can translate these reports into actionable steps.

Watch his step-by-step tutorial and begin leveraging Blameless to create incident tickets in ServiceNow from your preferred chat tool or in the Blameless interface.

BetterCloud worked with Blameless and an incident training agency to establish a sustainable and scalable process.

In this video, Paul Chu, Head of Customer Success at Blameless, teaches you how to build a custom report in our Reliability Insights. He'll show you how to pull data from various sources and configure a chart that expresses the data meaningfully. Watch the full video to follow along and hear tips from Paul based on what he's learned about how other reliability teams build their own custom reports.

Managing incidents, follow up actions, Jira tickets, and comms in Blameless

Starting an incident, assigning roles, and updating task checklists via Slack

In this episode, Liz Fong-Jones, Developer Advocate at Honeycomb.io, and Jean Clermont, SRE at Flatiron, join Matt Davis and Kurt Andersen from the Blameless team to search the dark and scary corners of technical debt. What's "dark debt"? Do you have haunted graveyards? And what's the difference between debt and toil? Register to watch the full conversation!

In this episode, Alyson van Hardenberg, Engineering Manager at Honeycomb.io, and Varun Pal, Staff SRE at Procore, talk to Matt Davis and Jake Englund from the Blameless team. Find out how their engineering orgs pick an incident commander. What type of technical knowledge should this person have about an incident? And does it take a certain type of personality to excel as incident commander? Register to watch the full conversation!

Our first episode covers a juicy topic to kick us off — What’s difficult about on-call? We invited Yvonne Lam, Engineering Manager at Kong, and Charles Cary, CTO at Shoreline.io, to chat with Kurt Andersen and Matt Davis from the Blameless team. Watch the full conversation, honest and unscripted, where the four discuss their personal experiences and learnings about on-call duty.

Communication can make or break the effectiveness of a team responding to an incident. On-call needs to know real-time status updates plus all relevant tasks. Support, customer success and functional owners across the business need assurance that it’s being handled. Blameless’ new powerful CommsFlow™, core to the platform, is designed to operate timely, context-rich and automated communications, from start to completion. The lift becomes much lighter, everyone is updated, and most importantly, engineers have time to focus.

Having good reliability means that incidents are nothing special, merely variations of our regular work. Such a perfect dream of Site Reliability Excellence means that there are clear paths to expertise and common grounding between teams happens frequently. To make this vision a reality, we run an open session at Blameless that builds on musical traditions of improvisation. Inspired by western jazz and Indonesian percussion orchestras, our weekly session seeks to build group intuition through a discipline of iterative collaboration. In this talk I introduce our approach to continuous learning, Practice of Practice Gamelan. What we've created is an opportunity for coming together in a collaborative way without the anxiety of performing under pressure. We share mental models through the telling of stories, playing of games, and riffing of ideas. Different areas of our socio-technical system are explored as seen through different eyes. By learning about how our coworkers view the system we operate together, we continuously build new connections through our newly shared perspectives. We not only learn how our teammates strive towards Reliability Excellence in their daily work, we also reduce unknowns about the system itself, giving us more flexibility to adapt around inevitable ambiguity. Come see how we want incidents to be just another time to get together and jam about some fascinating part of the system that has suddenly revealed itself as a wrong note we can learn by.

On call doesn’t have to suck. Right? Some vendors even have this as a tagline. Truth is some on-calls do suck (a little) especially if it’s a Sev0. Ever been in a chaotic on-call environment? Unclear on who’s doing what and how to run the playbook. Is there a playbook? This session shares approaches that simply work and more importantly how teams learn and improve over time. From living through inevitable incidents and reviewing retrospectives, everyone up-levels and learns. Happier DevOps teams, happier customers. Presented by Paul Chu, Head of Customer Success at Blameless.

In SRE, a core message is that failure is inevitable. No matter how much you prepare, there will always be incidents you can't foresee. This doesn't mean preparation is useless, though. This talk will focus on one extremely valuable type of preparedness: having backups and restoration processes for the worst disasters. When your system experiences a total outage, an effective option is often to switch to a backup system before trying to solve the issue itself. This will restore service as fast as possible. However, just making backup systems isn't enough. This talk reveals complacency and blind spots when it comes to backup systems. Many organizations feel comforted by having created backups, but aren't actually prepared to use them. There will be practical advice given on how to improve backup systems for organizations of all sizes. The talk will cover looking at backup systems from the perspectives making them more reliable, more robust, and more resilient - based on the definitions given by Dr. David D. Woods. In order to make the advice inclusive, there won't be much technical detail. Instead, the focus will be on mindsets and strategies. Black swan events are highly impactful incidents that are so unlikely or unimaginable that effort isn’t made to prepare for them. You'll learn how to conduct thought experiments of "meteor strikes" and other worst-case scenarios, such as ransomware, to feel ready for other problems you can't yet imagine. You'll also see how backup systems can still be useful for such disasters. This is how a resilient backup system is created - one that can still handle what falls outside your expectations.

Alerts go off. Acknowledge it’s real! Start an Incident. Blameless and PagerDuty work brilliantly together. Blameless has a bi-directional PagerDuty integration to access on-call assignments and escalation policies from within the Blameless product to find and allocate the right engineers on-call. Bring your questions and energy. Excited to share new Blameless updates!

How are you doing with service reliability? How well are teams responding to and resolving incidents? What are your MTTx metrics and how efficient is your process?

Service level objectives (SLOs) define what “success” means for engineering orgs, based on the user experience. It’s a whole new way of understanding your production services that helps keep customers satisfied and boosts development velocity. Another benefit to using SLOs is being able to align multiple teams on shared goals — from product to dev, ops, sales, and so on.

A retrospective (or post-mortem) is a critical part of Incident Response. If you don’t document, share, and learn, everyone misses out on a critical learning opportunity. Retrospectives are designed to help teams collectively learn, iterate, and improve. It involves taking the time to story-tell exactly how an incident took place and discussing how to do better next time. The key is to shine new light on processes, tools, and systems so that you know how to fine-tune.

A retrospective (or post-mortem) is a critical part of Incident Response. If you don’t document, share, and learn, everyone misses out on a critical learning opportunity. Retrospectives are designed to help teams collectively learn, iterate, and improve. It involves taking the time to story-tell exactly how an incident took place and discussing how to do better next time. The key is to shine new light on processes, tools, and systems so that you know how to fine-tune.

Every field of endeavor has its leading edge where the answers are unclear and active exploration is warranted. Although the phrase "here be dragons" might be an appropriate warning, this panel of intrepid adventurers will venture into that unknown territory.

How do you reconcile the ideal of blamelessness with the demand for blame? When is it constructive to hold someone accountable, and how? To change a blameful culture, we must empathize with those that point the finger and see how their goals align with our own. We'll show you how to communicate that their goals can be achieved blamelessly. Lastly, we'll share how to hold true accountability well.

What can we learn about working through and analyzing IT incidents from the high tempo, very high consequence world of medical emergency rooms? What does vulnerability have to do with psychological safety, creativity, and team collaboration? It turns out that we can learn quite a lot. We can all benefit from drawing connections across different fields of study and this panel will explore some of these connections. DevOps Enterprise Summit Virtual - US 2021

On call doesn’t have to suck. Right? Some vendors even have this as a tagline. Truth is some on-calls do suck (a little) especially if it’s a Sev0. Ever been in a chaotic on-call environment? Unclear on who’s doing what and how to run the playbook. Is there a playbook? This session shares approaches that simply work and more importantly how teams learn and improve over time. From living through inevitable incidents and reviewing retrospectives, everyone up-levels and learns. Happier DevOps teams, happier customers. Presented by Paul Chu, Head of Customer Success at Blameless.

The networked software systems we build are increasing in complexity every moment. Today the most successful builders and operators are embracing complexity through CI/CD, Chaos Engineering, and innovation in Incident Response. They realize that the adaptive world around us is advancing at such a breakneck speed, it is leaving our capacity to understand it in the dust. That humans and technology must race a gauntlet of automation surprises and collaboration challenges as a team, learning and improving along the way. This session showcases methods of deploying, running, and navigating complexity. It offers a practical view of how software systems can scale and remain robust to failure (like fallbacks or high availability), achieve highly reliable socio-technical operations (via runbooks and game days), and adapt to surprise through techniques of resilience engineering (graceful extensibility and building for adaptation).

Reaching higher levels of the organization is essential to achieving broader adoption of SRE. Without the right buy-in even if parts of SRE are rolled out, behaviors will regress. How then do you get this support? It starts with finding out the incentives needed by key levels of leadership and how to effectively speak to those. It also requires listening to the resistance to SRE adoption, and then effectively address that resistance with reason and metrics. In this talk, I share how to persuade the right level of leadership so that an organization can progress the adoption of SRE through key stages. I provide case studies of how real companies have succeeded or failed with their SRE adoption. This talk aims to equip the audience with the tools to promote SRE adoption through a grassroots/bottoms-up approach.