The blameless Resources

Videos

Varieties of Incident Response

Have you ever wondered if there was a better way to respond to incidents? When you are in the midst of an incident, does the process help you and your teammates or is it more of a burden? There have been a variety of approaches to organizing people and teams over the 30+ years of online services. Each of them have benefits and drawbacks. This talk will dive into a representative set of these approaches to examine them and help the audience to have a wider context by which they can evaluate their own arrangements for incident response. The talk will also look at incident response from a more abstract, task/intent-focused perspective to give a framework against which processes can be examined and adjusted to be more enabling, less burdensome. (And no, this is not a lite beer commercial ;-))

Varieties of Incident Response

Have you ever wondered if there was a better way to respond to incidents? When you are in the midst of an incident, does the process help you and your teammates or is it more of a burden? There have been a variety of approaches to organizing people and teams over the 30+ years of online services. Each of them have benefits and drawbacks. This talk will dive into a representative set of these approaches to examine them and help the audience to have a wider context by which they can evaluate their own arrangements for incident response. The talk will also look at incident response from a more abstract, task/intent-focused perspective to give a framework against which processes can be examined and adjusted to be more enabling, less burdensome. (And no, this is not a lite beer commercial ;-))

Resilience Thrives in the Practice of Practice

Having good reliability means that incidents are nothing special, merely variations of our regular work. Such a perfect dream of Site Reliability Excellence means that there are clear paths to expertise and common grounding between teams happens frequently. To make this vision a reality, we run an open session at Blameless that builds on musical traditions of improvisation. Inspired by western jazz and Indonesian percussion orchestras, our weekly session seeks to build group intuition through a discipline of iterative collaboration. In this talk I introduce our approach to continuous learning, Practice of Practice Gamelan. What we've created is an opportunity for coming together in a collaborative way without the anxiety of performing under pressure. We share mental models through the telling of stories, playing of games, and riffing of ideas. Different areas of our socio-technical system are explored as seen through different eyes. By learning about how our coworkers view the system we operate together, we continuously build new connections through our newly shared perspectives. We not only learn how our teammates strive towards Reliability Excellence in their daily work, we also reduce unknowns about the system itself, giving us more flexibility to adapt around inevitable ambiguity. Come see how we want incidents to be just another time to get together and jam about some fascinating part of the system that has suddenly revealed itself as a wrong note we can learn by.

Resilience Thrives in the Practice of Practice

Having good reliability means that incidents are nothing special, merely variations of our regular work. Such a perfect dream of Site Reliability Excellence means that there are clear paths to expertise and common grounding between teams happens frequently. To make this vision a reality, we run an open session at Blameless that builds on musical traditions of improvisation. Inspired by western jazz and Indonesian percussion orchestras, our weekly session seeks to build group intuition through a discipline of iterative collaboration. In this talk I introduce our approach to continuous learning, Practice of Practice Gamelan. What we've created is an opportunity for coming together in a collaborative way without the anxiety of performing under pressure. We share mental models through the telling of stories, playing of games, and riffing of ideas. Different areas of our socio-technical system are explored as seen through different eyes. By learning about how our coworkers view the system we operate together, we continuously build new connections through our newly shared perspectives. We not only learn how our teammates strive towards Reliability Excellence in their daily work, we also reduce unknowns about the system itself, giving us more flexibility to adapt around inevitable ambiguity. Come see how we want incidents to be just another time to get together and jam about some fascinating part of the system that has suddenly revealed itself as a wrong note we can learn by.

Survival Guide: Black Swan Events

In SRE, a core message is that failure is inevitable. No matter how much you prepare, there will always be incidents you can't foresee. This doesn't mean preparation is useless, though. This talk will focus on one extremely valuable type of preparedness: having backups and restoration processes for the worst disasters. When your system experiences a total outage, an effective option is often to switch to a backup system before trying to solve the issue itself. This will restore service as fast as possible. However, just making backup systems isn't enough. This talk reveals complacency and blind spots when it comes to backup systems. Many organizations feel comforted by having created backups, but aren't actually prepared to use them. There will be practical advice given on how to improve backup systems for organizations of all sizes. The talk will cover looking at backup systems from the perspectives making them more reliable, more robust, and more resilient - based on the definitions given by Dr. David D. Woods. In order to make the advice inclusive, there won't be much technical detail. Instead, the focus will be on mindsets and strategies. Black swan events are highly impactful incidents that are so unlikely or unimaginable that effort isn’t made to prepare for them. You'll learn how to conduct thought experiments of "meteor strikes" and other worst-case scenarios, such as ransomware, to feel ready for other problems you can't yet imagine. You'll also see how backup systems can still be useful for such disasters. This is how a resilient backup system is created - one that can still handle what falls outside your expectations.

Survival Guide: Black Swan Events

In SRE, a core message is that failure is inevitable. No matter how much you prepare, there will always be incidents you can't foresee. This doesn't mean preparation is useless, though. This talk will focus on one extremely valuable type of preparedness: having backups and restoration processes for the worst disasters. When your system experiences a total outage, an effective option is often to switch to a backup system before trying to solve the issue itself. This will restore service as fast as possible. However, just making backup systems isn't enough. This talk reveals complacency and blind spots when it comes to backup systems. Many organizations feel comforted by having created backups, but aren't actually prepared to use them. There will be practical advice given on how to improve backup systems for organizations of all sizes. The talk will cover looking at backup systems from the perspectives making them more reliable, more robust, and more resilient - based on the definitions given by Dr. David D. Woods. In order to make the advice inclusive, there won't be much technical detail. Instead, the focus will be on mindsets and strategies. Black swan events are highly impactful incidents that are so unlikely or unimaginable that effort isn’t made to prepare for them. You'll learn how to conduct thought experiments of "meteor strikes" and other worst-case scenarios, such as ransomware, to feel ready for other problems you can't yet imagine. You'll also see how backup systems can still be useful for such disasters. This is how a resilient backup system is created - one that can still handle what falls outside your expectations.

LISA21 - Groove with Ambiguity: The Robust, the Reliable, and the Resilient

The networked software systems we build are increasing in complexity every moment. Today the most successful builders and operators are embracing complexity through CI/CD, Chaos Engineering, and innovation in Incident Response. They realize that the adaptive world around us is advancing at such a breakneck speed, it is leaving our capacity to understand it in the dust. That humans and technology must race a gauntlet of automation surprises and collaboration challenges as a team, learning and improving along the way. This session showcases methods of deploying, running, and navigating complexity. It offers a practical view of how software systems can scale and remain robust to failure (like fallbacks or high availability), achieve highly reliable socio-technical operations (via runbooks and game days), and adapt to surprise through techniques of resilience engineering (graceful extensibility and building for adaptation).

LISA21 - Groove with Ambiguity: The Robust, the Reliable, and the Resilient

The networked software systems we build are increasing in complexity every moment. Today the most successful builders and operators are embracing complexity through CI/CD, Chaos Engineering, and innovation in Incident Response. They realize that the adaptive world around us is advancing at such a breakneck speed, it is leaving our capacity to understand it in the dust. That humans and technology must race a gauntlet of automation surprises and collaboration challenges as a team, learning and improving along the way. This session showcases methods of deploying, running, and navigating complexity. It offers a practical view of how software systems can scale and remain robust to failure (like fallbacks or high availability), achieve highly reliable socio-technical operations (via runbooks and game days), and adapt to surprise through techniques of resilience engineering (graceful extensibility and building for adaptation).

How to Champion SRE Investment to Different Levels of Leadership

Reaching higher levels of the organization is essential to achieving broader adoption of SRE. Without the right buy-in even if parts of SRE are rolled out, behaviors will regress. How then do you get this support? It starts with finding out the incentives needed by key levels of leadership and how to effectively speak to those. It also requires listening to the resistance to SRE adoption, and then effectively address that resistance with reason and metrics. In this talk, I share how to persuade the right level of leadership so that an organization can progress the adoption of SRE through key stages. I provide case studies of how real companies have succeeded or failed with their SRE adoption. This talk aims to equip the audience with the tools to promote SRE adoption through a grassroots/bottoms-up approach.

How to Champion SRE Investment to Different Levels of Leadership

Reaching higher levels of the organization is essential to achieving broader adoption of SRE. Without the right buy-in even if parts of SRE are rolled out, behaviors will regress. How then do you get this support? It starts with finding out the incentives needed by key levels of leadership and how to effectively speak to those. It also requires listening to the resistance to SRE adoption, and then effectively address that resistance with reason and metrics. In this talk, I share how to persuade the right level of leadership so that an organization can progress the adoption of SRE through key stages. I provide case studies of how real companies have succeeded or failed with their SRE adoption. This talk aims to equip the audience with the tools to promote SRE adoption through a grassroots/bottoms-up approach.