When we think of reliability tools, we may overlook the humble checklist. While tools like SLOs represent the cutting edge of SRE, checklists have been recommended in many industries such as surgery and aviation for almost a century. But checklists owe this long and widespread adoption to their usefulness.
Checklists can also help limit errors when deploying code to production. In this blog post, we’ll cover:
Production checklists should be holistic. They should cover everything from launch logistics to contingency plans for failure. Let’s break down what you’ll need for a thorough checklist.
To determine how thorough your checklist should be, consider what level of reliability your customers need.. You may be tempted to be as comprehensive as possible with every checklist, but that costs time and may be unnecessary. At Mercari, the service level is determined based on the service’s SLO. Services that are critical to business success are scrutinized more than niche services.
List all major components of your service. These components may be under the ownership of various teams. For example, you’ll likely need to consult server management teams, testing teams, and many others. It’s important to know as soon as possible whom you’ll need to consult. Some areas to consider include:
Each of these areas contains many issues, and requires data to answer. Your checklist should ask for each piece of data. Here’s an example of how certain sections could be broken down:
You may also want to include information on who to consult to check off each item, and the timeframe for being able to check it. Build your checklist and check items off as development progresses. Double check to ensure that items are ready to go. Right before launch, do a final check through the whole list, just in case.
As you develop, you’ll likely find more areas you want to vet prior to launch. To keep your checklist from becoming too long, you’ll need a system to make sure new additions are helpful. At Google, teams have two criteria for adding an item to the checklist:
You can determine criteria based on the service level you’ve assigned. It’s better to have an unnecessary item than to lack one you need. It’s okay to start with a big checklist, then remove items after each launch that proved to not be useful.
Production checklists can seemingly add overhead to engineers’ jobs. However, the upfront work can save teams from future problems and ensure a successful launch. Production checklists help:
You will need to review and revise your checklists periodically to keep them useful. Be sure to revisit them at these times:
When development on a new service starts. When mapping out a new service, consider which production checklist to use when it launches. Based on the type of service and service level, find the closest checklist you have. Review it to make sure it follows the processes and architecture you currently use. Add any service-specific requirements as you develop.
After a launch. Take a look at the production checklist after you launch the new service. Were there any problems with the launch? Could they have been checked for beforehand? Look for checklist items that were misunderstood and filled out incorrectly. Revise these items to ensure the checklist lines up with the reality of development.
After an incident. If an incident impacts the new service, see if any of the contributing factors could have been addressed with the checklist If so, try to capture those items on future checklists. This task can be incorporated into your incident retrospectives.
As part of regular review cycles. Set a schedule to review tools like runbooks and production checklists. Make sure to invite all team members who will be required to use these runbooks or checklists. Each of these people can provide insight on what to improve moving forward.
To get the most from your checklists, you need to integrate them into your workflows. Here’s how Blameless can help:
To see more of how Blameless helps you be your most reliable, check out a demo.
If you enjoyed this blog post, check out these resources: