Effective communication is critical during incidents. In order to minimize the impact of an incident and resolve it quickly, it's important that all stakeholders are kept informed and updated throughout the incident response process. However, communicating during an incident can be challenging, especially when dealing with multiple stakeholders and a high level of stress. On-call engineers can have their focus disrupted by switching out of their diagnostic tools to issue communications. Getting stakeholders up to speed can be a major time drain.
Blameless CommsFlow is our answer to this challenge. With CommsFlow(™), you can set up templated messages with specific recipients that send automatically when the incident moves through different statuses. For example, you can have a CommsFlow(™) template that automatically alerts management whenever a new incident is declared, automatically filling itself out with the information of that particular incident. Let’s look at other examples of helpful templates.
Automated Status Updates
One example of a helpful messaging template is an automated status update. This template can be set up to automatically send updates to all stakeholders, such as managers, developers, and customers, whenever the incident status changes. This could include updates when the incident is first identified, when it's being investigated, when a fix has been implemented, and when it's been resolved.
Make sure to explain what each step means for each stakeholder. For example, does this update mean developers could still be needed to come work on the incident, or is the incident past the point where that escalation is needed? Do customer-facing teams need to announce progress to customers with this status update?
Critical Incident Alerts
When something goes very wrong, which requires escalation or delays resolution estimates, it’s important to let people know without slowing down the response process. Setting up templates that trigger when the incident severity changes makes sure that all the relevant stakeholders know the update without needing them to seek out updates or distract on-call engineers. Then they can prepare messaging to customers or allocate other resources as necessary.
Customer Impact Alerts
Sometimes incidents will change from being internal to customer-facing. For example, you could detect an incident that causes some servers to be unavailable, but the service could still have enough servers for customers to use it. Then, during the process of fixing the servers, demand could increase or more servers could fail to the extent that customers start experiencing outages.
When this happens, the severity of the incident can be easily changed, which can trigger CommsFlow(™) to automatically send a customer impact alert. Customer success and PR teams may need to start writing responses any time customers are affected, so they need to know as soon as possible.
Incident Summary Reports
Another helpful template is an incident summary report. This template can be set up to automatically send a summary of the incident to all stakeholders once the incident has been resolved. The report could include information such as the root cause, how the incident was resolved, and any actions that are being taken to prevent similar incidents from happening in the future. This can also be sent to management, engineering, and other departments for further analysis.
To enhance the information provided by this update, link the corresponding incident retrospective to give people the full story of the incident. Talk to people who review retrospectives and see what info they’re going there most often for, and consider including that information in the CommsFlow template for ease of access. You could also highlight reliability insights dashboards that were impacted by the incident to give stakeholders context in how this incident compares to others.
Incident Retrospective Review Invitations
Finally, retrospective review invitations can be set up to automatically invite stakeholders to participate in a review of the incident once it has been resolved. This can help ensure that all stakeholders have an opportunity to provide feedback and suggestions for how the incident could have been handled better in the future. These can be triggered when the incident’s status changes to “resolved”.
Ultimately, you want each incident to lead to systemic changes that mitigate future incidents like it. Having post-incident review meetings and retrospective documents give you an opportunity to identify and assign follow-up tasks.
Using CommsFlow(™), teams can set up custom messaging templates that are tailored to the specific needs of their organization. This can help ensure that stakeholders are kept informed and updated during incidents, while reducing the stress and workload on the incident response team. By using these messaging templates, teams can communicate more effectively during incidents and ultimately minimize the impact of an incident and resolve it more quickly.
To see CommsFlow(™) in action, start your free trial today!