When you stop to think about it for a moment, why does a movement for things like “blameless postmortems” exist?
We’ve, as a community, spent a large amount of effort and time to talk about these and things like The Five Why’s.
They’re great to help foster a learning environment and to help build a sense of team and psychological safety.
We know this, and it’s why we try to build these things into our engineering cultures.
But, it takes and has taken a tremendous effort.
Why is that?
Because most software engineering – and corporate – cultures default to a blame-centric shitshow of finger-pointing, scapegoating and cover-your-ass behavior.
When something goes wrong, somebody has to go tell the customer, the business, or the team what went wrong and why.
And those folks will often have a knee-jerk question of “Who did that?” when it comes down to it.
As an engineering leader, part of the job is to handle these inquiries – to address blame and responsibility at a team or personal level, while protecting your team members from experiencing this firsthand.
And if you bungle this, you can quickly become the answer to “Who did that?”
This can be a surprise if you’ve come from individual contributor roles where you have been successfully shielded from this – it means you’ve had good managers and leaders above you who are handling this cultural interface without much knowledge of or impact to you.
When you move into management or leadership roles, however, you are quickly exposed to this rift, and it can be a jarring realization as you begin to navigate corporate environments like this.
When things go wrong in an engineering organization – a security incident, an outage, a plain old, ordinary bug – and it rises to the level of the wider business, this quickly becomes your bailiwick to manage.
Essentially, you go blameless down to your team – mistakes happen, it’s not anybody’s fault, we’ll get ’em next time. And you go blameful up from your team – this mistake will never happen again, these are the steps we’re taking to ensure that, here’s exactly what went wrong, and these are the root causes.
Some tips and best practices:
- Get out in front: There’s going to be an instinct to delay communication about whatever it is that went down – maybe you’ll fix it before anybody notices, or maybe the fix is mere moments away. However, by the time someone asks you about how it’s going, it’s already too late for you to have communicated. The communication objective is to let the rest of the business and other stakeholders know that everything is being handled and under control. You can’t do this effectively by going into a cave and becoming silent. You need to be frequently transparent about what you and the team are working on, the status of things, and estimated time to resolution and without being asked. This is true during the incident, of course, but it’s also true afterward. Let stakeholders know what the next steps are, what the theories are, what steps are already being taken to monitor, mitigate, and mechanize.
- Take ownership: Point One above is in service of this point. You need to take ownership of the issue quickly, before someone else decides to do so. If you can maintain control of the situation, you earn the right to control the outcome. Sensing a lack of ownership means others will try to fill that void, and they are unlikely to have the same blameless stance as you have, given the larger corporate culture at hand in many American businesses.
- The process takes the blame: When the situation is resolved, you now get to deal with the aftermath, which will involve the same sort of flow. Stakeholders will want to know what happened, when it happened, why it happened, what the short and long term impact is, and what steps you and your team are taking to ensure this doesn’t happen again. If you have ownership over the situation, you can dictate the outcome and control the messaging. This allows you to avoid the “who” question conspicuously absent from the previous list of questions. The blame must be laid on the process that got you here, and the mitigation steps must outline how that process will change to prevent this in the future. Perhaps your code review was rushed, or your monitoring was lacking, or your process skipped a step. Those are the things to fix and to blame, not the junior engineer who pushed the code, the senior engineer who rubber stamped the pull request for the DevOps guy who let a log file go unrotated. Process, not people, is to blame. Because _something_ will need to be blamed.
- Barring that, you take the blame: Sometimes, all the messaging in the world won’t work, and in some corporate cultures, there will need to be a head to assign the blame to. If there must be a head, let it be yours publicly. After all, the process you tried to blame, and are blaming, is your responsibility. If the process was wrong or flawed, it is you to blame. Privately, whether it got to this point or not, you can hash things out with the responsible parties – the ones who pushed the buttons, skipped the steps, and made the mistakes – but publicly, it’s you that screwed the pooch.
All the bits in this are hard because in the moment, you might not be thinking clearly or logically. Many times in my career, moments like this have been frantic and filled with emotion – both my own and others’.
If need be, take a moment, take a deep breath, and then go own the situation.
Because if you get nothing else from this:
If you own the situation, you earn the right to own the outcome.