Any software engineer of nearly any level of experience shares a similar story: the time they brought down production.
Or the time they emailed an entire customer base.
We tell these horror stories over pints of beer, delivery pizza, and coffee.
What we don’t talk about as much is what happens after we leave a giant crater in our wake.
The runbacks, the process, the procedures to ensure such a thing never happens again.
Until the next time it happens and the cycle begins anew with some new layer of scrutiny and thick film of internal or customer mistrust.
Over time, these incidents and the associated fear and mistrust internal to the company encrust processes like a layer of barnacles on the hull of a speed boat – creating friction and slowing everything down.
One of the biggest unlocks in this situation is to introduce the concept of “blast radius” to the engineering work.
Not every change has the same risk profile.
A one-line bug fix, well-reviewed, is not likely to drop customer data.
A couple lines of CSS to style a button aren’t likely to delete a database.
A new controller endpoint that isn’t exposed to the frontend yet isn’t likely to crash the mobile app.
And so on.
But, in many engineering processes, these small blast radius changes go through the same process pipe as the more impactful and dangerous stuff.
Or, the company starts to think of every deployment as a high blast radius event, so they bundle these small fixes in with the more impactful stuff into some regular release cycle.
And, in software engineering – where each and every change contains the distinct possibility of introducing a bug – bundling together changes just means you’re bundling together risk.
And unlike in insurance, bundling risk here compounds it rather than spreading it.
Getting out of this morass and speeding up delivery velocity requires two efforts:
Define your risk tolerance
Any software engineering culture is trying to develop clean, correct, bug free software.
This is an impossible goal.
No matter who writes it – senior, junior, AI or agent, there will be bugs.
Sometimes, these are bugs of requirements understanding, or customer expectation, edge cases or typos.
Despite an industry’s best effort, bugs will happen.
The job of an engineering organization is to understand how sensitive the business and customer base are to these inevitable bugs.
The next step is to determine which classes of bugs are more sensitive than others.
Some of these are fairly obvious – downtime, data loss, security and the like.
Some can also be sneaky – corners of the application that are heavily used, features highly prized by important customers.
Some are from scar tissue – library updates that cause downtime, database migrations that bog everything down.
This sort of classification happens naturally as engineers get experience with a code base and a company.
Create buckets that map these intrinsic classifications into no more than two or three categories: Low Blast Radius, Medium Blast Radius and High Blast Radius at the most.
Then, you can map work into the buckets at the pull request level – GitHub labels and similar work fine for this.
Different levels of blast radius then get different procedures for deployment to production:
- Low Blast Radius items can go live at any time.
- Medium only on Mondays, Tuesdays, and Wednesdays during the day
- High Blast Radius at night or on weekends with at least two engineers monitoring the release and aftermath
These are examples. Your level of risk tolerance might require more stringent for the relatively safe changes, or less for the high end.
After all, financial institutions and health care have much more downside impact of bugs than your average marketing technology startup.
Lower the overall blast radius
Now that you have the buckets, the next question for an engineering leader to ask themselves is two-fold:
- How do I make more of our work low blast radius?
- How do I lower the blast radius of even our highest blast radius changes?
The answer to both questions leads to familiar places:
- Make changes smaller: Small amounts of change are by definition a lower surface area for inevitable bugs. Small pull requests, frequently merged, also create fewer merge conflicts later. Long-lived, large feature branches are almost definitionally High Blast Radius change sets.
- Code review every change: A parable from my newspaper days is that one never copy edits their own writing – you always get a second set of eyes on it. In my own career, the most confident code – the code I was so sure of that I rushed through – almost always breaks something. Every change gets reviewed by at least one other engineer. And the burden for code review rises as the Blast Radius increases.
- Automated tests: Some of the most pernicious bugs are regression bugs – where something that once worked no longer works – especially when they’re in a part of an application your change theoretically shouldn’t be impacting. This is where having a good practice of writing automated tests can lower the overall Blast Radius. They’re not a replacement for manual testing (more later …), but provide a good safety net to catch failures in expected-to-work code.
- Make bugs easy to recover from: Look for tooling and techniques that can make it easier to deploy changes, rollback changes or hide changes. Blue-green deployments, continuous integration, feature flags and API versioning can all help to deploy changes without impacting a wide swath of customers before your team can verify things are working properly, or make it easy to push corrections and changes when those expectations are not met.
- Manual testing: All the automated testing and all the tooling to protect yourself in the world is still no replacement for the engineer who did the work verifying that it works as they expect in the production environment. Requiring a verification pass as part of the process will hopefully catch bugs before a customer can – especially if the work is deployed behind a feature flag.
- Catch errors before customers: In addition to manual tests, adding error reporting infrastructure – not just logging, but a service that notifies you when a 500 or similar error occurs – will allow your team to find and squash bugs more quickly. The ideal is that you can get into a flow where you’re fixing bugs before customers can report them.
These techniques taken as a whole represent a cohesive working philosophy built around speed and safety.
Because speed without safety considerations – “move fast and break shit” – is reckless and can kill a business just as much as safety without speed – a bogged down morass of mistrust and fear.
If anything, speed without safety can kill it quicker and probably cause more societal damage.
Finding the balance between these two poles – speed and safety – defines and reflects your engineering culture.
It is the most important and impactful work you can do.
This is part of my Editorial framework, a set of modular tactics and strategies that help software companies ship with speed, alignment, and safety at the same time.