Risks in Software Development: How to Match Your Caution to What’s Actually at Stake

On July 19, 2024, CrowdStrike shipped a routine update to its security software. One bad file took down about 8.5 million Windows machines and grounded flights, froze banks, and knocked hospital systems offline for the better part of a day.

The engineer who pushed it almost certainly thought it was a small change.

That’s the whole problem with how we usually talk about risk. The standard “risks in software development” checklist (budget, schedule, scope creep, technical and security risk) is a project manager’s list, the kind that gets reviewed once at a kickoff and filed away. It won’t help you on a random Tuesday afternoon when you’re staring at a one-line change that could quietly take down production.

Because here’s the question nobody teaches you to ask before you touch anything: what’s the worst thing that happens if I’m wrong here? A config tweak and a change to the code that charges a customer’s credit card can look basically identical in a pull request. They are not even close to the same risk, and knowing which is which is most of the job.

Every change you make has a blast radius. A good engineer reads it before they write a line and lets it decide how careful to be. This is a guide to reading it, written for our own engineers at Full Scale as much as anybody, because it’s the whole difference between a developer who moves your business forward and one who just closes tickets.

Where the risks in software development actually live

Risk isn’t something you assess once at kickoff and file away. It’s a question you answer every single time you open a pull request, and it comes down to two things: how much damage this change can do if it’s wrong, and how hard it is to undo.

Think about how wildly different the software inside one company is. Code that launches a rocket has to be flawless, because you can’t patch a spacecraft that’s already halfway to orbit. Code that runs some internal dashboard three people glance at on a slow Friday can be broken all week and nobody notices. Same language, same craft, but the stakes aren’t in the same universe.

The skill is matching how carefully you work to how much damage a mistake can actually do.

The easiest way to think about the reversibility half is a one-way door versus a two-way door. A two-way door you can walk back through: a feature behind a flag, a config tweak, a deploy you roll back with one click. A one-way door you don’t get back from without a ton of pain: the core architecture everything else is built on, a migration that rewrites millions of rows, a giant cutover that swaps everything at once. Almost all your day-to-day work is a two-way door. It’s nowhere near as scary as it feels, and you should be flying through it. Save the white-knuckle caution for the one-way doors.

The damage half runs on a spectrum, and roughly three levels of it each earn a different amount of care.

The stuff that can’t be wrong even once touches money or fires messages at scale: billing code that double-charges a card, a job that emails your whole list, the CrowdStrike-class update that repeats one bug ten thousand times before a human looks up. This earns your deepest testing, a real review from someone who knows the system, and a slow staged rollout.

The stuff you can recover from, but it’ll sting is where most of your real work lives, and it’s the hardest to judge: a customer-facing feature, an API change a few clients depend on, a data migration. The trap is that a migration can look reversible and quietly corrupt data you won’t notice for weeks. Test the one or two paths that actually matter (on a checkout change, the part that takes the payment, not the order-history label), get a second set of eyes on the tricky logic, and make sure you can re-run it safely if it dies halfway.

The stuff nobody will remember by Friday is the throwaway script, the internal dashboard, the feature behind a flag only your team can see. Worst case, you fix it in twenty minutes and nobody outside the building knew. Move fast and skip the ceremony; piling process onto this is pure waste.

So before you start, run the change through a few quick questions. What’s the worst case if I’m wrong: money, data, customer trust, or just a quick fix? Can I undo it in seconds, or am I committed once it ships? How many people does it touch if it breaks? And do I actually understand how this behaves in production, not just on my laptop? The answers set your rigor. High-stakes changes earn deep testing on the paths that actually matter, a real review, and a staged rollout. The low-stakes stuff earns almost none of that, and piling heavy process onto it is pure waste.

This is also exactly how much testing each piece deserves, which is why we treat testing depth as a leadership decision, not a QA checklist and security as practices baked into the work rather than a step bolted on at the end.

The dangerous changes are the ones that look like nothing

The hardest part of reading a blast radius is that the riskiest changes often look like the safest. This is where developers who’ve never really lived in production get wrecked.

Adding an index to a table feels like nothing on your laptop. The table’s got a few thousand rows, the change runs instantly, you commit it and move on. Then it hits production, where that same table has 500 million rows, and that “instant” change locks the whole table, grinds for hours, and drags the app to a crawl during your busiest traffic of the day. What was a one-line migration on your machine needed a planned maintenance window and a careful rollout in the real world.

The risk was never in the code. It was in the gap between what your laptop looks like and what production actually looks like.

You can’t judge a blast radius you’ve never seen. A developer who’s never watched a system buckle under real load, with real data and real traffic, will keep mistaking grenades for water balloons. That’s the most common way genuinely good engineers cause genuinely bad outages, and it’s why understanding the production environment isn’t optional.

The most reliable way to lower risk is boring

When something breaks, the cause is almost always whatever just changed. A system that worked yesterday and faceplants today doesn’t need you spelunking through the whole codebase like a detective. It needs you to ask what changed since the last time it worked. But that only saves you if the answer is small. Ship one tight change and you find the culprit in minutes. Ship a giant release with forty things crammed in and you’re interrogating forty suspects while production is on fire.

So the single most reliable way to lower your risk is dull as hell: ship lots of small changes instead of a few big ones. A small change is easy to understand and easy to roll back. A big release buries the cause of every problem under a mountain of noise.

The rollout is the other half. You don’t have to flip a risky change on for everyone at the same second. Feature flags let you split shipping the code from releasing the feature, so you turn it on for a tiny group, watch what happens, and widen only once it holds.

Bugs are going to happen no matter how good you are, so the smart play is to catch them while the blast radius is tiny. Picture Tesla pushing an update to every car on the road in one night. Something’s wrong, and the whole fleet is broken by breakfast. Now picture them shipping to a few thousand cars first, watching, then widening. Same update, but the risk is completely different.

The risks that cost the most aren’t in the code

But all of that only helps if you actually calibrate in the first place, and fear pushes most people the wrong way. It isn’t their company, and a mistake feels like it could torch their reputation or their job, so they over-review, over-test, and beg for permission on changes that carry no real risk. The whole team slows down to guard against downsides that were never going to be a big deal.

I notice this because I don’t feel it. I’m the founder. If I break something, I’ll fix it, and I’m not about to fire myself over it. An employee doesn’t get to feel that loose, and that’s fair. I’m not telling anyone to care less. The trick is knowing when to white-knuckle a change and when to ease off. And if you’re new to the system or just not sure, treat the change as scarier than it looks and get a second read. You earn the right to move fast by watching real systems break, not on day one.

But over-caution isn’t even the expensive mistake. The expensive one is pouring all that rigor into the wrong thing. We needed a quick internal tool a while back, a dirty little MCP server, the kind of thing that should’ve taken two or three days. Instead the guys on our team decided to architect it properly and spent six weeks on it. When they finally shipped, it didn’t even work right, because in six weeks of building it properly not one of them let a real person try it. The risk they should’ve been sweating wasn’t “is this elegant enough.” It was “are we even building the right thing.”

That’s the biggest risk in all of software development, and it sits underneath every other one: that the work isn’t worth doing at all. You can write flawless, beautifully tested, perfectly rolled-out code for a feature nobody asked for. At Stackify, we once dropped $10,000 sponsoring a developer conference to launch a product and walked away with exactly zero new customers. The product just wasn’t what people needed, and no amount of clean code was going to fix that. We learned it a quieter way too: we assumed more usage meant more value and optimized to get people logging in more often, then we actually listened, and customers told us the opposite. “I just need to know why production is broken.” They didn’t want to live in our app. The most valuable thing we could build was the thing that got them out of it faster.

Shipping the wrong thing efficiently is still shipping the wrong thing. Getting real feedback early beats almost any other risk control there is, which is the whole heart of asking why before you build, something I bang on about constantly in Product Driven.

Reading risk is the developer’s job now

For a long time the industry treated developers like order takers. Requirements rolled downhill, you built what the ticket said, and judgment was somebody else’s department. That model is dead.

Now that AI cranks out so much of the code we ship, the bottleneck isn’t typing anymore. It’s judgment. A developer has to understand the business problem, weigh the risk of the different ways to solve it, and go solve it without a permission slip at every step. The 2025 DORA report put it well: AI amplifies whatever’s already there. Strong teams get faster and better with it, while weak teams just ship their problems quicker. Either way, the judgment comes from you, not the tool.

This is the real difference between a developer who multiplies a team and one who quietly drains it, and it has nothing to do with speed or lines of code. It’s whether they can read risk and move accordingly, flooring it where it’s safe and easing off where it isn’t. The one who treats every change like he’s launching a rocket is as much of an anchor as the one who treats a billing change like a throwaway script.

That judgment is also exactly what you’re paying for when you hire a senior engineer, and exactly what you’re not getting when you chase the cheapest warm body you can find. We call that mistake cheapshoring. A cheap dev can type code. Reading the blast radius of a change, in your system, against your business, is the part that takes real experience.

Build a team that reads risk well

Managing software development risk well is a learned skill, and it’s the whole difference between an engineer who speeds your company up and one who quietly slows it down. It’s exactly what we train for at Full Scale, where our engineers work as part of your team through our staff augmentation model instead of as some vendor taking orders. If you want to scale with developers who actually own that judgment, book a call and let’s talk.