DORA Metrics: Are We Measuring the Right Things?

There's an old line, usually attributed to an economist, that goes: when a measure becomes a target, it ceases to be a good measure. Engineering leadership has spent the last decade learning this one the expensive way, one dashboard at a time.

DORA metrics — deployment frequency, lead time for changes, change failure rate, and mean time to recovery — started as genuinely good ideas. They came out of real research into what separates high-performing engineering teams from the rest. For a while, they were a mirror: hold them up to your org and see yourself honestly.

Then they became targets. And the mirror became a funhouse.

How good metrics go bad

Take deployment frequency. The research says elite teams deploy on demand, sometimes multiple times a day. That's a real signal of a healthy pipeline and a team that isn't afraid of its own release process.

But the moment a VP puts "deploy daily" on a slide, something changes. I've watched teams split one sensible release into eleven meaningless ones to hit the number. Nobody got faster. The change didn't get smaller in any meaningful way. The dashboard turned green and the engineering didn't change at all.

Lead time for changes is worse. It's supposed to measure how long it takes an idea to go from commit to production — a proxy for how much friction lives in your pipeline. But once it's a KPI, watch what happens: the clock starts at the commit, so engineers start committing earlier, in smaller fragments, sometimes before the code is even coherent. The metric improves. The review process gets harder. Nobody wins except the chart.

The perverse incentives nobody talks about

Change failure rate is the sneakiest of the four. It measures what fraction of your deployments cause a failure in production. Sounds reasonable. But "failure" is doing a lot of heavy lifting in that sentence.

Who defines a failure? If your team defines it as "a deploy that triggered a rollback," then congratulations, you've just incentivized never rolling back. If it's "a deploy that caused an incident," you've incentivized downgrading incidents. The teams with the most honest postmortems will always look worse than the teams with the most creative definitions. The metric punishes exactly the culture it was supposed to reward.

And mean time to recovery? Lovely idea. Also trivially gamed. If the clock starts when the incident is declared, then the rational move is to declare it later. Nobody does this maliciously. It just... happens. The definitions drift, the numbers improve, and everyone politely agrees not to notice.

Measure anyway, just differently

I'm not arguing against metrics. Flying blind isn't a strategy. The DORA four are still the best starting framework we have, and a team that ignores them entirely is probably ignoring other things too.

But here's the thing I keep coming back to: metrics are for diagnosis, not for judgment. The moment a metric is attached to a performance review, a bonus, or a promotion decision, it stops measuring the system and starts measuring the team's ability to game the measurement. This isn't cynicism. It's just how incentives work on actual humans.

The teams I've seen use DORA well all share one trait: the metrics live with the teams, not above them. The team looks at its own lead time and asks "where's the friction?" — and then fixes the friction. The number is a conversation starter, not a verdict. Nobody's getting a bad review because their change failure rate ticked up in a quarter where they were doing genuinely hard work.

There's also a quiet fifth metric nobody dashboards, and it's the one that matters most: do the engineers trust the measurement? The day your team starts arguing about definitions instead of arguing about engineering, you've lost the plot. The metric has eaten the thing it was supposed to measure.

So yes, measure deployment frequency. Track your lead time. Watch your failure rate and your recovery time. Just keep them in the team's hands, keep the definitions honest, and the next time someone proposes putting them on an executive scorecard, ask what exactly they're hoping to learn — and what they're afraid they'll incentivize instead.

Goodhart's law doesn't care about your dashboard. It just waits.

Ransomware's Mature Playbook

There's a comforting story defenders tell themselves about ransomware: that each attack is a novel crisis, a unique adversary, a situation requiring fresh thinking. The ransomware crews would like to thank us for this story. It makes their lives much easier. Because on their side of the glass, there is nothing novel happening at all. They're running a business, and businesses run playbooks.

Understand the org chart and the mystique evaporates. Modern ransomware operations have a division of labor that would look familiar to any mid-size company. Initial access brokers specialize in getting the first foothold — phished credentials, exploited edge devices — and sell that access wholesale. Affiliates buy the access and deploy the ransomware-as-a-service platform, which is exactly what it sounds like: the core developers lease out their malware the way Salesforce leases out CRM. The developers take a cut of every ransom. Some operations even run help desks — negotiation portals with support staff who will patiently walk your incident response team through the payment process. They have QA: decryptors get tested, because a decryptor that doesn't work is bad for repeat business. They have marketing: leak sites with countdown timers and press-release aesthetics, designed to create exactly the urgency they're supposed to create.

This is not a metaphor. It's the literal structure. Law enforcement takedowns keep confirming it — when a major operation gets disrupted, the affiliates don't retire. They shop for a new platform, rebrand, and get back to work. The individuals are replaceable. The business model isn't.

And the business model keeps evolving in rational, predictable ways. Double extortion — encrypt the data and threaten to leak it — became standard because encryption alone stopped working once backups got better. Defenders adapted; the business adapted back. That's not innovation in the heroic sense. It's product-market fit.

Here's what this maturity means for defenders: stop preparing for the last attack's specifics and start preparing for the business model. The crews don't care about your industry, your tech stack, or your threat intel reports with their names on them. They care about three things: can they get in, can they move laterally, and can they make recovery painful enough that paying looks rational. Every defensive investment should be measured against those three questions, because that's the only exam that matters.

Get in: that's phishing-resistant MFA, patching internet-facing systems, and killing the VPN concentrator from 2019 that everyone forgot about. Move laterally: that's network segmentation, privileged access controls, and actually monitoring east-west traffic instead of assuming the perimeter held. Make recovery survivable: that's backups that are tested, offline, and immutable — tested being the word most people skip, right up until the restore fails during the incident.

Notice what's not on that list: knowing which crew it is. Attribution is interesting. It's occasionally useful. It is almost never the thing that determines the outcome. The affiliate running last month's playbook against you doesn't care what you call them, and your recovery doesn't go faster because you identified the strain. The industry's obsession with naming and tracking crews is mostly theater — useful for journalists, marginal for defenders.

The uncomfortable truth is that ransomware works because the economics work. As long as enough victims pay, the business funds itself, hires talent, and improves its product. Every paid ransom is R&D funding for the next campaign. I'm not moralizing about individual decisions made under existential pressure — if paying is the difference between the company surviving and not, that's a business decision, and I wasn't in the room. But collectively, the payments are the revenue that keeps the machine running, and no amount of takedowns changes the arithmetic.

So here's the mature defender's playbook, to match theirs: assume breach, segment ruthlessly, back up like your business depends on it — because it does — and practice recovery until it's boring. The crews are running a business. The only way to beat a business is to make it unprofitable. Make getting in hard, make moving laterally harder, and make recovery a non-event, and their unit economics fall apart.

They're not geniuses. They're not ghosts. They're a supply chain with a playbook. Learn the playbook, and the mystique goes with it.