Just this week, two major outages were blamed on fixes. But this isn't a new problem.
For example, on November 10, 2024, the Linux kernel merged a fix for a security hole reported by Google's Project Zero. Two reviewers approved it. A third developer tested it. It worked.
Seventeen days later, a user found that it had also broken a feature its own author had built the year before.
Nothing about that fix was careless. And there is nothing unusual about a fix that breaks something. In the Linux kernel, a fix is about as likely to need fixing as any other change, and when it does, it is usually because it broke something new. We traced the kernel's fix graph, the record of which change each fix repairs, and found that roughly one in thirty runtime kernel fixes broke something that then needed a fix of its own within a year. And for some, one repair was not the end of it.
Linux kernel · fixes to runtime code
One fix in thirty needed a fix of its own
About one kernel fix in thirty introduced a failure repaired within a year. Among the 7,478 fixes that a later fix repaired, one in ten started a chain of repairs and one in six drew two or more separate repairs.
3.40% broke something that needed its own fix within a year · sampling interval 3.04%–3.73%
When a fix needs a fix, it often does not stop at one.
Shares of the fixes that needed a fix: the 7,478 that a later fix repaired at any point in the kernel's history.
Chains · the repair also needed a repair
A is repaired by B, B by C, C by D
1 in 10 of fixes that needed a fix started a chain (743 fixes)
- Two repairs deep
- 657
- Three deep
- 73
- Four deep
- 13
Branches · one fix, several repairs
A is repaired by B, C, D and E
1 in 6 of fixes that needed a fix drew several repairs (1,289 fixes)
- Two repairs
- 959
- Three
- 218
- Four or more
- 112
One in thirty: 5,157 of 97,933 tagged fixes were named by a later fix within a year; in a sample of 200 of those, 129 showed a new failure the fix introduced or set off. Model-assisted audit of commit messages and diffs. Chains and branches count every recorded repair link at any distance in time, so they describe a wider group than the one in thirty and are not screened for cause in the same way.
The kernel's own maintainers see the same thing. In 2023 Sasha Levin, who maintains the stable kernels, counted 9,422 fixes in one stable series and found that 360 of them, 3.8%, had gone on to need a fix themselves.
The half of fix verification with no failing test
A fix arrives with half of its verification already written. There is a bug report and a failing case, and the fix makes the failing case pass. That is an experiment with a clear result, and good teams run it every time. The kernel fix passed it.
The other half has no failing case to start from. What else depended on the code that changed? Nobody can list every downstream effect of a change, so mostly nobody tries. The answer shows up later, in someone else's bug report.
Often much later.
Linux kernel · fixes named by a later fix
The longer a fix has been in the kernel, the more likely it has needed another
Share of kernel fixes named by a later fix: 3.44 percent within 90 days, 4.26 within 180 days, 5.10 within a year, 5.84 within two years, 6.62 within four years.
- 90 days
- 180 days
- 1 year
- 2 years
- 4 years
Share of fixes named by a later fix within each window
Each window includes only fixes old enough for the full follow-up, so the groups differ. A recorded repair alone does not establish a new defect.
And sometimes more than once. A fix can need several repairs, and a repair can need its own. For example, a 2022 network-driver fix was repaired that year and again in 2024. In 2025, the second repair needed one too.
Linux kernel · one network-driver fix
One fix, two repairs, and a repair of a repair
A 2022 kernel fix drew two separate repairs, in November 2022 and May 2024; the second needed a repair of its own in January 2025.
Original fix · Jan 2022
Release the semaphore after freeing the slot
Prevents a race when a command slot is allocated.
8e715cd613a1
Two branches Two separate repairs
Repair A · Nov 2022
Allow a waiting thread to yield the CPU
Repairs a single-CPU deadlock.
2808b37b5928
Repair B · May 2024
Wait for a slot before starting the completion timer
Repairs access through an invalid index.
485d65e13571
Another generation One repair needed its own
Repair of B · Jan 2025
Signal completion on the early error return
Repairs a command that can hang.
0e2909c6bec9
Links are recorded repair tags, checked against the source. A selected case from the mlx5 network driver; no failure was reproduced.
Of course, breaking something is occasionally a deliberate trade. You land a hotfix knowing what it will break, because the alternative is worse. But that is rarely what the record shows. In a sample of 129 kernel fixes that went on to break something, the commit message of exactly one saw the failure coming.
The problem has become much more important because fixes are landing faster. Last month Trail of Bits wrote that bug disclosures now "arrive at an unprecedented speed and scale," and every one of them needs a fix. For example, on November 17, the Director of National Intelligence said an outage on the NSA's classified network came from a patch for critical vulnerabilities: some systems, he said, did "not come back up when you patch them as expected." (The agency itself described scheduled maintenance.) Two days earlier, Microsoft traced a day-long email outage to a defect introduced by a fix it had deployed in July. By the time the defect surfaced the change was months old, and Microsoft's usual rollback tools could not quickly revert it.
The break nobody knew about
Checking that a fix works is not enough. Fix verification has to be defensive: it has to go looking for what the fix broke, before anyone knows there is anything to find. The 2024 kernel fix shows what happens otherwise.
That fix was not a minor cleanup. It closed a kernel vulnerability, CVE-2024-53096, which the National Vulnerability Database scores 7.8 out of 10 and Red Hat 6.1, and it was backported to four long-term kernel series. A week after it merged, it went out in the Linux 6.12 release, and the break nobody knew about went out with it.
The initial fix changed the kernel function that maps files into a program's memory, a function the author's own commit message calls "somewhat terrifying." Before the fix, the function sometimes refused a request too late, after it had already torn down whatever was mapped there. A late refusal could leak resources and leave memory in an inconsistent state, which is what made it a security hole. To prevent this, the fix moved every check that could refuse a request to the front.
One of those checks guards sealed files: files locked against writing so that programs can share them safely. But a sealed file is still supposed to be readable and whether the kernel allowed that depended on the check running second. The fix made it run first, and read-only mappings of sealed files were refused.
Linux 6.12 · mapping a write-sealed file
How the fix changed the order of two steps
Before the fix, the file's setup step runs first and the seal check passes. With the fix, the seal check runs first and refuses the mapping; the setup step never runs.
Before the fix
Step 1
The file's own setup
Marks this mapping as one that can never become writable.
Step 2
The seal check
Could anything write through this mapping? No.
Mapping allowed Read-only, as documented
With the fix
Step 1 · moved earlier
The seal check
Could anything write through this mapping? Not ruled out yet.
Mapping refused Operation not permitted
Step 2
The file's own setup
Never runs.
The fix moved every check that can refuse a request to the front, so a refusal can no longer leave damage behind. A write-sealed file relied on this one check coming second. Simplified from the kernel's file-mapping path.
Seventeen days after the fix merged, and ten days after it shipped in Linux 6.12, a user reported the break. Even then, the fix stayed in: it was closing a security hole in kernels people were already running. So the repair kept the fix and rebuilt the sealed-file feature around it. It reached the kernel eight weeks after the fix did. Every Linux 6.12 release from 6.12.0 through 6.12.8 shipped with the break.
The older long-term kernels that received the fix were spared, because at the time they did not have the sealed-file feature to break. Ubuntu's 6.8 kernel, the one in its 24.04 long-term release, did. It picked up the break in April 2025 through its own backport of the fix, and one mis-backported line kept it there until a one-line patch in early 2026.
Had anyone known about the break on November 10, it could have been fixed in the same patch, before any of this happened.
Our work on fix verification
We are working on fix verification to make it easier for teams that maintain systems code to land a fix without breaking something else.
Our system examines a fix, identifies what it corrects, and establishes the standing obligations the changed code must continue to meet. It then searches for violations of those obligations: for each one it constructs an executable test, states the expected outcome in advance, and runs it on the code before and after the fix. It works on user-space code and, as here, inside the operating-system kernel itself.
We ran it on the 2024 fix with only what a reviewer had on the day it merged: the patch, and the source it applied to.
Fix verification · one kernel fix, start to finish
It confirmed the fix. Then it found what the fix broke.
Our system read the fix, mapped what depends on the changed code, wrote 8 tests and reported 2 repaired, 2 broken and 4 unchanged. All 8 pass on the later repair.
1 · Read the fix
What is it for?
Every check that can refuse a mapping request moves to the front, before the kernel has changed anything.
The patch
mm: resolve faulty mmap_region() error path
2 · Map its obligations
What depends on it?
Must be repaired
- A refused request leaves the existing mapping intact
Must still work
- Sealed files stay mappable read-only
- Private mappings of sealed files work
- Ordinary file mappings behave as before
3 · Write defensive tests
Does each one hold?
8 executable tests, written for this fix
Each test states what should happen, then runs on the kernel before the fix and on the kernel with it.
Both kernels are built from source and booted in a virtual machine.
4 · Report what changed
What it found
2 repaired
Failed before the fix, pass with it
2 broken
Passed before the fix, fail with it:
a sealed file can no longer be mapped read-only
4 unchanged
Pass before and after
The same 8 tests, run on the repair the kernel later shipped: 8 of 8 pass.
Our system was given the patch and the kernel source it applied to, and nothing later. Results are from executed tests on the kernel immediately before and after the fix, booted in a virtual machine.
It wrote eight tests. Two confirmed the fix: a refused request no longer destroys what was already mapped there. Two failed: a sealed file could no longer be mapped read-only, with or without a fixed address. The other four checked neighbouring behaviour, such as private mappings of sealed files and mappings of ordinary files, and passed before and after.
That failure is the regression the user reported seventeen days after the merge. Our system found it from the patch alone, in under six minutes of model and test time, and said so:
The candidate preserves existing mappings when the tested MAP_FIXED requests are rejected, but it also rejects read-only shared mappings of F_SEAL_WRITE memfds that succeed on the parent.
From the system's report on this fix
For the person deciding whether to ship, that changes the conversation. "Moving this check might affect sealed files" is an opinion, and it invites a debate. "On your patch a sealed file can no longer be mapped read-only; on the previous kernel it can; here is the test" is a result.
In a subset of our early results, all on Linux kernel fixes, our system identified 4 of 6 real breaks (67%) and 7 of 10 planted defects (70%).
Fix verification · a subset of our preliminary results
Breaks found, on real kernel fixes and on planted defects
Our system found the break in 4 of 6 kernel fixes that were later repaired upstream and in 7 of 10 fixes with a newly planted defect, and flagged none of the same 10 fixes left untouched. Each fix is listed by its upstream commit.
Real breaks
Six kernel fixes that upstream later had to repair
Both misses were quiet: fix confirmed, nothing reported.
4 of 6 found
- Found: 5de195060b2e mm: mmap_region error paths
- Found: 1c24a186398f fs: fd table sizing
- Found: 2174a08db80d sch_netem: qdisc lock
- Found: efa56305908b nvmet-tcp: invalid PDU length
- Missed: 3e453522593d md: resources freed on stop
- Missed: de6d25924c2a sch_ets: classes beyond nbands
Planted defects
Ten other kernel fixes, a new defect planted in each
Written by an independent AI agent on the day of the test.
7 of 10 found
- Found: 0200679fc795 tmpfs: uid/gid mount options
- Found: c04fe8e32f90 pipe: non-blocking write
- Found: 6beb489b2eed md/raid10: safe-mode delay
- Found: 96398560f26a net/sched: noqueue for classes
- Found: 935d44acf621 memfd: seals on create
- Found: 151e887d8ff9 veth: transmit status
- Found: 158810b261d0 sch_qfq: lmax bound for MTU
- Missed: c9a82bec02c3 cls_api: chain flush
- Missed: f1a3b283f852 sch_fq: attribute validation
- Missed: bb430b694226 loop: uevents for partitions
False alarms
The same ten fixes, left as they shipped
Our system reported no break in any of them.
0 of 10 flagged
One run per fix, on the Linux kernel. Each fix is listed by its upstream commit. No model was trained on the planted defects: they did not exist before the test.
One of the four it caught is a 2022 fix by Linus Torvalds to how the kernel copies a process's table of open files, which left newly forked processes with only their first 64 open files. He repaired it himself eight hours later, before it shipped in a release. Three of the tests the system wrote passed on the old kernel and failed on the fixed one, and from those failures it identified the cause.
The system takes only the fix and the code before it as input, with no network access. Still, we cannot rule out that its detections on historical fixes draw partly on information the model saw during training. To test for that, we also run controlled experiments in which an independent AI agent plants a new defect in a kernel fix that has no known problems. Each planted defect is modeled on a way real kernel fixes have broken things, and none existed when the model was trained. Its detection rate on them was about the same as on the real breaks. Six and ten are small samples, and we are preparing a larger evaluation.
Before it lands
A security fix can't wait, and once it ships it can't simply be rolled back. The best time to learn what it breaks is before it lands.
The kernel learned what this one broke seventeen days after the merge, from a user's bug report. Our system found it in minutes, from the patch.
Fix verification is what we are working on at IR Labs. If it would help your team, request early access.
How we counted one in thirty: of 97,933 tagged fixes to kernel runtime code with a year of history after them, 5,157 were named by a later fix within that year. We sampled 200 of those; 129 showed a new failure the fix introduced or set off. Together that is 3.40%, with a sampling interval of 3.04% to 3.73%. A model screened the commit messages and diffs, and 20 cases were checked against the source. For comparison, a later fix named 5.3% of tagged fixes and 5.4% of other commits to runtime code within a year, counting from 2013. Sasha Levin's 3.8% is the same kind of count, fixes later named by another fix, taken on the 5.15 stable series alone.
