Skip to content

The fix can't wait. What breaks when it lands?

Joe Hoover · November 19, 2026

Just this week, two major outages were blamed on fixes. But this isn't a new problem.

For example, on November 10, 2024, the Linux kernel merged a fix for a security hole reported by Google's Project Zero. Two reviewers approved it. A third developer tested it. It worked.

Seventeen days later, a user found that it had also broken a feature its own author had built the year before.

Nothing about that fix was careless. And there is nothing unusual about a fix that breaks something. In the Linux kernel, a fix is about as likely to need fixing as any other change, and when it does, it is usually because it broke something new. We traced the kernel's fix graph, the record of which change each fix repairs, and found that roughly one in thirty runtime kernel fixes broke something that then needed a fix of its own within a year. And for some, one repair was not the end of it.

Linux kernel · fixes to runtime code

One fix in thirty needed a fix of its own

About one kernel fix in thirty introduced a failure repaired within a year. Among the 7,478 fixes that a later fix repaired, one in ten started a chain of repairs and one in six drew two or more separate repairs.

3.40% broke something that needed its own fix within a year · sampling interval 3.04%–3.73%

When a fix needs a fix, it often does not stop at one.

Shares of the fixes that needed a fix: the 7,478 that a later fix repaired at any point in the kernel's history.

Chains · the repair also needed a repair

A is repaired by B, B by C, C by D

1 in 10 of fixes that needed a fix started a chain (743 fixes)

Two repairs deep
657
Three deep
73
Four deep
13

Branches · one fix, several repairs

A is repaired by B, C, D and E

1 in 6 of fixes that needed a fix drew several repairs (1,289 fixes)

Two repairs
959
Three
218
Four or more
112

One in thirty: 5,157 of 97,933 tagged fixes were named by a later fix within a year; in a sample of 200 of those, 129 showed a new failure the fix introduced or set off. Model-assisted audit of commit messages and diffs. Chains and branches count every recorded repair link at any distance in time, so they describe a wider group than the one in thirty and are not screened for cause in the same way.

The kernel's own maintainers see the same thing. In 2023 Sasha Levin, who maintains the stable kernels, counted 9,422 fixes in one stable series and found that 360 of them, 3.8%, had gone on to need a fix themselves.

The half of fix verification with no failing test

A fix arrives with half of its verification already written. There is a bug report and a failing case, and the fix makes the failing case pass. That is an experiment with a clear result, and good teams run it every time. The kernel fix passed it.

The other half has no failing case to start from. What else depended on the code that changed? Nobody can list every downstream effect of a change, so mostly nobody tries. The answer shows up later, in someone else's bug report.

Often much later.

Linux kernel · fixes named by a later fix

The longer a fix has been in the kernel, the more likely it has needed another

Share of kernel fixes named by a later fix: 3.44 percent within 90 days, 4.26 within 180 days, 5.10 within a year, 5.84 within two years, 6.62 within four years.

  1. 90 days 3.44% 4,230 of 122,918
  2. 180 days 4.26% 5,075 of 119,185
  3. 1 year 5.10% 5,685 of 111,557
  4. 2 years 5.84% 5,686 of 97,420
  5. 4 years 6.62% 4,556 of 68,867

Share of fixes named by a later fix within each window

Each window includes only fixes old enough for the full follow-up, so the groups differ. A recorded repair alone does not establish a new defect.

And sometimes more than once. A fix can need several repairs, and a repair can need its own. For example, a 2022 network-driver fix was repaired that year and again in 2024. In 2025, the second repair needed one too.

Linux kernel · one network-driver fix

One fix, two repairs, and a repair of a repair

A 2022 kernel fix drew two separate repairs, in November 2022 and May 2024; the second needed a repair of its own in January 2025.

Original fix · Jan 2022

Release the semaphore after freeing the slot

Prevents a race when a command slot is allocated.

8e715cd613a1

Two branches Two separate repairs

Repair A · Nov 2022

Allow a waiting thread to yield the CPU

Repairs a single-CPU deadlock.

2808b37b5928

Repair B · May 2024

Wait for a slot before starting the completion timer

Repairs access through an invalid index.

485d65e13571

Another generation One repair needed its own

Repair of B · Jan 2025

Signal completion on the early error return

Repairs a command that can hang.

0e2909c6bec9

Links are recorded repair tags, checked against the source. A selected case from the mlx5 network driver; no failure was reproduced.

Of course, breaking something is occasionally a deliberate trade. You land a hotfix knowing what it will break, because the alternative is worse. But that is rarely what the record shows. In a sample of 129 kernel fixes that went on to break something, the commit message of exactly one saw the failure coming.

The problem has become much more important because fixes are landing faster. Last month Trail of Bits wrote that bug disclosures now "arrive at an unprecedented speed and scale," and every one of them needs a fix. For example, on November 17, the Director of National Intelligence said an outage on the NSA's classified network came from a patch for critical vulnerabilities: some systems, he said, did "not come back up when you patch them as expected." (The agency itself described scheduled maintenance.) Two days earlier, Microsoft traced a day-long email outage to a defect introduced by a fix it had deployed in July. By the time the defect surfaced the change was months old, and Microsoft's usual rollback tools could not quickly revert it.

The break nobody knew about

Checking that a fix works is not enough. Fix verification has to be defensive: it has to go looking for what the fix broke, before anyone knows there is anything to find. The 2024 kernel fix shows what happens otherwise.

That fix was not a minor cleanup. It closed a kernel vulnerability, CVE-2024-53096, which the National Vulnerability Database scores 7.8 out of 10 and Red Hat 6.1, and it was backported to four long-term kernel series. A week after it merged, it went out in the Linux 6.12 release, and the break nobody knew about went out with it.

The initial fix changed the kernel function that maps files into a program's memory, a function the author's own commit message calls "somewhat terrifying." Before the fix, the function sometimes refused a request too late, after it had already torn down whatever was mapped there. A late refusal could leak resources and leave memory in an inconsistent state, which is what made it a security hole. To prevent this, the fix moved every check that could refuse a request to the front.

One of those checks guards sealed files: files locked against writing so that programs can share them safely. But a sealed file is still supposed to be readable and whether the kernel allowed that depended on the check running second. The fix made it run first, and read-only mappings of sealed files were refused.

Linux 6.12 · mapping a write-sealed file

How the fix changed the order of two steps

Before the fix, the file's setup step runs first and the seal check passes. With the fix, the seal check runs first and refuses the mapping; the setup step never runs.

Before the fix

Step 1

The file's own setup

Marks this mapping as one that can never become writable.

Step 2

The seal check

Could anything write through this mapping? No.

Mapping allowed Read-only, as documented

With the fix

Step 1 · moved earlier

The seal check

Could anything write through this mapping? Not ruled out yet.

Mapping refused Operation not permitted

Step 2

The file's own setup

Never runs.

The fix moved every check that can refuse a request to the front, so a refusal can no longer leave damage behind. A write-sealed file relied on this one check coming second. Simplified from the kernel's file-mapping path.

Seventeen days after the fix merged, and ten days after it shipped in Linux 6.12, a user reported the break. Even then, the fix stayed in: it was closing a security hole in kernels people were already running. So the repair kept the fix and rebuilt the sealed-file feature around it. It reached the kernel eight weeks after the fix did. Every Linux 6.12 release from 6.12.0 through 6.12.8 shipped with the break.

The older long-term kernels that received the fix were spared, because at the time they did not have the sealed-file feature to break. Ubuntu's 6.8 kernel, the one in its 24.04 long-term release, did. It picked up the break in April 2025 through its own backport of the fix, and one mis-backported line kept it there until a one-line patch in early 2026.

Had anyone known about the break on November 10, it could have been fixed in the same patch, before any of this happened.

Our work on fix verification

We are working on fix verification to make it easier for teams that maintain systems code to land a fix without breaking something else.

Our system examines a fix, identifies what it corrects, and establishes the standing obligations the changed code must continue to meet. It then searches for violations of those obligations: for each one it constructs an executable test, states the expected outcome in advance, and runs it on the code before and after the fix. It works on user-space code and, as here, inside the operating-system kernel itself.

We ran it on the 2024 fix with only what a reviewer had on the day it merged: the patch, and the source it applied to.

Fix verification · one kernel fix, start to finish

It confirmed the fix. Then it found what the fix broke.

Our system read the fix, mapped what depends on the changed code, wrote 8 tests and reported 2 repaired, 2 broken and 4 unchanged. All 8 pass on the later repair.

1 · Read the fix

What is it for?

Every check that can refuse a mapping request moves to the front, before the kernel has changed anything.

The patch

mm: resolve faulty mmap_region() error path

2 · Map its obligations

What depends on it?

Must be repaired

  • A refused request leaves the existing mapping intact

Must still work

  • Sealed files stay mappable read-only
  • Private mappings of sealed files work
  • Ordinary file mappings behave as before

3 · Write defensive tests

Does each one hold?

8 executable tests, written for this fix

Each test states what should happen, then runs on the kernel before the fix and on the kernel with it.

Both kernels are built from source and booted in a virtual machine.

4 · Report what changed

What it found

2 repaired

Failed before the fix, pass with it

2 broken

Passed before the fix, fail with it:

a sealed file can no longer be mapped read-only

4 unchanged

Pass before and after

The same 8 tests, run on the repair the kernel later shipped: 8 of 8 pass.

Our system was given the patch and the kernel source it applied to, and nothing later. Results are from executed tests on the kernel immediately before and after the fix, booted in a virtual machine.

It wrote eight tests. Two confirmed the fix: a refused request no longer destroys what was already mapped there. Two failed: a sealed file could no longer be mapped read-only, with or without a fixed address. The other four checked neighbouring behaviour, such as private mappings of sealed files and mappings of ordinary files, and passed before and after.

That failure is the regression the user reported seventeen days after the merge. Our system found it from the patch alone, in under six minutes of model and test time, and said so:

The candidate preserves existing mappings when the tested MAP_FIXED requests are rejected, but it also rejects read-only shared mappings of F_SEAL_WRITE memfds that succeed on the parent.

From the system's report on this fix

For the person deciding whether to ship, that changes the conversation. "Moving this check might affect sealed files" is an opinion, and it invites a debate. "On your patch a sealed file can no longer be mapped read-only; on the previous kernel it can; here is the test" is a result.

In a subset of our early results, all on Linux kernel fixes, our system identified 4 of 6 real breaks (67%) and 7 of 10 planted defects (70%).

Fix verification · a subset of our preliminary results

Breaks found, on real kernel fixes and on planted defects

Our system found the break in 4 of 6 kernel fixes that were later repaired upstream and in 7 of 10 fixes with a newly planted defect, and flagged none of the same 10 fixes left untouched. Each fix is listed by its upstream commit.

Real breaks

Six kernel fixes that upstream later had to repair

Both misses were quiet: fix confirmed, nothing reported.

4 of 6 found

Planted defects

Ten other kernel fixes, a new defect planted in each

Written by an independent AI agent on the day of the test.

7 of 10 found

False alarms

The same ten fixes, left as they shipped

Our system reported no break in any of them.

0 of 10 flagged

One run per fix, on the Linux kernel. Each fix is listed by its upstream commit. No model was trained on the planted defects: they did not exist before the test.

One of the four it caught is a 2022 fix by Linus Torvalds to how the kernel copies a process's table of open files, which left newly forked processes with only their first 64 open files. He repaired it himself eight hours later, before it shipped in a release. Three of the tests the system wrote passed on the old kernel and failed on the fixed one, and from those failures it identified the cause.

The system takes only the fix and the code before it as input, with no network access. Still, we cannot rule out that its detections on historical fixes draw partly on information the model saw during training. To test for that, we also run controlled experiments in which an independent AI agent plants a new defect in a kernel fix that has no known problems. Each planted defect is modeled on a way real kernel fixes have broken things, and none existed when the model was trained. Its detection rate on them was about the same as on the real breaks. Six and ten are small samples, and we are preparing a larger evaluation.

Before it lands

A security fix can't wait, and once it ships it can't simply be rolled back. The best time to learn what it breaks is before it lands.

The kernel learned what this one broke seventeen days after the merge, from a user's bug report. Our system found it in minutes, from the patch.

Fix verification is what we are working on at IR Labs. If it would help your team, request early access.

How we counted one in thirty: of 97,933 tagged fixes to kernel runtime code with a year of history after them, 5,157 were named by a later fix within that year. We sampled 200 of those; 129 showed a new failure the fix introduced or set off. Together that is 3.40%, with a sampling interval of 3.04% to 3.73%. A model screened the commit messages and diffs, and 20 cases were checked against the source. For comparison, a later fix named 5.3% of tagged fixes and 5.4% of other commits to runtime code within a year, counting from 2013. Sasha Levin's 3.8% is the same kind of count, fixes later named by another fix, taken on the 5.15 stable series alone.