Zero-Downtime Silicon Surgery: Authoring Linux Kernel Livepatches (klp & kpatch-build) to Remediate Ring-0 CVEs Without Rebooting

Imagine you are operating a fleet of 128 bare-metal GPU nodes running a 72-hour distributed training job—or a high-frequency KVM hypervisor cluster holding 4 TB of pinned guest memory—when an urgent local privilege escalation CVE drops in net/netfilter/nf_tables_api.c or fs/io_uring.c.

Rebooting those nodes to boot a new vmlinuz kernel image means tearing down active RDMA Queue Pairs, flushing terabytes of warm page/KV caches, and incurring hours of cluster orchestration downtime. Leaving the nodes unpatched exposes Ring-0 to container escape.

Linux Kernel Livepatching (CONFIG_LIVEPATCH / klp) solves this dilemma by performing surgical, in-memory function replacement on the running kernel—splicing patched C functions into Ring-0 execution paths in microseconds without dropping a single TCP socket, CUDA context, or KVM guest thread.

Here is the internal engineering breakdown of how klp hijacks function prologues via ftrace, how the kernel’s Hybrid Per-Task Consistency Model prevents torn state across SMP cores, how to add new struct fields to live kernel objects using Shadow Variables (klp_shadow_alloc), how Atomic Cumulative Replacement (.replace = true) upgrades stacked patches, and how to compile and load your own custom livepatch module with kpatch-build.

1. How CONFIG_LIVEPATCH Redirects Execution Inside Ring-0 (ftrace + NOP Sleds)

How can the Linux kernel replace an arbitrary compiled C function—say, nf_tables_newrule()—while 128 CPU cores are actively executing inside the kernel?

The secret lies in a compiler flag enabled on every modern enterprise and upstream Linux kernel (CONFIG_DYNAMIC_FTRACE_WITH_REGS=y / CONFIG_DYNAMIC_FTRACE_WITH_ARGS=y with CONFIG_HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS):

1. When the kernel is compiled with -pg -mfentry (or -fpatchable-function-entry=5), GCC/Clang inserts a 5-byte call __fentry__ instruction at the very first byte of every non-inlined kernel function (immediately after endbr64 if Intel CET CONFIG_X86_KERNEL_IBT is active).
2. At boot time (ftrace_init), the kernel overwrites all of those 5-byte call __fentry__ instructions with a single hardware 5-byte NOP instruction (0f 1f 44 00 00, nopl 0x0(%rax,%rax,1)) so normal execution pays zero cycle overhead.
3. When you load a Livepatch module (insmod livepatch-cve-2026.ko), klp registers an ftrace_ops handler with FTRACE_OPS_FL_IPMODIFY on the target function’s address and atomically text-pokes (text_poke_bp) that 5-byte NOP into a call klp_ftrace_handler:

+-------------------------------------------------------------------------------+
|          LINUX KERNEL LIVEPATCH (KLP) FTRACE TRAMPOLINE REDIRECTION           |
+-------------------------------------------------------------------------------+
|                                                                               |
|  [ BEFORE LIVEPATCH (Zero Overhead) ]                                         |
|    vulnerable_func():                                                         |
|      0xffffffff81402100:  0f 1f 44 00 00    nopl   0x0(%rax,%rax,1)           |
|      0xffffffff81402105:  55                push   %rbp                       |
|      0xffffffff81402106:  48 89 e5          mov    %rsp,%rbp                  |
|      ... (executes vulnerable C code) ...                                     |
|                                                                               |
|  ---------------------------------------------------------------------------  |
|                                                                               |
|  [ AFTER LIVEPATCH LOADED ]                                                   |
|    vulnerable_func():                                                         |
|      0xffffffff81402100:  e8 7b d4 e2 3e    call   klp_ftrace_handler         |
|                                                      |                        |
|      +-----------------------------------------------+                        |
|      v                                                                        |
|    klp_ftrace_handler(ip, parent_ip, fops, fregs):                            |
|      ├── Reads RCU-protected `func = list_first_or_null_rcu(&ops->func_stack)`|
|      ├── Checks `current->patch_state` (`TIF_PATCH_PENDING`)                  |
|      ├── Rewrites instruction pointer: `ftrace_regs_set_instruction_pointer`  |
|      └── `ret` jumps directly into `klp_patched_func()`!                      |
|                                                                               |
+-------------------------------------------------------------------------------+

When klp_ftrace_handler() (kernel/livepatch/patch.c) intercepts the function entry, it inspects the calling task’s patch_state and modifies the saved instruction pointer (ftrace_regs_set_instruction_pointer(fregs, (unsigned long)func->new_func)) to point to your replacement function inside the livepatch .ko module. When klp_patched_func() returns, it returns directly to parent_ip—skipping the original vulnerable function body completely!

2. The Hybrid Per-Task Consistency Model (TIF_PATCH_PENDING & ORC Unwinding)

What if CPU 4 is already halfway through executing vulnerable_func() (or sleeping on a mutex inside a callee of vulnerable_func()) at the exact microsecond CPU 0 loads a livepatch that patches both vulnerable_func() and its sibling helper_func()?

If CPU 4 resumed inside vulnerable_func() using the old locking or reference-counting semantics and then called helper_func() executing the new semantics, the kernel would immediately panic or corrupt slab memory.

To prevent mixed-version execution within any thread’s call chain, Linux klp enforces a Hybrid Per-Task Consistency Model (kernel/livepatch/transition.c) combining kGraft-style syscall boundary tracking with kpatch-style stack unwinding:

1. Initialize Transition (klp_init_transition): When a livepatch starts transitioning toward KLP_TRANSITION_PATCHED, the kernel sets each task’s task->patch_state = KLP_TRANSITION_UNPATCHED and sets the thread flag TIF_PATCH_PENDING on every single task_struct in the system (for_each_process_thread).
2. Per-Thread Execution Universe (OLD vs. NEW): Inside klp_ftrace_handler(), the kernel reads patch_state = current->patch_state. As long as a task has TIF_PATCH_PENDING set (KLP_TRANSITION_UNPATCHED), klp_ftrace_handler() directs that task to the OLD functions. The moment a task clears TIF_PATCH_PENDING, its patch_state flips to KLP_TRANSITION_PATCHED and it permanently enters the NEW patched function universe.
3. Three Mechanisms to Clear TIF_PATCH_PENDING:
   • Syscall / Trap Return Boundary (klp_update_patch_state): Whenever any userspace thread exits the kernel back to Ring-3 (exit_to_user_mode_loop), its kernel call stack is guaranteed to be 100% empty! It cannot be inside any patched kernel function, so the kernel clears TIF_PATCH_PENDING and switches the task to KLP_TRANSITION_PATCHED in O(1) time.
   • Reliable Stack Unwinding (klp_check_stack + ORC Unwinder): What about idle CPUs (swapper/N), kernel threads (kworker, ksoftirqd, vhost-net), or tasks sleeping in epoll_wait()? A periodic workqueue (klp_transition_work) invokes task_call_func() to freeze each non-running task’s stack and runs arch_stack_walk_reliable() (backed by compile-time objtool ORC unwind tables via CONFIG_HAVE_RELIABLE_STACKTRACE=y). If none of the functions on the task’s stack are in the patch’s klp_func list, the task is safely switched to KLP_TRANSITION_PATCHED!
   • Fake Signal Wakeup (klp_send_signals) & force: If a task remains stalled in TIF_PATCH_PENDING for more than 15 seconds (for example, blocked in an interruptible sleep or tight vhost loop), klp sends a benign fake signal (SIGSTOP/TIF_SIGPENDING wakeup without delivering a fatal signal) to nudge the thread across the kernel/user boundary.

Once every task in the system clears TIF_PATCH_PENDING, klp_complete_transition() runs, /sys/kernel/livepatch/<patch>/transition flips from 1 -> 0, and old superseded livepatch modules (when .replace = true) are automatically unlinked!

3. Writing a Livepatch in C: klp_patch, klp_shadow_alloc, and .replace = true

What if fixing a CVE (such as a double-free or missing generation check in struct nft_rule or struct file) requires adding a new bool or refcount_t field to an existing kernel struct? You cannot modify the C struct definition directly in a livepatch because thousands of struct nft_rule objects are already allocated with the old size in the kernel’s SLUB allocator!

Instead, CONFIG_LIVEPATCH provides Shadow Variables (<linux/livepatch.h>) and State Callbacks (struct klp_callbacks):

• klp_shadow_get_or_alloc(obj, id, size, gfp_flags, ctor, ctor_data) attaches an RCU-protected sidecar structure to any existing kernel pointer obj inside the global klp_shadow_hash table (kernel/livepatch/shadow.c).
• Setting .replace = true on struct klp_patch enables Atomic Cumulative Livepatching: the new livepatch atomically replaces all previously loaded livepatches in a single transition, creating dynamic nop trampolines to revert any previously patched functions that are no longer modified by the cumulative patch.

Here is a complete, production-structured C livepatch (livepatch_cve_2026_fix.c) demonstrating Shadow Variables, Pre/Post-Patch Callbacks, and Cumulative Atomic Replace:

// SPDX-License-Identifier: GPL-2.0
// livepatch_cve_2026_fix.c — Ring-0 Livepatch with Shadow Variables & Callbacks
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt

#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/livepatch.h>
#include <linux/seq_file.h>
#include <linux/slab.h>
#include <linux/jiffies.h>

#define KLP_SHADOW_CVE_2026_ID 0x20261030UL

struct seq_cve_shadow {
    u64 first_access_jiffies;
    pid_t creator_pid;
    bool sanitized;
};

/*
 * Replacement function for proc_cmdline_show().
 * Attaches an out-of-band shadow struct to `struct seq_file *m` via RCU hash
 * lookup without altering sizeof(struct seq_file) in the SLUB cache.
 */
static int klp_proc_cmdline_show(struct seq_file *m, void *v)
{
    struct seq_cve_shadow *shadow;

    shadow = klp_shadow_get_or_alloc(m, KLP_SHADOW_CVE_2026_ID,
                                     sizeof(*shadow), GFP_ATOMIC,
                                     NULL, NULL);
    if (shadow && !shadow->sanitized) {
        shadow->first_access_jiffies = get_jiffies_64();
        shadow->creator_pid = current->pid;
        shadow->sanitized = true;
    }

    seq_printf(m, "%s klp_cve_2026_remediated=1\n", saved_command_line);

    /* Free shadow metadata when object lifecycle completes */
    klp_shadow_free(m, KLP_SHADOW_CVE_2026_ID, NULL);
    return 0;
}

/* Pre-patch and post-unpatch lifecycle callbacks */
static int klp_pre_patch_cb(struct klp_object *obj)
{
    pr_info("Preparing Ring-0 ftrace transition for object: %s\n",
            obj->name ? obj->name : "vmlinux");
    return 0;
}

static void klp_post_unpatch_cb(struct klp_object *obj)
{
    /* Sweep any remaining shadow variables matching KLP_SHADOW_CVE_2026_ID */
    klp_shadow_free_all(KLP_SHADOW_CVE_2026_ID, NULL);
    pr_info("Cleaned up all shadow variables for KLP_SHADOW_CVE_2026_ID\n");
}

static struct klp_func vmlinux_funcs[] = {
    {
        .old_name = "proc_cmdline_show",
        .new_func = klp_proc_cmdline_show,
    },
    { }
};

static struct klp_object patch_objs[] = {
    {
        /* NULL targets core vmlinux; set to e.g. "nf_tables" for a .ko module */
        .name = NULL,
        .funcs = vmlinux_funcs,
        .callbacks = {
            .pre_patch = klp_pre_patch_cb,
            .post_unpatch = klp_post_unpatch_cb,
        },
    },
    { }
};

static struct klp_patch livepatch_cve_2026 = {
    .mod = THIS_MODULE,
    .objs = patch_objs,
    /* Atomic replace: supersedes any older loaded livepatches cleanly */
    .replace = true,
};

static int __init livepatch_cve_init(void)
{
    return klp_enable_patch(&livepatch_cve_2026);
}

static void __exit livepatch_cve_exit(void)
{
}

module_init(livepatch_cve_init);
module_exit(livepatch_cve_exit);
MODULE_LICENSE("GPL");
MODULE_INFO(livepatch, "Y");

4. Automated Binary ELF Section Diffing with kpatch-build & .klp.rela Relocations

In production, hand-writing struct klp_func wrappers breaks down as soon as a patched C function accesses a file-scoped static variable or calls an unexported static helper inside vmlinux—because a normal loadable kernel module (.ko) can only resolve EXPORT_SYMBOL globals!

How does Red Hat’s kpatch-build solve unexported static symbols and inlined callees automatically?

1. Dual Compilation with Function/Data Sections: kpatch-build intercepts Kbuild and compiles the target kernel twice with -ffunction-sections -fdata-sections—placing every individual C function into its own .text.<func_name> ELF section (orig.o vs. patched.o).
2. Binary ELF Comparison (create-diff-object): create-diff-object parses the DWARF debug info and ELF relocation tables of orig.o and patched.o, automatically pulling in any callers whose inlined functions changed.
3. Special .klp.rela.<objname> Sections (SHF_RELA_LIVEPATCH): For any unexported static symbol referenced by the patched function, kpatch-build emits a special ELF relocation section .klp.rela.vmlinux..text.klp_func using symbol names formatted as .klp.sym.vmlinux.<symbol_name>,<sympos> (where <sympos> disambiguates duplicate static symbol names across different .c files in kallsyms). When the module loads (or when a delayed target module like nf_tables.ko loads later), the kernel’s klp_write_section_relocs() resolves those unexported addresses directly from kallsyms!

#!/usr/bin/env bash
set -euo pipefail

# 1. Build a cumulative livepatch .ko directly from a unified git diff (cve-2026-nft.patch)
#    against the exact debuginfo vmlinux of your running kernel:
kpatch-build \
  --sourcerpm /var/cache/kernel-src/linux-6.12.8.src.rpm \
  --vmlinux /usr/lib/debug/lib/modules/$(uname -r)/vmlinux \
  --jobs $(nproc) \
  cve-2026-nft.patch

# 2. Inspect the synthesized .klp.rela symbol relocations and klp_funcs inside the .ko
readelf -SW livepatch-cve-2026-nft.ko | grep -E "\.klp\.rela|\.text\."

# 3. Sign the livepatch module with your fleet MOK / SecureBoot X.509 key
/usr/src/kernels/$(uname -r)/scripts/sign-file \
  sha512 /etc/keys/mok_priv.der /etc/keys/mok_pub.der \
  livepatch-cve-2026-nft.ko

# 4. Load the livepatch and monitor per-task consistency convergence in sysfs
insmod livepatch-cve-2026-nft.ko
cat /sys/kernel/livepatch/livepatch_cve_2026_nft/enabled     # 1
cat /sys/kernel/livepatch/livepatch_cve_2026_nft/transition  # 0 (All tasks converged!)

# 5. Diagnose any pinned thread holding TIF_PATCH_PENDING during a slow transition:
grep -rn "1" /proc/[0-9]*/task/[0-9]*/patch_state 2>/dev/null || echo "0 pending tasks"

5. Production Comparison: Reboot vs. User-Space kexec vs. CONFIG_LIVEPATCH

How does CONFIG_LIVEPATCH compare against alternative kernel remediation strategies across a 128-node GPU or KVM hypervisor fleet?

With CONFIG_LIVEPATCH + kpatch-build, you can neutralize critical Ring-0 vulnerabilities across hundreds of bare-metal servers in milliseconds—preserving 100% workload uptime until your next scheduled maintenance window.