Daily Tech Now

Daily News on AI, Big Tech, Linux, Security & Gadgets

Kmorph Keeps a Spare Linux Kernel Running to Take Over After a Crash

A Linux server that hits a kernel panic normally needs a full reboot before it serves anyone again. A new open-source tool called kmorph tries to skip that step. It keeps a second kernel already running on the same machine, ready to take over when the first one dies. The code lives in the kmorph…

Linux kernel crash recovery with kmorph, where a spare successor kernel takes over a server after its main kernel crashes

A Linux server that hits a kernel panic normally needs a full reboot before it serves anyone again. A new open-source tool called kmorph tries to skip that step. It keeps a second kernel already running on the same machine, ready to take over when the first one dies.

The code lives in the kmorph repository on GitHub, under the Multikernel Technologies organization. Its README says kmorph moves “a running multikernel Linux machine from one kernel into another without a reboot.” The project’s documentation puts the gap at about one second from detecting a crash to a running replacement kernel.

Hardware Busters, in its report on kmorph, says Phoronix first covered the tool. It cites an nginx test in which the service came back after about two seconds of downtime.

How kmorph works

Kmorph uses two names for the two kernels. The kernel running the machine is the predecessor. The spare kernel is the successor, and it boots beside the predecessor on its own small share of CPUs and memory. The README’s minimal example gives it one CPU and 128 MB of RAM.

Two programs handle the job. The kmorph command runs on the main system. It builds the successor’s image, arms or disarms it, and keeps its copy of the network settings current. The kmorphd daemon runs inside the successor and watches the main kernel.

By default, kmorphd checks on the main kernel ten times a second. Five missed checks in a row mark it as dead, which takes about half a second. The successor then stops the old kernel’s CPUs and claims the CPUs, memory and devices for itself. It can also save the crashed kernel’s memory as a crash dump first.

The project comes from Cong Wang, founder and CEO of Multikernel Technologies. In his SCaLE 23x presentation on multikernel Linux, he described running several Linux kernels on one machine at the same time. Each kernel gets its own CPUs, memory and devices, with no hypervisor or virtual machines involved.

What changes compared with kdump

Most Linux servers today rely on kdump after a crash. According to the kernel’s kdump documentation, kdump uses kexec to boot a separate dump-capture kernel when the system crashes. That kernel exists to save the failed kernel’s memory for later study. The kexec manual page explains that kexec boots a new kernel from the running one and skips the firmware stage.

Hardware Busters notes that kdump only collects evidence and the machine then reboots anyway. Kmorph turns the spare kernel into the running system instead. The table below sums up the difference using the projects’ own documentation.

kdump kmorph
Job of the spare kernel Capture a memory dump Become the machine’s kernel
Spare kernel before a crash Loaded into reserved memory, not booted Already booted and watching
After a crash Reboot into the capture kernel Takeover of CPUs, memory and devices, no reboot
Crash dump Exposed as /proc/vmcore Optional vmcore in the same format
Planned kernel upgrades Not its purpose Yes, with kmorph upgrade
Resources set aside Memory reserved with crashkernel= CPUs and memory for the successor
Kernel support Documented in the mainline kernel Needs a multikernel-enabled kernel

Crash analysis does not have to suffer. Kmorph writes its dump in the format of kdump’s /proc/vmcore, so the standard crash and makedumpfile tools can read it. The dump can also stream over the network when the spare kernel has a network card but no disk.

Kernel upgrades without a reboot

Crash recovery is only half the story. The README also describes a kmorph upgrade command that hands a healthy machine to a successor running a newer kernel. On a distribution, the documented flow is three steps: upgrade the kernel package, restart the kmorph service, then run kmorph upgrade.

Before the handover, kmorph freezes every writable filesystem on a block device. The successor then mounts them with nothing in flight. No crash dump is written during an upgrade. If the handover does not complete within ten seconds, kmorph thaws the filesystems and the old kernel stays in charge.

What it takes to try kmorph

Kmorph is not a drop-in package for a stock kernel. Hardware Busters reports that the multikernel architecture was proposed as an RFC in September 2025 and is not part of the mainline kernel. Admins who want to test it need to check these points first:

  • A multikernel Linux kernel built with CONFIG_MULTIKERNEL=y and CONFIG_MULTIKERNEL_VSOCKETS=y, plus the host tree interface.
  • An x86-64 or arm64 machine.
  • CONFIG_CRASH_DUMP=y, if CPU registers and VMCOREINFO should appear in the dump.
  • Root access on the main system.
  • Enough spare CPUs and memory to dedicate to the successor while it is armed.

Packaging follows the kdump model. The install adds a dracut module, an initramfs-tools hook, a kmorph.service unit, and a hook for kernel-install and Debian’s postinst.d. That hook rebuilds the successor image when a new kernel lands. After the operator writes a short config file, one command enables protection: systemctl enable --now kmorph.

Standby services

Restarting every service after a takeover takes time. Kmorph can start chosen systemd units, such as nginx.service, inside the successor ahead of time. Those services keep running through the takeover. The README sets a firm rule, though. A standby service must not write state that has to persist, or reach the network, before the takeover.

Technical details: fencing, devices and the network

Stopping the old kernel

The successor must be sure the dead kernel is no longer running. It stops every CPU it does not own, using NMI and INIT on x86 or a stop IPI and PSCI on arm64. It then confirms each CPU halted and refuses to continue with a partial fence. A slow kernel is not treated as dead, because only silence over several checks counts.

Resetting devices

Hardware left running by a crashed kernel can keep writing to memory through DMA. By default, kmorphd resets each adopted PCI device before its driver loads. If a device cannot be reset, kmorphd turns off its ability to start memory transfers. That way, stray DMA stops before the old kernel’s memory is reused.

Keeping the same network identity

Kmorph records each network interface’s name, MAC address, addresses and routes in advance. After a takeover, it puts them back on the same hardware. The MAC address does not change, so neighbors do not need to learn a new one. A physical network card still has to renegotiate its link after the reset, which can take a second or more.

Limits and open questions

The README lists clear limits. A takeover is one-shot. The new kernel stays unprotected until it arms a successor of its own, which kmorph.service does on a distribution. Adopted devices lose secondary bus reset, hotplug and AER error reporting. A device added after arming stays as the crash left it.

Memory is another trade-off. The successor claims memory in whole chunks, and some chunks near reserved regions stay unclaimed. With 4 MB chunks, that is a few megabytes. On a 2 GB machine using 128 MB memory blocks, the README estimates about 500 MB.

The bigger open question is the kernel itself. Mainline Linux already has related tools. The kernel documents Kexec HandOver (KHO) and the Live Update Orchestrator (LUO) for carrying state into a new kernel through kexec. Both expect a working kernel to prepare the handover. Kmorph also targets the case where the old kernel cannot cooperate because it has crashed. Whether the multikernel patches it depends on reach mainline will decide how widely that idea spreads.

Independent numbers are also scarce. Beyond the project’s own timings and the reported nginx test, there are no published results on large production servers yet.

About the Author

Trang Nguyen Avatar

Leave a Reply

Your email address will not be published. Required fields are marked *