Skip to main content

Recovering From a Kernel Panic in Linux

Media Recovering From a Kernel Panic in Linux

A Linux kernel panic is one of the most serious failures a Linux system can experience. Instead of an application crashing or a service stopping, the operating system’s kernel reaches a condition from which it cannot safely continue.

The result may be a frozen screen filled with diagnostic text, an automatic reboot, or a server that stops responding completely.

A kernel panic can look alarming, but it does not always mean the machine is permanently damaged. Many panics are caused by faulty drivers, broken kernel updates, unstable hardware, filesystem corruption or incorrect boot configuration. With a structured recovery process, it is often possible to get the system running again and identify the underlying cause.

This guide explains what a kernel panic is, how to recover access to the system and how to reduce the risk of it happening again.

What Is a Kernel Panic?

The Linux kernel sits between the hardware and the rest of the operating system. It controls memory, processors, storage devices, drivers, networking and system processes.

When the kernel encounters a serious error that could corrupt data or leave the machine in an unsafe state, it may stop the system deliberately. This is called a kernel panic.

A typical panic message may include wording such as:

Kernel panic - not syncing: Fatal exception

You may also see:

Unable to mount root fs

or:

VFS: Cannot open root device

The exact message is important because it often points towards the failing component.

Common Causes of Kernel Panics

Kernel panics can be triggered by software, hardware or configuration problems.

Common causes include:

  • A faulty or incompatible kernel module
  • A recently installed kernel update
  • A broken initramfs image
  • Incorrect bootloader configuration
  • A missing root filesystem
  • Filesystem corruption
  • Failing RAM
  • A failing SSD or hard drive
  • CPU overheating or instability
  • Unsupported hardware
  • Incorrect overclocking settings
  • Problems caused by third-party security or storage drivers
  • Virtual machine configuration faults

The last change made before the panic is often an important clue.

For example, if the problem started immediately after installing a new kernel, the previous kernel may still work. If it started after adding a graphics driver, that driver may be involved.

Record the Panic Message

Before rebooting, take a photograph of the screen or record the exact error message.

Important details may include:

  • The first line containing “Kernel panic”
  • Any named kernel module
  • The device or partition mentioned
  • References to the root filesystem
  • Error codes
  • The kernel version
  • The final few lines before the system stopped

The screen may contain a large amount of technical output, but the first clear error is often more useful than the long stack trace that follows.

If the machine reboots too quickly to read the message, disable automatic rebooting temporarily from the bootloader or recovery environment if practical.

First Step: Try a Normal Reboot

Some panics are temporary and do not immediately reappear.

Restart the system and observe what happens:

  • Does it boot normally?
  • Does the panic occur at the same point?
  • Does it happen only under load?
  • Does it occur after a particular service starts?
  • Does it happen only with a specific device connected?

If the machine boots successfully, do not assume the problem has gone away. Review the logs and investigate the likely cause.

Boot an Older Kernel

If the panic began after a kernel update, booting the previous kernel is usually the best first recovery step.

Most Linux distributions keep at least one older kernel installed.

Using GRUB

Restart the machine and display the GRUB menu.

Depending on the system, you may need to hold or repeatedly press:

Shift

or:

Esc

during startup.

Select:

Advanced options for Linux

You should see multiple kernel versions.

Choose an older kernel, preferably one that does not include “recovery mode” initially.

For example:

Linux 6.x.x

Linux 6.x.x-previous

If the older kernel boots successfully, the newer kernel or one of its modules is likely responsible.

Keep the Working Kernel Installed

Once the system is running with the older kernel, confirm which version is active:

uname -r

List installed kernels on Debian or Ubuntu systems:

dpkg --list | grep linux-image

On Fedora, Rocky Linux or Red Hat-based systems:

rpm -q kernel

Do not remove the working kernel until the new one has been repaired or replaced.

A common mistake is deleting older kernels too aggressively. Keeping one known-good kernel provides an important recovery route.

Boot Into Recovery Mode

If a normal older kernel does not work, try recovery mode.

From the GRUB menu:

  1. Select Advanced options for Linux.
  2. Choose a kernel entry marked recovery mode.
  3. Use the recovery menu to inspect filesystems, networking or root access.

Recovery mode may allow you to:

  • Repair packages
  • Check filesystems
  • Rebuild boot files
  • Remove a faulty driver
  • Disable a broken service
  • Access a root shell

The root filesystem may initially be mounted read-only.

Remount it read/write with:

mount -o remount,rw /

Use recovery mode carefully because you are working with elevated privileges.

Check the Previous Boot Logs

If the system booted after the panic, review the logs from the previous boot:

sudo journalctl -b -1

To show kernel messages only:

sudo journalctl -k -b -1

Search for panic-related entries:

sudo journalctl -k -b -1 | grep -iE "panic|oops|fatal|error|fail"

You can also review the current kernel log:

sudo dmesg -T

Look for messages involving:

  • I/O errors
  • Memory faults
  • Filesystem errors
  • Driver failures
  • GPU faults
  • NVMe timeouts
  • SATA errors
  • Machine check exceptions
  • Out-of-memory conditions

Be aware that logs may not always be saved if the system stopped too quickly.

Check for a Kernel Oops

A kernel oops is a serious kernel error that does not always stop the system immediately.

Repeated oops messages may eventually lead to a panic or unstable behaviour.

Search the journal:

sudo journalctl -k | grep -i oops

If a specific module appears repeatedly, it may be the cause.

Example:

BUG: unable to handle kernel paging request

Modules linked in: example_driver

A third-party module named near the failure should be investigated carefully.

Rebuild the initramfs

The initramfs is a temporary filesystem loaded early during boot. It contains drivers and tools needed to locate and mount the real root filesystem.

If it becomes corrupted or does not include the required storage driver, the machine may panic with messages such as:

Unable to mount root fs

On Debian and Ubuntu systems, rebuild it with:

sudo update-initramfs -c -k all

Or update existing images:

sudo update-initramfs -u -k all

On Fedora, Rocky Linux and Red Hat-based systems, use:

sudo dracut --force

To rebuild for a specific kernel:

sudo dracut --force /boot/initramfs-$(uname -r).img $(uname -r)

After rebuilding the initramfs, update the bootloader if required.

Update GRUB

A damaged or outdated GRUB configuration can point to the wrong kernel, initramfs or root partition.

On Debian and Ubuntu systems:

sudo update-grub

On many Red Hat-based BIOS systems:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

On UEFI systems, the path may be different.

For example:

sudo grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg

The exact command depends on the distribution and boot configuration, so confirm the correct path before overwriting anything.

Check the Root Filesystem Entry

A panic mentioning the root filesystem may be caused by an incorrect UUID or missing device.

Check the current filesystem layout:

lsblk -f

Check UUIDs:

sudo blkid

Review /etc/fstab:

cat /etc/fstab

Check the kernel boot parameters:

cat /proc/cmdline

The root filesystem specified by GRUB should match the actual root partition.

A mismatch may occur after:

  • Cloning a disk
  • Replacing a drive
  • Restoring a backup
  • Changing partitions
  • Editing GRUB manually
  • Converting between storage controllers

Check for Filesystem Corruption

Filesystem corruption can prevent Linux from mounting the root volume and may lead to a panic.

Do not run a repair tool on a mounted filesystem.

Boot from recovery media, a live Linux environment or another system, then identify the affected partition:

lsblk -f

For an ext4 filesystem:

sudo fsck -f /dev/sdXn

Replace /dev/sdXn with the correct partition.

For XFS:

sudo xfs_repair /dev/sdXn

XFS may require the filesystem to be unmounted first.

For Btrfs, start with a non-destructive check:

sudo btrfs check --readonly /dev/sdXn

Do not use aggressive repair options unless you understand the consequences and have a backup.

Check Disk Health

A failing storage device may cause read errors, filesystem corruption and kernel panics.

Install SMART tools if necessary:

sudo apt install smartmontools

or:

sudo dnf install smartmontools

Check a SATA drive:

sudo smartctl -a /dev/sda

Check an NVMe drive:

sudo smartctl -a /dev/nvme0

Look for:

  • Reallocated sectors
  • Pending sectors
  • Uncorrectable errors
  • Media errors
  • CRC errors
  • NVMe critical warnings
  • Excessive temperature
  • Failed self-tests

Run a short self-test:

sudo smartctl -t short /dev/sda

After the test completes:

sudo smartctl -a /dev/sda

If the drive appears unhealthy, prioritise data recovery and replacement rather than repeated repair attempts.

Test the Memory

Faulty RAM can cause unpredictable kernel panics, corrupted data and random application crashes.

Many GRUB menus include a memory test option such as Memtest86+.

Run multiple passes, ideally for several hours.

Any reported memory error should be treated seriously.

For systems with multiple memory modules:

  1. Power off the machine.
  2. Reseat the modules.
  3. Test one module at a time.
  4. Test different slots.
  5. Disable memory overclocking.
  6. Restore the manufacturer’s recommended settings.

On servers, also check the hardware management interface for ECC memory errors.

Check CPU Temperature and Stability

Overheating or unstable processor settings can trigger machine-check exceptions and kernel panics.

Check temperatures using:

sensors

Install the tool if needed:

sudo apt install lm-sensors

or:

sudo dnf install lm_sensors

Detect sensors:

sudo sensors-detect

Then run:

sensors

Inspect:

  • CPU temperature
  • Fan operation
  • Thermal throttling
  • Dust buildup
  • Failed cooling fans
  • Poor heatsink contact

Return BIOS or UEFI settings to defaults if the system is overclocked or undervolted.

Investigate Third-Party Kernel Modules

Third-party modules are a common source of panics after kernel updates.

Examples include:

  • Proprietary graphics drivers
  • Antivirus modules
  • Backup agents
  • Storage drivers
  • VPN drivers
  • Virtualisation tools
  • Hardware monitoring software
  • Out-of-tree network drivers

List loaded modules:

lsmod

Check information about a module:

modinfo module_name

Check DKMS status:

dkms status

DKMS rebuilds third-party modules for new kernels. A failed DKMS build may leave a module incompatible with the updated kernel.

Review DKMS logs:

sudo find /var/lib/dkms -name make.log -print

Remove or rebuild the problematic module before booting the affected kernel again.

Temporarily Blacklist a Faulty Module

If a particular module is causing the panic, you can prevent it from loading.

Add it to a blacklist file:

echo "blacklist module_name" | sudo tee /etc/modprobe.d/blacklist-module_name.conf

Rebuild the initramfs:

sudo update-initramfs -u

or:

sudo dracut --force

Then reboot.

Do not blacklist essential storage, filesystem or network modules without understanding the impact.

Test With a Kernel Boot Parameter

GRUB allows temporary kernel parameters to be added for troubleshooting.

At the GRUB menu:

  1. Highlight the kernel.
  2. Press e.
  3. Find the line beginning with linux.
  4. Add the test option to the end.
  5. Press Ctrl+X or F10 to boot.

Possible troubleshooting options include:

nomodeset

This disables normal graphics mode setting and may help when the graphics driver is involved.

Another option is:

systemd.unit=multi-user.target

This boots into a text-based multi-user environment rather than the graphical desktop.

For a minimal emergency shell:

systemd.unit=emergency.target

These changes apply only to that boot unless added permanently to the GRUB configuration.

Remove a Broken Kernel

If one kernel consistently panics and an older kernel works, the broken kernel can be removed and reinstalled.

On Debian or Ubuntu, identify it first:

dpkg --list | grep linux-image

Then remove the exact problematic package:

sudo apt remove linux-image-VERSION

Reinstall it if appropriate:

sudo apt install --reinstall linux-image-VERSION

Update GRUB:

sudo update-grub

On Fedora or Red Hat-based systems, list kernels:

rpm -q kernel

Remove a specific package carefully:

sudo dnf remove kernel-core-VERSION

Never remove the kernel that is currently running.

Confirm the active version first:

uname -r

Reinstall the Kernel Packages

If the kernel or its modules are damaged, reinstalling the package may repair missing files.

On Ubuntu or Debian:

sudo apt install --reinstall linux-image-$(uname -r) linux-modules-$(uname -r)

On some systems, additional module packages may be required:

sudo apt install --reinstall linux-modules-extra-$(uname -r)

On Fedora or Red Hat-based systems:

sudo dnf reinstall kernel-core kernel-modules

After reinstalling, rebuild the initramfs and bootloader configuration.

Use a Live Linux Environment

If the installed system will not boot at all, use a live USB or recovery ISO.

From the live environment, you can:

  • Inspect the disk
  • Copy important files
  • Repair filesystems
  • Review logs
  • Edit /etc/fstab
  • Rebuild the initramfs
  • Repair GRUB
  • Chroot into the installed system

Identify the root partition:

lsblk -f

Mount it:

sudo mount /dev/sdXn /mnt

Mount the boot partition if separate:

sudo mount /dev/sdXm /mnt/boot

For UEFI systems:

sudo mount /dev/sdXp /mnt/boot/efi

Bind the virtual filesystems:

sudo mount --bind /dev /mnt/dev

sudo mount --bind /proc /mnt/proc

sudo mount --bind /sys /mnt/sys

sudo mount --bind /run /mnt/run

Enter the installed system:

sudo chroot /mnt

You can then repair packages, rebuild the initramfs or update GRUB as though the installed system had booted normally.

Exit when finished:

exit

Unmount the filesystems before rebooting.

Kernel Panic in a Virtual Machine

When the affected Linux system is virtual, check the hypervisor as well as the guest.

Look for changes to:

  • Virtual disk controller type
  • CPU compatibility mode
  • Assigned memory
  • Secure Boot
  • Virtual hardware version
  • Snapshot state
  • Disk attachment order
  • Pass-through devices
  • Hypervisor tools

A guest may panic if its root disk was previously connected through one controller type and is now presented through another without the required driver in the initramfs.

Check the hypervisor logs for storage timeouts or host hardware issues.

Kernel Panic During Installation

If the panic occurs while installing Linux:

  • Verify the installation image checksum.
  • Recreate the USB drive.
  • Try a different USB port.
  • Disconnect unnecessary peripherals.
  • Disable overclocking.
  • Test the RAM.
  • Update the BIOS or UEFI.
  • Try safe graphics mode.
  • Use a newer or older installer kernel.
  • Check whether the hardware requires special boot parameters.

Installation panics often indicate media corruption, unsupported hardware or hardware instability.

Configuring Automatic Reboot After a Panic

Servers may be configured to reboot automatically after a kernel panic.

Check the current setting:

sysctl kernel.panic

A value of 0 means the system will not automatically reboot.

To reboot after 30 seconds:

sudo sysctl -w kernel.panic=30

To make it persistent, create a configuration file:

echo "kernel.panic = 30" | sudo tee /etc/sysctl.d/99-kernel-panic.conf

Apply the configuration:

sudo sysctl --system

Automatic rebooting can restore service temporarily, but it can also create a reboot loop. It does not fix the underlying cause.

Preserving Crash Information With kdump

On important servers, configure kernel crash dumping.

The kdump service reserves a small amount of memory for a secondary kernel. If the main kernel panics, the secondary kernel can save a memory dump for later analysis.

Depending on the distribution, the package may be called:

kdump-tools

or:

kexec-tools

A crash dump can provide much more detail than a photograph of the panic screen.

It may reveal:

  • The faulting thread
  • The kernel stack
  • The loaded modules
  • The CPU state
  • Memory structures
  • The exact code path that failed

Crash dumps can be large and may contain sensitive information, so they should be stored securely.

Distinguishing a Panic From Other Failures

Not every frozen Linux machine has experienced a kernel panic.

Similar symptoms may be caused by:

  • A graphical desktop crash
  • A deadlocked application
  • Memory exhaustion
  • Storage timeouts
  • A power supply fault
  • A hardware reset
  • A watchdog reboot
  • A hypervisor issue
  • A complete network outage

A genuine kernel panic usually displays an explicit panic message or leaves evidence in logs or crash dumps.

If the screen is frozen but keyboard input still works, try switching to another console:

Ctrl+Alt+F3

If remote access still works, the kernel may still be running and only the graphical session may have failed.

Common Recovery Mistakes

Reinstalling Linux Immediately

A reinstall may hide the cause without resolving faulty hardware.

Check the disk, memory and logs first.

Removing Every Older Kernel

Older kernels provide a valuable fallback.

Keep at least one known-good version installed.

Repairing a Mounted Filesystem

Running filesystem repair against a mounted filesystem can cause further damage.

Use recovery mode or live media.

Assuming the Latest Kernel Must Be Correct

A newly released kernel can expose a driver or hardware compatibility issue.

Boot the previous version to confirm.

Ignoring Third-Party Modules

A kernel update may be stable while an external driver is not.

Check DKMS and out-of-tree modules.

Rebooting Repeatedly Without Recording the Error

Repeated restarts can overwrite useful logs and make the original error harder to capture.

Record the panic message before attempting multiple fixes.

Treating Automatic Reboot as a Solution

Automatic rebooting may reduce downtime, but recurring panics still require investigation.

A Practical Recovery Order

When a Linux machine experiences a kernel panic, use this order:

  1. Photograph or record the panic message.
  2. Disconnect recently added hardware where appropriate.
  3. Boot an older kernel from GRUB.
  4. Try recovery mode.
  5. Review the previous boot’s kernel logs.
  6. Check for failing modules or DKMS errors.
  7. Rebuild the initramfs.
  8. Update the bootloader configuration.
  9. Verify the root filesystem UUID and /etc/fstab.
  10. Check and repair the filesystem from recovery media.
  11. Test disk health.
  12. Test the memory.
  13. Check temperatures and BIOS settings.
  14. Use a live environment if the system remains unbootable.
  15. Configure crash dumping on important systems.

This order starts with the least destructive actions and moves towards deeper recovery work.

Preventing Future Kernel Panics

No system can be protected from every kernel panic, but the risk can be reduced.

Good practices include:

  • Keep a known-good older kernel installed.
  • Apply kernel updates in a controlled manner.
  • Test updates on non-critical systems first.
  • Maintain current backups.
  • Monitor disk health.
  • Use tested RAM.
  • Avoid unnecessary third-party kernel modules.
  • Keep firmware and BIOS updates under review.
  • Avoid unstable overclocking.
  • Configure kdump on critical servers.
  • Document storage and boot configuration.
  • Maintain remote console access.
  • Monitor repeated kernel warnings before they become panics.

For business systems, kernel updates should ideally be combined with a rollback plan and a tested maintenance window.

Final Thoughts

A kernel panic is serious because the operating system has decided it cannot continue safely. It is not, however, a diagnosis by itself.

The real task is to determine whether the panic was caused by:

  • A broken kernel
  • A faulty driver
  • A damaged boot environment
  • Filesystem corruption
  • Failing storage
  • Defective memory
  • Unstable hardware

Start by preserving the error message, then boot a known-good kernel and investigate methodically. Avoid destructive repairs until the system’s data is backed up and the likely cause is understood.

With a working older kernel, recovery mode, live media and good logs, many kernel panic incidents can be resolved without reinstalling the entire operating system.

Need Help Recovering a Linux System?

Hamilton Group can help diagnose Linux kernel panics, boot failures, damaged filesystems, driver problems and unstable server hardware.

We can restore access, protect critical data and identify the root cause before the same fault causes further downtime.

Call 0330 043 0069 or visit hgmssp.com to speak with one of our IT specialists.