Recovering From a Kernel Panic in Linux
A Linux kernel panic is one of the most serious failures a Linux system can experience. Instead of an application crashing or a service stopping, the operating system’s kernel reaches a condition from which it cannot safely continue.
The result may be a frozen screen filled with diagnostic text, an automatic reboot, or a server that stops responding completely.
A kernel panic can look alarming, but it does not always mean the machine is permanently damaged. Many panics are caused by faulty drivers, broken kernel updates, unstable hardware, filesystem corruption or incorrect boot configuration. With a structured recovery process, it is often possible to get the system running again and identify the underlying cause.
This guide explains what a kernel panic is, how to recover access to the system and how to reduce the risk of it happening again.
What Is a Kernel Panic?
The Linux kernel sits between the hardware and the rest of the operating system. It controls memory, processors, storage devices, drivers, networking and system processes.
When the kernel encounters a serious error that could corrupt data or leave the machine in an unsafe state, it may stop the system deliberately. This is called a kernel panic.
A typical panic message may include wording such as:
Kernel panic - not syncing: Fatal exception
You may also see:
Unable to mount root fs
or:
VFS: Cannot open root device
The exact message is important because it often points towards the failing component.
Common Causes of Kernel Panics
Kernel panics can be triggered by software, hardware or configuration problems.
Common causes include:
- A faulty or incompatible kernel module
- A recently installed kernel update
- A broken initramfs image
- Incorrect bootloader configuration
- A missing root filesystem
- Filesystem corruption
- Failing RAM
- A failing SSD or hard drive
- CPU overheating or instability
- Unsupported hardware
- Incorrect overclocking settings
- Problems caused by third-party security or storage drivers
- Virtual machine configuration faults
The last change made before the panic is often an important clue.
For example, if the problem started immediately after installing a new kernel, the previous kernel may still work. If it started after adding a graphics driver, that driver may be involved.
Record the Panic Message
Before rebooting, take a photograph of the screen or record the exact error message.
Important details may include:
- The first line containing “Kernel panic”
- Any named kernel module
- The device or partition mentioned
- References to the root filesystem
- Error codes
- The kernel version
- The final few lines before the system stopped
The screen may contain a large amount of technical output, but the first clear error is often more useful than the long stack trace that follows.
If the machine reboots too quickly to read the message, disable automatic rebooting temporarily from the bootloader or recovery environment if practical.
First Step: Try a Normal Reboot
Some panics are temporary and do not immediately reappear.
Restart the system and observe what happens:
- Does it boot normally?
- Does the panic occur at the same point?
- Does it happen only under load?
- Does it occur after a particular service starts?
- Does it happen only with a specific device connected?
If the machine boots successfully, do not assume the problem has gone away. Review the logs and investigate the likely cause.
Boot an Older Kernel
If the panic began after a kernel update, booting the previous kernel is usually the best first recovery step.
Most Linux distributions keep at least one older kernel installed.
Using GRUB
Restart the machine and display the GRUB menu.
Depending on the system, you may need to hold or repeatedly press:
Shift
or:
Esc
during startup.
Select:
Advanced options for Linux
You should see multiple kernel versions.
Choose an older kernel, preferably one that does not include “recovery mode” initially.
For example:
Linux 6.x.x
Linux 6.x.x-previous
If the older kernel boots successfully, the newer kernel or one of its modules is likely responsible.
Keep the Working Kernel Installed
Once the system is running with the older kernel, confirm which version is active:
uname -r
List installed kernels on Debian or Ubuntu systems:
dpkg --list | grep linux-image
On Fedora, Rocky Linux or Red Hat-based systems:
rpm -q kernel
Do not remove the working kernel until the new one has been repaired or replaced.
A common mistake is deleting older kernels too aggressively. Keeping one known-good kernel provides an important recovery route.
Boot Into Recovery Mode
If a normal older kernel does not work, try recovery mode.
From the GRUB menu:
- Select Advanced options for Linux.
- Choose a kernel entry marked recovery mode.
- Use the recovery menu to inspect filesystems, networking or root access.
Recovery mode may allow you to:
- Repair packages
- Check filesystems
- Rebuild boot files
- Remove a faulty driver
- Disable a broken service
- Access a root shell
The root filesystem may initially be mounted read-only.
Remount it read/write with:
mount -o remount,rw /
Use recovery mode carefully because you are working with elevated privileges.
Check the Previous Boot Logs
If the system booted after the panic, review the logs from the previous boot:
sudo journalctl -b -1
To show kernel messages only:
sudo journalctl -k -b -1
Search for panic-related entries:
sudo journalctl -k -b -1 | grep -iE "panic|oops|fatal|error|fail"
You can also review the current kernel log:
sudo dmesg -T
Look for messages involving:
- I/O errors
- Memory faults
- Filesystem errors
- Driver failures
- GPU faults
- NVMe timeouts
- SATA errors
- Machine check exceptions
- Out-of-memory conditions
Be aware that logs may not always be saved if the system stopped too quickly.
Check for a Kernel Oops
A kernel oops is a serious kernel error that does not always stop the system immediately.
Repeated oops messages may eventually lead to a panic or unstable behaviour.
Search the journal:
sudo journalctl -k | grep -i oops
If a specific module appears repeatedly, it may be the cause.
Example:
BUG: unable to handle kernel paging request
Modules linked in: example_driver
A third-party module named near the failure should be investigated carefully.
Rebuild the initramfs
The initramfs is a temporary filesystem loaded early during boot. It contains drivers and tools needed to locate and mount the real root filesystem.
If it becomes corrupted or does not include the required storage driver, the machine may panic with messages such as:
Unable to mount root fs
On Debian and Ubuntu systems, rebuild it with:
sudo update-initramfs -c -k all
Or update existing images:
sudo update-initramfs -u -k all
On Fedora, Rocky Linux and Red Hat-based systems, use:
sudo dracut --force
To rebuild for a specific kernel:
sudo dracut --force /boot/initramfs-$(uname -r).img $(uname -r)
After rebuilding the initramfs, update the bootloader if required.
Update GRUB
A damaged or outdated GRUB configuration can point to the wrong kernel, initramfs or root partition.
On Debian and Ubuntu systems:
sudo update-grub
On many Red Hat-based BIOS systems:
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
On UEFI systems, the path may be different.
For example:
sudo grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
The exact command depends on the distribution and boot configuration, so confirm the correct path before overwriting anything.
Check the Root Filesystem Entry
A panic mentioning the root filesystem may be caused by an incorrect UUID or missing device.
Check the current filesystem layout:
lsblk -f
Check UUIDs:
sudo blkid
Review /etc/fstab:
cat /etc/fstab
Check the kernel boot parameters:
cat /proc/cmdline
The root filesystem specified by GRUB should match the actual root partition.
A mismatch may occur after:
- Cloning a disk
- Replacing a drive
- Restoring a backup
- Changing partitions
- Editing GRUB manually
- Converting between storage controllers
Check for Filesystem Corruption
Filesystem corruption can prevent Linux from mounting the root volume and may lead to a panic.
Do not run a repair tool on a mounted filesystem.
Boot from recovery media, a live Linux environment or another system, then identify the affected partition:
lsblk -f
For an ext4 filesystem:
sudo fsck -f /dev/sdXn
Replace /dev/sdXn with the correct partition.
For XFS:
sudo xfs_repair /dev/sdXn
XFS may require the filesystem to be unmounted first.
For Btrfs, start with a non-destructive check:
sudo btrfs check --readonly /dev/sdXn
Do not use aggressive repair options unless you understand the consequences and have a backup.
Check Disk Health
A failing storage device may cause read errors, filesystem corruption and kernel panics.
Install SMART tools if necessary:
sudo apt install smartmontools
or:
sudo dnf install smartmontools
Check a SATA drive:
sudo smartctl -a /dev/sda
Check an NVMe drive:
sudo smartctl -a /dev/nvme0
Look for:
- Reallocated sectors
- Pending sectors
- Uncorrectable errors
- Media errors
- CRC errors
- NVMe critical warnings
- Excessive temperature
- Failed self-tests
Run a short self-test:
sudo smartctl -t short /dev/sda
After the test completes:
sudo smartctl -a /dev/sda
If the drive appears unhealthy, prioritise data recovery and replacement rather than repeated repair attempts.
Test the Memory
Faulty RAM can cause unpredictable kernel panics, corrupted data and random application crashes.
Many GRUB menus include a memory test option such as Memtest86+.
Run multiple passes, ideally for several hours.
Any reported memory error should be treated seriously.
For systems with multiple memory modules:
- Power off the machine.
- Reseat the modules.
- Test one module at a time.
- Test different slots.
- Disable memory overclocking.
- Restore the manufacturer’s recommended settings.
On servers, also check the hardware management interface for ECC memory errors.
Check CPU Temperature and Stability
Overheating or unstable processor settings can trigger machine-check exceptions and kernel panics.
Check temperatures using:
sensors
Install the tool if needed:
sudo apt install lm-sensors
or:
sudo dnf install lm_sensors
Detect sensors:
sudo sensors-detect
Then run:
sensors
Inspect:
- CPU temperature
- Fan operation
- Thermal throttling
- Dust buildup
- Failed cooling fans
- Poor heatsink contact
Return BIOS or UEFI settings to defaults if the system is overclocked or undervolted.
Investigate Third-Party Kernel Modules
Third-party modules are a common source of panics after kernel updates.
Examples include:
- Proprietary graphics drivers
- Antivirus modules
- Backup agents
- Storage drivers
- VPN drivers
- Virtualisation tools
- Hardware monitoring software
- Out-of-tree network drivers
List loaded modules:
lsmod
Check information about a module:
modinfo module_name
Check DKMS status:
dkms status
DKMS rebuilds third-party modules for new kernels. A failed DKMS build may leave a module incompatible with the updated kernel.
Review DKMS logs:
sudo find /var/lib/dkms -name make.log -print
Remove or rebuild the problematic module before booting the affected kernel again.
Temporarily Blacklist a Faulty Module
If a particular module is causing the panic, you can prevent it from loading.
Add it to a blacklist file:
echo "blacklist module_name" | sudo tee /etc/modprobe.d/blacklist-module_name.conf
Rebuild the initramfs:
sudo update-initramfs -u
or:
sudo dracut --force
Then reboot.
Do not blacklist essential storage, filesystem or network modules without understanding the impact.
Test With a Kernel Boot Parameter
GRUB allows temporary kernel parameters to be added for troubleshooting.
At the GRUB menu:
- Highlight the kernel.
- Press e.
- Find the line beginning with linux.
- Add the test option to the end.
- Press Ctrl+X or F10 to boot.
Possible troubleshooting options include:
nomodeset
This disables normal graphics mode setting and may help when the graphics driver is involved.
Another option is:
systemd.unit=multi-user.target
This boots into a text-based multi-user environment rather than the graphical desktop.
For a minimal emergency shell:
systemd.unit=emergency.target
These changes apply only to that boot unless added permanently to the GRUB configuration.
Remove a Broken Kernel
If one kernel consistently panics and an older kernel works, the broken kernel can be removed and reinstalled.
On Debian or Ubuntu, identify it first:
dpkg --list | grep linux-image
Then remove the exact problematic package:
sudo apt remove linux-image-VERSION
Reinstall it if appropriate:
sudo apt install --reinstall linux-image-VERSION
Update GRUB:
sudo update-grub
On Fedora or Red Hat-based systems, list kernels:
rpm -q kernel
Remove a specific package carefully:
sudo dnf remove kernel-core-VERSION
Never remove the kernel that is currently running.
Confirm the active version first:
uname -r
Reinstall the Kernel Packages
If the kernel or its modules are damaged, reinstalling the package may repair missing files.
On Ubuntu or Debian:
sudo apt install --reinstall linux-image-$(uname -r) linux-modules-$(uname -r)
On some systems, additional module packages may be required:
sudo apt install --reinstall linux-modules-extra-$(uname -r)
On Fedora or Red Hat-based systems:
sudo dnf reinstall kernel-core kernel-modules
After reinstalling, rebuild the initramfs and bootloader configuration.
Use a Live Linux Environment
If the installed system will not boot at all, use a live USB or recovery ISO.
From the live environment, you can:
- Inspect the disk
- Copy important files
- Repair filesystems
- Review logs
- Edit /etc/fstab
- Rebuild the initramfs
- Repair GRUB
- Chroot into the installed system
Identify the root partition:
lsblk -f
Mount it:
sudo mount /dev/sdXn /mnt
Mount the boot partition if separate:
sudo mount /dev/sdXm /mnt/boot
For UEFI systems:
sudo mount /dev/sdXp /mnt/boot/efi
Bind the virtual filesystems:
sudo mount --bind /dev /mnt/dev
sudo mount --bind /proc /mnt/proc
sudo mount --bind /sys /mnt/sys
sudo mount --bind /run /mnt/run
Enter the installed system:
sudo chroot /mnt
You can then repair packages, rebuild the initramfs or update GRUB as though the installed system had booted normally.
Exit when finished:
exit
Unmount the filesystems before rebooting.
Kernel Panic in a Virtual Machine
When the affected Linux system is virtual, check the hypervisor as well as the guest.
Look for changes to:
- Virtual disk controller type
- CPU compatibility mode
- Assigned memory
- Secure Boot
- Virtual hardware version
- Snapshot state
- Disk attachment order
- Pass-through devices
- Hypervisor tools
A guest may panic if its root disk was previously connected through one controller type and is now presented through another without the required driver in the initramfs.
Check the hypervisor logs for storage timeouts or host hardware issues.
Kernel Panic During Installation
If the panic occurs while installing Linux:
- Verify the installation image checksum.
- Recreate the USB drive.
- Try a different USB port.
- Disconnect unnecessary peripherals.
- Disable overclocking.
- Test the RAM.
- Update the BIOS or UEFI.
- Try safe graphics mode.
- Use a newer or older installer kernel.
- Check whether the hardware requires special boot parameters.
Installation panics often indicate media corruption, unsupported hardware or hardware instability.
Configuring Automatic Reboot After a Panic
Servers may be configured to reboot automatically after a kernel panic.
Check the current setting:
sysctl kernel.panic
A value of 0 means the system will not automatically reboot.
To reboot after 30 seconds:
sudo sysctl -w kernel.panic=30
To make it persistent, create a configuration file:
echo "kernel.panic = 30" | sudo tee /etc/sysctl.d/99-kernel-panic.conf
Apply the configuration:
sudo sysctl --system
Automatic rebooting can restore service temporarily, but it can also create a reboot loop. It does not fix the underlying cause.
Preserving Crash Information With kdump
On important servers, configure kernel crash dumping.
The kdump service reserves a small amount of memory for a secondary kernel. If the main kernel panics, the secondary kernel can save a memory dump for later analysis.
Depending on the distribution, the package may be called:
kdump-tools
or:
kexec-tools
A crash dump can provide much more detail than a photograph of the panic screen.
It may reveal:
- The faulting thread
- The kernel stack
- The loaded modules
- The CPU state
- Memory structures
- The exact code path that failed
Crash dumps can be large and may contain sensitive information, so they should be stored securely.
Distinguishing a Panic From Other Failures
Not every frozen Linux machine has experienced a kernel panic.
Similar symptoms may be caused by:
- A graphical desktop crash
- A deadlocked application
- Memory exhaustion
- Storage timeouts
- A power supply fault
- A hardware reset
- A watchdog reboot
- A hypervisor issue
- A complete network outage
A genuine kernel panic usually displays an explicit panic message or leaves evidence in logs or crash dumps.
If the screen is frozen but keyboard input still works, try switching to another console:
Ctrl+Alt+F3
If remote access still works, the kernel may still be running and only the graphical session may have failed.
Common Recovery Mistakes
Reinstalling Linux Immediately
A reinstall may hide the cause without resolving faulty hardware.
Check the disk, memory and logs first.
Removing Every Older Kernel
Older kernels provide a valuable fallback.
Keep at least one known-good version installed.
Repairing a Mounted Filesystem
Running filesystem repair against a mounted filesystem can cause further damage.
Use recovery mode or live media.
Assuming the Latest Kernel Must Be Correct
A newly released kernel can expose a driver or hardware compatibility issue.
Boot the previous version to confirm.
Ignoring Third-Party Modules
A kernel update may be stable while an external driver is not.
Check DKMS and out-of-tree modules.
Rebooting Repeatedly Without Recording the Error
Repeated restarts can overwrite useful logs and make the original error harder to capture.
Record the panic message before attempting multiple fixes.
Treating Automatic Reboot as a Solution
Automatic rebooting may reduce downtime, but recurring panics still require investigation.
A Practical Recovery Order
When a Linux machine experiences a kernel panic, use this order:
- Photograph or record the panic message.
- Disconnect recently added hardware where appropriate.
- Boot an older kernel from GRUB.
- Try recovery mode.
- Review the previous boot’s kernel logs.
- Check for failing modules or DKMS errors.
- Rebuild the initramfs.
- Update the bootloader configuration.
- Verify the root filesystem UUID and /etc/fstab.
- Check and repair the filesystem from recovery media.
- Test disk health.
- Test the memory.
- Check temperatures and BIOS settings.
- Use a live environment if the system remains unbootable.
- Configure crash dumping on important systems.
This order starts with the least destructive actions and moves towards deeper recovery work.
Preventing Future Kernel Panics
No system can be protected from every kernel panic, but the risk can be reduced.
Good practices include:
- Keep a known-good older kernel installed.
- Apply kernel updates in a controlled manner.
- Test updates on non-critical systems first.
- Maintain current backups.
- Monitor disk health.
- Use tested RAM.
- Avoid unnecessary third-party kernel modules.
- Keep firmware and BIOS updates under review.
- Avoid unstable overclocking.
- Configure kdump on critical servers.
- Document storage and boot configuration.
- Maintain remote console access.
- Monitor repeated kernel warnings before they become panics.
For business systems, kernel updates should ideally be combined with a rollback plan and a tested maintenance window.
Final Thoughts
A kernel panic is serious because the operating system has decided it cannot continue safely. It is not, however, a diagnosis by itself.
The real task is to determine whether the panic was caused by:
- A broken kernel
- A faulty driver
- A damaged boot environment
- Filesystem corruption
- Failing storage
- Defective memory
- Unstable hardware
Start by preserving the error message, then boot a known-good kernel and investigate methodically. Avoid destructive repairs until the system’s data is backed up and the likely cause is understood.
With a working older kernel, recovery mode, live media and good logs, many kernel panic incidents can be resolved without reinstalling the entire operating system.
Need Help Recovering a Linux System?
Hamilton Group can help diagnose Linux kernel panics, boot failures, damaged filesystems, driver problems and unstable server hardware.
We can restore access, protect critical data and identify the root cause before the same fault causes further downtime.
Call 0330 043 0069 or visit hgmssp.com to speak with one of our IT specialists.