After update, kernel panic at boot with error: Unable to mount root fs on unknown-block(0,0)

Environment

• Red Hat Enterprise Linux 5
• Red Hat Enterprise Linux 6
• Red Hat Enterprise Linux 7

Issue

• After a fresh install and reboot, or after a kernel update, the system fails to boot with the following message:

  Raw

  VFS: Cannot open root device XXX or unknown-block(0,0)
  Please append a correct "root=" boot option; here are the available partitions:
  Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
  ...
  Call Trace:
    [<ffffffff81527213>] ? panic+0xa7/0x16f
    [<ffffffff81c27432>] ? mount_block_root+0x216/0x2cb
    [<ffffffff81002930>] ? bstat+0x2b0/0x980
    [<ffffffff81c2753d>] ? mount_root0x56/0x5a
             

• OS unable to boot after kernel upgrade on RHEL 7

  

• When upgrading the system (and the kernel in the process), the server doesn't reboot anymore.

Resolution

1. Use the GRUB menu to boot into an older kernel:
   • Wait for the 

     Booting Red Hat Enterprise Linux Server in X seconds

      countdown to begin and press any key.
   • On the next screen, using the arrow keys, select a different kernel to boot from other than the top selected kernel. Once a kernel has been highlighted press 'Enter' to boot the selected kernel.
   • Note: If no older kernels are available, boot into rescue mode and then chroot into the root filesystem and proceed with Step 2.
2. Ensure the filesystem mounted at 

   /boot

    and 

   /tmp/

    have free space

   Raw

   df -h /boot /tmp
              

   If it's virtually full, inspect it and remove any files that shouldn't be there

   If unsure what to remove, use the following to inspect the installed kernel versions

   Raw

   rpm --last -q kernel{,-xen,-PAE}
              

   Then, use 

   yum remove

    against the oldest one to free up some space
3. Remove and then reinstall the newest kernel as shown by the previous 

   rpm

    command

   For example:

   Raw

   yum remove kernel-2.6.18-348.el5; yum install kernel-2.6.18-348.el5
              

4. After the kernel reinstall, ensure that an initrd/initramfs file was created in 

   /boot

    for the kernel in question

   Check the title statement for the newly-reinstalled kernel in 

   /boot/grub/grub.conf

    to ensure that said initrd/initramfs file is mentioned on the "initrd" line
5. If the initrd/initramfs file was not created in 

   /boot

   , please rebuild by manually, for more see: How to rebuild the initial ramdisk image in Red Hat Enterprise Linux

Warning: When removing and reinstalling kernels, it's always a good idea to keep at least one known-good kernel installed and available as a fall-back.

Root Cause

• Either the initrd declaration is missing from the new kernel's title statement in 

  /boot/grub/grub.conf

  , or the initrd/initramfs file itself is missing from the 

  /boot

   directory.
• Either of these could happen due to an incomplete install of a kernel package -- perhaps due to a system hang/crash or perhaps due to the 

  /boot

   or 

  /tmp/

   filesystem running out of space.
• Check any third party modules present in initrd/initramfs , sometimes third party modules can cause such kind of issue.

Diagnostic Steps

1. Boot the system from an older kernel version or from rescue mode
2. Verify whether initrd/initramfs is present in 

   /boot

    directory
3. Verify whether initrd statement is present for all kernels in 

   /boot/grub/grub.conf

4. Use 

   df /boot /tmp

    to verify the 

   /boot

    and 

   /tmp

    partitions are not ~100% full

Fix: unable to mount root fs on an unknown block

Since the arrival of an unable to mount root fs on an unknown block error portends you being unable to boot your GNU/Linux installation, it can be a rather concerning error to get. You’ll most often see it after an update has gone wrong or some other kind of system irregularity has interfered with the normal operation of your kernel.

Many users would prefer to not have to work with the Linux kernel, since it can be rather annoying to do so. Fortunately, though, this is an easy enough problem to fix provided that you can either boot your machine using another kernel stored on it or have access to a different computer to make a bootable thumb drive.

Remounting and Repairing the initramfs Module

This error usually happens because you’re missing the initramfs for the kernel you’re working with. Since you can’t boot your system properly, you have two options to get to a shell. You’ll have noticed it as part of a black screen with either a few lines or many lines of text when you’re trying to boot. The system will appear hung after you get it.

Reboot and then try pushing shift right after you see the BIOS or UEFI logo when you turn your machine on. If you see the Grub menu come up, then you can select the Advanced Options setting and try an older kernel. You might be able to get into your system this way.

Should this not have worked, then you’ll need to boot from a live CD, USB or SD card. You can make one by downloading the install media for your distribution on another machine as, unfortunately, you can’t get in if the Grub boot menu didn’t help.

Either way, once you’re booted open up a terminal with Ctrl+Alt+T regardless of the method you used to start your machine. List the partitions you have on your machine with sudo fdisk -l and look for the one that’s your root file system. Once you find it, then type sudo mkdir /mnt/fs; sudo mount /dev/sda1 /mnt/fs, though you’ll naturally have to replace /dev/sda1 with the right partition name if that’s not it. That should be the root partition on many Linux Mint, Debian and Ubuntu systems, however.

You’ll need to mount your EFI partition if you have one. Assuming that you’re using a legacy installation booting from an MBR partitioned disk, you won’t have to do this. If you do have an EFI system, then run sudo mkdir /mnt/boot/efi; sudo mount /dev/sda2 /mnt/boot/efi while again replacing /dev/sda2 with the right partition number. The previous fdisk command should have listed all of the partitions.

Now you’ll need to run a list of commands to get what they call chroot access into your install. You’ll see these posted many times, as this is a common error, but things might differ slightly due to future updates. It seems that the following commands should work:

sudo mkdir -p /mnt/dev/pts; sudo mount -o bind /dev /mnt/dev
sudo mount -o bind /dev/pts /mnt/dev/pts
sudo mkdir /mnt/sys; sudo mount -t sysfs /sys /mnt/sys
sudo mkdir /mnt/proc; sudo mount -t proc /proc /mnt/proc
sudo chroot /mnt      

Run dpkg –list | grep linux-image to find which Linux image you’re running, and then run update-initramfs -u -k 4.10.0-38-generic, making sure to replace the label with what you’re actually running. We used the name from a test installation of Xubuntu we were running, but your image might be very different. Once you’re done, run update-grub; exit and you’ll be back at your first prompt. Try sudo umount /mnt/boot/efi if you were on any kind of UEFI system.

You don’t have to do that if you’re on a legacy system, though. In any case, you will want to run sudo umount /mnt; reboot and then boot like normal.

If everything has gone well, then you’ll be completely back to normal. It might seem like a lot of playing around, but if these commands work you won’t have to actually adjust any configuration files or anything like that. You’ll simply need to run them and reboot to get a working system again.

Once you’ve gotten things working again, you may wish to update with the new kernel.

I had the same problem after kernel update. Please mount relevant ISO or CD/DVD disk and do the rescue. For example, I mounted the CentOS7 DVD and do this:

mount --bind /proc /mnt/sysimage/proc
mount --bind /dev  /mnt/sysimage/dev
mount --bind /sys  /mnt/sysimage/sys
chroot /mnt/sysimage
           

Тhen finding the last initramfs from /boot and regenerate it, well in this case:

dracut -f /boot/initramfs-2.6.32-754.14.2.el6.x86_64.img initramfs-2.6.32-754.14.2.el6.x86_64
           

After reboot, everything works wine.