How to Fix unbootable after kernel upgrade for 6.x Physical

Problem Descriptions

Description: Provide a brief description of the project or initiative causing the change. Note that the author should limit the verbiage to the relevancy of the document and needed changes. Give a short summary of the project overview, background and objectives.

After applying the latest kernel on a HP Physical server, the server cannot boot up with the following errors in iLO:

 

 

Steps

1. Make sure HP iLO access is working

2. Start remote console in ILO

3. Reset the server through iLO

4. When Linux start to boot up, the Kernel info shows, press Space key to have the following menu appears. There are about 3-5 secs Window to press Space key

5. Choose the old kernel and press Enter

6. Log in to server as root when it’s up

7. Open the file /etc/lvm/lvm.conf

vi /etc/lvm/lvm.conf

Make sure the following filter is uncomment, and comment out the other filter

filter = [ "a/.*/" ]

#filter = [ "a/sda[1-9]$/", "a/cciss/c0d0*/", "a/dev/mapper/.*/", "a/emcpower*/", "a/dm-*/", "r/.*/" ]

Save the file

8. Then rebuild the new kernel image (the red part is different depending on kernel versions)

cd /boot

Find the new kernel info in red (may vary depending on the update date)

[root@apm-bcclq1aba01 boot]# ls

config-2.6.32-358.el6.x86_64 lost+found

config-2.6.32-431.29.2.el6.x86_64 symvers-2.6.32-358.el6.x86_64.gz

efi symvers-2.6.32-431.29.2.el6.x86_64.gz

grub System.map-2.6.32-358.el6.x86_64

initramfs-2.6.32-358.el6.x86_64.img System.map-2.6.32-431.29.2.el6.x86_64

initramfs-2.6.32-431.29.2.el6.x86_64.img vmlinuz-2.6.32-358.el6.x86_64

initrd-2.6.32-358.el6.x86_64kdump.img vmlinuz-2.6.32-431.29.2.el6.x86_64

Run the following command to rebuild the image

dracut -f /boot/initramfs-2.6.32-431.29.2.el6.x86_64.img 2.6.32-431.29.2.el6.x86_64

reboot the server

9. The server should boot up to new kernel by default

10. Verify the PVs, and verify if rootvg has any changes. Sometimes the boot device may change from sda to sdb or other disks

[root@fmsbccluadb08 init.d]# pvs

Found duplicate PV xEKfbOLuz29IF3z6haxQruPZyFHiov2o: using /dev/sdb1 not /dev/sdq1

PV VG Fmt Attr PSize PFree

/dev/emcpowera1 dbvg1 lvm2 a-- 249.99g 0

/dev/emcpowerb1 dbvg2 lvm2 a-- 249.99g 0

/dev/emcpowerc1 dbvg3 lvm2 a-- 249.99g 0

/dev/emcpowerd1 dbvg4 lvm2 a-- 249.99g 0

/dev/emcpowere1 dbvg5 lvm2 a-- 65.00g 3.00g

/dev/sdb2 rootvg lvm2 a-- 29.97g 15.97g

/dev/sdb3 systemvg lvm2 a-- 29.97g 0

/dev/sdb4 wilyvg lvm2 a-- 161.00g 1.00g

Roll back the filter in /etc/lvm/lvm.conf (make sure filter covers the rootvg and systemvg’s disks. See example highlighted in red)

# By default we accept every block device:

#filter = [ "a/.*/" ]

filter = [ "a/sda[1-9]$/" "a/sdb[1-9]$/" "a/emcpower*/" "r/.*/" ]

 

• If for some reasons, you forgot to change lVM filter and server becomes unbootable, you need to do the followings:

And then rebuild the new kernel image (the red part is different depending on kernel versions)

cd /boot

dracut -f /boot/initramfs-2.6.32-431.20.5.el6.x86_64.img 2.6.32-431.20.5.el6.x86_64

reboot