Troubleshooting the system in rescue mode

We all make mistakes and this is especially true for novice Linux system administrators. Linux can have a steep learning curve and sooner or later there will be a point in your career where your CentOS installation does not start up due to broad number of reasons, including hardware problems or human mistakes such as configuration errors. If this has happened to you then you can use the CentOS rescue mode in order to boot an otherwise unbootable system and try to undo your mistakes or find out the root of the problems. In this recipe, we will show you three common use cases when to use this option:

• Accessing the filesystem for recovering important data or undoing changes to configuration files if CentOS is not booting up
• Changing the root password if you forgot it
• Re-installing the boot loader which can be damaged when installing another operating system on the same harddisk where CentOS is installed

Getting ready

To complete this recipe, you will require a standard installation media (CD/DVD or USB device) of the CentOS 7 operating system. For recovering the data from the system, you will need to connect some sort of external storage device to the system, such as an external hard disk or a working network connection to another computer to copy all your precious data to a different location.

How to do it...

To begin this recipe, you should boot your server from the CentOS installation CD/DVD or the USB device and wait until the first welcome splash screen appears with the cursor waiting at the Test this media & install CentOS 7 menu option.

Reaching rescue mode

1. From the main menu, use the down arrow key to select Troubleshooting and then press the Return key to proceed.
2. On the Troubleshooting screen, use the down arrow key to highlight Rescue a CentOS system. When you are ready, press the Return key to proceed.
3. After some loading time, we enter the rescue screen, which includes various confirmation sub-screens. To begin this section, use the left and right arrow keys to choose Continue and press the Return key to proceed.
4. On the first sub-screen, choose OK and press the Return key to proceed.
5. Again, in the following sub-screen, choose OK and press the Return key to proceed.
6. On the next screen, choose the Start shell and by using the Tab key, highlight OK and press the Return key to proceed.
7. By completing the preceding steps, you will launch a shell session. You will notice this at the bottom of your display. The current status of the shell session will read as follows:

   bash-4.2#_
   

8. At the prompt, type the following instruction to change the root filesystem, before pressing the Return key to complete your request:

   chroot /mnt/sysimage
   

9. Congratulations! You just reached the rescue mode. To exit it at any time, simply type the following command and then press the Return key to complete your request (don't do this right now as this will restart the system):

   reboot
   

10. After the basic rescue mode is reached, we have the following options, depending on the type of problem.

Accessing the filesystem

If you are now in the rescue mode and need to backup important files from the filesystem, you need a destination location for the data transfer. For transferring the data we want to recover from the server to another computer please physically connect an external USB device to it. You can also use network storages for the recovery. For example, you could import an NFS server share and copy data to it. Refer to the Working with NFS recipe in Chapter 7, Building a Network.

1. On the rescue mode command line, type in the following command, which will show you all the current partitions connected to the system, and then press the Return key to complete your request:

   fdisk -l
   

2. You now need to find out the right device name with the partition number of your connected device; comparing the total size or the filesystem output of the various devices with the specifications from your stick can help you in this process. You can also try the following trick: run the fdisk -l command twice, first with the plugged-in USB device and then again with the USB device unplugged, and compare the output of both the commands. It should be different by one device name which you are searching for!
3. If you have found the right device name in the list, create a directory to mount the stick to the filesystem:

   mkdir /mnt/hdd-recovery
   

4. Next, mount the disk partition to this folder. Here we assume that the USB device of interest has the device name sdd1 (please change if different on your system):

   mount /dev/sdd1 /mnt/hdd-recovery
   

5. The original system's hard disk's root partition has been mounted under a specific folder by the rescue system automatically (under /mnt/sysimage), if you need to access it for example to change configuration files which caused startup problems or make a full or partial backup. For example, if you need to backup your Apache webserver configuration files, use:

   cp -r /mnt/sysimage/etc/http /mnt/hdd-recovery
   

6. If you need to access the data that lives on partitions other than the currently mounted root partition, use fdisk -l to identify the partition of interest. Then create a directory and mount the partition to it and change to that directory to access your data similar you did when mounting the USB device.
7. To finish backing up the files, type:

   reboot
   

Accessing the filesystem

1. If you are in the rescue mode for changing the root password, just use the following command and provide a new password:

   passwd
   

2. To complete changing the password, type:

   reboot
   

Re-install the CentOS boot loader

1. We will now use the fdisk command to find the name of all the current partitions. To do this, type the following instruction and then press the Return key to complete your request:

   fdisk –l
   

2. Now run the following command:

   dmesg | grep -Fq "EFI v"
   

3. If the preceding command does not produce any output look for the * symbol in the fdisk listing in the boot column to find the correct start partition, and assuming that your boot disk is on /dev/sda1 (change this as required), type the following:

   grub2-install /dev/sda
   

4. Otherwise, if there is an output, run instead:

   yum reinstall grub2-efi shim
   

5. If no error is reported, the console should respond as follows:

   # this device map was generated by anaconda
   (hd0) /dev/sda
   

6. The console output from the last step has confirmed that GRUB has now been successfully restored.
7. To reboot the computer, type:

   reboot
   

How it works...

There are a broad variety of problems which can be resolved by the tools provided through the rescue mode environment. Often these problems refer to booting problems but can also be from different types, such as forgetting the root password. Rescue mode can be a life-saver and an understanding of it is a very important skill to learn. It was felt that such a recipe should thus remain close at hand.

Tip

Remember to always be careful when working with bootloader commands as improper use can make your operating system unbootable.