Ubuntu 20.04 Server Edition Rescue Tips
or
When Things Go Horribly Wrong

Updated 28 July 2021

• You can no longer use sudo(8)
• System will not boot
• Single-user boot fails
• Mount LVM partition filesystems
• fsck(8) LVM partition filesystems

People make mistakes. It happens. Lord knows I made quite a number of them in my trek to build a gaggle of servers. This page describes techniques I learned the hard way to recover from my blunders without having to start the server build all over again. (But sometimes, it's a shorter path to start from scratch, because the effort to effect a rescue can take longer than dumping and going again.)

• You can no longer use sudo(8): Mis-adjusting the groups for a non-root login can wipe out the ability to use the sudo(8) command or to perform other administrative operations.

  Get a root shell another way ("su -l") — you did set a root password, right? — and edit the file /etc/group. Here is a list of the groups to which you should re-add your non-root login, to regain administrator access (replace $USER with your non-root login name):

  adm:x:4:syslog,$USER
  cdrom:x:24:$USER
  sudo:x:27:$USER
  dip:x:30:$USER
  www-data:x:33:$USER
  plugdev:x:46:$USER
  kvm:x:108:$USER
  lxd:x:116:$USER
  lpadmin:x:120:$USER
  lxd:x:131:$USER
  

  NOTE: the group IDs may be different than presented here. Also, not all groups listed here may be present in your group file.

• System will not boot: Some problems will prevent the system from booting. File system problems can drop you into single-user mode.

  Errors in netplan(5,8), that you don't catch, will stop things cold. Other mistakes can trigger a kernel panic. Reboot in single-user mode to fix. (You did modify the /etc/default/grub file, right?)

  Booting into single-user mode requires you to have console access to the system; SSH won't cut it. Assuming you have a console, try these steps:
  • Reboot using the power button, or the reboot toggleon your AWS/Azure control panel
  • Watch the screen; when the grub menu appears, press the "e" key. (You have a two-second window to do this. Pay attention.)
  • Enter your root password (you defined one, right?)

  You now have a root shell, so you can fix your problems.

• Single-user boot fails: If your server is in the cloud, tough noogies. Cry, if that helps you. You will have to rebuild unless your cloud service offers an alternative. (I'm not aware of such.)

  If you are dealing with physical hardware, you have an out. You will need a USB drive or DVD-ROM with the ISO image for Live Ubuntu 20.04 LTS.

  • Boot the Live Ubuntu ISO.
  • After a coffee-break interval, you will get a desktop with two options: "Try Ubuntu" and "Install Ubuntu". Click on the "Try Ubuntu button." This gets you an Ubuntu desktop.
  • Launch Terminal.
  • Get a root shell by keying "su -l" (no password should be required).

  Follow the instructions in the next bullet, How To Mount LVM file systems.

• Mount a LVM partition filesystem: If your server has a Logical Volume Management partition, you can mount the filesystems in that LVM partition by following these instructions:

  # mkdir /mnt/$mountpoint
    [repeat for each needed filesystem in the LVM.]
  # vgscan
  # vgchange -ay $volume_group
  # lvs
  # mount /dev/$volume_group/$logical_volume>   /mnt/$mountpoint
    [repeat for each logical filesystem in the LVM as needed.]
  

  Most of the time, you only need to mount the root filesystem, to fix your configuration errors. The /var filesystem may not have anything you need to change.

• fsck(8) a LVM partition filesystem: If you need to fix a file system in a Logical Volume Management partition file system, the steps are similar.

  # mkdir /mnt/$mountpoint
    [repeat for each filesystem in the LVM]
  # vgscan
  # vgchange -ay $volume_group
  # lvs
  # mount /dev/$volume_group/$logical-volume   /mnt/$mountpoint
    [repeat for each filesystem in the LVM]
  # mount
    [Note the type of filesystem; for root and /var, these should be ext4.]
  # umount /mnt/$mountpoint
  # fsck.ext4 /dev/$volume_group/$logical_volume
  

Comments, suggestions, and error reports are welcome.
Send them to: spamfilter (at) satchell (dot) net)
© 2021 Stephen Satchell, Reno NV