Updated 28 July 2021
To err is human;
to really foul things up requires a computer.
— Bill Vaughn
People make mistakes. It happens. Lord knows I made quite a number of them in my trek to build a gaggle of servers. This page describes techniques I learned the hard way to recover from my blunders without having to start the server build all over again. (But sometimes, it's a shorter path to start from scratch, because the effort to effect a rescue can take longer than dumping and going again.)
Get a root shell another way ("su -l") — you did set a
root password, right? — and edit the file
/etc/group. Here is a list of the
groups to which you should re-add your non-root login, to
regain administrator access (replace $USER with your non-root
adm:x:4:syslog,$USER cdrom:x:24:$USER sudo:x:27:$USER dip:x:30:$USER www-data:x:33:$USER plugdev:x:46:$USER kvm:x:108:$USER lxd:x:116:$USER lpadmin:x:120:$USER lxd:x:131:$USER
NOTE: the group IDs may be different than presented here. Also, not all groups listed here may be present in your group file.
System will not boot: Some problems will prevent the system from booting. File system problems can drop you into single-user mode.
Errors in netplan(5,8), that you don't catch, will stop things cold. Other mistakes can trigger a kernel panic. Reboot in single-user mode to fix. (You did modify the /etc/default/grub file, right?)Booting into single-user mode requires you to have console access to the system; SSH won't cut it. Assuming you have a console, try these steps:
You now have a root shell, so you can fix your problems.
If you are dealing with physical hardware, you have an out. You will need a USB drive or DVD-ROM with the ISO image for Live Ubuntu 20.04 LTS.
Follow the instructions in the next bullet, How To Mount LVM file systems.
Mount a LVM partition filesystem: If your server has a Logical Volume Management partition, you can mount the filesystems in that LVM partition by following these instructions:
# mkdir /mnt/$mountpoint [repeat for each needed filesystem in the LVM.] # vgscan # vgchange -ay $volume_group # lvs # mount /dev/$volume_group/$logical_volume> /mnt/$mountpoint [repeat for each logical filesystem in the LVM as needed.]
Most of the time, you only need to mount the root filesystem, to fix your configuration errors. The /var filesystem may not have anything you need to change.
fsck(8) a LVM partition filesystem: If you need to fix a file system in a Logical Volume Management partition file system, the steps are similar.
# mkdir /mnt/$mountpoint [repeat for each filesystem in the LVM] # vgscan # vgchange -ay $volume_group # lvs # mount /dev/$volume_group/$logical-volume /mnt/$mountpoint [repeat for each filesystem in the LVM] # mount [Note the type of filesystem; for root and /var, these should be ext4.] # umount /mnt/$mountpoint # fsck.ext4 /dev/$volume_group/$logical_volume
Comments, suggestions, and error reports are welcome.
Send them to:
spamfilter (at) satchell (dot)
© 2021 Stephen Satchell, Reno NV