Ubuntu 20.04 LTS Server Edition
I Stopped Fighting The “evil” Systemd
or
How I Learned To Stop Worrying and Love the Bomb

Updated 31 July 2021

I loved Stanley Kubric’s movie Dr. Strangelove. It makes sense that I would “steal” the tag line from the movie and apply it to the likes of Red Hat’s contribution to Linux initialization, systemd(8). Part of my problem is that there was never a concise explanation of exactly how the facility took over so many functions from their “rightful” owners.

Side note: After I finished writing the initial version of this page, I ran across an interesting YouTube video that explains the history of the Unix system initialization process, and how the systemd project was conceived to fix some of the problems. The tragedy of systemd (the talk given by Benno Rice, a long-time Free BSD person, by the way) frames the systemd story in the frame of a classic five-act tragedy. The talk provides the rationale for systemd, and points out some of the advances in Linux system management (not just initialization) that the project provides. The talk was quite general in scope. What the rest of this page does is provide some practical analytic pointers, and how to deal with specific issues that affected the servers I integrate into my server farm.

(This section re-done 31 July 2021)

The key to understanding systemd(8) is to look at the list of service units and timer units -- these two sets of units describe the running state of the system as managed by systemd. The documentation within the units is terse, or just plain cryptic, so expect to spend time with a search engine finding out exactly what each service does, and how to access any configuration files to bend systemd to your will.

This is where you start: a list of all the services that are known to systemd in your running system. This will be a long listing, which will be presented in a pager similar to less(1). (It's frankly too long to present here.) A number of the units will be tagged as "inactive", "dead", "exited" and more. That means that the service is run only once, or the service is triggered by something else, like a timer. More on the latter later.

Remember that some service units are triggered by timers? Here is the list of time-based triggers. (The descriptions are better, at least when I issued the command on my Web server.)

From the list of services, one can trace down the configuration files that need attention. What I change as a rule:

/etc/systemd/resolved.conf
/etc/systemd/timesyncd.conf

resolved.conf: In resolved.conf, I set the DNS servers I want systemd(8)’s domain name resolversto use. My network has its own DNS server, that presents private zones for my internal network, as well as a huge cache that reduces DNS traffic for mail servers. It’s a two-line change:

timesyncd.conf: In timesyncd.conf, I point to my local NTP server. I leave the fallback to the default. We are therefore talking about a single-line change:

(Yes, my LAN services server offers domain and ntp service. For 8.7 watts of power in my fanless computing appliance, I like the separation of function. I’m showing what I do in my network; you can adjust to your own situation.)

Prior version of material

The key to understanding systemd(8) is looking at the list of service units, and understanding the configuration files for those units. And how to deal with those. When first building a system with Ubuntu server edition (or any distribution that uses systemd(8)) there is one command that extracts the Core of the Beast:

(Leave it to Reddit to show a better way to display this information:

systemctl list-units --type=service

Learning never stops. One advantage of using this form of command is that it will catch non-running as well as running services. This distinction is why I missed seeing the unit fstrim.service while looking at everything. Why isn't this serviced running? It's periodically triggered by unit fstrim.timer -- once a week!)

With this one command, all the distractions are removed, and you are left with a synopsis of what systemd(8) is exactly running in your shiny new server.

Here is what my Web server reports:

  accounts-daemon.service        loaded active running   Accounts Service
  acpid.service                  loaded active running   ACPI event daemon
  avahi-daemon.service           loaded active running   Avahi mDNS/DNS-SD Stack
  bluetooth.service              loaded active running   Bluetooth service
  clamav-freshclam.service       loaded active running   ClamAV virus database updater
  colord.service                 loaded active running   Manage, Install and Generate Color Profiles
  cron.service                   loaded active running   Regular background program processing daemon
  cups-browsed.service           loaded active running   Make remote CUPS printers available locally
  cups.service                   loaded active running   CUPS Scheduler
  dbus.service                   loaded active running   D-Bus System Message Bus
  fwupd.service                  loaded active running   Firmware update daemon
  gdm.service                    loaded active running   GNOME Display Manager
  geoclue.service                loaded active running   Location Lookup Service
  irqbalance.service             loaded active running   irqbalance daemon
  kerneloops.service             loaded active running   Tool to automatically collect and submit kernel crash signatures
  libvirtd.service               loaded active running   Virtualization daemon
  ModemManager.service           loaded active running   Modem Manager
  mongodb.service                loaded active running   An object/document-oriented database
  networkd-dispatcher.service    loaded active running   Dispatcher daemon for systemd-networkd
  NetworkManager.service         loaded active running   Network Manager
  polkit.service                 loaded active running   Authorization Manager
  rpcbind.service                loaded active running   RPC bind portmap service
  rsyslog.service                loaded active running   System Logging Service
  rtkit-daemon.service           loaded active running   RealtimeKit Scheduling Policy Service
  smartmontools.service          loaded active running   Self Monitoring and Reporting Technology (SMART) Daemon
  snapd.service                  loaded active running   Snap Daemon
  ssh.service                    loaded active running   OpenBSD Secure Shell server
  switcheroo-control.service     loaded active running   Switcheroo Control Proxy service
  systemd-journald.service       loaded active running   Journal Service
  systemd-logind.service         loaded active running   Login Service
  systemd-machined.service       loaded active running   Virtual Machine and Container Registration Service
  systemd-networkd.service       loaded active running   Network Service
  systemd-resolved.service       loaded active running   Network Name Resolution
  systemd-timesyncd.service      loaded active running   Network Time Synchronization
  systemd-udevd.service          loaded active running   udev Kernel Device Manager
  tftpd-hpa.service              loaded active running   LSB: HPA's tftp server
  udisks2.service                loaded active running   Disk Manager
  unattended-upgrades.service    loaded active running   Unattended Upgrades Shutdown
  unifi.service                  loaded active running   unifi
  upower.service                 loaded active running   Daemon for power management
  user@1000.service              loaded active running   User Manager for UID 1000
  virtlogd.service               loaded active running   Virtual machine log manager
  whoopsie.service               loaded active running   crash report submission daemon
  wpa_supplicant.service         loaded active running   WPA supplicant
  

NOTE: extra space has been removed to allow this listing, an similar listings, to be seen on mobile devices.

From that list, one can trace down the configuration files you need to pay attention to. What I change as a rule:

resolved.conf: In resolved.conf, I set the DNS servers I want systemd(8)’s domain name resolversto use. My network has its own DNS server, that presents private zones for my internal network, as well as a huge cache that reduces DNS traffic for mail servers. It’s a two-line change:

timesyncd.conf: In timesyncd.conf, I point to my local NTP server. I leave the fallback to the default. We are therefore talking about a single-line change:

(Yes, my LAN services server offers domain and ntp service. For 8.7 watts of power in my fanless computing appliance, I like the separation of function. I’m showing what I do in my network; you can adjust to your own situation.)

But there's more. Looking at another set of lines output from systemctl list-units, we can see all the targets. A target encodes information about a target unit of systemd, which is used for grouping units and as well-known synchronization points during start-up. When a particular part of the system is stable, a target has been reached. This is used within unit definitions to provide a hint as to what a particular unit needs to have active before provisioning that unit, and what that unit provides after provisioning. For example, before ssh provisioning can be started, the network.target (and auditd.service) needs to be finished being provisioned. This is how systemd determines the order in which to activate the various units.

  basic.target            loaded active active    Basic System
  cloud-config.target     loaded active active    Cloud-config availability
  cloud-init.target       loaded active active    Cloud-init target
  cryptsetup.target       loaded active active    Local Encrypted Volumes
  getty.target            loaded active active    Login Prompts
  graphical.target        loaded active active    Graphical Interface
  local-fs-pre.target     loaded active active    Local File Systems (Pre)
  local-fs.target         loaded active active    Local File Systems
  multi-user.target       loaded active active    Multi-User System
  network-online.target   loaded active active    Network is Online
  network-pre.target      loaded active active    Network (Pre)
  network.target          loaded active active    Network
  nss-lookup.target       loaded active active    Host and Network Name Lookups
  nss-user-lookup.target  loaded active active    User and Group Name Lookups
  paths.target            loaded active active    Paths
  remote-fs-pre.target    loaded active active    Remote File Systems (Pre)
  remote-fs.target        loaded active active    Remote File Systems
  slices.target           loaded active active    Slices
  sockets.target          loaded active active    Sockets
  sound.target            loaded active active    Sound Card
  swap.target             loaded active active    Swap
  sysinit.target          loaded active active    System Initialization
  time-set.target         loaded active active    System Time Set
  time-sync.target        loaded active active    System Time Synchronized
  timers.target           loaded active active    Timers
  

This section looks at the rest of the output. The command below sorts the output by type, then by name. (Not perfectly, so I had to do some manual effort to get this in the right order.) Unfortunately, the systemctl utility doesn't allow this sort of grouping natively, so we resort to the UNIX rule: use the other tools. Looking at each type:

• .automount: from freedesktop.com: "[E]ncodes information about a file system automount point controlled and supervised by systemd."
• .device: Used by the systemd-udev.service, this describes the characeristics of the given device, and how to integrate the device in the system. There is very little of interest to the casual user here.
• .mount: Describes how a systemd-controlled and -supervised partitions are mounted and unmounted from the file system. Note that the unit files include snap modules that are mounted at boot time.
• .path: (Starting to see a pattern here?) encodes information about a path monitored by systemd, for path-based activation. Not many of those here, and some point to /dev/null.
• .scope: From freedesktop.org: "Scopes units manage a set of system processes. ... The main purpose of scope units is grouping worker processes of a system service for organization and for managing resources." Unlike other entries here, there is no specific file; instead, the other unit files can have "[Scope]" sections in them.
• .slice: From freedesktop.org: "[E]ncodes information about a slice unit. A slice unit is a concept for hierarchically managing resources of a group of processes. This management is performed by creating a node in the Linux Control Group (cgroup) tree. Units that manage processes (primarily scope and service units) may be assigned to a specific slice. For each slice, certain resource limits may be set that apply to all processes of all units contained in that slice." Unlike other entries here, there is no specific file; instead, the specification for slices are written into the source of the program that is to be included in a CGROUP. (I think.)

  (And if you understand all that, you are a better person than I am. Assuming I got any of this right!)

• .socket: From freedesktop.org: "[E]ncodes information about an IPC or network socket or a file system FIFO controlled and supervised by systemd, for socket-based activation. ... For each socket unit, a matching service unit must exist, describing the service to start on incoming traffic on the socket". The unit files contain the port numbers on which services may listen; this may have ramifications when ensuring a server is sufficiently hardened against intrustion from outside. Specifically, watch out for unwanted binding to a public-facing interface. Pay attention to the "BindToDevice" attribute; the default is not specified. This little problem if present, will show up when you run netstat -tanp" and netstat -uanp.
• .swap: From freedesktop.org: "[E]ncodes information about a swap device or file for memory paging controlled and supervised by systemd." In Ubuntu, entries in /etc/fstab can create this kind of unit. Can you have multiple swap devices? The documentation in Ubuntu is silent on this question.
• .timer: From freedesktop.org: "[E]ncodes information about a timer controlled and supervised by systemd, for timer-based activation." The wiki.arglinux.org goes a little farther: "Timers can be used as an alternative to cron". The opensource.com page is more explicit: "Timers provide finer-grained control of events than cronjobs. ... For example, a timer can trigger a script or program to run a specific amount of time after an event such as boot, startup, completion of a previous task, or even the previous completion of the service unit called by the timer." Interesting.

Again from my web server, here is the (slightly edited) output:

ssystemctl list-units | egrep '^[ ]' | grep -v '.service' | sort -t. -k4,4 -k3,3 -k2,2 -k1,1

  proc-sys-fs-binfmt_misc.automount                   loaded active waiting   Arbitrary Executable File Formats File System Automount Point

  sys-devices-pci0000:00-0000:00:13.0-ata1-host0-target0:0:0-0:0:0:0-block-sda.device      loaded active plugged   TS32GMSA370
  sys-devices-pci0000:00-0000:00:13.0-ata1-host0-target0:0:0-0:0:0:0-block-sda-sda1.device loaded active plugged   TS32GMSA370 1
  sys-devices-pci0000:00-0000:00:13.0-ata1-host0-target0:0:0-0:0:0:0-block-sda-sda2.device loaded active plugged   TS32GMSA370 2
  sys-devices-pci0000:00-0000:00:13.0-ata1-host0-target0:0:0-0:0:0:0-block-sda-sda3.device loaded active plugged   TS32GMSA370 3
  sys-devices-pci0000:00-0000:00:1b.0-sound-card0.device                                   loaded active plugged   Atom Processor Z36xxx/Z37xxx Series High Definition Audio Controller
  sys-devices-pci0000:00-0000:00:1c.0-0000:01:00.0-net-enp1s0.device                       loaded active plugged   82583V Gigabit Network Connection
  sys-devices-pci0000:00-0000:00:1c.1-0000:02:00.0-net-enp2s0.device                       loaded active plugged   82583V Gigabit Network Connection
  sys-devices-platform-serial8250-tty-ttyS10.device   loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS10
  sys-devices-platform-serial8250-tty-ttyS11.device   loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS11
  sys-devices-platform-serial8250-tty-ttyS12.device   loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS12
  sys-devices-platform-serial8250-tty-ttyS13.device   loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS13
  sys-devices-platform-serial8250-tty-ttyS14.device   loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS14
  sys-devices-platform-serial8250-tty-ttyS15.device   loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS15
  sys-devices-platform-serial8250-tty-ttyS16.device   loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS16
  sys-devices-platform-serial8250-tty-ttyS17.device   loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS17
  sys-devices-platform-serial8250-tty-ttyS18.device   loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS18
  sys-devices-platform-serial8250-tty-ttyS19.device   loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS19
  sys-devices-platform-serial8250-tty-ttyS1.device    loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS1
  sys-devices-platform-serial8250-tty-ttyS20.device   loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS20
  sys-devices-platform-serial8250-tty-ttyS21.device   loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS21
  sys-devices-platform-serial8250-tty-ttyS22.device   loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS22
  sys-devices-platform-serial8250-tty-ttyS23.device   loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS23
  sys-devices-platform-serial8250-tty-ttyS24.device   loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS24
  sys-devices-platform-serial8250-tty-ttyS25.device   loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS25
  sys-devices-platform-serial8250-tty-ttyS26.device   loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS26
  sys-devices-platform-serial8250-tty-ttyS27.device   loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS27
  sys-devices-platform-serial8250-tty-ttyS28.device   loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS28
  sys-devices-platform-serial8250-tty-ttyS29.device   loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS29
  sys-devices-platform-serial8250-tty-ttyS2.device    loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS2
  sys-devices-platform-serial8250-tty-ttyS30.device   loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS30
  sys-devices-platform-serial8250-tty-ttyS31.device   loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS31
  sys-devices-platform-serial8250-tty-ttyS3.device    loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS3
  sys-devices-platform-serial8250-tty-ttyS4.device    loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS4
  sys-devices-platform-serial8250-tty-ttyS5.device    loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS5
  sys-devices-platform-serial8250-tty-ttyS6.device    loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS6
  sys-devices-platform-serial8250-tty-ttyS7.device    loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS7
  sys-devices-platform-serial8250-tty-ttyS8.device    loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS8
  sys-devices-platform-serial8250-tty-ttyS9.device    loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS9
  sys-devices-pnp0-00:05-tty-ttyS0.device             loaded active plugged   /sys/devices/pnp0/00:05/tty/ttyS0
  sys-devices-virtual-block-dm\x2d0.device            loaded active plugged   /sys/devices/virtual/block/dm-0
  sys-devices-virtual-block-dm\x2d1.device            loaded active plugged   /sys/devices/virtual/block/dm-1
  sys-devices-virtual-block-loop0.device              loaded active plugged   /sys/devices/virtual/block/loop0
  sys-devices-virtual-block-loop1.device              loaded active plugged   /sys/devices/virtual/block/loop1
  sys-devices-virtual-block-loop2.device              loaded active plugged   /sys/devices/virtual/block/loop2
  sys-devices-virtual-block-loop3.device              loaded active plugged   /sys/devices/virtual/block/loop3
  sys-devices-virtual-block-loop4.device              loaded active plugged   /sys/devices/virtual/block/loop4
  sys-devices-virtual-block-loop5.device              loaded active plugged   /sys/devices/virtual/block/loop5
  sys-devices-virtual-block-loop6.device              loaded active plugged   /sys/devices/virtual/block/loop6
  sys-devices-virtual-block-loop7.device              loaded active plugged   /sys/devices/virtual/block/loop7
  sys-devices-virtual-block-loop8.device              loaded active plugged   /sys/devices/virtual/block/loop8
  sys-devices-virtual-misc-rfkill.device              loaded active plugged   /sys/devices/virtual/misc/rfkill
  sys-devices-virtual-tty-ttyprintk.device            loaded active plugged   /sys/devices/virtual/tty/ttyprintk
  sys-module-configfs.device                          loaded active plugged   /sys/module/configfs
  sys-module-fuse.device                              loaded active plugged   /sys/module/fuse
  sys-subsystem-net-devices-enp1s0.device             loaded active plugged   82583V Gigabit Network Connection
  sys-subsystem-net-devices-enp2s0.device             loaded active plugged   82583V Gigabit Network Connection

  -.mount                         loaded active mounted   Root Mount
  boot-efi.mount                  loaded active mounted   /boot/efi
  boot.mount                      loaded active mounted   /boot
  dev-hugepages.mount             loaded active mounted   Huge Pages File System
  dev-mqueue.mount                loaded active mounted   POSIX Message Queue File System
  run-snapd-ns-lxd.mnt.mount      loaded active mounted   /run/snapd/ns/lxd.mnt
  run-snapd-ns.mount              loaded active mounted   /run/snapd/ns
  run-user-1000.mount             loaded active mounted   /run/user/1000
  snap-certbot-1280.mount         loaded active mounted   Mount unit for certbot, revision 1280
  snap-core18-2066.mount          loaded active mounted   Mount unit for core18, revision 2066
  snap-core18-2074.mount          loaded active mounted   Mount unit for core18, revision 2074
  snap-core20-1026.mount          loaded active mounted   Mount unit for core20, revision 1026
  snap-core20-1081.mount          loaded active mounted   Mount unit for core20, revision 1081
  snap-lxd-20326.mount            loaded active mounted   Mount unit for lxd, revision 20326
  snap-lxd-21029.mount            loaded active mounted   Mount unit for lxd, revision 21029
  snap-snapd-12159.mount          loaded active mounted   Mount unit for snapd, revision 12159
  snap-snapd-12398.mount          loaded active mounted   Mount unit for snapd, revision 12398
  sys-fs-fuse-connections.mount   loaded active mounted   FUSE Control File System
  sys-kernel-config.mount         loaded active mounted   Kernel Configuration File System
  sys-kernel-debug.mount          loaded active mounted   Kernel Debug File System
  sys-kernel-tracing.mount        loaded active mounted   Kernel Trace File System
  var.mount                       loaded active mounted   /var

  systemd-ask-password-console.path loaded active waiting   Dispatch Password Requests to Console Directory Watch
  systemd-ask-password-wall.path  loaded active waiting   Forward Password Requests to Wall Directory Watch

  session-130.scope               loaded active running   Session 130 of user satch
  init.scope                      loaded active running   System and Service Manager

  -.slice                         loaded active active    Root Slice
  system.slice                    loaded active active    System Slice
  system-lvm2\x2dpvscan.slice     loaded active active    system-lvm2\x2dpvscan.slice
  system-modprobe.slice           loaded active active    system-modprobe.slice
  system-getty.slice              loaded active active    system-getty.slice
  user.slice                      loaded active active    User and Session Slice
  user-1000.slice                 loaded active active    User Slice of UID 1000

  dbus.socket                     loaded active running   D-Bus System Message Bus Socket
  dm-event.socket                 loaded active listening Device-mapper event daemon FIFOs
  iscsid.socket                   loaded active listening Open-iSCSI iscsid Socket
  lvm2-lvmpolld.socket            loaded active listening LVM2 poll daemon socket
  multipathd.socket               loaded active running   multipathd control socket
  snapd.socket                    loaded active running   Socket activation for snappy daemon
  snap.lxd.daemon.unix.socket     loaded active listening Socket unix for snap application lxd.daemon
  syslog.socket                   loaded active running   Syslog Socket
  systemd-initctl.socket          loaded active listening initctl Compatibility Named Pipe
  systemd-journald-audit.socket   loaded active running   Journal Audit Socket
  systemd-journald-dev-log.socket loaded active running   Journal Socket (/dev/log)
  systemd-journald.socket         loaded active running   Journal Socket
  systemd-networkd.socket         loaded active running   Network Service Netlink Socket
  systemd-rfkill.socket           loaded active listening Load/Save RF Kill Switch Status /dev/rfkill Watch
  systemd-udevd-control.socket    loaded active running   udev Control Socket
  systemd-udevd-kernel.socket     loaded active running   udev Kernel Socket
  uuidd.socket                    loaded active listening UUID daemon activation socket

  swap.img.swap                   loaded active active    /swap.img

  apt-daily.timer                 loaded active waiting   Daily apt download activities
  apt-daily-upgrade.timer         loaded active waiting   Daily apt upgrade and clean activities
  systemd-tmpfiles-clean.timer    loaded active waiting   Daily Cleanup of Temporary Directories
  man-db.timer                    loaded active waiting   Daily man-db regeneration
  logrotate.timer                 loaded active waiting   Daily rotation of log files
  fstrim.timer                    loaded active waiting   Discard unused blocks once a week
  motd-news.timer                 loaded active waiting   Message of the Day
  e2scrub_all.timer               loaded active waiting   Periodic ext4 Online Metadata Check for All Filesystems
  fwupd-refresh.timer             loaded active waiting   Refresh fwupd metadata regularly
  ua-messaging.timer              loaded active waiting   Ubuntu Advantage update messaging
  snap.certbot.renew.timer        loaded active waiting   Timer renew for snap application certbot.renew
  

Comments, suggestions, and error reports are welcome.
Send them to: spamfilter (at) satchell (dot) net)
© 2021 Stephen Satchell, Reno NV