Ubuntu 20.04 LTS Server Edition
Core Server Install Check List
Step By Step

Updated 30 July 2021

WARNING: This is my build check list. There are many details, like IP addresses and interface names, that are specific to my network. You may want to copy this check list, and customize it to match your networks.

Updated 30 July 2021 to correct SSD information.
Updated 25 July 2021 to remove the installation and configuration of NTP. See this page for why.

This check list covers the common configuration for all my servers. Configuration of specific additional functions, such as email, web, and VPN, are covered separately on other pages (as they become available).

These instructions assume that one is installing Ubuntu Server Edition on physical hardware. Adjust the procedures when using a virtual instance, such as in VMWare, KVM, AWS, Azure, or other provider.

NOTE: In this check list, the lines prefixed with the pound (#) character indicates the command needs to be input as root. Either prefix the command with "sudo", or enter root for multiple commands using "sudo /bin/bash" command.

What You Need Before Starting:

NOTE: Unlike some other distributions, Ubuntu Server Edition does not include a "rescue" option when booting the ISO. To be prepared, you should have a desktop ISO you can use as the ultimate rescue vehicle, in case your server will not boot at all.

These links were valid as of the writing of this page, 14 July 2021:

Specifications:

Begin here:

Install From DVD-ROM or USB Thumb Drive:
Start by installing from the Ubuntu 20.04 LTS server edition ISO. The instructions show how to answer each of the questions the Ubiquity installer asks of you. The information in lower- or mixed-case are the parameter, the all-uppercase information is the button you press to continue.

Select keyboard language
  us
  DONE
Network
  IP setup, &c; DHCP should be OK
  DONE
Proxy
  DONE
Mirror
  DONE
Storage
  all disk
  check LVM
  DONE
Edit storage
  edit / -> 9.4G
    SAVE
  LVM Group create
    /var -> 16G
    CREATE
  DONE
CONTINUE
REBOOT

Housekeeping:
After the initial installation and reloading from the boot volume, install all available updates. Install all the tools needed to maintain the device and troubleshoot network issues. Remove unneeded modules. Always set a root password. Set the timezone as appropriate; I'm on the West Coast so I set Pacific time.

# apt update
# apt upgrade
# apt install net-tools traceroute
# apt install ufw
# apt autoremove
# passwd root
# timedatectl set-timezone America/Los_Angeles
NOTE: ufw may be replaced with another firewall package at a later date.

Fix GRUB to show menu at boot:
Sometimes you will need to boot your system into single-user mode. To accomplish this task, you have to tell the GRUB boot loader to give you a chance to modify the kernel load command to add "1" to the end. This edit to the GRUB configuration makes this available:

--------------------/etc/default/grub----------------
  [edit these lines]
GRUB_TIMEOUT_STYLE=menu
GRUB_TIMEOUT=2
------------------------(end)------------------------

# update-grub

More Housekeeping:
The following steps are to be sure you have basic configuration files that may or may not be set up by the installer. Included here is my custom network configuration; if the default is good for you, that's fine. The sysctl.conf is optimized for a gigabit network, where the gateway to the Internet is on a separate IP address. The entry for vm.swappiness is set to ten (10) in order to discourage the kernel from writing to the swap store. (The value may need to be reduced to 5, or perhaps even to 1, to better improve SSD life. Warning: a value of 0 may cause processes to be killed when RAM is filled.)

--------------------/etc/securetty-------------------
tty1
tty2
tty3
tty4
ttyS1
------------------------(end)------------------------

--------------------/etc/netplan/00-installer-config.yaml
(NOTE: customize as required.  VLAN only if needed)
  version: 2
  renderer: networkd
  ethernets:
    enp2s0:
      dhcp4: no
      addresses: [ $LAN_CIDR_1, $LAN_CIDR_2 ]
      link-local: []
      gateway4: $LAN_GATEWAY
      nameservers:
        addresses: [ 127.0.0.1 ]
        search:
        - satchell.net
  vlans:
    enp2s0.2:
      id: 2
      link: enp2s0
      dhcp4: no
      addresses: [ $LAN_CIDR_3 ]
      link-local: []
------------------------(end)------------------------

# netplan try
   [if problems, fix them and redo the command]
# reboot

--------------------/etc/sysctl.conf-----------------
net.core.rmem_max                   = 16777216
net.core.wmem_max                   = 16777216
net.core.rmem_default               = 204800
net.core.wmem_default               = 204800
net.core.optmem_max                 = 40960
net.core.default_qdisc              = fq
net.core.netdev_max_backlog         = 50000

net.ipv4.tcp_rmem                   = 4096 87380 16777216
net.ipv4.tcp_wmem                   = 4096 65536 16777216
net.ipv4.tcp_no_metrics_save        = 1


net.ipv4.conf.default.rp_filter     = 2
net.ipv4.conf.enp1s0.rp_filter      = 2
net.ipv4.conf.all.rp_filter         = 0

net.ipv6.conf.all.disable_ipv6      = 1
net.ipv6.conf.default.disable_ipv6  = 1
net.ipv6.conf.all.forwarding        = 1

net.ipv4.conf.all.accept_redirects  = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.log_martians      = 1
net.ipv4.conf.all.send_redirects    = 0
net.ipv4.conf.all.send_redirects    = 0
net.ipv4.ip_forward                 = 0
net.ipv4.tcp_congestion_control     = htcp
net.ipv4.tcp_syncookies             = 1

vm.swappiness                       = 10
------------------------(end)------------------------

# sysctl -p
# for f in /proc/sys/net/ipv4//conf/*/rp_filter; \
     do echo $f=`cat $f`; done
# mkdir /home/$USER/.ssh
# chown $USER.$USER /home/$USER/.ssh
# chmod 700 /home/$USER/ssh

NOTE: This text about alternatives to using the Linux kernel reverse-path filtering is for information only. It's here for completeness. Application of this stuff depends on which firewall you use, and is beyond the scope of this configuration checklist at this time.

Reverse-Path Filter Information:
rp_filter - INTEGER

Current recommended practice in RFC3704 is to enable strict mode to prevent IP spoofing from DDos attacks. If using asymmetric routing or other complicated routing, then loose mode is recommended.

The max value from conf/{all,interface}/rp_filter is used when doing source validation on the {interface}.

Default value is 0. Note that some distributions enable it in startup scripts.

Alternative to using the Linux kernel reverse path filtering:

/sbin/iptables -t raw -A PREROUTING -i enp0s1 \
     -m addrtype ! --src-type UNICAST -j DROP
/sbin/iptables -t raw -A PREROUTING -s 127.0.0.1/32 -i enp0s1 -j DROP
/sbin/iptables -t raw -A PREROUTING -s $LAN_CIDR_1  -i enp0s1 -j DROP
/sbin/iptables -t raw -A PREROUTING -s $LAN_CIDR_2  -i enp0s1 -j DROP
/sbin/iptables -t raw -A PREROUTING -s $LAN_CIDR_3  -i enp0s1 -j DROP
  (repeat this last for each netblock in your network)
Another alternative to using the Linux kernel reverse path filtering: 
/sbin/iptables  -t raw -A PREROUTING -i enp0s1 -m rpfilter \
     --loose --invert -j DROP
/sbin/ip6tables -t raw -A PREROUTING -i emps01 -m rpfilter \
     --loose --invert -j DROP

SSHD (SSH server) Settings:
In my network, all the servers use the same public/private key pair in order to access all the servers. Perhaps not the best practice, but it saves me from polluting my ~/.ssh directory and config file. The SSHD configuration turns off the use of PAM (username/password authentication) so that ne'er-do-wells can try all they like to break in. I also add firewall rules using the ufw(8) [uncomplicated fire wall] facility, describe in the next section. SSHD is limited to IPv4; this may change in the future.

------------/home/$USER/.ssh/authorized_keys---------
     [insert public key(s) here]
------------(end)------------------------------------

# chown $USER.$USER /home/$USER/.ssh/authorized_keys
# chmod 600 /home/$USER/.ssh/authorized_keys

------------------------/etc/default/ssh-------------
SSHD_OPTS=-4
------------------------(end)------------------------

-------------------------/etc/ssh/sshd_config--------
# --add to end of file:
# --my "improvements"
AddressFamily                   inet
PermitRootLogin                 no
PermitEmptyPasswords            no
IgnoreRhosts                    yes
PasswordAuthentication          no
ChallengeResponseAuthentication no
UsePAM                          no
DenyUsers                       root
# *** Add DenyUsers for “role” account
DenyUsers  nobody
# Interesting usernames
DenyUsers  news sshd
DenyUsers  guest administrator pi mailman ftpuser admin
DenyUsers  system user git gituser postgres oracle ansible
DenyUsers  ec2-user test ubuntu demo spark debian
DenyUsers  ftpadmin webadmin student www
DenyUsers  webmaster postmaster operator

# Now back to reality...
DenyUsers  @*
AllowUsers $USER@$LAN_NETBLOCK  # Local network
   (repeat for each trusted LAN netblock)
AllowUsers $USER@$TRUSTED_IP_1  # Work
AllowUsers $USER@$TRUSTED_IP_2  # Convention-network
AllowUsers $USER@$TRUSTED_IP_3  # Other-remote-site
------------------------(end)------------------------

# systemctl restart sshd
# systemctl status sshd

UFW (uncomplicated fire wall) Configuration:: Modify the ICMP Echo Request rule to allow a limited number of ping requests through from the Internet. This change can be done without harm for servers that don't have a public IP address or connection. The intent here is to limit the damage that a ne'er-do-well can do with a ping flood.

------------------------/etc/ufw/before.rules-------------
-A ufw-before-input -p icmp --icmp-type echo-request -i enp1s0 \
     -m limit --limit 3/second --limit-burst 2  -j ACCEPT
-A ufw-before-input -p icmp --icmp-type echo-request -i enp1s0 -j DROP
------------------------(end)------------------------

# ufw allow in on enp2s0 from $LAN_NETBLOCK to $LAN_IP port 22 \
     comment local-SSH

If there is no public IP network on the WAN port, skip over this section. Otherwise, adjust IP addresses accordingly.

# ufw deny in on enp1s0 to $WAN_IP port 53  comment kill-external-DNS-queries
# ufw deny in on enp1s0 to $WAN_IP port 123 comment kill-external-NTP-queries
# ufw route deny in on enp1s0 out on enp2s0 comment deny-forwarding
# ufw route deny in on enp2s0 out on enp1s0 comment deny-forwarding

# ufw allow in on enp1s0 proto tcp from $TRUSTED_IP1 to $WAN_IP port 22 \
     comment trusted-ip-1-ssh
# ufw allow in on enp1s0 proto tcp from $TRUSTED_IP2 to $WAN_IP port 22 \
     comment trusted-ip-2-ssh
# ufw allow in on enp1s0 proto tcp from $TRUSTED_IP3 to $WAN_IP port 22 \
     comment trusted-ip-3-ssh
# ufw allow in on enp1s0 proto tcp from $TRUSTED_IP4 to $WAN_IP port 22 \
     comment trusted-ip-4-ssh
# ufw allow in on enp1s0 proto tcp from $TRUSTED_IP5 to $WAN_IP port 22 \
     comment trusted-ip-5-ssh

Fixing up DNS configuration:
The system initialization daemon cluster, "systemd", is quite invasive compared to the old method with init.d and family. In particular, the systemd system has its own DNS resolver, systemd-resolved, that needs to be pointed to appropriate DNS servers. In my network, I have a caching DNS server (with a LARGE cache, useful with mail servers) that I point to. So this part of the check list updates the systemd resolver.

# nano /etc/systemd/resolved.conf
    DNS=$DNS_IP
    ^X
# service systemd-resolved restart
# resolvectl status

Fixing Up Network Time Protocol (NTP) configuration:
If you do not have a local NTP server, you can skip this step; systemd-timesyncd will use compiled-in defaults for network time service.

This section assumes you have a NTP server configured elsewhere on your LAN, and this server will need to synchronize the system clock with that server.

In my case, I have a stratum two NTP server, synchronized with a GPS receiver that is ten cable feet away, as well as synchronized with stratum 1 servers within 400 miles of the local site. 
# nano /etc/systemd/timesyncd.conf
    NTP=$NTP_IP
    ^X
# service systemd-timesyncd restart
# datetimectl status

Routing Table Configuration For Reserved IP netblocks:
Reverse-path filtering doesn't work well if the server doesn't know what the bad networks are. Many people put this information in the firewall; I prefer to use the routing table (Forward Information Base, or FIB) so that the server is blocked from sending to the reserved addresses as well as blocking packets from reserved addresses. Don't worry, the kernel will add routes for networks defined in the interfaces.

# cd /etc/netplan
# wget http://www.satchell.net/01-blackhole-unroutable-config.yaml
# netplan try

Saving Configuration information:
Configurating a server is a tedious procedure, and can be difficult to get exactly right. So adding this script to your server (adjusted for your situation) lets you save working configurations quickly and easily, so that recovery is that much shorter.

------------------------/root/config.backup.sh-------
#!/bin/bash
if [ "$EUID" -ne 0 ]; then
  echo You must be root to run this script
  exit 1
  fi
id=`ifconfig | grep -o 'inet [0-9.]\+' | head -1 \
        | cut -d\  -f2 | cut -d. -f4`
echo
echo Backing up configs for $id
echo
cd /root
DIRS="\
/etc/apache2
/etc/bind
/etc/default
/etc/dhcp
/etc/dovecot
/etc/fail2ban
/etc/fwknop
/etc/hostname
/etc/netplan
/etc/ntp.conf
/etc/openvpn
/etc/postfix
/etc/ssh/sshd_config
/etc/ssh/sshd_config.d
/etc/ssl
/etc/systemd/resolved.conf
/etc/sysctl.conf
/etc/timezone
/etc/ufw
/home/$USER/.ssh/authorized_keys
"
DIRLIST=`
for f in $DIRS
  do
    if test -e $f
      then echo $f
      else echo Skipped $f 1>&2
    fi
  done
`
rm -f   /root/"$id"configs.tar
tar -cf /root/"$id"configs.tar $DIRLIST
scp /root/"$id"configs.tar $NOCUSER@$NOC_IP:Desktop/NOC/Configs/.
------------------------(end)------------------------

# bash /root/config.backup.sh

Finished.

Comments, suggestions, and error reports are welcome.
Send them to: spamfilter (at) satchell (dot) net)
© 2021 Stephen Satchell, Reno NV