Education › Security › Stage 2: Securing systems

Linux hardening

Users, sudo, SSH, file permissions, services, auditd and CIS benchmarks for a server you can defend.

Intermediate ~35 min read Module 5 of 16

A fresh Linux server is a reasonable starting point and a poor finishing one: password logins enabled, a dozen services listening, every user able to read most of the filesystem, no record of who ran what. Hardening is the process of closing the distance between that default and a host you could defend during an incident. It is not exotic. Most of it is removing what you do not need, restricting what remains, and making sure the machine tells you when something changes. This module walks the standard checklist in the order that matters and shows how to check your work against a published benchmark.

After this module you can
  • Lock down SSH and administrative access: keys only, no root login, MFA or an access proxy in front
  • Apply least privilege to users, sudo and file permissions, and find the exceptions
  • Reduce the attack surface by removing services, tightening the host firewall and keeping packages patched automatically
  • Enable audit logging and integrity monitoring so that changes are recorded and visible
  • Measure a host against a CIS benchmark and fix the findings that matter

Start with access: SSH and sudo

The first thing an attacker tries on a Linux host is logging in. Make that path narrow. Disable password authentication and root login over SSH, allow only key-based logins for named users or groups, and prefer an access layer (cloud session manager, an identity-aware proxy, a bastion with MFA) so that port 22 is not exposed to the internet at all. Modern keys are Ed25519; anything RSA under 3072 bits or DSA should be retired.

The lines that matter in /etc/ssh/sshd_config. Validate with sshd -t, then systemctl reload sshd, and keep a second session open while you test.
text
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AllowGroups ssh-users
MaxAuthTries 3
LoginGraceTime 30
X11Forwarding no
AllowAgentForwarding no
ClientAliveInterval 300
ClientAliveCountMax 2
LogLevel VERBOSE

Administrative rights go through sudo, never a shared root password. Grant sudo to a group, require a password (or MFA via a PAM module) for elevation, and log every sudo command; the default sudo logs to the auth log, which should ship off the host. Avoid NOPASSWD: ALL for humans; where automation needs it, scope the rule to the exact command.

A scoped sudoers rule for an automation account, placed in /etc/sudoers.d/ and validated with visudo -cf.
text
# deploy may restart the app service and nothing else
deploy ALL=(root) NOPASSWD: /usr/bin/systemctl restart app.service, /usr/bin/systemctl status app.service
Watch out

Lock yourself out drills happen. Before reloading sshd with new settings, open a second SSH session and keep it open; only close it after a fresh login with the new configuration succeeds.

Users, permissions and the filesystem

Every account is an entry point. Remove or lock accounts that nobody uses, give service accounts no shell (/usr/sbin/nologin), and make sure no account other than root has UID 0. Set a restrictive default umask (027) so new files are not world-readable, and review the files that are: world-writable files and directories, and setuid binaries, which run as their owner and are the classic path from a normal user to root.

Finding the exceptions worth reviewing.
bash
# accounts with a login shell
awk -F: '$7 !~ /(nologin|false)$/ {print $1, $3, $7}' /etc/passwd

# any account with UID 0 that is not root
awk -F: '$3 == 0 && $1 != "root"' /etc/passwd

# world-writable files and directories outside /proc and /tmp-like paths
find / -xdev \( -path /proc -o -path /tmp -o -path /var/tmp \) -prune -o \
  -perm -0002 -type f -print 2>/dev/null

# setuid and setgid binaries: compare with the distribution's expected list
find / -xdev \( -perm -4000 -o -perm -2000 \) -type f -print 2>/dev/null

Mount options are hardening too: noexec, nosuid and nodev on /tmp, /var/tmp, /dev/shm and removable media stop a downloaded binary from running from the obvious places. Keep secrets in files readable only by the service that needs them (mode 0600 or 0640 with a dedicated group), and never in world-readable environment files or in shell history.

Tip

Run services as dedicated, unprivileged users with systemd sandboxing (ProtectSystem=strict, PrivateTmp=yes, NoNewPrivileges=yes, CapabilityBoundingSet=). It costs a few lines in the unit file and removes most of the damage a compromised process can do.

Shrink the surface: services, firewall, patches

Every listening service is a potential entry point and every installed package is potential vulnerable code. List what listens, stop and disable what is not needed, and remove packages you do not use. Then add a host firewall that denies inbound by default and allows only the service ports; even behind a cloud security group, the host firewall catches mistakes in the group and traffic from inside the same subnet.

Inventory, prune, and a default-deny host firewall with nftables (Debian/Ubuntu: apt install nftables).
bash
# what listens, and which process owns each socket
sudo ss -tulnp

# disable a service you do not need
sudo systemctl disable --now avahi-daemon

# minimal nftables policy: allow loopback, established, SSH from the admin network, HTTPS from anywhere
sudo tee /etc/nftables.conf >/dev/null <<'EOF'
flush ruleset
table inet filter {
  chain input {
    type filter hook input priority 0; policy drop;
    iif lo accept
    ct state established,related accept
    ip saddr 10.20.0.0/16 tcp dport 22 accept
    tcp dport 443 accept
    icmp type echo-request limit rate 5/second accept
  }
  chain forward { type filter hook forward priority 0; policy drop; }
  chain output  { type filter hook output priority 0; policy accept; }
}
EOF
sudo nft -f /etc/nftables.conf && sudo systemctl enable --now nftables

Patching is the single most effective hardening step and the one most often skipped. Enable unattended security updates for the operating system, schedule reboots for kernel updates within a defined window (or use live patching where available), and rebuild container images and machine images on a cadence rather than patching them in place. A host that has been up for four hundred days is a host running a four-hundred-day-old kernel.

Automatic security updates on Debian and Ubuntu; check the log afterwards.
bash
sudo apt install -y unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades
# verify it is active and see what it did
systemctl status unattended-upgrades --no-pager
sudo tail -n 20 /var/log/unattended-upgrades/unattended-upgrades.log

Know what happened: auditing and integrity

Hardening without visibility only delays discovery. The Linux audit subsystem (auditd) records system calls matching rules you define: who executed which commands, changes to /etc/passwd and sudoers, access to secret files, module loads. Ship the audit log and the auth log off the host to your logging pipeline the moment they are written; an attacker with root will otherwise edit them.

Audit rules worth having on every server, in /etc/audit/rules.d/hardening.rules.
text
# identity and privilege changes
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/sudoers -p wa -k privilege
-w /etc/sudoers.d/ -p wa -k privilege
-w /etc/ssh/sshd_config -p wa -k sshd

# every command run with elevated privileges
-a always,exit -F arch=b64 -S execve -F euid=0 -F auid>=1000 -F auid!=unset -k root_commands

# kernel module loading
-w /sbin/insmod -p x -k modules
-w /sbin/modprobe -p x -k modules

# make the rules immutable until reboot
-e 2

File integrity monitoring (AIDE, or the equivalent in your endpoint agent) takes a baseline of binaries and configuration and reports what changed. Combined with immutable infrastructure, where a host is replaced rather than modified, unexpected change becomes a rare and meaningful event. Finally, make the system clock trustworthy (chrony to reliable NTP sources): every log line and every certificate check depends on it.

Note

Logs that stay on the host are evidence that the attacker controls. Central, append-only log storage with retention is part of hardening, and the detection module covers what to do with it.

Measure against a benchmark

The CIS Benchmarks are consensus hardening standards for each major distribution, organised into levels: Level 1 is the safe baseline for any server, Level 2 is stricter and may break things. Auditors ask for them, and they are a good checklist even when nobody is asking. Automated scanners such as OpenSCAP with the SCAP Security Guide, or Lynis for a quick pass, score a host against the benchmark and list failures with remediation.

A quick audit with Lynis, then a CIS Level 1 scan with OpenSCAP (Ubuntu package names shown).
bash
sudo apt install -y lynis && sudo lynis audit system --quick

sudo apt install -y openscap-scanner ssg-base
sudo oscap xccdf eval \
  --profile xccdf_org.ssgproject.content_profile_cis_level1_server \
  --report /tmp/cis-report.html \
  /usr/share/xml/scap/ssg/content/ssg-ubuntu2404-ds.xml

Do not chase a perfect score. Fix the findings that map to real threats first: access, privilege, exposure, patching, logging. Record the exceptions you consciously keep (with a reason and an owner), bake the fixes into your machine image or configuration management so that every new host starts hardened, and re-scan on a schedule so that drift shows up as a number going the wrong way.

Hands-on practice

Harden a throwaway VM and score it

  1. Create a small Ubuntu VM (local with Multipass or a cloud instance you will delete). Run sudo lynis audit system --quick and note the hardening index as your baseline.
  2. Apply the SSH settings from the lesson: create an ssh-users group, add your user, disable passwords and root login, sshd -t, reload, and confirm a fresh login works while a second session stays open.
  3. Run the account, world-writable and setuid finds. Lock any account you do not recognise with sudo usermod -L, and note which setuid binaries are expected on Ubuntu.
  4. List listening services with ss -tulnp, disable at least one you do not need, and install the nftables policy (adjust the admin network to your own IP). Confirm you can still connect.
  5. Enable unattended-upgrades and install auditd with the rules from the lesson; run sudo ausearch -k identity after editing a test user and confirm the event is recorded.
  6. Run Lynis again and compare the index; then run the OpenSCAP CIS Level 1 profile and open the HTML report. Pick three failed rules, decide fix or accept, and write the reason for each.
  7. Delete the VM.
Cheat sheet

Linux hardening — at a glance

Main things to focus on

  • Access first: keys only, no root login, sudo via a group with logging, port 22 not on the internet
  • Least privilege on the box: no extra UID 0, service accounts without shells, umask 027, review setuid and world-writable
  • Shrink the surface: disable unused services, default-deny host firewall, automatic security updates and scheduled reboots
  • Visibility: auditd rules for identity, privilege and root commands; ship logs off the host; trustworthy time
  • Measure with CIS Level 1 via OpenSCAP or Lynis; fix what maps to threats, record exceptions
  • Bake hardening into images so every new host starts compliant

SSH and sudo

PermitRootLogin no / PasswordAuthentication noKeys only, never root directly
AllowGroups ssh-usersOnly named group members may log in
sshd -t && systemctl reload sshdValidate then apply; keep a session open
ssh-keygen -t ed25519Modern key type
visudo -cf /etc/sudoers.d/FILEValidate a sudoers drop-in
user ALL=(root) NOPASSWD: /usr/bin/CMDScoped passwordless rule for automation

Accounts and files

awk -F: '$3 == 0' /etc/passwdFind UID 0 accounts
usermod -L USER / usermod -s /usr/sbin/nologin USERLock an account / remove its shell
find / -xdev -perm -0002 -type fWorld-writable files
find / -xdev -perm -4000 -type fSetuid binaries
umask 027New files not readable by others (set in /etc/login.defs and profile)
mount -o noexec,nosuid,nodev /tmpNo execution from temp locations

Services, firewall, patches

ss -tulnpWhat listens and which process owns it
systemctl disable --now SERVICEStop and prevent start
nft -f /etc/nftables.confLoad a default-drop input policy
nft list rulesetShow the active rules
unattended-upgradesAutomatic security patches (Debian/Ubuntu); dnf-automatic on RHEL
needrestart / checkrestartWhich services still run old libraries after a patch

systemd sandboxing

User=app DynamicUser=yesRun as an unprivileged, even ephemeral, user
ProtectSystem=strict ProtectHome=yesRead-only filesystem except declared paths
PrivateTmp=yes NoNewPrivileges=yesOwn /tmp, no setuid escalation
CapabilityBoundingSet= AmbientCapabilities=Drop all capabilities, add back only what is needed
systemd-analyze security app.serviceScore a unit's exposure

Audit and benchmark

-w /etc/sudoers -p wa -k privilegeAudit writes and attribute changes to a file
-a always,exit -F arch=b64 -S execve -F euid=0 ...Record commands run as root
ausearch -k KEY / aureport -xQuery audit events by key / summary of executions
lynis audit system --quickFast hardening index with suggestions
oscap xccdf eval --profile ...cis_level1_serverCIS scan with an HTML report

Common pitfalls

  • Reloading sshd with a typo and no second session open; the host is now unreachable.
  • Granting NOPASSWD: ALL to a human or a shared account because a script needed it once.
  • Hardening the firewall in the cloud console but leaving the host trusting everything on its own subnet.
  • Never rebooting for kernel updates; the patch is installed and the vulnerable kernel is still running.
  • Keeping audit and auth logs only on the host, where an attacker with root can edit them.
  • Chasing a 100% benchmark score and breaking the application, instead of fixing findings that map to real threats.
Quiz

Check your understanding

5 questions · 4 to pass · answers are explained as you go. Your best score is saved on this device only.

Progress and quiz scores are saved in this browser only. Back up or restore on the hub.

Was this lesson useful? Tell me what to improve →