Linux & the command line

The filesystem is a single tree

Linux has no drive letters. Everything hangs off one root directory, /, and extra disks, network shares, and even kernel information are mounted somewhere inside that tree. The layout is a convention called the Filesystem Hierarchy Standard, and learning it means you can sit down at any server and know where to look.

Paths that start with / are absolute; anything else is relative to your current directory. Scripts and cron jobs should use absolute paths, because you cannot be sure which directory they start in.

pwd                       # where am I?
ls -lah /var/log          # long listing, all files, human-readable sizes
cd /etc && ls | head      # move, then peek at the first ten entries
df -h                     # free space per mounted filesystem
du -sh /var/log/* | sort -h | tail -5   # the five biggest things in /var/log

Users, permissions, and ownership

Every file has an owning user, an owning group, and three sets of permission bits: for the user, the group, and everyone else. Each set can grant read, write, and execute. ls -l shows them as a ten-character string such as -rwxr-x---: a type flag, then three triplets.

The octal form is the same information as numbers: read is 4, write is 2, execute is 1, and you add them up per triplet. So rwx is 7, r-x is 5, r-- is 4, and -rwxr-x--- is 750. On a directory, execute means permission to enter it and read means permission to list it, which is why a directory with r-- alone is nearly useless.

ls -l deploy.sh                 # -rw-r--r-- 1 alice dev 412 Mar  3 10:12 deploy.sh
chmod u+x deploy.sh             # symbolic: add execute for the owning user
chmod 640 secrets.env           # octal: owner rw, group r, others nothing
chown alice:dev secrets.env     # change owner and group (needs root)
chmod -R g+rX /srv/app          # capital X: execute only on directories
id                              # which user am I, and which groups am I in?

The root user bypasses all of these checks, which is exactly why you should not work as root. Use sudo to run one command with elevated rights: it is logged, it can be limited per user, and it forces a moment of thought.

Processes and signals

A running program is a process with a numeric ID (PID), a parent, an owning user, and an exit code when it finishes. An exit code of 0 means success and anything else means failure. CI pipelines, shell && chains, and Kubernetes probes all depend on that one convention.

ps aux | head                   # every process, with owner, CPU and memory
ps -ef --forest | less          # the same, drawn as a parent/child tree
pgrep -a nginx                  # PIDs and command lines matching a name
top                             # live view; press q to quit (htop is friendlier)
ss -tulpn                       # which process is listening on which port
echo $?                         # exit code of the last command

You do not kill a process, you send it a signal. SIGTERM (15) is a polite request to shut down, and a well-written program catches it, finishes in-flight work, and exits. SIGKILL (9) cannot be caught: the kernel removes the process immediately, with no chance to flush data or release locks. Always try SIGTERM first.

kill 4242            # SIGTERM by default
kill -9 4242         # SIGKILL: last resort only
kill -HUP 4242       # many daemons reload their config on SIGHUP
sleep 600 &          # trailing & runs a job in the background
jobs                 # list background jobs of this shell
fg %1                # bring job 1 back to the foreground

Services with systemd

On nearly every modern distribution, long-running services are supervised by systemd. It starts them at boot in dependency order, restarts them when they crash, and captures their output. You talk to it with systemctl, and each service is described by a unit file ending in .service.

systemctl status nginx          # running? since when? last log lines
sudo systemctl restart nginx    # stop, then start
sudo systemctl reload nginx     # re-read config without dropping connections
sudo systemctl enable --now nginx   # start at boot AND start right now
systemctl list-units --type=service --state=failed
systemctl cat nginx             # show the unit file that is actually in effect

start and enable are different things. start affects this moment; enable affects the next boot. A service that was started but never enabled works perfectly until the server reboots, a classic cause of an outage after routine patching.

[Unit]
Description=My web app
After=network-online.target

[Service]
User=myapp
WorkingDirectory=/srv/myapp
EnvironmentFile=/etc/myapp/env
ExecStart=/srv/myapp/bin/server --port 8080
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

After creating or editing a unit file, run sudo systemctl daemon-reload so systemd re-reads it. Note the User= line: the service runs as an unprivileged account, not as root.

Pipes, redirection, and the text toolkit

The Unix design idea is many small programs that each do one thing, connected by pipes. Every process has three standard streams: stdin (0), stdout (1), and stderr (2). The | operator connects one program's stdout to the next one's stdin; > and >> send a stream to a file.

grep -ri "timeout" /etc/nginx/          # recursive, case-insensitive search
grep -c " 500 " access.log               # count matching lines
tail -f /var/log/syslog                  # follow a file as it grows

# top 5 client IPs in a web access log
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -5

./backup.sh > backup.log 2>&1            # stdout AND stderr into one file
./backup.sh 2> /dev/null                 # discard errors only
find /var/log -name "*.gz" -mtime +30    # files older than 30 days

Read that log pipeline left to right: take column one, sort so identical lines sit together, collapse them with a count, sort numerically in reverse, keep five. Nearly every investigation you do on a server is some variation of filter, extract, count, sort.

Finding out why it broke

systemd collects the output of every service in the journal. journalctl queries it, and its filters are what turn a wall of text into an answer.

journalctl -u nginx --since "10 min ago"   # one service, recent window
journalctl -u nginx -f                      # follow live
journalctl -p err -b                        # errors and worse, since this boot
journalctl -k | tail -50                    # kernel messages (OOM kills show here)
dmesg -T | grep -i "out of memory"          # did the kernel kill something?

A reliable order of attack for "the service is down":

1. systemctl status NAME: is it running, failed, or restarting in a loop? Note the exit code.
2. journalctl -u NAME --since "15 min ago": read the last lines before it stopped. The real error is usually the first one, not the last.
3. Check resources: df -h for disk, free -h for memory, journalctl -k for OOM kills.
4. Check the network: is it listening (ss -tulpn), and can you reach it locally (curl -v localhost:8080)?
5. Only then change something, one thing at a time, and watch the log while you do it.