Interactive Guide to systemd

The Engine of Your Linux System

This interactive guide explores `systemd`, the foundational system and service manager for modern Linux. More than just an init system, `systemd` is a comprehensive suite of tools that manages nearly all aspects of the operating system after the kernel boots. At its core, `systemd` uses "units" as standardized building blocks to manage everything from services and hardware to filesystems and scheduled tasks. Understanding these units and their "dependencies"—the rules that connect them—is key to mastering system administration.

By defining clear relationships, `systemd` enables robust, on-demand activation of resources and aggressive parallelization of startup tasks, leading to a more efficient and reliable system. This application allows you to explore each unit type in detail and visualize how dependencies create order and reliability.

Modular Units

`systemd` treats every system resource as a "unit," providing a consistent way to manage services, sockets, mount points, and more.

Defined Dependencies

Units declare their relationships, ensuring that services start in the correct order and only when their prerequisites are met.

Parallel Execution

By understanding dependencies, `systemd` can start independent units simultaneously, leading to faster boot times.

Unit Explorer

`systemd` manages many different types of resources, each defined by a specific unit type. This standardized approach allows for consistent management across the entire system. Select a unit from the list below to learn about its purpose, see common configuration directives, and understand its typical use case. This section helps you break down the system into its fundamental components.

Select a unit type above to see its details.

Dependency Visualizer

Dependencies are the rules that govern how units interact, forming the core of `systemd`'s orchestration logic. This tool helps you visualize these critical relationships. Select two units and a dependency type to see a plain-language explanation and a simple diagram illustrating how `systemd` will handle the interaction. This is the best way to understand the practical difference between a critical `Requires=` and a more flexible `Wants=`.

Dependent Unit (Unit A)

Dependency Type

Target Unit (Unit B)

Dependency Relationships

Not all dependencies are created equal. `systemd` distinguishes between critical requirements and optional "wants", as well as simple ordering rules. This chart provides a quick visual reference for the most common dependency types, illustrating their primary function: either linking units together (Requirement) or simply controlling their start sequence (Ordering). Use the tooltips to see a description of each dependency's role.

Unit Files & Overrides (Where + How)

systemd loads unit files from multiple locations with a clear precedence. The safest way to customize a service is usually via drop-in snippet files (overrides), not by editing vendor files directly.

Unit file locations (and precedence)

/etc/systemd/system/ — Admin overrides (highest priority; you control it)
/run/systemd/system/ — Runtime/transient units (temporary; created at runtime)
/usr/lib/systemd/system/ — Vendor units (package-provided)
Note: On some distros vendor units can be in /lib/systemd/system/.

Rule of thumb: if the same unit name exists in multiple places, systemd uses the highest-precedence one.

# See where the "main" unit file comes from
systemctl show -p FragmentPath nginx.service

# See what drop-in snippet files affect it
systemctl show -p DropInPaths nginx.service

Drop-in “snippet” files (.d/*.conf)

A drop-in (snippet) is a small .conf file placed in:

/etc/systemd/system/<unit>.d/*.conf (persistent override)
/run/systemd/system/<unit>.d/*.conf (runtime override)
/usr/lib/systemd/system/<unit>.d/*.conf (vendor drop-ins)

systemd merges them (lexicographic order). This is why a common override file name is: override.conf.

# Create a persistent drop-in snippet safely
sudo systemctl edit nginx.service

# Open full effective unit (base + drop-ins)
systemctl cat nginx.service

Overriding ExecStart (the correct rule)

For directives like ExecStart=, you can’t “append” safely. You usually must reset it first with an empty assignment, then set your new one.

# /etc/systemd/system/myapp.service.d/override.conf
[Service]
# Reset the original ExecStart from vendor unit:
ExecStart=
# Set your own:
ExecStart=/usr/local/bin/myapp --config /etc/myapp/config.yml

# Apply changes:
sudo systemctl daemon-reload
sudo systemctl restart myapp.service

Runtime overrides vs persistent overrides

Persistent: saved under /etc/systemd/system/... (survives reboot)
Runtime: saved under /run/systemd/system/... (lost after reboot)

# Runtime-only drop-in editor (temporary)
sudo systemctl edit --runtime myapp.service

# Set a property temporarily (runtime):
sudo systemctl set-property --runtime myapp.service MemoryMax=1G

Masking (hard block)

Masking prevents a unit from starting at all (manual or automatic). Internally it is commonly implemented by a symlink to /dev/null in /etc/systemd/system.

sudo systemctl mask bluetooth.service
sudo systemctl unmask bluetooth.service

See what you changed (diff tools)

# Show overrides vs vendor defaults
systemd-delta

# Remove all overrides and go back to vendor default
sudo systemctl revert myapp.service

systemd-delta is perfect for explaining “why does this service behave differently than default?”

Systemd Tools & Key Config Files

systemd is not just systemctl. It ships many tools (ctl commands) that talk to system services (daemons). This section maps “tool → daemon → config files → useful commands”.

resolvectl (DNS)

Controls / queries systemd-resolved (DNS resolver).

/etc/systemd/resolved.conf (+ resolved.conf.d/*.conf)
/run/systemd/resolve/resolv.conf (generated)
/etc/resolv.conf is often a symlink (depends on distro setup)

resolvectl status
resolvectl query example.com
resolvectl flush-caches

timedatectl (time/NTP)

Controls time settings; NTP is typically via systemd-timesyncd.

/etc/systemd/timesyncd.conf (+ timesyncd.conf.d/*.conf)

timedatectl status
sudo timedatectl set-timezone Europe/Amsterdam
sudo timedatectl set-ntp true

journalctl (logs)

Queries logs from systemd-journald.

/etc/systemd/journald.conf (+ journald.conf.d/*.conf)

journalctl -b
journalctl -u ssh.service -f
journalctl --disk-usage

loginctl (sessions)

Talks to systemd-logind (users/sessions/seats).

/etc/systemd/logind.conf (+ logind.conf.d/*.conf)

loginctl list-sessions
loginctl show-session <id>
sudo loginctl terminate-session <id>

hostnamectl / localectl

System identity & locale management (via systemd services).

hostnamectl status
sudo hostnamectl set-hostname myhost

localectl status
sudo localectl set-locale LANG=en_US.UTF-8

systemd-networkd tools (if used)

If your distro uses systemd-networkd, you manage links/networks with networkctl.

/etc/systemd/network/*.network, *.netdev, *.link

networkctl status
networkctl list
networkctl status eth0

coredumpctl (crashes)

Inspect crashes collected by systemd-coredump.

/etc/systemd/coredump.conf (+ coredump.conf.d/*.conf)

coredumpctl list
coredumpctl info <PID>
coredumpctl gdb <PID>

systemd-tmpfiles (temp rules)

Create/clean temp files/dirs using tmpfiles rules.

/etc/tmpfiles.d/*.conf (admin)
/usr/lib/tmpfiles.d/*.conf (vendor)
/run/tmpfiles.d/*.conf (runtime)

systemd-tmpfiles --create
systemd-tmpfiles --clean

system-wide defaults

Global systemd manager settings (rarely changed but important).

/etc/systemd/system.conf (system manager)
/etc/systemd/user.conf (user manager)

# Example: inspect current manager settings
systemctl show --property=DefaultTimeoutStartSec

systemd-journald (Central Logging)

systemd-journald is the logging service that collects logs from the kernel, boot process, and services (including stdout/stderr from .service units). It stores logs in a structured “journal” with rich metadata (unit name, PID, user, boot ID, timestamps), making filtering and troubleshooting much easier than plain text logs.

How it works (in practice)

Collects service output (stdout/stderr), kernel messages, and events during boot.
Stores entries in a binary journal format with indexing (fast queries).
Adds metadata like _SYSTEMD_UNIT, _PID, _UID, _BOOT_ID, etc.
Retention is controlled by size/time limits and vacuum settings.

Logs are stored either in memory (volatile) or on disk (persistent):

/run/log/journal/ → volatile (lost after reboot)
/var/log/journal/ → persistent (survives reboot)

Service + config locations

Service: systemd-journald.service
Main config: /etc/systemd/journald.conf
Drop-in snippets: /etc/systemd/journald.conf.d/*.conf
Vendor defaults: usually under /usr/lib/systemd/ (don’t edit)

# Check journald status
systemctl status systemd-journald

# See current effective journald settings (most useful properties)
systemctl show systemd-journald --no-pager | grep -E "LogLevel|StatusText|ActiveState"

Most useful journalctl commands (daily work)

# Follow logs (like tail -f)
journalctl -f

# Logs for one service
journalctl -u nginx.service

# Follow logs for one service
journalctl -u nginx.service -f

# Logs for current boot / previous boot
journalctl -b
journalctl -b -1

# Time filtering
journalctl --since "1 hour ago"
journalctl --since "2025-12-10 08:00" --until "2025-12-10 09:00"

# Priority filtering (0=emerg .. 7=debug)
journalctl -p err..alert

# Show newest entries first
journalctl -u ssh.service -r

# Show fields to learn metadata keys
journalctl -u nginx.service -n 1 -o verbose

Tip: If you need non-root read access, add your user to systemd-journal group:

sudo usermod -aG systemd-journal $USER

Maintenance commands (size/cleanup/rotation)

# Journal disk usage
journalctl --disk-usage

# Rotate journals (start a new file)
sudo journalctl --rotate

# Vacuum by time (delete older than 7 days)
sudo journalctl --vacuum-time=7d

# Vacuum by size (keep max 500MB)
sudo journalctl --vacuum-size=500M

These are common “daily admin” tasks when logs grow too much or disks fill up.

Common configuration (practical recipes)

Best practice: use a drop-in file instead of editing the main config directly. Create /etc/systemd/journald.conf.d/override.conf.

A) Enable persistent logs + reasonable limits

# /etc/systemd/journald.conf.d/override.conf
[Journal]

# Store logs on disk in /var/log/journal (survives reboot). Other option: volatile (RAM only).
Storage=persistent

# Compress older journal files to save disk space.
Compress=yes

# Hard cap: journald will not use more than this total disk space for persistent logs.
SystemMaxUse=1G

# Always try to keep at least this much free space on the filesystem that holds the journal.
SystemKeepFree=500M

# Keep journal entries for at most this duration (older entries become eligible for removal).
MaxRetentionSec=7day

# Rate limiting window: within this time window, journald applies the burst limit below.
RateLimitIntervalSec=30s

# Maximum number of messages allowed within RateLimitIntervalSec before journald starts dropping/suppressing logs.
RateLimitBurst=10000

B) Apply changes (what to run)

# Ensure persistent directory exists (important on some systems)
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal

# Restart journald to apply config changes
sudo systemctl restart systemd-journald

C) Forward to syslog (if you still use rsyslog/syslog-ng)

# /etc/systemd/journald.conf.d/forward.conf
[Journal]
ForwardToSyslog=yes

Useful when you want journald + classic text logs at the same time.

D) Make logs more verbose for troubleshooting (temporary)

# Example: temporarily increase logging for a service
sudo systemctl set-environment SYSTEMD_LOG_LEVEL=debug
sudo systemctl restart systemd-journald

# Undo after troubleshooting
sudo systemctl unset-environment SYSTEMD_LOG_LEVEL
sudo systemctl restart systemd-journald

Use temporarily—debug logging can be noisy.

systemd-tmpfiles (Create & Clean Files/Dirs Automatically)

systemd-tmpfiles is a rules-based tool used to create, fix (permissions/ownership), and clean/remove files and directories in a consistent way—especially for paths that must exist after boot (like /run) and for cleaning old cache/temp content.

How it works (mental model)

Rules live in tmpfiles.d/*.conf files.
Create pass (--create): makes directories/files/symlinks and fixes perms/owners.
Clean pass (--clean): deletes files based on the AGE column (retention).
Remove pass (--remove): removes paths/contents that rules mark as removable (stronger than clean).
Boot-only rules can be marked with ! and applied when tmpfiles is invoked in boot mode.

Config locations (precedence)

/etc/tmpfiles.d/*.conf — admin rules (highest priority)
/run/tmpfiles.d/*.conf — runtime rules (temporary)
/usr/lib/tmpfiles.d/*.conf — vendor/package rules

Best practice: put your custom rules in /etc/tmpfiles.d/.

When does it run automatically?

systemd typically runs tmpfiles automatically using system units:

systemd-tmpfiles-setup.service: runs during boot to apply “create” rules (prepare paths like /run).
systemd-tmpfiles-clean.timer → systemd-tmpfiles-clean.service: runs periodically to do cleanup (--clean).

# Check boot setup
systemctl status systemd-tmpfiles-setup.service

# Check cleanup timer schedule
systemctl list-timers --all | grep tmpfiles

# See what they execute
systemctl cat systemd-tmpfiles-setup.service
systemctl cat systemd-tmpfiles-clean.timer
systemctl cat systemd-tmpfiles-clean.service

Want to change cleanup frequency? Override the timer:

sudo systemctl edit systemd-tmpfiles-clean.timer
sudo systemctl daemon-reload
sudo systemctl restart systemd-tmpfiles-clean.timer

Most useful commands (daily ops)

# Apply all "create" rules now
sudo systemd-tmpfiles --create

# Apply "create" for one rule file
sudo systemd-tmpfiles --create /etc/tmpfiles.d/myapp.conf

# Run cleanup based on AGE rules
sudo systemd-tmpfiles --clean

# Strong cleanup/remove (use carefully)
sudo systemd-tmpfiles --remove

# Boot-mode run (also applies rules marked with "!")
sudo systemd-tmpfiles --create --boot

The most common real-life use: a service fails because /run/myapp doesn’t exist → run tmpfiles create and restart the service.

Rule format

Each line is:

TYPE PATH MODE USER GROUP AGE ARGUMENTS

Notes:
• AGE controls cleanup eligibility (only matters for --clean).
• If you don’t need a field, use -.
• Add ! after the type (example: r!) to mark “boot-only”.

Most used types (with practical meaning)

d: create a directory (if missing). If AGE is set, its contents may be cleaned when old.
D: create a directory (if missing) and treat it as removable for “remove mode” workflows (strong cleanup scenarios).
e: “empty/cleanup managed directory” — do not rely on it to create new directories in all cases; use it mainly to apply cleanup policy to an existing directory and keep it clean by AGE.
v: create a btrfs subvolume (if possible); otherwise behaves like a normal directory rule.
q: btrfs subvolume like v, but attaches it to the same qgroup/quota context as the parent (when qgroups are enabled).
Q: btrfs subvolume like v, but creates a new “subtree” qgroup/quota context for that subvolume (when qgroups are enabled).
f/f+: create file if missing; f+ truncates/replaces content (ARGUMENTS can provide content).
w/w+: write/append to an existing file; w+ may create if missing depending on usage.
L/L+: create a symlink; L+ replaces an existing path if necessary.
z/Z: fix ownership/mode/labels; Z applies recursively.
a/A: set ACLs; A recursively.
t/T: set xattrs; T recursively.
r/R: remove a path (or glob); R is recursive (dangerous if misused).
x/X: exclude from cleanup (x is recursive ignore, X is non-recursive ignore).

If you’re not on btrfs or don’t use qgroups, you can ignore v/q/Q. Most admins live in d, z/Z, L, and cleanup via e + AGE.

Practical examples (copy/paste patterns)

1) Create /run directory for a service (most common)

# /etc/tmpfiles.d/myapp.conf
d /run/myapp 0755 myapp myapp - -

sudo systemd-tmpfiles --create /etc/tmpfiles.d/myapp.conf
sudo systemctl restart myapp.service

2) Ensure a persistent dir exists with strict perms

# /etc/tmpfiles.d/myapp.conf
d /var/lib/myapp 0750 myapp myapp - -

3) Enforce perms recursively (after deployment)

# /etc/tmpfiles.d/myapp-perms.conf
Z /var/lib/myapp 0750 myapp myapp - -

sudo systemd-tmpfiles --create /etc/tmpfiles.d/myapp-perms.conf

4) Boot-only removal of stale lock files

# /etc/tmpfiles.d/myapp-locks.conf
r! /var/lib/myapp/*.lock - - - - -

sudo systemd-tmpfiles --create --boot /etc/tmpfiles.d/myapp-locks.conf

5) Symlink for compatibility

# /etc/tmpfiles.d/myapp-links.conf
L /var/run/myapp - - - - /run/myapp

6) Cleanup policy example (keep cache tidy)

Use a cleanup-managed directory policy: keep cache files only for a limited time (AGE).

# /etc/tmpfiles.d/myapp-cache.conf
# Apply cleanup policy to existing cache dir (retention 7 days)
e /var/cache/myapp - - - 7d -

sudo systemd-tmpfiles --clean /etc/tmpfiles.d/myapp-cache.conf

systemd-networkd (Network Management)

systemd-networkd is a lightweight network manager for servers and minimal systems. It configures interfaces based on declarative files and reacts to link/device events (interfaces appearing/disappearing). It’s commonly paired with systemd-resolved for DNS and systemd-networkd-wait-online for “network is ready” semantics.

How it decides what to apply

Match first: it evaluates [Match] in .network files.
First match wins: the first matching file (by filename sort) is applied; later matches are ignored.
Tip: prefix filenames with numbers like 10-, 20- to control ordering (commonly keep < 70).

Config file types

.network: assign addresses, DHCP client/server, routes, DNS, NAT.
.netdev: create virtual devices (bridge, bond, vlan, veth, wireguard, …).
.link: low-level link settings (naming rules, MTU, MAC policy, etc.).

Where files live (precedence)

/etc/systemd/network/ (admin, highest priority)
/run/systemd/network/ (runtime/temporary)
/usr/lib/systemd/network/ (vendor/package defaults)

Same filename in /etc overrides vendor files. You can also use drop-ins like foo.network.d/*.conf for overrides.

Core commands you’ll actually use

# Enable + start networkd
sudo systemctl enable --now systemd-networkd

# See links and their states (managed/unmanaged, addresses, DHCP, routes)
networkctl list
networkctl status
networkctl status 

# Re-apply configs after editing files
sudo networkctl reload
sudo networkctl reconfigure 

# Helpful troubleshooting
systemctl status systemd-networkd -l
journalctl -u systemd-networkd -b --no-pager

# “Wait until network is configured” (important for services depending on network-online.target)
systemctl status systemd-networkd-wait-online

If another network manager (like NetworkManager) is active, don’t run both on the same interface.

Example 1: Simple DHCP client on a WAN interface

Get an IPv4 address from upstream DHCP on enp1s0.

# /etc/systemd/network/10-wan.network
[Match]
Name=enp1s0

[Network]
DHCP=ipv4

[DHCPv4]
UseDNS=true
UseRoutes=true
UseDomains=false

Example 2: Static IP + gateway + DNS

Put a server on 10.10.10.0/24 with explicit routing and DNS.

# /etc/systemd/network/20-static.network
[Match]
Name=enp2s0

[Network]
Address=10.10.10.50/24
Gateway=10.10.10.1
DNS=10.10.10.1
DNS=1.1.1.1
Domains=corp.local

Example 3: VLAN (netdev + network)

Create VLAN 20 on top of enp3s0 and assign a static address to the VLAN interface.

# /etc/systemd/network/10-vlan20.netdev
[NetDev]
Name=vlan20
Kind=vlan

[VLAN]
Id=20

# /etc/systemd/network/11-enp3s0.network
[Match]
Name=enp3s0

[Network]
VLAN=vlan20

# /etc/systemd/network/12-vlan20.network
[Match]
Name=vlan20

[Network]
Address=192.168.20.1/24

Example 4: Bridge (common on servers/VM hosts)

Create br0, attach enp4s0 to it, and put DHCP on the bridge.

# /etc/systemd/network/10-br0.netdev
[NetDev]
Name=br0
Kind=bridge

# /etc/systemd/network/11-enp4s0.network
[Match]
Name=enp4s0

[Network]
Bridge=br0

# /etc/systemd/network/12-br0.network
[Match]
Name=br0

[Network]
DHCP=ipv4

Comprehensive Example: DHCP Server with systemd-networkd (router-style)

Scenario: this host has two interfaces:
WAN = enp1s0 (gets internet via DHCP)
LAN = enp2s0 (serves clients on 192.168.50.0/24)
It will: assign a static LAN IP, run a DHCPv4 server on LAN, and NAT LAN traffic out to WAN.

# /etc/systemd/network/10-wan.network
[Match]
Name=enp1s0

[Network]
DHCP=ipv4

# /etc/systemd/network/20-lan.network
[Match]
Name=enp2s0

[Network]
# Our LAN interface address (this is also the default DHCP server/router address)
Address=192.168.50.1/24

# Enable built-in DHCPv4 server on this link
DHCPServer=yes

# If this box routes traffic to the WAN, enable NAT (masquerade) for IPv4
IPMasquerade=ipv4

# Optional explicit forwarding (IPMasquerade=ipv4 implies forwarding, but you can be explicit)
IPv4Forwarding=yes

[DHCPServer]
# Choose server address explicitly (useful if multiple Address= exist)
ServerAddress=192.168.50.1/24

# Lease pool inside 192.168.50.0/24:
# PoolOffset=50 => start at .50
# PoolSize=150  => hand out .50 .. .199 (approx)
PoolOffset=50
PoolSize=150

# Lease times (good defaults for home/lab LAN)
DefaultLeaseTimeSec=1h
MaxLeaseTimeSec=12h

# Tell clients which DNS servers to use:
# Option A) Use public resolvers directly:
DNS=1.1.1.1
DNS=8.8.8.8
EmitDNS=yes

# Router option (gateway) handed to clients:
EmitRouter=yes
Router=192.168.50.1

# Persist leases to storage (default is typically yes).
# If you want to avoid waiting for persistent storage, you can use:
# PersistLeases=runtime

# Apply changes
sudo networkctl reload
sudo networkctl reconfigure enp1s0
sudo networkctl reconfigure enp2s0

# Verify DHCP server is running and handing leases
networkctl status enp2s0
journalctl -u systemd-networkd -b --no-pager | tail -n 200

Notes for real environments:
• DHCPServer=yes is DHCPv4 server (not a full “enterprise DHCP suite”).
• If your distro uses lease persistence, DHCP server start can depend on networkd persistent storage readiness; using PersistLeases=runtime avoids waiting (at the cost of losing leases on reboot).

Command Quick Reference

Interaction with `systemd` is primarily handled through `systemctl` and other related utilities. This section provides a searchable list of common commands for managing units and the system itself. Use the search box to quickly find the command you need for tasks like starting services, checking logs, or analyzing boot performance.

Systemd Guide