tengel/papyri
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
papyri/md/systemd_mechanics.md
tengel e8fb7b288e initial import
2024-03-20 11:40:22 -05:00

389 lines
31 KiB
Markdown
Raw Permalink Blame History

# systemd Mechanics
## Contents
- [Overview](#overview)
- [Configuration](#configuration)
- [Common Tools](#common-tools)
- [Boot Parameters](#boot-parameters)
- [Runlevels/Targets](#runlevelstargets)
- [Manipulating Runlevels](#manipulating-runlevels)
- [Manipulating Services](#manipulating-services)
- [Customizing Services](#customizing-services)
- [Instanced Units (.service, .socket, etc...)](#instanced-units-service-socket-etc)
- [Path Units (.path)](#path-units-path)
- [Mount Units (.mount)](#mount-units-mount)
- [Example Bind Mount - /var/tmp to /tmp](#example-bind-mount---vartmp-to-tmp)
- [Service Resource Limits](#service-resource-limits)
- [Service Types](#service-types)
- [Operational Commands](#operational-commands)
- [Power Management](#power-management)
- [Journal Commands](#journal-commands)
- [Notes](#notes)
- [References](#references)
## Overview
From the [systemd project homepage](http://www.freedesktop.org/wiki/Software/systemd/):
> `systemd` is a system and service manager for Linux, compatible with SysV and LSB init scripts. systemd provides aggressive parallelization capabilities, uses socket and D-Bus activation for starting services, offers on-demand starting of daemons, keeps track of processes using Linux cgroups, supports snapshotting and restoring of the system state, maintains mount and automount points and implements an elaborate transactional dependency-based service control logic. It can work as a drop-in replacement for sysvinit.
## Configuration
| **File / Directory** | **Function** |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `/etc/hostname` | The host name for the system (yes, it's part of systemd now) |
| `/etc/os-release` | Standardization of the various distribution ID files like /etc/redhat-release and similar |
| `/etc/vconsole.conf` | Configuration of the default keyboard mapping and console font |
| `/etc/locale.conf` | Configuration of the system-wide locale |
| `/etc/machine-id` | Machine ID file, superseding D-Bus' machine ID file. Guaranteed to exist and be valid on a systemd system |
| `/etc/machine-info` | Metadata about a host, like a pretty host name and an icon name; maintained by systemd-hostnamed |
| `/etc/systemd/system.conf` | Read when systemd is run as a system instance |
| `/etc/systemd/user.conf` | Read when systemd is run as a non-system service |
| `/etc/systemd/journald.conf` | Configures the systemd-journald system service |
| `/etc/systemd/logind.conf` | Configures various parameters of the systemd login manager systemd-logind |
| `/etc/modules-load.d/*.conf` | Drop-in directory for kernel modules to statically load at boot |
| `/etc/sysctl.d/\*.conf` | Drop-in directory for kernel sysctl parameters, extending /etc/sysctl.conf |
| `/etc/tmpfiles.d/*.conf` | Drop-in directory for configuration of runtime files that need to be removed/created/cleaned up at boot and during uptime |
| `/etc/binfmt.d/*.conf` | Drop-in directory for registration of additional binary formats for systems like Java, Mono and WINE |
| `/etc/systemd/` | Main configuration directory for customizations |
| `/var/log/journal/` | Location of the systemd-journald persistent log data |
| `/usr/lib/systemd/` | Main directory of distro-provided systemd components |
| `/var/lib/systemd/` | Main directory of runtime systemd components |
## Common Tools
Primary commands used in the day-to-day systemd world:
| **Command** | **Purpose** |
| ---------------- | --------------------------------------------------------------------------------------------- |
| `systemctl` | Used to introspect and control the state of the systemd system and service manager |
| `journalctl` | Query the contents of the systemd journal as written by systemd-journald.service |
| `hostnamectl` | Used to query and change the system hostname and related settings |
| `timedatectl` | Query and change the system clock and its settings |
| `localectl` | Control the system locale and keyboard layout settings |
| `loginctl` | Control the systemd login manager systemd-logind |
| `systemd-cat` | Connect a pipeline or program's output with the journal (like **logger**) |
| `systemd-cgls` | Recursively shows the contents of the selected Linux control group hierarchy in a tree |
| `systemd-cgtop` | Show top control groups by their resource usage; useful when using cgroups to limit resources |
| `systemd-delta` | Identify and compare configuration files in /etc that override default counterparts in /usr |
| `kernel-install` | Add and remove kernel and initramfs images to and from /boot |
## Boot Parameters
On boot **systemd** activates (by default) the target unit `default.target` whose job is to activate services and other units by pulling them in via dependencies. To override the unit to activate, **systemd** parses its own kernel command line arguments via the `systemd.unit=` command line option. This may be used to temporarily boot into a different boot unit. The classic runlevels are replaced as following:
| **Parameter** | **Purpose** |
| -------------------------------- | --------------------------------------------------------------------------------------------------- |
| `systemd.unit=rescue.target` | A special target unit for setting up the base system and a rescue shell (**similar to runlevel 1**) |
| `systemd.unit=emergency.target` | Very similar to passing `init=/bin/sh` but with the option to boot the full system from there |
| `systemd.unit=multi-user.target` | Setting up a non-graphical multi-user system |
| `systemd.unit=graphical.target` | Setting up a graphical login screen |
| **Parameter** | **Type** | **Default** | **Purpose** |
| ------------------------ | ------------- | --------------- | --------------------------------------------------------------------------------------------------- |
| `systemd.unit=` | string | default.target | Overrides the unit to activate on boot. Example: rescue.target or emergency.target |
| `systemd.dump_core=` | boolean | true | If true systemd dumps core when it crashes. Otherwise no core dump is created |
| `systemd.crash_shell=` | boolean | false | If true systemd spawns a shell when it crashes. Otherwise no core dump is created |
| `systemd.crash_chvt=` | int | \-1 | If positive systemd activates the specified virtual terminal when it crashes |
| `systemd.confirm_spawn=` | boolean | false | If true asks for confirmation when spawning processes |
| `systemd.show_status=` | boolean | true | If true shows terse service status updates on the console during bootup |
| `systemd.sysv_console=` | boolean | true | If true output of SysV initscripts will be directed to the console |
| `systemd.log_target=` | string | journal-or-kmsg | Set log target. One of: console, journal, syslog, kmsg, journal-or-kmsg, syslog-or-kmsg, null |
| `systemd.log_level=` | int, constant | info | Set log level. Numerical log level or one of: emerg, alert, crit, err, warning, notice, info, debug |
| `systemd.log_color=` | boolean | true | Highlight important log messages |
| `systemd.log_location=` | boolean | true | Include code location in log messages. This is mostly relevant for debugging purposes |
For details about these special systemd boot units, view the [systemd.special](http://www.freedesktop.org/software/systemd/man/systemd.special.html) man page.
## Runlevels/Targets
Systemd has a concept of _targets_ which serve a similar purpose as runlevels but act a little different. Each _target_ is named instead of numbered and is intended to serve a specific purpose. Some _targets_ are implemented by inheriting all of the services of another _target_ and adding additional services to it. There are systemd _targets_ that mimic the common sysvinit runlevels so you can still switch _targets_ using the familiar `telinit RUNLEVEL` command. The runlevels that are assigned a specific purpose on vanilla systemd-enabled RHEL/Fedora installs; 0, 1, 3, 5, and 6; have a 1:1 mapping with a specific systemd *target*.
| **SysV Runlevel** | **systemd Target** | **Notes** |
| ----------------- | ----------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| 0 | runlevel0.target, poweroff.target | Halt the system. |
| 1, s, single | runlevel1.target, rescue.target | Single user mode. |
| 2, 4 | runlevel2.target, runlevel4.target, multi-user.target | User-defined/Site-specific runlevels. By default, identical to 3. |
| 3 | runlevel3.target, multi-user.target | Multi-user, non-graphical. Users can usually login via multiple consoles or via the network. |
| 5 | runlevel5.target, graphical.target | Multi-user, graphical. Usually has all the services of runlevel 3 plus a graphical login. |
| 6 | runlevel6.target, reboot.target | Reboot |
| emergency | emergency.target | Emergency shell |
## Manipulating Runlevels
| **SysVinit Command** | **systemd Command** | **Notes** |
| ------------------------------- | -------------------------------------------- | --------------------------------------------- |
| `grep initdefault /etc/inittab` | `systemctl list-units --type=target` | Show current runlevel |
| `telinit 3` | `systemctl isolate multi-user.target` | Change to multi-user runlevel |
| `vi /etc/inittab` (initdefault) | `systemctl enable multi-user.target --force` | Set to use multi-user runlevel on next reboot |
## Manipulating Services
Note that multiple service can be specified after the command, so `systemctl start foo.service bar.service baz@foo.service` is a valid command.
| **SysVinit Command** | **systemd Command** | **Notes** |
| ------------------------------ | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| `service (foobar) start` | `systemctl start (foobar).service` | Used to start a service (not reboot persistent) |
| `service (foobar) stop` | `systemctl stop (foobar).service` | Used to stop a service (not reboot persistent) |
| `service (foobar) restart` | `systemctl restart (foobar).service` | Used to stop and then start a service |
| `service (foobar) reload` | `systemctl reload (foobar).service` | When supported, reloads the config file without interrupting pending operations. |
| `service (foobar) condrestart` | `systemctl condrestart (foobar).service` | Restarts if the service is already running. |
| `service (foobar) status` | `systemctl status (foobar).service` | Tells whether a service is currently running. |
| `ls /etc/rc.d/init.d/` | `systemctl list-unit-files --type=service` | Used to list the services that can be started or stopped; Used to list all the services and other units |
| `chkconfig (foobar) on` | `systemctl enable (foobar).service` | Turn the service on, for start at next boot, or other trigger. |
| `chkconfig (foobar) off` | `systemctl disable (foobar).service` | Turn the service off for the next reboot, or any other trigger. |
| `chkconfig (foobar)` | `systemctl is-enabled (foobar).service` | Used to check whether a service is configured to start or not in the current environment. |
| `chkconfig --list` | `systemctl list-unit-files --type=service` | Print a table of services that lists which runlevels each is configured on or off |
| `chkconfig (foobar) --list` | `ls /etc/systemd/system/*.wants/(foobar).service` | Used to list what levels this service is configured on or off |
| `chkconfig (foobar) --add` | `systemctl daemon-reload` | Used when you create a new service file or modify any configuration |
> All `/sbin/service` and `/sbin/chkconfig` lines listed above continue to work on Fedora (and likely RHEL7), and will be translated to native equivalents as necessary. The only exception is chkconfig --list.
> In SysVinit, services can define arbitrary commands. Examples would be `service iptables panic`, or `service httpd graceful`. Native systemd services do not have this ability; any service that defines an additional command in this way would need to define some other - service-specific - way to accomplish this task when writing a native systemd service definition. Check the package-specific release notes for any services that may have done this.
## Customizing Services
The best way to customize unit files is to add `/etc/systemd/system/`foobar.service.d`/*.conf` where _foobar.service.d_ is the name of the service you want to customize. If a directory doesn't already exist, create one and add a conf file with the settings you want to override.
- [systemd.unit](http://www.freedesktop.org/software/systemd/man/systemd.unit.html)
- [systemd.directives](http://www.freedesktop.org/software/systemd/man/systemd.directives.html)
**Example**: raise the number of Open Files for MariaDB (previously configured in /etc/security/limits.conf):
```
/etc/systemd/system/mariadb.service.d/limits.conf
[Service]
LimitNOFILE=10000
LimitMEMLOCK=100000
```
**Example**: automatically restart Apache if it dies/killed/etc.:
```
/etc/systemd/system/httpd.service.d/restart.conf
[Service]
Restart=always
RestartSec=30
```
Alternatively, you can copy the distribution provided unit file from /lib/systemd/system to /etc/systemd/system since the latter has higher precedence. If a line starts with .include followed by a file name, the specified file will be parsed at this point.
```
/etc/systemd/system/httpd.service
.include /lib/systemd/system/httpd.service
[Service]
Nice=-5
```
> Don't forget to reload systemd daemon using `systemctl daemon-reload` and `systemctl restart (foobar)` after editing a unit file where _foobar_ is the name of the unit. Also note that you can `systemd-delta` to list the unit files which have been customized and also the precise differences
> Be careful when using `.include` together with directives that can be defined multiple times like `EnvironmentFile`, since we can only add new directives, but we can't remove already defined ones. You have to copy the whole file from /lib/systemd/system to /etc/systemd/system in this case
## Instanced Units (.service, .socket, etc...)
systemd allows you to set up _instanced_ services, or services that you can run multiple times in parallel with a small change to the unit file, such as getty, dhcpcd, sshd, and more. The way you define these services is slightly different from normal service files such as sshd.service or _foobar_.socket. Instead of _name_._type_, you would call the file _name_@._type_, and this would become an instanced unit file. The internals of the unit file are also somewhat different:
```
/usr/lib/systemd/system/dhcpcd@.service
[Unit]
Description=dhcpcd on %I
Wants=network.target
Before=network.target
BindsTo=sys-subsystem-net-devices-%i.device
After=sys-subsystem-net-devices-%i.device
[Service]
Type=forking
PIDFile=/run/dhcpcd-%I.pid
ExecStart=/usr/bin/dhcpcd -q -w %I
ExecStop=/usr/bin/dhcpcd -x %I
[Install]
WantedBy=multi-user.target
# # TO run the service you would use:
# systemctl start dhcpcd@eth1.service
# # enabling would also be the same, unit@instance.type
# systemctl enable dhcpcd@eth1.service
```
These commands would start or enable the dhcpcd service with the instance 'eth1'. All of the %i and %I variables would be replaced with the instance, in this case eth1. This allows you to have a generic service file for dhcpcd or other units, instead of writing a new service file for each instance of the daemon.
The %i and %I are two of the different ways to specify the instance name in a service file. You can see more here: [systemd.unit](http://www.freedesktop.org/software/systemd/man/systemd.unit.html#Specifiers)
## Path Units (.path)
systemd includes many types of unit files. The .path unit file allows systemd to watch a path with inotify and execute a service according to the .path file. You can find a full manual here: [systemd.path](http://www.freedesktop.org/software/systemd/man/systemd.path)
## Mount Units (.mount)
Most mounts are specified in `/etc/fstab` and the `systemd-generator` creates the mount units dynamically upon boot, however there are cases where it does not work as expected or needed. One of those types is a bind mount - a bind mount needs to have a unique mount file created for it outside of fstab. See the upstream [systemd.mount](https://www.freedesktop.org/software/systemd/man/systemd.mount.html) documentation for full details on all possible options and features that can be used.
### Example Bind Mount - /var/tmp to /tmp
One of the common bind mounts to create is for `/var/tmp` to `/tmp` as it is a line item in the [CIS Hardening Guidelines](https://benchmarks.cisecurity.org/downloads/benchmarks/) as a requirement. The name of the mount unit file **must be the name of the destination** with the leading slash removed, and all internal slashes converted to hyphens; it is placed in the expected runtime location if created by hand (`/etc/systemd/system`) and not delivered via a package (`/usr/lib/systemd/system`).
```
/etc/systemd/system/var-tmp.mount
# https://www.freedesktop.org/software/systemd/man/systemd.mount.html
# -> Automatic Dependencies
[Unit]
DefaultDependencies=yes
[Mount]
What=/tmp
Where=/var/tmp
Type=none
Options=bind
[Install]
WantedBy=local-fs.target
```
The unit must be enabled as per any other unit type:
```
# systemctl daemon-reload
# systemctl enable var-tmp.mount
```
With this particular bind mount, if anything is currently using `/var/tmp` those services must be stopped first to activate (`systemctl start ...`) the unit. It is possible a full reboot is needed depending on which services are using the directory.
## Service Resource Limits
The integration of cgroups into systemd allows us to limit resource usage for a service; the limiting of CPU, Memory and Block I/O are the primary concern to most system administrators. These limits are a per-cgroup limit, not per-process, so for Apache it would only allow the whole cgroup to use 1G of memory, no matter how many tiny processes it spawns.
By default all processes get an even value of CPU time (1024); by raising this number per-service more CPU shares are given over those at the default 1024 level:
```
/etc/systemd/system/httpd.service
.include /lib/systemd/system/httpd.service
[Service]
CPUShares=1500
```
Memory is limited in the exact same way; this setting understands K, M, G, T suffixes (base 1024)
```
/etc/systemd/system/httpd.service
.include /lib/systemd/system/httpd.service
[Service]
MemoryLimit=1G
```
Block I/O is _weighted_; default weight is 1000, valid range is from 10 to 1000 and can be used by service, by block device or by named directory:
```
/etc/systemd/system/httpd.service
.include /lib/systemd/system/httpd.service
[Service]
BlockIOWeight=500
BlockIOWeight=/dev/disk/by-id/dm-name-vg_local-lv_home 250
BlockIOWeight=/var/www/html 750
```
## Service Types
| **Type** | **Definition** |
| -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| simple | (default) _systemd_ considers the service to be started up immediately. The process must not fork. Do not use this type if other services need to be ordered on this service, unless it is socket activated. |
| forking | _systemd_ considers the service started up once the process forks and the parent has exited. Specify `PIDFile=` as well so systemd can keep track of the main process |
| oneshot | Useful for scripts that do a single job and then exit. Set `RemainAfterExit=yes` as well so that systemd still considers the service as active after the process has exited |
| notify | Identical to `Type=simple`, but with the stipulation that the daemon will send a signal to systemd when it is ready |
| dbus | The service is considered ready when the specified BusName appears on DBus's system bus |
## Operational Commands
| **Command** | **Purpose** |
| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `systemctl --failed` | List failed units only |
| `systemctl list-jobs` | Look for jobs "running" (boot waits for completion) and "waiting" (executed only after those which are "running" are completed) |
| `systemctl list-units -t service --all` | List all available services and their current status |
| `systemctl list-units -t service` | Show all active services |
| `systemctl status` _foobar_.service | Examine the current runtime status of a service |
| `systemctl list-units -t target --all` | Show all available targets. |
| `systemctl list-units -t target` | Show all active targets |
| `systemctl show -p "Wants"` multi-user.target | See which services a target pulls in |
| `systemctl kill` httpd.service | Send the *kill* (SIGTERM) signal to all processes of a service |
| `systemctl kill -s SIGKILL` httpd.service | Send the specified signal to all processes of a service |
| `systemctl kill -s HUP --kill-who=main` httpd.service | Send the HUP signal to only the parent process of the service |
| `systemd-analyze blame` | List systemd unit initialization times at boot |
| `systemd --test --system --unit=`multi-user.target | Examine what gets started when when booted into a specific target |
## Power Management
| **Command** | **Purpose** |
| ------------------------ | ----------------------------------------------------------- |
| `systemctl reboot` | Shut down and reboot the system |
| `systemctl poweroff` | Shut down and power-off the system |
| `systemctl suspend` | Suspend the system |
| `systemctl hibernate` | Put the system into hibernation |
| `systemctl hybrid-sleep` | Put the system into hybrid-sleep state (or suspend-to-both) |
## Journal Commands
Compatibility with classic syslog implementations is provided via a socket `/run/systemd/journal/syslog`, to which all messages are forwarded. To make the syslog daemon work with the journal, it has to bind to this socket instead of `/dev/log`.
| **Command** | **Purpose** |
| ------------------------------------------------ | --------------------------------------------------------------------------------- |
| `journalctl -b -0` | Show all messages from the current boot (-1 = previous, -2 = two boots ago, etc.) |
| `journalctl -b -p err` | Show only priority ERROR level boot log messages |
| `journalctl --since=yesterday` | Show only logs since yesterday if rebooting is seldom (servers) |
| `journalctl -f` | Follow the journal like `tail -f /var/log/messages` |
| `journalctl /usr/sbin/httpd` | Show all messages by a specific binary |
| `journalctl /usr/sbin/vpnc /usr/sbin/dhclient` | Show messages interleaved between two binaries |
| `journalctl _PID=1234` | Show all messages by a specific process ID |
| `journalctl -u httpd` | Show all messages by a specific unit |
| `journalctl -u httpd --since=00:00 --until=9:30` | Show all messages for a unit and timeframe |
| `journalctl _TRANSPORT=kernel` | Show kernel ring buffer |
| `journalctl /dev/sdc` | Show messages related to a specific device |
## Notes
- RHEL7 will not have kdbus, which is going to make `system --user` require `$DISPLAY` variable because of dbus. kdbus is something that has been added very recently to the kernel and systemd.
## References
- <http://www.freedesktop.org/software/systemd/man/bootup.html>
- <http://en.wikipedia.org/wiki/Systemd>
- <https://en.wikipedia.org/wiki/Cgroups>
- <http://www.freedesktop.org/wiki/Software/systemd/FrequentlyAskedQuestions/>
- <http://www.freedesktop.org/wiki/Software/systemd/TipsAndTricks/>
- <http://www.freedesktop.org/wiki/Software/systemd/Debugging/>
- <http://www.freedesktop.org/software/systemd/man/>
- <http://0pointer.de/blog/projects/systemd-docs.html>
- <http://0pointer.de/blog/projects/journalctl.html>
- <https://www.kernel.org/doc/Documentation/cgroups/cgroups.txt>
- <https://fedoraproject.org/wiki/SysVinit_to_Systemd_Cheatsheet>
- <https://fedoraproject.org/wiki/How_to_debug_Systemd_problems>
- <https://fedoraproject.org/wiki/Systemd>
- <https://wiki.debian.org/systemd>
- <https://wiki.ubuntu.com/systemd>
- <https://wiki.archlinux.org/index.php/Systemd>
Reference in a new issue View git blame Copy permalink