systemd

What is systemd?

systemd is a system and service manager for Linux, compatible with SysV and LSB init scripts. systemd provides aggressive parallelization capabilities, uses socket and D-Bus activation for starting services, offers on-demand starting of daemons, keeps track of processes using Linux control groups, supports snapshotting and restoring of the system state, maintains mount and automount points and implements an elaborate transactional dependency-based service control logic. It can work as a drop-in replacement for sysvinit.

systemd is a replacement for the Linux init daemon, (either System V or BSD-style). It is intended to provide a better framework for expressing services' dependencies, allow more work to be done in parallel (concurrently) at system startup, and to reduce shell overhead. The name comes from the Unix convention of suffixing the names of system daemons (background processes) with the letter "d".
systemd is maintained by Lennart Poettering and Kay Sievers, with many other contributors. It is published as free software under the terms of the GNU Lesser General Public License version 2.1 or later.

NOTE: the root process of the system is still /sbin/init, however it is a symbolic link to /lib/systemd/systemd.

systemd configuration files

Here's a little overview over these new common configuration files systemd supports on all distributions:

• /etc/hostname
  the host name for the system. One of the most basic and trivial system settings. Nonetheless previously all distributions used different files for this. Fedora used /etc/sysconfig/network, OpenSUSE /etc/HOSTNAME. We chose to standardize on the Debian configuration file /etc/hostname.

• /etc/vconsole.conf
  configuration of the default keyboard mapping and console font.

• /etc/locale.conf
  configuration of the system-wide locale.

• /etc/modules-load.d/*.conf
  a drop-in directory for kernel modules to statically load at boot (for the very few that still need this).

• /etc/sysctl.d/*.conf
  a drop-in directory for kernel sysctl parameters, extending what you can already do with /etc/sysctl.conf.

• /etc/tmpfiles.d/*.conf
  a drop-in directory for configuration of runtime files that need to be removed/created/cleaned up at boot and during uptime.

• /etc/binfmt.d/*.conf
  a drop-in directory for registration of additional binary formats for systems like Java, Mono and WINE.

• /etc/os-release
  a standardization of the various distribution ID files like /etc/fedora-release and similar. Really every distribution introduced their own file here; writing a simple tool that just prints out the name of the local distribution usually means including a database of release files to check. The LSB tried to standardize something like this with the lsb_release tool, but quite frankly the idea of employing a shell script in this is not the best choice the LSB folks ever made. To rectify this we just decided to generalize this, so that everybody can use the same file here.

• /etc/machine-id
  a machine ID file, superseding D-Bus' machine ID file. This file is guaranteed to be existing and valid on a systemd system, covering also stateless boots. By moving this out of the D-Bus logic it is hopefully interesting for a lot of additional uses as a unique and stable machine identifier.

• /etc/machine-info
  a new information file encoding meta data about a host, like a pretty host name and an icon name, replacing stuff like /etc/favicon.png and suchlike. This is maintained by systemd-hostnamed.

Reference

http://0pointer.de/blog/projects/the-new-configuration-files

Concepts

Systemd introduces the concept of systemd units. These units are represented by unit configuration files located in one of the directories listed in Table “Systemd Unit Locations”, and encapsulate information about system services, listening sockets, saved system state snapshots, and other objects that are relevant to the init system. For a complete list of available systemd unit types, see Table “Available systemd Unit Types”.

As well as services (name.service) and targets (name.target), other types of unit that you can manage in systemd include devices (name.device), file system mount points (name.mount), and sockets (name.socket).

Targets, meanwhile, are logical groups of units that provide a set of services. Think of a target as a wrapper in which you can place multiple units, making a tidy bundle to work with.

Unit files are built from several configurable sections, including unit descriptions and dependencies. Systemd also allows administrators to explicitly define a service's dependencies and load them before the given service starts by editing the unit files. Each unit file has a line that starts After= that can be used to define what service is required before the current service can start. WantedBy=lines specify that a target requires a given unit.

Targets have more meaningful names than those used in SysV-based systems. A name like graphical.target gives admins an idea of what a file will provide! To see the current target at which the system is residing, use the command systemctl get-default. To set the default target, use the command systemctl set-default targetname.target. targetname can be the following:

• basic.target
• rescue.target
• multi-user.target
• graphical.target
• reboot.target
• suspend.target

You don't want to set the last 2 as default ;-D

To get the default target specified via default.target link

systemctl get-default
multi-user.target

# OR if /etc/systemd/system/default.target exists
ls -l /etc/systemd/system/default.target
lrwxrwxrwx 1 root root 41 Aug  6 11:16 /etc/systemd/system/default.target -> /usr/lib/systemd/system/multi-user.target

Get a list of available targets

systemctl list-units --type target
systemctl list-units -t target

Set default target to boot into. Command links default.target to the given unit.

systemctl set-default name.target

List dependencies

show the required and wanted units of the specified unit. If no unit is specified, default.target is implied. Target units are recursively expanded. When --all is passed, all other units are recursively expanded as well.

systemctl list-dependencies graphical.target

Available systemd Unit Types

Analyzing the system state

systemctl commands

# List running units
systemctl
systemctl list-units

# List failed units
systemctl --failed

# List unit files
systemctl list-unit-files
systemctl list-unit-files -t service
systemctl list-unit-files --type=socket
# NOTE: --state is also available, arguments can be LOAD, SUB or ACTIVE.

Execute operations or manage services on remote hosts

systemctl -H user@host
 
# check sshd.service status on a remote host
systemctl status sshd.service -H username@hostname

Analyze system boot-up performance

systemd-analyze may be used to determine system boot-up performance statistics and retrieve other state and tracing information from the system and service manager, and to verify the correctness of unit files.

Prints the time spent in the kernel before userspace has been reached, the time spent in the initial RAM disk (initrd / initramfs) before normal system userspace has been reached and the time normal system userspace took to initialized.

NOTE: These measurements simply measure the time passed up to the point where all system services have been spawned, but not necessarily until they fully finished initialization or the disk is idle.

# equivalent to systemd-analyze
systemd-analyze time

Startup finished in 3.910s (kernel) + 14.711s (userspace) = 18.621s

systemd-analyze blame - prints a list of all running units, ordered by the time they took to initialize. This information may be used to optimize boot-up times.

NOTE: the output might be misleading as the initialization of one service might be slow simply because it waits for the initialization of another service to complete.

# systemd-analyze blame
         10.102s updatedb.service
          6.999s man-db.service
          2.041s logrotate.service
          1.392s systemd-modules-load.service
          1.073s systemd-journal-flush.service
           939ms systemd-fsck-root.service
           889ms shadow.service
           814ms systemd-logind.service
           731ms vboxservice.service
           729ms tmp.mount
           727ms dev-hugepages.mount
           697ms systemd-vconsole-setup.service
           696ms dhcpcd.service
           500ms systemd-udev-trigger.service
           375ms systemd-user-sessions.service
           373ms sys-kernel-debug.mount
           288ms dev-mqueue.mount
           129ms systemd-tmpfiles-clean.service
           109ms systemd-tmpfiles-setup.service
           108ms systemd-tmpfiles-setup-dev.service
           100ms kmod-static-nodes.service
            97ms systemd-update-utmp.service
            82ms systemd-fsck@dev-sda2.service
            81ms user@0.service
            67ms systemd-random-seed.service
            55ms systemd-sysctl.service
            47ms systemd-udevd.service
            31ms home.mount
            31ms systemd-remount-fs.service
            24ms user@1000.service
             7ms sys-kernel-config.mount

systemd-analyze critical-chain [UNIT...] - prints a tree of the time-critical chain of units (for each of the specified UNITs or for the default target otherwise). The time after the unit is active or started is printed after the "@" character. The time the unit takes to start is printed after the "+" character. Note that the output might be misleading as the initialization of one service might depend on socket activation and because of the parallel execution of units.

# systemd-analyze critical-chain
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.
multi-user.target @5.179s
└─getty.target @5.153s
  └─getty@tty1.service @5.128s
    └─systemd-user-sessions.service @4.725s +375ms
      └─basic.target @3.812s
        └─timers.target @3.786s
          └─systemd-tmpfiles-clean.timer @3.759s
            └─sysinit.target @3.682s
              └─systemd-update-utmp.service @3.582s +97ms
                └─systemd-tmpfiles-setup.service @3.463s +109ms
                  └─local-fs.target @3.461s
                    └─home.mount @3.427s +31ms
                      └─systemd-fsck@dev-sda2.service @3.290s +82ms
                        └─dev-sda2.device @3.286s

systemd-analyze plot - prints an SVG graphic detailing which system services have been started at what time, highlighting the time they spent on initialization.

systemd-analyze dot - generates textual dependency graph description in dot format for further processing with the GraphViz dot(1) tool. Use a command line like systemd-analyze dot | dot -Tsvg > systemd.svg to generate a graphical dependency tree. Unless --order or --require is passed, the generated graph will show both ordering and requirement dependencies. Optional pattern globbing style specifications (e.g. *.target) may be given at the end. A unit dependency is included in the graph if any of these patterns match either the origin or destination node.

systemd-analyze dump - outputs a (usually very long) human-readable serialization of the complete server state. Its format is subject to change without notice and should not be parsed by applications.

systemd-analyze set-log-level LEVEL - changes the current log level of the systemd daemon to LEVEL (accepts the same values as --log-level= described in systemd(1)).

systemd-analyze verify - will load unit files and print warnings if any errors are detected. Files specified on the command line will be loaded, but also any other units referenced by them. This command works by prepending the directories for all command line arguments at the beginning of the unit load path, which means that all units files found in those directories will be used in preference to the unit files found in the standard locations, even if not listed explicitly.

SysVinit to systemd Cheat Sheet

runlevel

Services

NOTE: all /sbin/service and /sbin/chkconfig lines listed above continue to work on systemd, and will be translated to native equivalents as necessary. The only exception is chkconfig --list.

Runlevels and targets

systemd has a concept of targets which serve a similar purpose as runlevels but act a little different. Each target is named instead of numbered and is intended to serve a specific purpose. Some targets are implemented by inheriting all of the services of another target and adding additional services to it.

There are systemd targets that mimic the common sysvinit runlevels so you can still switch targets using the familiar telinit RUNLEVEL command. The runlevels that are assigned a specific purpose on vanilla Fedora installs; 0, 1, 3, 5, and 6; have a 1:1 mapping with a specific systemd target. Unfortunately, there's no good way to do the same for the user-defined runlevels like 2 and 4. If you make use of those it is suggested that you make a new named systemd target as /etc/systemd/system/$YOURTARGET that takes one of the existing runlevels as a base (you can look at /lib/systemd/system/graphical.target as an example), make a directory /etc/systemd/system/$YOURTARGET.wants, and then symlink the additional services that you want to enable into that directory. (The service unit files that you symlink live in /lib/systemd/system).

Reference: Working with systemd targets

Changing runlevel

NOTE: Since RHEL 7 / CentOS 7 / Oracle Linux 7 /etc/inittab is NO longer used due to the switch to systemd.

Full Cheat Sheet

systemd Upstart SysVinit

NOTE: service nginx {start | stop | restart} works for Debian, Ubuntu and RHEL/Oracle.

Journal - System Logs

systemd has its own logging system called the journal; therefore, running a syslog daemon is NO LONGER required. To read the log, use:

journalctl

In Arch Linux, the directory /var/log/journal/ is a part of the systemd package, and the journal (when Storage= is set to auto in /etc/systemd/journald.conf) will write to /var/log/journal/.

 If you or some program delete that directory, systemd will NOT recreate it automatically; however, it will be recreated during the next update of the systemd package. Until then, logs will be written to /run/systemd/journal, and logs will be LOST on reboot.

 Tip: If /var/log/journal/ resides in a btrfs file system, you should consider disabling Copy-on-Write for the directory.

 There is NO /var/log/message ;-)

Filtering Output using journalctl

journalctl allows you to filter the output by specific fields. Be aware that if there are many messages to display or filtering of large time span has to be done, the output of this command can be delayed for quite some time.

Show all messages from this boot:

journalctl -b

NOTE: However, often one is interested in messages NOT from the current, but from the previous boot (e.g. if an unrecoverable system crash happened). This is possible through optional offset parameter of the -b flag: journalctl -b -0 shows messages from the current boot, journalctl -b -1 from the previous boot, journalctl -b -2 from the second previous and so on. See man 1 journalctl for full description, the semantics is much more powerful.

Follow new messages (tail -f):

journalctl -f

Show all messages by a specific executable:

# systemd
journalctl /usr/lib/systemd/systemd
# sshd
journalctl -f `which sshd`

Show all messages of priority levels ERROR and worse, from the current boot:

journalctl -b -p err

Show all messages by a specific process:

journalctl _PID=1

Show all messages by a specific unit:

journalctl -u netcfg 

Show kernel ring buffer for current boot:

journalctl _TRANSPORT=kernel -b 

See man 1 journalctl, man 7 systemd.journal-fields, or Lennert's blog post for details.

Core Dump Collection Behavior

By default systemd stores core dumps in its journal

[root@tux kernel]# cat core_pattern
|/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e

Access the core dumps uisng systemd-coredumpctl

This behaviour can be disabled with a simple "hack":

$ ln -s /dev/null /etc/sysctl.d/coredump.conf
$ /lib/systemd/systemd-sysctl # or possibly a reboot

As always, the size of core dumps has to be equal or higher than the size of the core that is being dumped, as done by for example ulimit -c unlimited.

http://stackoverflow.com/questions/2065912/core-dumped-but-core-file-is-not-in-current-directory

journalctl - Query the systemd journal

# some examples
journalctl -b
journalctl -b | grep fsck

Set an upper file size for the systemd journal by editing /etc/systemd/journald.conf

systemd_journal_max_size: 100M

NOTE: If the journal is persistent (non-volatile), its size limit is set to a default value of 10% of the size of the respective file system. For example, with /var/log/journal located on a 50 GiB root partition this would lead to 5 GiB of journal data. The maximum size of the persistent journal can be controlled by SystemMaxUse in /etc/systemd/journald.conf. Run man journald.conf for more information.

NOTES for Arch Linux users

systemd 214 - core dumps are NO longer sent to the journal by default.

To re-enable

echo "|/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e" > /proc/sys/kernel/core_pattern
 
# or
 
echo "|/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e" | sudo tee /proc/sys/kernel/core_pattern

To make it persistent across reboots

echo "kernel.core_pattern=|/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e" > /etc/sysctl.d/50-coredump.conf

and run 

sysctl -p /etc/sysctl.d/50-coredump.conf

systemd 216 - made changes to core dump again, collection behavior can now be tuned in /etc/systemd/coredump.conf

Core dumps are stored in /var/lib/systemd/coredump if Storage=external is set.

Viewing Xorg.log with journalctl

GNOME 3.12 -> NO Xorg.log any more, GDM now starts the X server so that it writes the log to the systemd journal. Thus, to get the log file use journalctl, NOT vim, cat, less or tail.

Run the following command:

journalctl -e /usr/bin/Xorg

-e  --pager-end

Immediately jump to the end of the journal inside the implied pager tool. This implies -n1000 to guarantee that the pager will NOT buffer logs of unbounded size.

Troubleshooting

Arch Linux x86_64 running as XenServer HVM guest (XenServer 6.2 / Xen 4.1.5)

Investigate systemd errors

systemctl --state=failed
  UNIT                         LOAD   ACTIVE SUB    DESCRIPTION
● systemd-modules-load.service loaded failed failed Load Kernel Modules

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

1 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

OR

journalctl -b

Error

Mar 31 09:17:24 archlinux.apac.lab systemd-modules-load[97]: Failed to find module 'vboxguest'
Mar 31 09:17:24 archlinux.apac.lab systemd-modules-load[97]: Failed to find module 'vboxsf'
Mar 31 09:17:24 archlinux.apac.lab systemd-modules-load[97]: Failed to find module 'vboxvideo'
Mar 31 09:17:24 archlinux.apac.lab systemd[1]: systemd-modules-load.service: main process exited, code=exited, status=1/FAILURE
Mar 31 09:17:24 archlinux.apac.lab systemd[1]: Failed to start Load Kernel Modules.
Mar 31 09:17:24 archlinux.apac.lab systemd[1]: Unit systemd-modules-load.service entered failed state.

Get more information about the service - systemd-modules-load.service

systemctl status systemd-modules-load.service
● systemd-modules-load.service - Load Kernel Modules
   Loaded: loaded (/usr/lib/systemd/system/systemd-modules-load.service; static)
   Active: failed (Result: exit-code) since Mon 2014-03-31 09:17:24 EST; 12min ago
     Docs: man:systemd-modules-load.service(8)
           man:modules-load.d(5)
  Process: 97 ExecStart=/usr/lib/systemd/systemd-modules-load (code=exited, status=1/FAILURE)
 Main PID: 97 (code=exited, status=1/FAILURE)

Mar 31 09:17:24 archlinux.apac.lab systemd-modules-load[97]: Inserted module 'e1000'
Mar 31 09:17:24 archlinux.apac.lab systemd[1]: systemd-modules-load.service: main process exited, code=exited, status=1/FAILURE
Mar 31 09:17:24 archlinux.apac.lab systemd[1]: Failed to start Load Kernel Modules.
Mar 31 09:17:24 archlinux.apac.lab systemd[1]: Unit systemd-modules-load.service entered failed state.
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

NOTE: If the Process ID is NOT listed, just restart the failed service with systemctl restart systemd-modules-load

Investigate further with the PID

journalctl -b _PID=97
-- Logs begin at Wed 2013-03-06 10:17:20 EST, end at Mon 2014-03-31 09:23:59 EST. --
Mar 31 09:17:24 archlinux.apac.lab systemd-modules-load[97]: Inserted module 'e1000'
Mar 31 09:17:24 archlinux.apac.lab systemd-modules-load[97]: Failed to find module 'vboxguest'
Mar 31 09:17:24 archlinux.apac.lab systemd-modules-load[97]: Failed to find module 'vboxsf'
Mar 31 09:17:24 archlinux.apac.lab systemd-modules-load[97]: Failed to find module 'vboxvideo'

kernel module config issues

/etc/modules-load.d/

ls -Al /etc/modules-load.d/
total 8
-rw-r--r-- 1 root root 6 Feb 21 2013 e1000.conf
-rw-r----- 1 root root 27 Sep 20 2013 virtualbox.conf

Edit virtualbox.conf, comment out the modules we do not need any more (VirtualBox -> XenServer)

# vboxguest
# vboxsf
# vboxvideo

Restart systemctl start systemd-modules-load.service

# systemctl status systemd-modules-load.service
● systemd-modules-load.service - Load Kernel Modules
   Loaded: loaded (/usr/lib/systemd/system/systemd-modules-load.service; static)
   Active: active (exited) since Mon 2014-03-31 09:37:18 EST; 2min 7s ago
     Docs: man:systemd-modules-load.service(8)
           man:modules-load.d(5)
  Process: 2183 ExecStart=/usr/lib/systemd/systemd-modules-load (code=exited, status=0/SUCCESS)
 Main PID: 2183 (code=exited, status=0/SUCCESS)

Mar 31 09:37:18 archlinux.apac.lab systemd[1]: Started Load Kernel Modules.

Fixed!

Reference

https://wiki.archlinux.org/index.php/Systemd

http://freedesktop.org/wiki/Software/systemd

https://www.linux.com/learn/tutorials/527639-managing-services-on-linux-with-systemd

Cheat Sheet http://fedoraproject.org/wiki/SysVinit_to_Systemd_Cheatsheet 

Systemd tutorial https://www.linux.com/learn/tutorials/527639-managing-services-on-linux-with-systemd 

• Rethinking PID 1

• systemd for Administrators, Part 1

• systemd for Administrators, Part II

• systemd for Administrators, Part III

• systemd for Administrators, Part IV

• systemd for Administrators, Part V

• systemd for Administrators, Part VI

• systemd for Administrators, Part VII

• systemd for Administrators, Part VIII

• systemd for Administrators, Part IX

• systemd for Administrators, Part X 

• systemd for Administrators, Part XI 

LWN articles

• systemd programming part 1: modularity and configuration
  https://lwn.net/Articles/584175/
• Systemd programming part 2: activation and language issues
  https://lwn.net/Articles/584176/