systemctl Command - Complete Reference Guide

Overview and Concepts

What is systemctl?

systemctl is the command-line interface to systemd, the init system and service manager used by most modern Linux distributions. It manages system services, monitors their state, and controls the boot process and system states.

Unit Types

systemd manages various types of units, each with a specific purpose:

Unit Type	Extension	Purpose	Example
Service	.service	Daemon/service management	httpd.service, sshd.service
Socket	.socket	IPC or network socket activation	docker.socket, cups.socket
Target	.target	Group of units (like runlevels)	multi-user.target, graphical.target
Timer	.timer	Scheduled task activation	logrotate.timer, backup.timer
Mount	.mount	Filesystem mount points	home.mount, tmp.mount
Automount	.automount	On-demand mounting	home.automount
Device	.device	Device management	dev-disk-sda1.device
Path	.path	File/directory monitoring	cups.path
Swap	.swap	Swap space management	dev-sda2.swap
Slice	.slice	Resource management groups	user.slice, system.slice
Scope	.scope	External process groups	session-1.scope

ℹ️ Unit Naming

The .service extension is often optional when using systemctl commands. For example, systemctl start httpd and systemctl start httpd.service are equivalent.

systemctl vs. Legacy Commands

Legacy Command	systemctl Equivalent	Purpose
service httpd start	systemctl start httpd	Start service
service httpd stop	systemctl stop httpd	Stop service
service httpd restart	systemctl restart httpd	Restart service
service httpd status	systemctl status httpd	Check status
chkconfig httpd on	systemctl enable httpd	Enable at boot
chkconfig httpd off	systemctl disable httpd	Disable at boot
chkconfig --list	systemctl list-unit-files	List services
init 3	systemctl isolate multi-user.target	Change runlevel
init 6	systemctl reboot	Reboot system
init 0	systemctl poweroff	Shutdown system

Basic Usage and Syntax

Command Syntax

systemctl [OPTIONS] COMMAND [UNIT...]

Common Global Options

Option	Description	Example
-a, --all	Show all units (including inactive)	systemctl list-units --all
-t, --type=TYPE	Filter by unit type	systemctl -t service
--state=STATE	Filter by unit state	systemctl --state=failed
-l, --full	Don't truncate output	systemctl status -l httpd
-n, --lines=N	Number of journal lines to show	systemctl status -n 50 httpd
--no-pager	Don't pipe output through pager	systemctl status --no-pager httpd
--no-legend	Suppress headers and footers	systemctl list-units --no-legend
-q, --quiet	Suppress output	systemctl -q is-active httpd
-H, --host=HOST	Execute on remote host	systemctl -H user@host status httpd
-M, --machine=CONTAINER	Execute in container	systemctl -M mycontainer status httpd
--user	Operate on user units	systemctl --user status myapp
--system	Operate on system units (default)	systemctl --system status httpd
--failed	Show only failed units	systemctl --failed
--plain	Plain output (no tree structure)	systemctl list-dependencies --plain
--now	Start/stop when enabling/disabling	systemctl enable --now httpd
--force	Override normal checks	systemctl disable --force httpd

Getting Help

# General help
systemctl --help

# List all commands
systemctl --help | grep "^  "

# Man pages
man systemctl
man systemd.unit
man systemd.service
man systemd.target

Service Management

Starting and Stopping Services

# Start a service
systemctl start httpd

# Stop a service
systemctl stop httpd

# Restart a service (stop then start)
systemctl restart httpd

# Reload configuration without stopping
systemctl reload httpd

# Try reload, fall back to restart if not supported
systemctl reload-or-restart httpd

# Restart only if currently running
systemctl try-restart httpd

# Reload or restart only if currently running
systemctl try-reload-or-restart httpd

Enabling and Disabling Services

# Enable service to start at boot
systemctl enable httpd

# Disable service from starting at boot
systemctl disable httpd

# Enable and start immediately
systemctl enable --now httpd

# Disable and stop immediately
systemctl disable --now httpd

# Check if service is enabled
systemctl is-enabled httpd

# Re-enable service (disable then enable)
systemctl reenable httpd

ℹ️ Enable vs. Start

enable creates symlinks so the service starts at boot, but doesn't start it now. start runs the service immediately, but doesn't affect boot behavior. Use --now to do both at once.

Checking Service Status

# Detailed status information
systemctl status httpd

# Show more log lines (default is 10)
systemctl status -n 50 httpd

# Full output without truncation
systemctl status -l httpd

# Simple running check (returns exit code)
systemctl is-active httpd

# Check if service failed
systemctl is-failed httpd

# Check if service is enabled
systemctl is-enabled httpd

# Show if system is running
systemctl is-system-running

Viewing Service Properties

# Show all properties
systemctl show httpd

# Show specific property
systemctl show httpd -p ActiveState
systemctl show httpd -p SubState
systemctl show httpd -p MainPID
systemctl show httpd -p LoadState

# Multiple properties
systemctl show httpd -p ActiveState -p SubState -p MainPID

# Show properties for multiple units
systemctl show httpd nginx sshd

Viewing Service Files

# Display service unit file
systemctl cat httpd

# Show where unit file is located
systemctl cat httpd | head -1

# List all unit file paths
systemctl list-unit-files

# Edit service file (creates override)
systemctl edit httpd

# Edit full service file
systemctl edit --full httpd

# Remove overrides
systemctl revert httpd

Reloading systemd Configuration

# Reload systemd manager configuration
# Required after modifying unit files
systemctl daemon-reload

# Reexecute systemd manager (rarely needed)
systemctl daemon-reexec

⚠️ Important: daemon-reload

Always run systemctl daemon-reload after creating or modifying unit files. Without this, systemd won't see your changes.

Unit States and Status

Unit Load States

State	Description
loaded	Unit file has been processed
error	Error reading unit file
masked	Unit file is masked and cannot be loaded
not-found	Unit file could not be found

Unit Active States

State	Description
active	Unit is active and running
inactive	Unit is not active
activating	Unit is in the process of being activated
deactivating	Unit is in the process of being deactivated
failed	Unit failed in some way
reloading	Unit is reloading its configuration
maintenance	Unit is being maintained

Service Sub-States

Sub-State	Description
running	Service is actively running
exited	Service process exited (oneshot services)
dead	Service is not running
failed	Service failed to start or crashed
start	Service is starting
stop	Service is stopping
auto-restart	Service is being automatically restarted

Unit File States

State	Description
enabled	Unit will start automatically at boot
disabled	Unit will not start automatically
static	Unit cannot be enabled/disabled (no [Install] section)
masked	Unit is completely disabled and cannot be started
generated	Unit file was generated dynamically
indirect	Unit enabled via another unit
bad	Unit file has errors

Listing Units by State

# List all loaded units
systemctl list-units

# List all units including inactive
systemctl list-units --all

# List only active units
systemctl list-units --state=active

# List failed units
systemctl list-units --state=failed
systemctl --failed

# List running services
systemctl list-units --type=service --state=running

# List enabled services
systemctl list-unit-files --state=enabled

# List disabled services
systemctl list-unit-files --state=disabled

# List masked units
systemctl list-unit-files --state=masked

Status Output Interpretation

# Example status output
$ systemctl status httpd

● httpd.service - Apache Web Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2024-11-07 10:30:15 EST; 2h 15min ago
     Docs: man:httpd.service(8)
 Main PID: 1234 (httpd)
   Status: "Running, listening on port 80"
    Tasks: 213 (limit: 4915)
   Memory: 256.5M
   CGroup: /system.slice/httpd.service
           ├─1234 /usr/sbin/httpd -DFOREGROUND
           ├─1235 /usr/sbin/httpd -DFOREGROUND
           └─1236 /usr/sbin/httpd -DFOREGROUND

Nov 07 10:30:15 server systemd[1]: Starting Apache Web Server...
Nov 07 10:30:15 server httpd[1234]: Server configured, listening on port 80
Nov 07 10:30:15 server systemd[1]: Started Apache Web Server.

ℹ️ Understanding Status Output

● symbol: Green = running, Red = failed, White = inactive
Loaded: Shows unit file location and whether it's enabled
Active: Current runtime state and how long it's been that way
Main PID: Primary process ID
Tasks: Number of processes/threads
Memory: Memory usage
CGroup: Control group hierarchy showing all processes
Logs: Recent journal entries for this unit

System Targets and Runlevels

Understanding Targets

Targets are groups of units that define system states. They replace the traditional runlevel concept in systemd.

Common Targets

Target	Old Runlevel	Description
poweroff.target	0	Shutdown and power off
rescue.target	1, s, single	Single-user rescue mode
multi-user.target	2, 3, 4	Multi-user text mode
graphical.target	5	Multi-user graphical mode
reboot.target	6	Reboot the system
emergency.target	-	Emergency shell

Additional Important Targets

Target	Description
basic.target	Basic system initialization
sysinit.target	System initialization
network.target	Network services
network-online.target	Network is fully online
local-fs.target	Local filesystems mounted
remote-fs.target	Remote filesystems mounted
time-sync.target	System time synchronized
shutdown.target	System shutdown

Target Management Commands

# Show current target
systemctl get-default

# Set default target for boot
systemctl set-default multi-user.target
systemctl set-default graphical.target

# Change to different target (doesn't change default)
systemctl isolate multi-user.target
systemctl isolate graphical.target

# Change to rescue mode
systemctl rescue

# Change to emergency mode
systemctl emergency

# List all available targets
systemctl list-units --type=target

# List all target unit files
systemctl list-unit-files --type=target

# Show what units are part of a target
systemctl list-dependencies multi-user.target

# Show which targets are currently active
systemctl list-units --type=target --state=active

System State Commands

# Reboot the system
systemctl reboot

# Power off the system
systemctl poweroff

# Halt the system
systemctl halt

# Suspend the system (RAM)
systemctl suspend

# Hibernate the system (disk)
systemctl hibernate

# Hybrid sleep (suspend + hibernate)
systemctl hybrid-sleep

# Suspend then hibernate
systemctl suspend-then-hibernate

# Check if system supports operation
systemctl can-suspend
systemctl can-hibernate
systemctl can-hybrid-sleep

💡 Target vs. Isolate

set-default changes what target the system boots into. isolate switches to a target immediately without changing the boot default.

Dependencies and Ordering

Understanding Dependencies

systemd manages complex dependency relationships between units to ensure proper startup and shutdown order.

Dependency Types

Directive	Type	Description
Requires=	Hard dependency	If dependency fails, this unit fails too
Wants=	Soft dependency	Preferred but not required
Requisite=	Hard dependency	Dependency must already be running
BindsTo=	Strong binding	If dependency stops, this stops too
PartOf=	Group membership	Stopped/restarted with dependency
Conflicts=	Negative dependency	Cannot run simultaneously
Before=	Ordering	Start before these units
After=	Ordering	Start after these units

Viewing Dependencies

# Show dependency tree
systemctl list-dependencies httpd

# Show all dependencies (not just targets)
systemctl list-dependencies httpd --all

# Show reverse dependencies (what depends on this)
systemctl list-dependencies httpd --reverse

# Show dependencies before starting
systemctl list-dependencies httpd --before

# Show dependencies after starting
systemctl list-dependencies httpd --after

# Plain list (no tree)
systemctl list-dependencies httpd --plain

# Show dependencies for specific type
systemctl list-dependencies --type=service

# Combine options
systemctl list-dependencies httpd --all --reverse --plain

Example Dependency Output

$ systemctl list-dependencies httpd

httpd.service
● ├─system.slice
● ├─basic.target
● │ ├─-.mount
● │ ├─microcode.service
● │ ├─paths.target
● │ ├─slices.target
● │ │ ├─-.slice
● │ │ └─system.slice
● │ ├─sockets.target
● │ │ ├─systemd-journald.socket
● │ │ └─systemd-udevd-control.socket
● │ ├─sysinit.target
● │ │ ├─dev-hugepages.mount
● │ │ ├─systemd-journald.service
● │ │ └─systemd-tmpfiles-setup.service
● │ └─timers.target
● ├─network.target
● │ └─NetworkManager.service
● └─sysinit.target

Analyzing Startup Order

# Show what must start before this unit
systemctl show httpd -p Before

# Show what must start after this unit
systemctl show httpd -p After

# Show what this unit requires
systemctl show httpd -p Requires

# Show what this unit wants
systemctl show httpd -p Wants

# Show what conflicts with this unit
systemctl show httpd -p Conflicts

Timers (systemd cron replacement)

Understanding Timers

systemd timers provide an alternative to cron for scheduled task execution. They offer more flexibility, better integration with systemd, and improved logging.

Timer Types

Type	Description	Example Directive
Realtime	Runs at specific calendar times	OnCalendar=daily
Monotonic	Runs relative to system events	OnBootSec=15min

Timer Directives

Directive	Description	Example
OnCalendar=	Calendar-based schedule	OnCalendar=daily OnCalendar=Mon,Fri 10:00
OnBootSec=	Time after boot	OnBootSec=15min
OnStartupSec=	Time after systemd startup	OnStartupSec=10min
OnActiveSec=	Time after timer activation	OnActiveSec=1h
OnUnitActiveSec=	Time after unit activation	OnUnitActiveSec=2h
OnUnitInactiveSec=	Time after unit deactivation	OnUnitInactiveSec=30min
AccuracySec=	Accuracy window	AccuracySec=1min
RandomizedDelaySec=	Random delay before execution	RandomizedDelaySec=5min
Persistent=	Run immediately if missed	Persistent=true

Calendar Expressions

# Common calendar expressions
OnCalendar=daily         # Every day at 00:00
OnCalendar=weekly        # Every Monday at 00:00
OnCalendar=monthly       # First day of month at 00:00
OnCalendar=yearly        # January 1st at 00:00
OnCalendar=hourly        # Every hour at :00

# Specific times
OnCalendar=10:00         # Every day at 10:00 AM
OnCalendar=*:0/15        # Every 15 minutes
OnCalendar=*-*-* 04:00:00  # Every day at 4 AM

# Specific days
OnCalendar=Mon 10:00     # Every Monday at 10:00
OnCalendar=Mon,Fri 10:00 # Monday and Friday at 10:00
OnCalendar=Mon..Fri 10:00  # Weekdays at 10:00

# Complex expressions
OnCalendar=Mon,Tue *-*-01..07 10:00  # First Monday/Tuesday of month
OnCalendar=*-*-15,30 02:00           # 15th and 30th at 2 AM

Testing Calendar Expressions

# Test calendar expression
systemd-analyze calendar "daily"
systemd-analyze calendar "Mon,Fri 10:00"
systemd-analyze calendar "*-*-* 04:00:00"

# Shows next several execution times
systemd-analyze calendar "Mon..Fri *:0/15"

Creating a Timer

Timers require two files: a .timer file and a corresponding .service file.

Example: Backup Timer

/etc/systemd/system/backup.service

[Unit]
Description=Daily Backup Service
Documentation=file:///usr/share/doc/backup/README

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
User=backup
Group=backup
StandardOutput=journal

/etc/systemd/system/backup.timer

[Unit]
Description=Daily Backup Timer
Documentation=file:///usr/share/doc/backup/README

[Timer]
# Run at 2 AM daily
OnCalendar=daily
OnCalendar=*-*-* 02:00:00

# If system was off, run on next boot
Persistent=true

# Add random delay up to 5 minutes
RandomizedDelaySec=5min

[Install]
WantedBy=timers.target

Managing Timers

# Enable and start timer
systemctl enable --now backup.timer

# Check timer status
systemctl status backup.timer

# List all timers
systemctl list-timers

# List all timers including inactive
systemctl list-timers --all

# Show when timer will run next
systemctl list-timers backup.timer

# Manually trigger the service
systemctl start backup.service

# View timer logs
journalctl -u backup.timer
journalctl -u backup.service

Timer List Output

$ systemctl list-timers

NEXT                         LEFT          LAST                         PASSED       UNIT                         ACTIVATES
Fri 2024-11-08 02:00:00 EST  10h left      Thu 2024-11-07 02:00:00 EST  13h ago      backup.timer                 backup.service
Fri 2024-11-08 00:00:00 EST  8h left       Thu 2024-11-07 00:00:00 EST  15h ago      logrotate.timer              logrotate.service
Fri 2024-11-08 15:23:14 EST  23h left      Thu 2024-11-07 15:23:14 EST  42min ago    systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service

Cron to systemd Timer Migration

Cron Expression	systemd Timer Equivalent
0 2 * * *	OnCalendar=--* 02:00:00
/15 * * *	OnCalendar=*:0/15
0 0 * * 0	OnCalendar=Sun --* 00:00:00
0 0 1 * *	OnCalendar=--01 00:00:00
@reboot	OnBootSec=5min
@daily	OnCalendar=daily
@hourly	OnCalendar=hourly

💡 Advantages of Timers over Cron

Better logging integration with journald
Can run missed jobs after system was powered off (Persistent=)
Resource management through cgroups
Dependencies on other system units
More flexible scheduling options
Randomized delays to prevent thundering herd

Socket Activation

Understanding Socket Activation

Socket activation allows systemd to listen on sockets and start services on-demand when connections arrive. This improves boot time and resource usage.

Benefits of Socket Activation

Services start only when needed
Faster boot times (services don't start until first use)
Automatic restart of crashed services on next connection
Better parallelization during boot
Zero downtime service updates

Socket Unit Example

/etc/systemd/system/myapp.socket

[Unit]
Description=My Application Socket
Documentation=man:myapp(8)

[Socket]
# Listen on port 8080
ListenStream=8080

# Accept connections and pass to service
Accept=no

# Socket permissions
SocketMode=0666
SocketUser=myapp
SocketGroup=myapp

[Install]
WantedBy=sockets.target

/etc/systemd/system/myapp.service

[Unit]
Description=My Application Service
Documentation=man:myapp(8)
Requires=myapp.socket

[Service]
Type=notify
ExecStart=/usr/bin/myapp --socket-activation
User=myapp
Group=myapp
StandardInput=socket

[Install]
WantedBy=multi-user.target

Socket Types

Directive	Description	Example
ListenStream=	TCP stream socket	ListenStream=8080
ListenDatagram=	UDP datagram socket	ListenDatagram=514
ListenSequentialPacket=	Sequential packet socket	ListenSequentialPacket=/run/app.sock
ListenFIFO=	FIFO pipe	ListenFIFO=/run/app.fifo
ListenSpecial=	Special file descriptor	ListenSpecial=/dev/console
ListenNetlink=	Netlink socket	ListenNetlink=kobject-uevent 1
ListenMessageQueue=	POSIX message queue	ListenMessageQueue=/myqueue

Managing Socket Units

# Enable and start socket
systemctl enable --now myapp.socket

# Check socket status
systemctl status myapp.socket

# List all socket units
systemctl list-units --type=socket

# List all socket unit files
systemctl list-unit-files --type=socket

# Show what service the socket will activate
systemctl show myapp.socket -p Triggers

# Start associated service manually
systemctl start myapp.service

# Stop both socket and service
systemctl stop myapp.socket myapp.service

Real-World Socket Activation Examples

SSH Socket Activation

# Check if sshd uses socket activation
systemctl status sshd.socket
systemctl list-dependencies sshd.socket

# Traditional sshd always runs
systemctl status sshd.service

# Socket-activated sshd only runs when connection arrives
# First connection starts service automatically

Docker Socket

# Docker commonly uses socket activation
systemctl status docker.socket

# Socket listens on /var/run/docker.sock
# Docker daemon starts when docker command is used
systemctl status docker.service

Socket Options

Option	Description
Accept=yes	Spawn service instance for each connection
Accept=no	Single service handles all connections (default)
MaxConnections=	Maximum simultaneous connections
MaxConnectionsPerSource=	Max connections per source IP
SocketMode=	Permission mode for Unix socket
SocketUser=	Owner of Unix socket
SocketGroup=	Group of Unix socket
Service=	Specify service name if different from socket

Masking and Disabling

Understanding Masking

Masking completely prevents a unit from being started, even manually. It's stronger than disabling.

Action	Can Start Manually?	Starts at Boot?	Use Case
Enabled	Yes	Yes	Normal operation
Disabled	Yes	No	Prevent auto-start but allow manual
Masked	No	No	Completely prevent service from running

Masking Commands

# Mask a unit (creates symlink to /dev/null)
systemctl mask httpd

# Mask and stop immediately
systemctl mask --now httpd

# Unmask a unit
systemctl unmask httpd

# Check if unit is masked
systemctl is-enabled httpd
# Output: masked

# List all masked units
systemctl list-unit-files --state=masked

When to Use Masking

Prevent conflicting services (e.g., httpd and nginx)
Security hardening (disable unnecessary services)
Prevent service from being started by dependencies
Override package defaults that auto-start services
Testing and troubleshooting

Example: Preventing Conflicts

# You want to use nginx, ensure httpd never starts
systemctl mask httpd
systemctl enable --now nginx

# Even if something tries to start httpd, it will fail
systemctl start httpd
# Error: Unit httpd.service is masked

# To re-enable httpd later
systemctl unmask httpd
systemctl mask nginx
systemctl enable --now httpd

⚠️ Masking vs. Disabling

Use disable when you want to prevent automatic startup but allow manual control. Use mask when you want to completely prevent the service from running, even if another service tries to start it.

Unit Properties and Configuration

Viewing All Properties

# Show all properties of a unit
systemctl show httpd

# Show specific properties
systemctl show httpd -p ActiveState
systemctl show httpd -p SubState
systemctl show httpd -p MainPID
systemctl show httpd -p ExecStart
systemctl show httpd -p User
systemctl show httpd -p Group

# Multiple properties
systemctl show httpd -p ActiveState,SubState,MainPID,User

# Show properties for multiple units
systemctl show httpd nginx sshd

Important Properties

Property	Description	Example Command
ActiveState	Active/inactive/failed	systemctl show httpd -p ActiveState
SubState	Running/dead/exited	systemctl show httpd -p SubState
LoadState	Loaded/masked/not-found	systemctl show httpd -p LoadState
UnitFileState	Enabled/disabled/masked	systemctl show httpd -p UnitFileState
MainPID	Main process ID	systemctl show httpd -p MainPID
ExecStart	Start command	systemctl show httpd -p ExecStart
User	User running service	systemctl show httpd -p User
MemoryCurrent	Current memory usage	systemctl show httpd -p MemoryCurrent
TasksCurrent	Number of tasks/threads	systemctl show httpd -p TasksCurrent
After	Units that must start before	systemctl show httpd -p After
Before	Units that must start after	systemctl show httpd -p Before
Requires	Hard dependencies	systemctl show httpd -p Requires
Wants	Soft dependencies	systemctl show httpd -p Wants

Setting Runtime Properties

# Set property for current session only (doesn't persist)
systemctl set-property httpd MemoryLimit=512M
systemctl set-property httpd TasksMax=256

# Set property permanently
systemctl set-property httpd CPUQuota=20%

# Check the new property
systemctl show httpd -p MemoryLimit
systemctl show httpd -p CPUQuota

# Reset to default
systemctl revert httpd

ℹ️ Property Modifications

Properties set with set-property create drop-in override files in /etc/systemd/system/UNIT.d/. These persist across reboots.

Performance Analysis

Boot Time Analysis

# Show total boot time
systemd-analyze

# Output example:
# Startup finished in 2.547s (kernel) + 8.145s (initrd) + 
# 23.643s (userspace) = 34.335s

# Show time taken by each service
systemd-analyze blame

# Output shows services sorted by initialization time
# Example:
#   5.234s mariadb.service
#   3.891s NetworkManager.service
#   2.456s firewalld.service

# Show critical chain (bottleneck analysis)
systemd-analyze critical-chain

# Show critical chain for specific service
systemd-analyze critical-chain httpd.service

# Plot boot sequence (creates SVG file)
systemd-analyze plot > boot.svg

# Create detailed timing graph
systemd-analyze dot | dot -Tsvg > dependencies.svg

Understanding Critical Chain Output

$ systemd-analyze critical-chain

The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

multi-user.target @23.643s
└─httpd.service @20.155s +3.487s
  └─network.target @20.143s
    └─NetworkManager.service @15.234s +4.908s
      └─basic.target @15.201s
        └─sockets.target @15.199s
          └─systemd-journald.socket @2.345s

Service Resource Usage

# Show resource usage for all services
systemd-cgtop

# Real-time resource monitoring (like top)
# Shows CPU, Memory, I/O per service
# Press 'q' to quit

# Show specific service resource usage
systemctl status httpd
# Shows: Memory, Tasks (processes/threads), CGroup info

# Get detailed resource statistics
systemctl show httpd -p MemoryCurrent
systemctl show httpd -p TasksCurrent
systemctl show httpd -p CPUUsageNSec

# List all processes in service cgroup
systemctl status httpd
# or
systemd-cgls httpd.service

Verifying Configuration

# Verify unit file syntax
systemd-analyze verify httpd.service

# Verify all unit files
systemd-analyze verify /etc/systemd/system/*.service

# Check for common issues
systemd-analyze verify --recursive /etc/systemd/system/

# Dot syntax check (outputs warnings)
systemd-analyze dot

Security Analysis

# Analyze service security settings
systemd-analyze security httpd

# Shows security score and recommendations
# Example output:
# ✗ PrivateTmp=no                         (0.5)
# ✓ ProtectSystem=strict                  (0.0)
# ✗ NoNewPrivileges=no                    (0.3)

# Compare security of multiple services
systemd-analyze security httpd nginx sshd

# Show only failed/weak security settings
systemd-analyze security httpd | grep -E '✗|MEDIUM|HIGH'

System State Verification

# Check overall system state
systemctl is-system-running

# Possible outputs:
# - initializing: Still booting
# - starting: Startup in progress
# - running: System operational
# - degraded: One or more units failed
# - maintenance: In rescue/emergency mode
# - stopping: Shutting down

# If degraded, find what failed
systemctl --failed

# Get exit code for scripts
systemctl is-system-running >/dev/null
echo $?
# 0 = running, >0 = problem

Performance Tuning Tips

💡 Optimizing Boot Time

Disable unnecessary services: systemctl disable SERVICE
Mask services you'll never use: systemctl mask SERVICE
Use socket activation for rarely-used services
Parallel startup: systemd does this by default, but check dependencies
Convert slow Type=forking services to Type=simple or Type=notify
Review critical-chain and optimize bottlenecks
Use timers instead of services for maintenance tasks

Troubleshooting

Common Issues and Solutions

Issue: Service Fails to Start

# Check detailed status
systemctl status httpd -l -n 50

# Look for specific errors
journalctl -u httpd -n 100 --no-pager

# Check unit file for syntax errors
systemd-analyze verify /etc/systemd/system/httpd.service

# Test starting manually
/usr/sbin/httpd -t   # Test configuration
/usr/sbin/httpd -k start  # Try starting directly

# Check dependencies
systemctl list-dependencies httpd --all
systemctl list-dependencies httpd --reverse

Issue: Service Keeps Restarting

# Check restart settings
systemctl show httpd -p Restart
systemctl show httpd -p RestartSec

# View restart history
journalctl -u httpd | grep -i restart

# Check for crash patterns
systemctl status httpd

# View recent failures
journalctl -u httpd --since "1 hour ago" -p err

# Temporarily disable auto-restart for debugging
systemctl set-property httpd Restart=no

Issue: Service Won't Stop

# Try normal stop
systemctl stop httpd

# If that hangs, check status in another terminal
systemctl status httpd

# Kill forcefully
systemctl kill httpd

# Kill with specific signal
systemctl kill -s SIGKILL httpd

# Check timeout settings
systemctl show httpd -p TimeoutStopSec

# Increase timeout if needed
systemctl set-property httpd TimeoutStopSec=60s

Issue: "Failed to get D-Bus connection"

# This usually means you're in a container or chroot

# Check if you're in a container
systemd-detect-virt

# In Docker/container, use:
# - --privileged flag
# - Proper volume mounts
# - Or manage services without systemctl

# Alternative: use service command (if available)
service httpd start

# Or call init script directly
/etc/init.d/httpd start

Issue: Unit File Not Found

# Search for unit file
find /etc/systemd /lib/systemd /usr/lib/systemd -name "httpd.service" 2>/dev/null

# List all available units
systemctl list-unit-files | grep httpd

# Check if it's an alias
systemctl list-unit-files | grep -i apache

# Reload systemd if you just created file
systemctl daemon-reload

# Verify file permissions
ls -l /etc/systemd/system/httpd.service
# Should be: -rw-r--r-- root root

Issue: Dependency Failures

# Check what dependencies failed
systemctl status httpd
systemctl --failed

# List all dependencies
systemctl list-dependencies httpd --all

# Check specific dependency
systemctl status network.target

# Try starting dependency manually
systemctl start network.target

# View dependency tree
systemctl list-dependencies httpd --reverse

Debugging Commands

# Show what systemd is doing
systemctl list-jobs

# Cancel pending jobs
systemctl cancel JOB_ID

# Show system boot messages
journalctl -b
journalctl -b -1  # Previous boot

# Show only errors
journalctl -p err -b

# Follow logs in real-time
journalctl -u httpd -f

# Show logs with explanation
journalctl -u httpd -x

# Export logs
journalctl -u httpd > httpd-logs.txt

# Show kernel messages
journalctl -k

# Show logs from specific time
journalctl --since "2024-11-07 10:00" --until "2024-11-07 11:00"
journalctl --since "1 hour ago"
journalctl --since yesterday

Emergency Recovery

# Boot into rescue mode (from GRUB)
# Add to kernel parameters: systemd.unit=rescue.target

# Boot into emergency mode
# Add to kernel parameters: systemd.unit=emergency.target

# From running system
systemctl rescue
systemctl emergency

# Return to normal operation
systemctl default

# If systemd is broken, use init=/bin/bash
# At GRUB, add: init=/bin/bash
# Then:
mount -o remount,rw /
# Fix issues
# Reboot

Log Analysis

# Most recent boot logs
journalctl -b 0

# Previous boot logs
journalctl -b -1
journalctl -b -2

# List all boots
journalctl --list-boots

# Service-specific logs
journalctl -u httpd
journalctl -u httpd -u nginx  # Multiple services

# Priority levels
journalctl -p emerg     # 0
journalctl -p alert     # 1
journalctl -p crit      # 2
journalctl -p err       # 3
journalctl -p warning   # 4
journalctl -p notice    # 5
journalctl -p info      # 6
journalctl -p debug     # 7

# Disk usage
journalctl --disk-usage

# Vacuum old logs
journalctl --vacuum-time=7d
journalctl --vacuum-size=500M

Verification Commands

# Verify systemd configuration
systemd-analyze verify

# Check all unit files for issues
systemd-analyze verify /etc/systemd/system/*.service

# Validate calendar expression
systemd-analyze calendar "Mon,Fri 10:00"

# Test unit file syntax
systemd-analyze verify httpd.service

# Show security weaknesses
systemd-analyze security httpd

# Check system state
systemctl is-system-running

# List failed units
systemctl list-units --state=failed --all

# Check for masked units preventing startup
systemctl list-unit-files --state=masked

⚠️ Critical Troubleshooting Tips

Always run daemon-reload after editing unit files
Check logs with journalctl -xe for detailed errors
Verify syntax with systemd-analyze verify
Check dependencies with list-dependencies
Use systemctl --failed to find problem units
Keep a backup of working unit files
Test changes in development before production

Quick Reference

Essential Commands

Task	Command
Start service	systemctl start SERVICE
Stop service	systemctl stop SERVICE
Restart service	systemctl restart SERVICE
Check status	systemctl status SERVICE
Enable at boot	systemctl enable SERVICE
Disable at boot	systemctl disable SERVICE
Enable and start now	systemctl enable --now SERVICE
Reload config	systemctl daemon-reload
List services	systemctl list-units --type=service
Show failed	systemctl --failed
View logs	journalctl -u SERVICE
Follow logs	journalctl -u SERVICE -f
Boot time	systemd-analyze
Mask service	systemctl mask SERVICE
Show dependencies	systemctl list-dependencies SERVICE

Table of Contents