Creating systemd Service Units

Overview of systemd Units

systemd uses unit files to define services, devices, mount points, and other system resources. Service units (ending in .service) are the most common type for managing daemons and applications.

Unit File Locations

Directory	Purpose	Priority
/etc/systemd/system/	Local system administrator units (custom services)	Highest
/run/systemd/system/	Runtime units (temporary, cleared on reboot)	Medium
/lib/systemd/system/	Distribution-installed units (package defaults)	Lowest

💡 Best Practice

Always place custom service files in /etc/systemd/system/ to ensure they take precedence over distribution defaults and survive system updates.

Service Unit File Structure

A systemd service unit file is an INI-style configuration file divided into sections. Each section contains directives (key-value pairs) that define the service behavior.

Basic Template

basic-service-template.service

[Unit]
Description=Service description here
Documentation=man:servicename(8) https://example.com/docs
After=network.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/path/to/executable --options
ExecStop=/path/to/stop-script
Restart=on-failure
User=username
Group=groupname

[Install]
WantedBy=multi-user.target

Unit File Sections Explained

[Unit] Section

The [Unit] section contains generic information about the unit that is not specific to the type of unit.

Directive	Description	Example
Description=	Human-readable description of the unit	Description=Apache Web Server
Documentation=	URLs or man pages for documentation	Documentation=man:httpd(8)
After=	Start after the specified units	After=network.target syslog.target
Before=	Start before the specified units	Before=multi-user.target
Requires=	Hard dependency (both start together)	Requires=postgresql.service
Wants=	Soft dependency (doesn't fail if dependency fails)	Wants=network-online.target
Conflicts=	Units that cannot run simultaneously	Conflicts=shutdown.target

[Service] Section

The [Service] section contains service-specific configuration options.

Service Types

Type	Description	When to Use
simple	Default. Process started by ExecStart is the main process	Processes that stay in foreground
forking	Process forks and parent exits; child becomes main process	Traditional daemons that fork
oneshot	Short-lived process that exits; systemd waits for completion	Scripts that run and exit
notify	Process sends notification when ready via sd_notify()	Services that use systemd notification protocol
dbus	Service acquires a D-Bus name when ready	Services that register with D-Bus
idle	Delays execution until all active jobs are complete	Services that should start after boot messages

Key Service Directives

Directive	Description	Example
ExecStart=	Command to start the service	ExecStart=/usr/bin/myapp
ExecStartPre=	Commands to run before ExecStart	ExecStartPre=/usr/bin/check-config
ExecStartPost=	Commands to run after ExecStart	ExecStartPost=/usr/bin/notify-ready
ExecStop=	Command to stop the service gracefully	ExecStop=/usr/bin/myapp --shutdown
ExecStopPost=	Commands to run after ExecStop	ExecStopPost=/usr/bin/cleanup
ExecReload=	Command to reload configuration without restart	ExecReload=/bin/kill -HUP $MAINPID
Restart=	When to restart service (no, always, on-success, on-failure, on-abnormal, on-abort, on-watchdog)	Restart=on-failure
RestartSec=	Delay before restart	RestartSec=5s
TimeoutStartSec=	Maximum time to wait for startup	TimeoutStartSec=30s
TimeoutStopSec=	Maximum time to wait for shutdown	TimeoutStopSec=10s
User=	User account to run service as	User=www-data
Group=	Group to run service as	Group=www-data
WorkingDirectory=	Directory to use as current working directory	WorkingDirectory=/var/www
Environment=	Set environment variables	Environment="PATH=/usr/local/bin"
EnvironmentFile=	Load environment from a file	EnvironmentFile=/etc/myapp/config
PIDFile=	Path to PID file (for Type=forking)	PIDFile=/var/run/myapp.pid
RemainAfterExit=	Consider service active after process exits	RemainAfterExit=yes

[Install] Section

The [Install] section defines how the unit should be enabled/disabled and under what circumstances it should start.

Directive	Description	Example
WantedBy=	Target that should start this service (creates symlink)	WantedBy=multi-user.target
RequiredBy=	Target that requires this service	RequiredBy=multi-user.target
Alias=	Additional names for the unit	Alias=myapp.service
Also=	Other units to enable/disable together	Also=myapp-worker.service

ℹ️ Understanding Targets

Common systemd targets:

multi-user.target - Multi-user text mode (equivalent to runlevel 3)
graphical.target - Graphical desktop mode (equivalent to runlevel 5)
network.target - Basic networking is available
network-online.target - Network is fully configured and online

Complete Examples

Example 1: Simple Web Application

A Node.js web application that should start after network is available and restart on failure.

/etc/systemd/system/webapp.service

[Unit]
Description=Node.js Web Application
Documentation=https://example.com/docs
After=network.target
Wants=network-online.target

[Service]
Type=simple
User=webapp
Group=webapp
WorkingDirectory=/opt/webapp
Environment="NODE_ENV=production"
Environment="PORT=3000"
ExecStart=/usr/bin/node /opt/webapp/server.js
Restart=on-failure
RestartSec=10s
StandardOutput=journal
StandardError=journal

# Security settings
NoNewPrivileges=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Example 2: Traditional Forking Daemon

A traditional daemon that forks into the background and creates a PID file.

/etc/systemd/system/mydaemon.service

[Unit]
Description=My Custom Daemon
Documentation=man:mydaemon(8)
After=network.target syslog.target
Requires=network.target

[Service]
Type=forking
PIDFile=/var/run/mydaemon.pid
User=daemon
Group=daemon
ExecStartPre=/usr/bin/test -d /var/run/mydaemon || /usr/bin/mkdir -p /var/run/mydaemon
ExecStart=/usr/sbin/mydaemon --config /etc/mydaemon/mydaemon.conf
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/bin/kill -TERM $MAINPID
Restart=on-failure
RestartSec=5s
TimeoutStartSec=30s
TimeoutStopSec=10s

[Install]
WantedBy=multi-user.target

Example 3: Database Backup Service (oneshot)

A script that runs once to perform a backup, typically triggered by a timer unit.

/etc/systemd/system/database-backup.service

[Unit]
Description=Database Backup Service
Documentation=file:///usr/share/doc/backup/README
After=postgresql.service
Requires=postgresql.service

[Service]
Type=oneshot
User=postgres
Group=postgres
ExecStartPre=/usr/bin/test -d /backup/db || /usr/bin/mkdir -p /backup/db
ExecStart=/usr/local/bin/backup-database.sh
StandardOutput=journal
StandardError=journal

# Allow backup to take up to 1 hour
TimeoutStartSec=3600s

[Install]
WantedBy=multi-user.target

Example 4: Java Application with Pre/Post Scripts

A Java application with pre-start validation and post-stop cleanup.

/etc/systemd/system/javaapp.service

[Unit]
Description=Java Application Server
Documentation=https://docs.example.com/javaapp
After=network.target
Wants=network-online.target

[Service]
Type=simple
User=javaapp
Group=javaapp
WorkingDirectory=/opt/javaapp

# Pre-start: Validate configuration and create necessary directories
ExecStartPre=/usr/bin/test -f /etc/javaapp/config.properties
ExecStartPre=/usr/bin/mkdir -p /var/log/javaapp
ExecStartPre=/usr/bin/chown javaapp:javaapp /var/log/javaapp

# Main service
ExecStart=/usr/bin/java \
    -Xmx2048m \
    -Xms512m \
    -jar /opt/javaapp/application.jar \
    --config=/etc/javaapp/config.properties

# Post-start: Verify service is responding
ExecStartPost=/bin/sleep 5
ExecStartPost=/usr/bin/curl -f http://localhost:8080/health

# Graceful shutdown
ExecStop=/usr/bin/curl -X POST http://localhost:8080/shutdown
ExecStop=/bin/sleep 3
ExecStop=/bin/kill -TERM $MAINPID

# Post-stop: Archive logs and cleanup
ExecStopPost=/usr/bin/tar -czf /var/log/javaapp/archive-$(date +%%Y%%m%%d-%%H%%M%%S).tar.gz /var/log/javaapp/*.log
ExecStopPost=/usr/bin/find /var/log/javaapp -name "*.log" -delete

# Restart behavior
Restart=on-failure
RestartSec=30s

# Timeouts
TimeoutStartSec=60s
TimeoutStopSec=30s

# Environment
Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk"
EnvironmentFile=-/etc/javaapp/environment

# Logging
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Example 5: Python Application with Virtual Environment

A Python application running in a virtual environment with comprehensive error handling.

/etc/systemd/system/pythonapp.service

[Unit]
Description=Python Application Service
Documentation=https://github.com/example/pythonapp
After=network.target postgresql.service
Wants=postgresql.service

[Service]
Type=simple
User=pythonapp
Group=pythonapp
WorkingDirectory=/opt/pythonapp

# Activate virtual environment and run application
ExecStart=/opt/pythonapp/venv/bin/python /opt/pythonapp/app.py

# Reload configuration without restart
ExecReload=/bin/kill -USR1 $MAINPID

# Environment variables
Environment="PYTHONUNBUFFERED=1"
Environment="PYTHONPATH=/opt/pythonapp"
EnvironmentFile=/etc/pythonapp/config.env

# Restart policy
Restart=always
RestartSec=10s

# Limits
LimitNOFILE=65536

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=pythonapp

[Install]
WantedBy=multi-user.target

Example 6: Complex Startup/Shutdown with Dependencies

An application server that requires multiple services and has complex startup/shutdown procedures.

/etc/systemd/system/appserver.service

[Unit]
Description=Application Server with Complex Dependencies
Documentation=man:appserver(8) https://docs.example.com
After=network-online.target postgresql.service redis.service
Wants=network-online.target
Requires=postgresql.service redis.service

# Don't start if conflicting service is running
Conflicts=appserver-dev.service

[Service]
Type=notify
User=appserver
Group=appserver
WorkingDirectory=/opt/appserver

# Pre-start checks and setup
ExecStartPre=-/bin/mkdir -p /var/run/appserver
ExecStartPre=-/bin/chown appserver:appserver /var/run/appserver
ExecStartPre=/usr/local/bin/check-dependencies.sh
ExecStartPre=/usr/local/bin/migrate-database.sh

# Start the main service
ExecStart=/usr/local/bin/appserver \
    --config /etc/appserver/server.conf \
    --notify-systemd

# Post-start verification
ExecStartPost=/bin/sleep 5
ExecStartPost=/usr/local/bin/health-check.sh

# Reload without full restart
ExecReload=/usr/local/bin/appserver --reload

# Graceful shutdown sequence
ExecStop=/usr/local/bin/appserver --shutdown graceful
ExecStop=/bin/sleep 10

# If graceful shutdown fails, force it
ExecStop=-/bin/kill -TERM $MAINPID

# Post-stop cleanup
ExecStopPost=/usr/local/bin/cleanup-sessions.sh
ExecStopPost=-/bin/rm -f /var/run/appserver/*.pid

# Restart configuration
Restart=on-failure
RestartSec=30s
StartLimitBurst=5
StartLimitIntervalSec=300

# Timeouts
TimeoutStartSec=120s
TimeoutStopSec=60s

# Resource limits
LimitNOFILE=100000
LimitNPROC=512

# Environment
EnvironmentFile=/etc/appserver/environment
Environment="PATH=/usr/local/bin:/usr/bin:/bin"

# Logging
StandardOutput=journal+console
StandardError=journal
SyslogIdentifier=appserver
SyslogLevel=info

# Security hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log/appserver /var/run/appserver

[Install]
WantedBy=multi-user.target
Alias=app.service

⚠️ Important Notes on ExecStart Commands

Commands must use absolute paths (e.g., /usr/bin/python, not python)
For Type=simple, the command must run in the foreground (not daemonize)
Use - prefix (e.g., ExecStartPre=-/bin/mkdir) to ignore failure of non-critical commands
Multiple ExecStartPre directives execute in order; failure stops the sequence unless prefixed with -

systemctl Commands

Creating and Managing Services

1. Create the Service File

# Create service file with your editor
sudo nano /etc/systemd/system/myservice.service

# Or copy from a template
sudo cp myservice.service /etc/systemd/system/

2. Reload systemd Configuration

After creating or modifying a service file, you must reload systemd's configuration:

sudo systemctl daemon-reload

3. Enable the Service

Enable the service to start automatically at boot:

sudo systemctl enable myservice.service

4. Start the Service

sudo systemctl start myservice.service

5. Check Service Status

# Detailed status
sudo systemctl status myservice.service

# Brief status
systemctl is-active myservice.service
systemctl is-enabled myservice.service
systemctl is-failed myservice.service

Common Management Commands

Command	Description
systemctl start SERVICE	Start a service immediately
systemctl stop SERVICE	Stop a running service
systemctl restart SERVICE	Restart a service (stop then start)
systemctl reload SERVICE	Reload service configuration (if supported)
systemctl enable SERVICE	Enable service to start at boot
systemctl disable SERVICE	Disable service from starting at boot
systemctl status SERVICE	Show detailed service status
systemctl cat SERVICE	Display the service unit file
systemctl edit SERVICE	Create override file for service
systemctl edit --full SERVICE	Edit the complete service file
systemctl list-dependencies SERVICE	Show service dependencies
systemctl mask SERVICE	Prevent service from being started
systemctl unmask SERVICE	Remove mask from service

Viewing Logs

# View service logs
sudo journalctl -u myservice.service

# Follow logs in real-time
sudo journalctl -u myservice.service -f

# Show logs since boot
sudo journalctl -u myservice.service -b

# Show last 100 lines
sudo journalctl -u myservice.service -n 100

# Show logs from last hour
sudo journalctl -u myservice.service --since "1 hour ago"

# Show logs with specific priority (0-7, where 3 is error)
sudo journalctl -u myservice.service -p err

System-Wide Commands

Command	Description
systemctl list-units	List all active units
systemctl list-units --all	List all units (active and inactive)
systemctl list-unit-files	List all unit files and their states
systemctl list-dependencies	Show dependency tree
systemctl daemon-reload	Reload systemd configuration
systemctl get-default	Show default target
systemctl set-default TARGET	Set default target

Troubleshooting

Common Issues and Solutions

Service Fails to Start

# Check detailed status
sudo systemctl status myservice.service

# Check logs for errors
sudo journalctl -u myservice.service -n 50

# Verify syntax
sudo systemd-analyze verify /etc/systemd/system/myservice.service

💡 Common Causes

Incorrect file permissions (service file should be 644)
Wrong executable path in ExecStart
Missing dependencies (check After= and Requires=)
Process exits immediately (check Type=)
User/Group doesn't exist or lacks permissions

Service Starts but Stops Immediately

For Type=simple, the process must stay in foreground
For Type=forking, verify PIDFile location and permissions
Check if the application is daemonizing when it shouldn't
Verify WorkingDirectory exists and is accessible

Debugging Startup Issues

# Run the command manually as the service user
sudo -u serviceuser /path/to/executable

# Check what systemd sees
sudo systemctl show myservice.service

# Increase logging verbosity
sudo systemctl set-property myservice.service LogLevel=debug
sudo systemctl restart myservice.service

Environment Variables Not Working

systemd runs in a minimal environment. Explicitly set all required variables:

[Service]
Environment="PATH=/usr/local/bin:/usr/bin:/bin"
Environment="LANG=en_US.UTF-8"
EnvironmentFile=/etc/myservice/environment

Permission Denied Errors

# Check file ownership and permissions
ls -l /path/to/executable
ls -l /path/to/config

# Fix permissions if needed
sudo chown root:root /etc/systemd/system/myservice.service
sudo chmod 644 /etc/systemd/system/myservice.service

# Verify user/group exists
id serviceuser
getent group servicegroup

Service Takes Too Long to Start

# Increase timeout
[Service]
TimeoutStartSec=300s

# Or set to infinity
TimeoutStartSec=infinity

Analyzing Boot Performance

# Show boot time
systemd-analyze

# Show service start times
systemd-analyze blame

# Show critical chain
systemd-analyze critical-chain

# Show critical chain for specific service
systemd-analyze critical-chain myservice.service

⚠️ Security Considerations

Never run services as root unless absolutely necessary
Use dedicated user accounts for each service
Set appropriate file permissions (644 for unit files)
Use security directives: NoNewPrivileges=, PrivateTmp=, ProtectSystem=
Restrict file access with ReadOnlyPaths= and ReadWritePaths=
Never store passwords or secrets in unit files; use EnvironmentFile instead

Validation and Testing

# Validate unit file syntax
sudo systemd-analyze verify /etc/systemd/system/myservice.service

# Check configuration
sudo systemctl show myservice.service

# Test start/stop cycle
sudo systemctl start myservice.service
sudo systemctl status myservice.service
sudo systemctl stop myservice.service

# Test reboot behavior
sudo systemctl enable myservice.service
sudo reboot

# After reboot
systemctl status myservice.service

Best Practices Checklist

✅ Service Creation Checklist

Write a clear, descriptive Description=
Set appropriate After= dependencies
Choose the correct Type= for your service
Use absolute paths in ExecStart=
Set User= and Group= (avoid root)
Configure appropriate Restart= policy
Set reasonable timeout values
Add WantedBy= in [Install] section
Test the service manually before enabling
Check logs after starting
Test enable/disable and reboot behavior
Document any special requirements

Additional Resources

Man Pages: man systemd.service, man systemd.unit, man systemd.exec
systemd Documentation: https://www.freedesktop.org/software/systemd/man/
ArchWiki: https://wiki.archlinux.org/title/Systemd

Table of Contents