🐧 Linux Server Management Guide

Comprehensive Day-to-Day Operations for System Administrators

Daily, Weekly, Monthly Tasks & Emergency Procedures

Morning Health Checks DAILY

Start each day with these essential checks to ensure your servers are healthy and running optimally.

Daily Morning Checklist

  • Check system uptime and recent reboots
  • Review disk space on all partitions
  • Verify backup completion status
  • Check for security updates
  • Review system logs for errors/warnings
  • Monitor CPU and memory usage
  • Check service status (web, database, etc.)
  • Review failed login attempts
  • Verify network connectivity
  • Check monitoring alerts

Quick Health Check Script

Daily Health Check Script
#!/bin/bash
# Save as: /usr/local/bin/morning-check.sh

echo "=========================================="
echo "SERVER HEALTH CHECK - $(date)"
echo "=========================================="
echo ""

# System Uptime
echo "=== UPTIME ==="
uptime
echo ""

# Disk Usage
echo "=== DISK USAGE ==="
df -h | grep -E '^/dev/'
echo ""

# Memory Usage
echo "=== MEMORY USAGE ==="
free -h
echo ""

# Load Average
echo "=== LOAD AVERAGE ==="
cat /proc/loadavg
echo ""

# Top 5 CPU Processes
echo "=== TOP 5 CPU PROCESSES ==="
ps aux --sort=-%cpu | head -6
echo ""

# Top 5 Memory Processes
echo "=== TOP 5 MEMORY PROCESSES ==="
ps aux --sort=-%mem | head -6
echo ""

# Failed Login Attempts (last 24 hours)
echo "=== FAILED LOGIN ATTEMPTS (Last 24h) ==="
grep "Failed password" /var/log/auth.log | tail -10
echo ""

# Service Status
echo "=== CRITICAL SERVICES ==="
for service in sshd nginx apache2 mysql postgresql docker; do
    if systemctl list-unit-files | grep -q "^${service}.service"; then
        systemctl is-active --quiet $service && echo "✓ $service: running" || echo "✗ $service: STOPPED"
    fi
done
echo ""

# Last 10 System Errors
echo "=== RECENT SYSTEM ERRORS ==="
journalctl -p err -n 10 --no-pager
echo ""

# Check for Security Updates
echo "=== SECURITY UPDATES ==="
if command -v apt &> /dev/null; then
    apt list --upgradable 2>/dev/null | grep -i security | head -5
elif command -v yum &> /dev/null; then
    yum list updates --security | head -10
fi
echo ""

echo "=========================================="
echo "Health check completed at $(date)"
echo "=========================================="
Make it executable and run it
# Make executable
sudo chmod +x /usr/local/bin/morning-check.sh

# Run the check
sudo /usr/local/bin/morning-check.sh

# Or add to cron for automated daily email
0 8 * * * /usr/local/bin/morning-check.sh | mail -s "Daily Server Health Report" admin@example.com

System Monitoring DAILY

Real-Time Monitoring Commands

System Load and Uptime
# View current load average and uptime
uptime

# Continuous monitoring (updates every 2 seconds)
watch -n 2 uptime

# Detailed load info
cat /proc/loadavg
# Output: 0.52 0.58 0.59 1/395 12345
# Meaning: 1min, 5min, 15min load averages, running/total processes, last PID
Process Monitoring
# Interactive process viewer (better than top)
htop

# Traditional top
top

# Top commands within top:
# P - Sort by CPU usage
# M - Sort by memory usage
# k - Kill a process
# q - Quit

# Show processes by CPU usage
ps aux --sort=-%cpu | head -20

# Show processes by memory usage
ps aux --sort=-%mem | head -20

# Monitor specific user's processes
ps aux | grep username

# Show process tree
pstree -p

# Find process by name
pgrep -a nginx

# Detailed process information
ps -ef | grep process_name
CPU Monitoring
# CPU information
lscpu

# Per-CPU statistics
mpstat -P ALL 1 5

# I/O wait time (high % = disk bottleneck)
iostat -x 1 5

# Real-time CPU usage per process
top -b -n 1 | head -20

# Check CPU temperature (if sensors available)
sensors

# CPU usage percentage
grep 'cpu ' /proc/stat | awk '{usage=($2+$4)*100/($2+$4+$5)} END {print usage "%"}'
Memory Monitoring
# Memory usage summary
free -h

# Detailed memory info
cat /proc/meminfo

# Memory usage by process
ps aux --sort=-%mem | awk '{printf "%-10s %-10s %-10s %s\n", $2, $3, $4, $11}' | head -20

# Check for memory leaks
watch -n 1 'free -h'

# Show swap usage
swapon --show

# Memory usage breakdown
vmstat 1 5

# Page cache usage
sync; echo 3 > /proc/sys/vm/drop_caches  # Clear cache (use carefully!)

Disk Management DAILY

Disk Space Monitoring
# Disk usage by filesystem
df -h

# Human-readable with inodes
df -ih

# Show specific filesystem type
df -h -t ext4

# Disk usage sorted by size
df -h | sort -k 5 -r

# Monitor disk usage in real-time
watch -n 5 'df -h'

# Alert if disk usage > 90%
df -h | awk '0+$5 >= 90 {print $0}' | mail -s "ALERT: Disk Usage Critical" admin@example.com
Finding Large Files and Directories
# Find largest directories in current location
du -h --max-depth=1 | sort -hr | head -20

# Find largest files in /var
find /var -type f -exec du -h {} + | sort -hr | head -20

# Find files larger than 100MB
find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null

# Find files larger than 1GB
find / -type f -size +1G -exec ls -lh {} \; 2>/dev/null

# Disk usage of specific directory
du -sh /var/log/*

# Disk usage with depth limit
du -h -d 2 /var | sort -hr | head -20

# Find and list large log files
find /var/log -type f -size +50M -exec ls -lh {} \;

# Check what's using deleted files (still using disk space)
lsof | grep deleted
Disk I/O Monitoring
# I/O statistics
iostat -x 1 5

# Disk I/O by process (requires iotop)
sudo iotop

# Simple I/O stats
vmstat 1 5

# Detailed disk stats
cat /proc/diskstats

# Monitor specific disk
iostat -x sda 1 5

# Check disk health (SMART)
sudo smartctl -a /dev/sda

# Test disk speed
sudo hdparm -tT /dev/sda
Cleaning Up Disk Space
# Clean package cache (Ubuntu/Debian)
sudo apt clean
sudo apt autoclean
sudo apt autoremove

# Clean package cache (RHEL/CentOS)
sudo yum clean all

# Clean journal logs (keep last 3 days)
sudo journalctl --vacuum-time=3d

# Clean old log files
sudo find /var/log -type f -name "*.gz" -mtime +30 -delete
sudo find /var/log -type f -name "*.log.*" -mtime +30 -delete

# Clean temporary files
sudo find /tmp -type f -atime +7 -delete
sudo find /var/tmp -type f -atime +7 -delete

# Clean old core dumps
sudo find / -name "core.*" -mtime +7 -delete

# Clean thumbnail cache
rm -rf ~/.cache/thumbnails/*

# Find and remove empty directories
find /var/log -type d -empty -delete

Log Management DAILY

Essential Log Locations
# System logs
/var/log/syslog          # General system log (Debian/Ubuntu)
/var/log/messages        # General system log (RHEL/CentOS)
/var/log/auth.log        # Authentication log (Debian/Ubuntu)
/var/log/secure          # Authentication log (RHEL/CentOS)
/var/log/kern.log        # Kernel log
/var/log/dmesg           # Boot messages
/var/log/cron            # Cron job log

# Service logs
/var/log/apache2/        # Apache web server
/var/log/nginx/          # Nginx web server
/var/log/mysql/          # MySQL database
/var/log/postgresql/     # PostgreSQL database
/var/log/docker/         # Docker logs

# Application logs
~/.pm2/logs/             # PM2 process manager
/var/log/application/    # Custom application logs
Viewing and Searching Logs
# Follow log in real-time (Ctrl+C to stop)
tail -f /var/log/syslog

# View last 100 lines
tail -n 100 /var/log/syslog

# View first 50 lines
head -n 50 /var/log/syslog

# Search for errors in syslog
grep -i error /var/log/syslog

# Search for errors in all logs (last hour)
grep -i error /var/log/* | grep "$(date +'%b %d %H')"

# Search with context (5 lines before and after)
grep -C 5 "error" /var/log/syslog

# Count errors by type
grep -i error /var/log/syslog | cut -d' ' -f5- | sort | uniq -c | sort -rn

# View logs from specific time range
awk '/Oct 31 08:00/,/Oct 31 09:00/' /var/log/syslog

# Case-insensitive search with line numbers
grep -in "failed" /var/log/auth.log

# Search multiple patterns
grep -E "error|fail|critical" /var/log/syslog
Systemd Journal (journalctl)
# View all logs
sudo journalctl

# Follow logs in real-time
sudo journalctl -f

# Show only errors
sudo journalctl -p err

# Show only errors and critical
sudo journalctl -p err -p crit

# Logs since boot
sudo journalctl -b

# Logs from previous boot
sudo journalctl -b -1

# Logs since specific time
sudo journalctl --since "2025-10-31 08:00:00"
sudo journalctl --since "1 hour ago"
sudo journalctl --since today
sudo journalctl --since yesterday

# Logs until specific time
sudo journalctl --until "2025-10-31 17:00:00"

# Logs for specific service
sudo journalctl -u nginx.service
sudo journalctl -u ssh.service -f

# Logs for specific user
sudo journalctl _UID=1000

# Show kernel messages
sudo journalctl -k

# Output in JSON format
sudo journalctl -o json

# Show disk usage by journal
sudo journalctl --disk-usage

# Clean old journal entries (keep last 3 days)
sudo journalctl --vacuum-time=3d

# Clean old journal entries (keep max 500MB)
sudo journalctl --vacuum-size=500M

# Verify journal integrity
sudo journalctl --verify
Log Analysis Commands
# Count failed SSH login attempts
grep "Failed password" /var/log/auth.log | wc -l

# Show unique IPs with failed SSH attempts
grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn

# Find most common errors
grep -i error /var/log/syslog | awk '{print $5}' | sort | uniq -c | sort -rn | head -10

# Monitor error rate over time
watch -n 5 'grep -c "error" /var/log/syslog'

# Find OOM (Out of Memory) kills
grep -i "killed process" /var/log/syslog

# Find segmentation faults
grep -i "segfault" /var/log/syslog

# Show disk full errors
grep -i "no space left" /var/log/syslog

# Analyze web server access log
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

# Count HTTP status codes
awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn

# Find slow web requests (>5 seconds)
awk '$NF > 5 {print $0}' /var/log/nginx/access.log
✓ Log Management Best Practices
  • Review logs daily for errors, warnings, and security issues
  • Implement log rotation to prevent disk space issues
  • Archive old logs before deleting (compress and move to backup)
  • Set up centralized logging for multiple servers (syslog-ng, ELK stack)
  • Configure log retention policies based on compliance requirements
  • Use log monitoring tools (fail2ban, logwatch, splunk)
  • Create alerts for critical log events
  • Regularly check for unusual patterns or anomalies

User Management AS NEEDED

Creating and Managing Users
# Create new user
sudo useradd -m -s /bin/bash username

# Create user with specific UID
sudo useradd -m -u 1500 -s /bin/bash username

# Set password
sudo passwd username

# Create user with home directory and set password in one go
sudo useradd -m -s /bin/bash username && sudo passwd username

# Delete user (keep home directory)
sudo userdel username

# Delete user and home directory
sudo userdel -r username

# Modify user's shell
sudo usermod -s /bin/zsh username

# Lock user account
sudo usermod -L username

# Unlock user account
sudo usermod -U username

# Change user's home directory
sudo usermod -d /new/home/path username

# Set account expiration date
sudo usermod -e 2025-12-31 username

# Add comment/description to user
sudo usermod -c "John Doe - Developer" username
Group Management
# Create new group
sudo groupadd developers

# Delete group
sudo groupdel developers

# Add user to group
sudo usermod -aG developers username

# Add user to multiple groups
sudo usermod -aG developers,docker,sudo username

# Remove user from group
sudo gpasswd -d username groupname

# View user's groups
groups username

# View all groups
cat /etc/group

# View members of a group
getent group developers

# Change user's primary group
sudo usermod -g newgroup username
Viewing User Information
# List all users
cat /etc/passwd

# List only usernames
cut -d: -f1 /etc/passwd

# View specific user info
id username

# View last login information
lastlog

# View currently logged in users
who
w

# View user's last logins
last username

# View failed login attempts
sudo lastb

# View password aging info
sudo chage -l username

# View sudo access
sudo -l -U username

# Find user's processes
ps aux | grep ^username

File Permissions & Ownership

Basic Permission Management
# View permissions
ls -l filename
ls -lh directory/

# Change file permissions (numeric)
chmod 644 file.txt    # rw-r--r-- (owner: rw, group: r, others: r)
chmod 755 script.sh   # rwxr-xr-x (owner: rwx, group: rx, others: rx)
chmod 600 private.key # rw------- (owner: rw, group: none, others: none)

# Change permissions (symbolic)
chmod u+x script.sh   # Add execute for user
chmod g+w file.txt    # Add write for group
chmod o-r file.txt    # Remove read for others
chmod a+r file.txt    # Add read for all

# Recursive permission change
chmod -R 755 /var/www/html/

# Change ownership
sudo chown username:groupname file.txt

# Change only owner
sudo chown username file.txt

# Change only group
sudo chown :groupname file.txt

# Recursive ownership change
sudo chown -R username:groupname /home/username/

# Change ownership to match another file
sudo chown --reference=file1 file2
Special Permissions
# Set SUID (Set User ID) - file runs as owner
chmod u+s /usr/bin/program
chmod 4755 /usr/bin/program

# Set SGID (Set Group ID) - file runs as group
chmod g+s /shared/directory
chmod 2755 /shared/directory

# Set Sticky Bit - only owner can delete (useful for /tmp)
chmod +t /shared/public/
chmod 1755 /shared/public/

# View special permissions
ls -l filename
# s = SUID/SGID
# t = Sticky bit

# Find files with SUID/SGID
find / -perm /6000 -type f 2>/dev/null

# Find files with Sticky bit
find / -perm /1000 -type d 2>/dev/null
Access Control Lists (ACL)
# View ACL
getfacl filename

# Set ACL for specific user
setfacl -m u:username:rwx filename

# Set ACL for specific group
setfacl -m g:groupname:rx filename

# Set default ACL for directory (inherited by new files)
setfacl -d -m u:username:rwx directory/

# Remove ACL
setfacl -x u:username filename

# Remove all ACLs
setfacl -b filename

# Copy ACL from one file to another
getfacl file1 | setfacl --set-file=- file2

# Recursive ACL
setfacl -R -m u:username:rwx directory/
⚠️ Permission Security Warning
  • Never use 777 permissions (rwxrwxrwx) - huge security risk!
  • Be careful with SUID/SGID - can be security vulnerability
  • Regularly audit SUID/SGID files
  • Use ACLs for granular permission control
  • Always use principle of least privilege

Sudo Management

Configuring Sudo Access
# Edit sudoers file (ALWAYS use visudo!)
sudo visudo

# Add user to sudo group (Ubuntu/Debian)
sudo usermod -aG sudo username

# Add user to wheel group (RHEL/CentOS)
sudo usermod -aG wheel username

# Grant full sudo access to user
# Add to /etc/sudoers:
username ALL=(ALL:ALL) ALL

# Grant sudo without password
username ALL=(ALL) NOPASSWD: ALL

# Grant specific command only
username ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart nginx

# Grant multiple specific commands
username ALL=(ALL) NOPASSWD: /usr/bin/systemctl, /usr/sbin/reboot

# Create sudo configuration file for user
echo "username ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/username

# View user's sudo privileges
sudo -l -U username

# Test sudo access as user
sudo -u username sudo -l
Sudo Logging and Auditing
# View sudo usage log
sudo cat /var/log/auth.log | grep sudo

# View recent sudo commands
sudo journalctl -u sudo

# View specific user's sudo usage
sudo cat /var/log/auth.log | grep "sudo.*username"

# Enable sudo logging (add to /etc/sudoers)
Defaults logfile=/var/log/sudo.log
Defaults log_input, log_output
Defaults iolog_dir=/var/log/sudo-io

# View sudo session recordings
sudo sudoreplay -l

# Replay sudo session
sudo sudoreplay 00/00/01

Service Management DAILY

Systemd Service Management
# View all services
systemctl list-units --type=service

# View only running services
systemctl list-units --type=service --state=running

# View all service status
systemctl status

# Check specific service status
systemctl status nginx
systemctl status apache2
systemctl status mysql

# Start service
sudo systemctl start nginx

# Stop service
sudo systemctl stop nginx

# Restart service
sudo systemctl restart nginx

# Reload service configuration (without restart)
sudo systemctl reload nginx

# Enable service (start at boot)
sudo systemctl enable nginx

# Disable service (don't start at boot)
sudo systemctl disable nginx

# Enable and start service
sudo systemctl enable --now nginx

# Check if service is enabled
systemctl is-enabled nginx

# Check if service is active
systemctl is-active nginx

# View service dependencies
systemctl list-dependencies nginx

# View service configuration file location
systemctl cat nginx

# Edit service configuration
sudo systemctl edit nginx

# Reload systemd after service file changes
sudo systemctl daemon-reload

# View failed services
systemctl --failed
Creating Custom Systemd Service
# Create service file
sudo nano /etc/systemd/system/myapp.service

# Example service file:
[Unit]
Description=My Application Service
After=network.target

[Service]
Type=simple
User=myuser
WorkingDirectory=/opt/myapp
ExecStart=/usr/bin/python3 /opt/myapp/app.py
Restart=always
RestartSec=10
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=myapp

[Install]
WantedBy=multi-user.target

# Reload systemd and start service
sudo systemctl daemon-reload
sudo systemctl enable myapp
sudo systemctl start myapp
sudo systemctl status myapp
Service Troubleshooting
# View service logs
sudo journalctl -u nginx.service

# Follow service logs in real-time
sudo journalctl -u nginx.service -f

# View last 100 lines of service log
sudo journalctl -u nginx.service -n 100

# View service logs since specific time
sudo journalctl -u nginx.service --since "1 hour ago"

# Check service failures
sudo journalctl -u nginx.service -p err

# View service startup script
systemctl cat nginx.service

# Test service configuration
sudo nginx -t  # For nginx
sudo apachectl configtest  # For Apache

# Kill service process (last resort)
sudo killall nginx
sudo pkill -9 nginx

# Find service PID
pidof nginx
pgrep nginx

# Restart all failed services
sudo systemctl reset-failed

Daily Security Checks DAILY

Daily Security Checklist

  • Review authentication logs for failed login attempts
  • Check for suspicious user activity
  • Verify no unauthorized users have been created
  • Review sudo usage logs
  • Check for rootkit infections
  • Verify firewall rules are intact
  • Check for security updates
  • Review listening network services
  • Check for suspicious processes
  • Verify file integrity (critical system files)
Security Monitoring Commands
# Check failed login attempts
sudo grep "Failed password" /var/log/auth.log | tail -20

# List unique IPs with failed attempts
sudo grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn

# Check for unauthorized users
awk -F: '$3 >= 1000 {print $1}' /etc/passwd

# View recently created users
sudo cat /var/log/auth.log | grep "new user"

# Check for new sudo users
grep -E '^sudo|^wheel' /etc/group

# View last logins
last | head -20

# Check for suspicious sudo usage
sudo cat /var/log/auth.log | grep "sudo" | tail -30

# List listening ports
sudo ss -tulpn
sudo netstat -tulpn

# Check for suspicious processes
ps aux | grep -E 'nc|ncat|socat|/tmp'

# Check for SUID files changed recently
sudo find / -perm /4000 -mtime -7 -ls 2>/dev/null

# List cron jobs for all users
sudo cat /etc/crontab
sudo ls -la /etc/cron.*
for user in $(cut -f1 -d: /etc/passwd); do echo "=== $user ==="; sudo crontab -u $user -l 2>/dev/null; done

# Check SSH authorized keys
find /home -name authorized_keys -exec ls -la {} \;
find /root -name authorized_keys -exec ls -la {} \;
Rootkit Detection
# Install rootkit detection tools
sudo apt install rkhunter chkrootkit  # Debian/Ubuntu
sudo yum install rkhunter chkrootkit  # RHEL/CentOS

# Run rootkit hunter
sudo rkhunter --check
sudo rkhunter --update

# Run chkrootkit
sudo chkrootkit

# Check system binaries integrity
sudo debsums -c  # Debian/Ubuntu
rpm -Va  # RHEL/CentOS

# Install and run Lynis security audit
sudo apt install lynis
sudo lynis audit system

Firewall Management

UFW (Uncomplicated Firewall) - Ubuntu/Debian
# Enable UFW
sudo ufw enable

# Disable UFW
sudo ufw disable

# View status
sudo ufw status verbose
sudo ufw status numbered

# Allow service by name
sudo ufw allow ssh
sudo ufw allow http
sudo ufw allow https

# Allow specific port
sudo ufw allow 8080/tcp
sudo ufw allow 53/udp

# Allow from specific IP
sudo ufw allow from 192.168.1.100

# Allow from specific IP to specific port
sudo ufw allow from 192.168.1.100 to any port 22

# Allow from subnet
sudo ufw allow from 192.168.1.0/24

# Deny connection
sudo ufw deny from 192.168.1.50

# Delete rule by number
sudo ufw delete 3

# Delete rule by specification
sudo ufw delete allow 80/tcp

# Reset firewall (delete all rules)
sudo ufw reset

# Set default policies
sudo ufw default deny incoming
sudo ufw default allow outgoing

# Enable logging
sudo ufw logging on
sudo ufw logging medium

# View logs
sudo tail -f /var/log/ufw.log
Firewalld - RHEL/CentOS
# Check firewalld status
sudo firewall-cmd --state

# Start/enable firewalld
sudo systemctl start firewalld
sudo systemctl enable firewalld

# View active zones
sudo firewall-cmd --get-active-zones

# View all zones
sudo firewall-cmd --get-zones

# List all rules
sudo firewall-cmd --list-all

# Add service
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https

# Add port
sudo firewall-cmd --permanent --add-port=8080/tcp
sudo firewall-cmd --permanent --add-port=53/udp

# Remove service
sudo firewall-cmd --permanent --remove-service=http

# Add rich rule
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.1.0/24" port port="22" protocol="tcp" accept'

# Block IP address
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.1.50" reject'

# Reload firewall (apply changes)
sudo firewall-cmd --reload

# View open ports
sudo firewall-cmd --list-ports

# View services
sudo firewall-cmd --list-services
iptables (Traditional Firewall)
# View current rules
sudo iptables -L -n -v

# View rules with line numbers
sudo iptables -L --line-numbers

# Allow SSH
sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT

# Allow HTTP/HTTPS
sudo iptables -A INPUT -p tcp --dport 80 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 443 -j ACCEPT

# Allow from specific IP
sudo iptables -A INPUT -s 192.168.1.100 -j ACCEPT

# Block specific IP
sudo iptables -A INPUT -s 192.168.1.50 -j DROP

# Delete rule by number
sudo iptables -D INPUT 3

# Save rules (Ubuntu/Debian)
sudo iptables-save > /etc/iptables/rules.v4

# Restore rules
sudo iptables-restore < /etc/iptables/rules.v4

# Flush all rules (careful!)
sudo iptables -F

# Set default policies
sudo iptables -P INPUT DROP
sudo iptables -P FORWARD DROP
sudo iptables -P OUTPUT ACCEPT

# Allow established connections
sudo iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow loopback
sudo iptables -A INPUT -i lo -j ACCEPT

System Updates DAILY

Ubuntu/Debian Updates
# Update package list
sudo apt update

# Upgrade all packages
sudo apt upgrade -y

# Full upgrade (may remove packages)
sudo apt full-upgrade -y

# Dist upgrade (for version upgrades)
sudo apt dist-upgrade -y

# Security updates only
sudo apt upgrade -y -o Dpkg::Options::="--force-confold"

# List available updates
apt list --upgradable

# List security updates
apt list --upgradable | grep -i security

# Auto remove unused packages
sudo apt autoremove -y

# Clean package cache
sudo apt clean
sudo apt autoclean

# Check for broken dependencies
sudo apt check

# Fix broken packages
sudo apt --fix-broken install

# Unattended upgrades (automatic security updates)
sudo apt install unattended-upgrades
sudo dpkg-reconfigure --priority=low unattended-upgrades
RHEL/CentOS Updates
# Update all packages
sudo yum update -y

# Update specific package
sudo yum update package-name

# Security updates only
sudo yum update --security

# List available updates
yum list updates

# List security updates
yum list updates --security

# Check for updates without installing
yum check-update

# Clean cache
sudo yum clean all

# Auto remove unused packages
sudo yum autoremove

# Update with DNF (newer systems)
sudo dnf update -y
sudo dnf upgrade -y

# Enable automatic updates
sudo yum install yum-cron
sudo systemctl enable yum-cron
sudo systemctl start yum-cron
✓ Update Management Best Practices
  • Always test updates in staging environment first
  • Take backup/snapshot before major updates
  • Schedule updates during maintenance windows
  • Enable automatic security updates for production servers
  • Monitor for failed updates and dependency issues
  • Keep kernel updated but plan reboots carefully
  • Document all updates in change management system
  • Subscribe to security mailing lists for your distribution

Network Configuration

Network Interface Information
# View all network interfaces
ip addr show
ip a

# View specific interface
ip addr show eth0

# View interface statistics
ip -s link show eth0

# Traditional ifconfig (deprecated but still useful)
ifconfig
ifconfig eth0

# View routing table
ip route show
route -n

# View ARP table
ip neigh show
arp -n

# View DNS configuration
cat /etc/resolv.conf

# View hostname
hostname
hostnamectl

# Set hostname
sudo hostnamectl set-hostname newname
Network Configuration Files
# Ubuntu/Debian - Netplan (newer versions)
# Config file: /etc/netplan/*.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: true
    eth1:
      addresses:
        - 192.168.1.10/24
      gateway4: 192.168.1.1
      nameservers:
        addresses: [8.8.8.8, 8.8.4.4]

# Apply netplan configuration
sudo netplan apply
sudo netplan --debug apply

# Ubuntu/Debian - /etc/network/interfaces (older)
auto eth0
iface eth0 inet static
    address 192.168.1.10
    netmask 255.255.255.0
    gateway 192.168.1.1
    dns-nameservers 8.8.8.8 8.8.4.4

# RHEL/CentOS - /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
IPADDR=192.168.1.10
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
DNS1=8.8.8.8
DNS2=8.8.4.4

# Restart networking
sudo systemctl restart networking  # Debian/Ubuntu
sudo systemctl restart network     # RHEL/CentOS
sudo systemctl restart NetworkManager
IP Address Management
# Add IP address to interface
sudo ip addr add 192.168.1.10/24 dev eth0

# Remove IP address
sudo ip addr del 192.168.1.10/24 dev eth0

# Bring interface up
sudo ip link set eth0 up

# Bring interface down
sudo ip link set eth0 down

# Add default gateway
sudo ip route add default via 192.168.1.1

# Add static route
sudo ip route add 10.0.0.0/8 via 192.168.1.254

# Delete route
sudo ip route del 10.0.0.0/8

# Flush IP addresses
sudo ip addr flush dev eth0

# Change MAC address
sudo ip link set eth0 address aa:bb:cc:dd:ee:ff

Network Troubleshooting

Connectivity Testing
# Ping test
ping -c 4 8.8.8.8
ping -c 100 google.com

# Test with specific packet size
ping -s 1472 -c 5 8.8.8.8

# Flood ping (careful!)
sudo ping -f 8.8.8.8

# Traceroute
traceroute google.com
traceroute -n 8.8.8.8  # Don't resolve hostnames

# MTR (better than traceroute)
mtr google.com
mtr -n -c 100 8.8.8.8

# TCP traceroute
sudo tcptraceroute google.com 443

# Test DNS resolution
nslookup google.com
nslookup google.com 8.8.8.8

# Dig (detailed DNS)
dig google.com
dig @8.8.8.8 google.com
dig google.com +short

# Host command
host google.com
Port and Service Testing
# Test TCP port connectivity
nc -zv 192.168.1.100 22
nc -zv google.com 443

# Test UDP port
nc -zvu 192.168.1.100 53

# Telnet to port
telnet 192.168.1.100 80

# Test HTTP/HTTPS
curl -I http://example.com
curl -I https://example.com
wget --spider http://example.com

# Show listening ports
sudo ss -tulpn
sudo netstat -tulpn

# Show established connections
ss -tn
netstat -tn

# Show all connections
ss -a
netstat -a

# Show connections for specific process
sudo ss -tp | grep nginx
sudo netstat -anp | grep :80

# Find which process is using a port
sudo lsof -i :80
sudo fuser 80/tcp
Packet Capture and Analysis
# Capture packets on interface
sudo tcpdump -i eth0

# Capture with more details
sudo tcpdump -i eth0 -vv

# Capture specific host
sudo tcpdump -i eth0 host 192.168.1.100

# Capture specific port
sudo tcpdump -i eth0 port 80

# Capture and save to file
sudo tcpdump -i eth0 -w capture.pcap

# Read capture file
tcpdump -r capture.pcap

# Capture HTTP traffic
sudo tcpdump -i eth0 -A 'tcp port 80'

# Capture SSH traffic
sudo tcpdump -i eth0 port 22

# Capture DNS queries
sudo tcpdump -i eth0 port 53

# Capture with filters
sudo tcpdump -i eth0 'tcp[tcpflags] & (tcp-syn) != 0'  # SYN packets
sudo tcpdump -i eth0 icmp  # ICMP only
sudo tcpdump -i eth0 not port 22  # Exclude SSH

# Limit capture size
sudo tcpdump -i eth0 -c 100 -w capture.pcap  # 100 packets
sudo tcpdump -i eth0 -G 60 -w capture.pcap   # 60 seconds

# Monitor bandwidth usage
sudo iftop -i eth0
sudo nethogs eth0
Common Network Issues
# Check if interface is up
ip link show eth0

# Check if IP is assigned
ip addr show eth0

# Check default gateway
ip route | grep default

# Verify DNS
cat /etc/resolv.conf
systemd-resolve --status

# Test local network connectivity
ping -c 4 192.168.1.1  # Gateway

# Test internet connectivity
ping -c 4 8.8.8.8  # Google DNS
ping -c 4 google.com  # Test DNS resolution

# Check for IP conflicts
sudo arping -I eth0 192.168.1.10

# View network errors/drops
ip -s link show eth0
ethtool -S eth0

# Check MTU
ip link show eth0 | grep mtu

# Test with different MTU
sudo ip link set eth0 mtu 1400

# Restart networking
sudo systemctl restart networking
sudo systemctl restart NetworkManager

# Flush DNS cache
sudo systemd-resolve --flush-caches
sudo systemctl restart systemd-resolved

Performance Monitoring & Analysis

System Performance Overview
# Quick performance check
vmstat 1 5

# CPU and memory summary
free -h && uptime

# I/O statistics
iostat -x 1 5

# Full system monitoring
dstat

# All-in-one monitoring
glances

# System activity reporter
sar -u 1 5  # CPU
sar -r 1 5  # Memory
sar -d 1 5  # Disk

# Performance Co-Pilot
pcp atop
Finding Performance Bottlenecks
# High CPU usage processes
ps aux --sort=-%cpu | head -10

# High memory usage processes
ps aux --sort=-%mem | head -10

# High I/O processes
sudo iotop -o

# High network usage processes
sudo nethogs

# Trace system calls (slow process)
sudo strace -p PID

# Profile CPU usage
sudo perf top

# Find what's causing high load
uptime
top -b -n 1 | head -20

# Check for zombie processes
ps aux | grep -w Z

# Find processes in uninterruptible sleep (D state)
ps aux | awk '$8=="D" {print $0}'

# Check context switches
vmstat 1 5

# Monitor interrupts
watch -n 1 'cat /proc/interrupts'

Backup Management DAILY

Creating Backups with tar
# Basic tar backup
tar -czf backup-$(date +%Y%m%d).tar.gz /path/to/backup

# Backup with exclusions
tar -czf backup.tar.gz --exclude='/path/to/exclude' /path/to/backup

# Backup multiple directories
tar -czf backup.tar.gz /etc /var/www /home

# Backup with verbose output
tar -czvf backup.tar.gz /path/to/backup

# Extract backup
tar -xzf backup.tar.gz

# Extract to specific directory
tar -xzf backup.tar.gz -C /restore/path/

# List backup contents
tar -tzf backup.tar.gz

# Incremental backup using timestamp
tar -czf backup-$(date +%Y%m%d-%H%M%S).tar.gz -N "2025-10-30" /path/to/backup
Rsync Backups
# Basic rsync backup
rsync -av /source/ /destination/

# Rsync to remote server
rsync -avz /source/ user@remote:/destination/

# Rsync with progress and delete
rsync -avz --progress --delete /source/ /destination/

# Exclude files
rsync -av --exclude='*.log' --exclude='cache/*' /source/ /destination/

# Dry run (test without changes)
rsync -av --dry-run /source/ /destination/

# Backup with bandwidth limit (KB/s)
rsync -av --bwlimit=1000 /source/ /destination/

# Incremental backup
rsync -av --link-dest=/previous/backup /source/ /current/backup/

# Backup only files changed in last 24 hours
rsync -av --update /source/ /destination/

Complete Backup Script

#!/bin/bash
# Comprehensive backup script
# Save as: /usr/local/bin/backup.sh

# Configuration
BACKUP_DIR="/backup"
BACKUP_NAME="server-backup-$(date +%Y%m%d-%H%M%S)"
LOG_FILE="/var/log/backup.log"
RETENTION_DAYS=30
REMOTE_HOST="backup-server"
REMOTE_USER="backup"
REMOTE_DIR="/backups"

# Directories to backup
BACKUP_PATHS=(
    "/etc"
    "/var/www"
    "/home"
    "/opt"
)

# Start logging
exec 1> >(tee -a "$LOG_FILE")
exec 2>&1

echo "=========================================="
echo "Backup started at $(date)"
echo "=========================================="

# Create backup directory
mkdir -p "$BACKUP_DIR"

# Create tar archive
echo "Creating backup archive..."
tar -czf "$BACKUP_DIR/$BACKUP_NAME.tar.gz" \
    --exclude='*.log' \
    --exclude='cache/*' \
    --exclude='tmp/*' \
    "${BACKUP_PATHS[@]}"

if [ $? -eq 0 ]; then
    echo "✓ Backup archive created successfully"
    
    # Calculate checksum
    cd "$BACKUP_DIR"
    sha256sum "$BACKUP_NAME.tar.gz" > "$BACKUP_NAME.sha256"
    
    # Copy to remote server
    echo "Copying backup to remote server..."
    scp "$BACKUP_NAME.tar.gz" "$BACKUP_NAME.sha256" \
        "$REMOTE_USER@$REMOTE_HOST:$REMOTE_DIR/"
    
    if [ $? -eq 0 ]; then
        echo "✓ Backup copied to remote server"
    else
        echo "✗ Failed to copy backup to remote server"
    fi
    
    # Clean old backups
    echo "Cleaning old backups..."
    find "$BACKUP_DIR" -name "server-backup-*.tar.gz" -mtime +$RETENTION_DAYS -delete
    find "$BACKUP_DIR" -name "server-backup-*.sha256" -mtime +$RETENTION_DAYS -delete
    
    echo "✓ Old backups cleaned"
else
    echo "✗ Backup failed!"
    exit 1
fi

echo "=========================================="
echo "Backup completed at $(date)"
echo "Backup size: $(du -h "$BACKUP_DIR/$BACKUP_NAME.tar.gz" | cut -f1)"
echo "=========================================="

# Optional: Send email notification
# echo "Backup completed successfully" | mail -s "Backup Success - $(hostname)" admin@example.com
Database Backups
# MySQL/MariaDB backup
mysqldump -u root -p database_name > backup.sql
mysqldump -u root -p --all-databases > all-databases.sql

# MySQL with compression
mysqldump -u root -p database_name | gzip > backup.sql.gz

# PostgreSQL backup
pg_dump -U postgres database_name > backup.sql
pg_dumpall -U postgres > all-databases.sql

# MongoDB backup
mongodump --out /backup/mongodb/

# Restore MySQL
mysql -u root -p database_name < backup.sql

# Restore PostgreSQL
psql -U postgres database_name < backup.sql

# Restore MongoDB
mongorestore /backup/mongodb/
✓ Backup Best Practices
  • Follow 3-2-1 rule: 3 copies, 2 different media, 1 offsite
  • Test restores regularly - backups are useless if you can't restore
  • Automate backups with cron jobs
  • Encrypt sensitive backups before storing
  • Monitor backup success/failure with alerts
  • Document backup and restore procedures
  • Keep backup retention policy based on requirements
  • Verify backup integrity with checksums

Weekly Maintenance Tasks WEEKLY

Weekly Maintenance Checklist

  • Review weekly system performance reports
  • Analyze disk space trends and plan for growth
  • Review and rotate logs if necessary
  • Update documentation for any changes made
  • Test backup restore procedures (random sample)
  • Review and update firewall rules if needed
  • Check for and install non-critical updates
  • Review user access and remove inactive accounts
  • Verify SSL certificate expiration dates
  • Review monitoring alerts and false positives
  • Clean up old temporary files and caches
  • Review cron jobs and scheduled tasks

Monthly Maintenance Tasks MONTHLY

Monthly Maintenance Checklist

  • Full system audit and security review
  • Review and update disaster recovery plan
  • Performance capacity planning review
  • Update system documentation
  • Review and optimize database performance
  • Test disaster recovery procedures
  • Review SSL certificates for upcoming expirations (90 days)
  • Audit user permissions and sudo access
  • Review backup retention and storage costs
  • Update emergency contact list
  • Review and update runbooks
  • Schedule and plan kernel updates/reboots

Automation Scripts

Disk Space Alert Script

#!/bin/bash
# Alert when disk usage exceeds threshold
# Save as: /usr/local/bin/disk-alert.sh

THRESHOLD=90
EMAIL="admin@example.com"

df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' | while read output;
do
    usage=$(echo $output | awk '{ print $1}' | cut -d'%' -f1)
    partition=$(echo $output | awk '{ print $2 }')
    
    if [ $usage -ge $THRESHOLD ]; then
        echo "ALERT: Disk usage on $partition is at ${usage}%" | \
        mail -s "Disk Space Alert on $(hostname)" $EMAIL
    fi
done

Service Monitoring Script

#!/bin/bash
# Monitor critical services and restart if down
# Save as: /usr/local/bin/service-monitor.sh

SERVICES=("nginx" "mysql" "ssh")
LOG="/var/log/service-monitor.log"

for service in "${SERVICES[@]}"; do
    if ! systemctl is-active --quiet $service; then
        echo "$(date): $service is down, attempting restart" >> $LOG
        systemctl start $service
        
        # Wait and check again
        sleep 5
        if systemctl is-active --quiet $service; then
            echo "$(date): $service restarted successfully" >> $LOG
        else
            echo "$(date): FAILED to restart $service" >> $LOG
            echo "$service restart failed on $(hostname)" | \
            mail -s "CRITICAL: Service Restart Failed" admin@example.com
        fi
    fi
done

Log Cleanup Script

#!/bin/bash
# Clean up old logs and compress recent ones
# Save as: /usr/local/bin/log-cleanup.sh

LOG_DIR="/var/log"
DAYS_TO_KEEP=30
DAYS_TO_COMPRESS=7

# Delete old log files
find $LOG_DIR -name "*.log.*" -mtime +$DAYS_TO_KEEP -delete
find $LOG_DIR -name "*.gz" -mtime +$DAYS_TO_KEEP -delete

# Compress recent logs
find $LOG_DIR -name "*.log.*" -mtime +$DAYS_TO_COMPRESS ! -name "*.gz" -exec gzip {} \;

# Clean journal logs (keep 7 days)
journalctl --vacuum-time=7d

echo "$(date): Log cleanup completed" >> /var/log/log-cleanup.log
Setting Up Cron Jobs
# Edit crontab
crontab -e

# View current crontab
crontab -l

# Example cron entries:

# Daily backup at 2 AM
0 2 * * * /usr/local/bin/backup.sh

# Hourly service check
0 * * * * /usr/local/bin/service-monitor.sh

# Daily disk space alert at 8 AM
0 8 * * * /usr/local/bin/disk-alert.sh

# Weekly log cleanup (Sunday at 3 AM)
0 3 * * 0 /usr/local/bin/log-cleanup.sh

# Update and reboot monthly (first Sunday at 4 AM)
0 4 1-7 * 0 /usr/local/bin/monthly-update.sh

# System health check every 5 minutes
*/5 * * * * /usr/local/bin/health-check.sh

# Cron format:
# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of the month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday)
# │ │ │ │ │
# * * * * * command to execute

Emergency Troubleshooting Guide

Server is Slow/High Load

Quick Diagnostic Steps
# 1. Check load average
uptime
# If load > number of CPU cores, system is overloaded

# 2. Find CPU hogs
ps aux --sort=-%cpu | head -10

# 3. Find memory hogs
ps aux --sort=-%mem | head -10

# 4. Check disk I/O
iostat -x 1 5

# 5. Check for high I/O processes
sudo iotop -o

# 6. Look for network issues
netstat -s | grep -i error
ip -s link

# 7. Check for zombie processes
ps aux | grep -w Z

# 8. Review recent logs
journalctl -p err -n 50

# 9. Check disk space
df -h

# 10. Look for runaway processes
top -b -n 1 | head -20

Server Out of Disk Space

Quick Space Recovery
# Find what's using space
du -h / --max-depth=1 | sort -hr | head -10

# Find large files
find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null | head -20

# Clear common space hogs
sudo apt clean  # Package cache
sudo journalctl --vacuum-size=100M  # Journal logs
docker system prune -af  # Docker (if installed)

# Remove old log files
sudo find /var/log -name "*.gz" -mtime +30 -delete

# Find and remove core dumps
sudo find / -name "core.*" -delete

# Clear user cache
rm -rf ~/.cache/*

# Check for deleted files still open
lsof | grep deleted | grep -v "/tmp"

Cannot SSH to Server

SSH Troubleshooting
# From another terminal/console:

# 1. Check if SSH service is running
systemctl status sshd

# 2. Check if SSH port is listening
ss -tulpn | grep :22

# 3. Check firewall
ufw status
iptables -L -n | grep 22

# 4. Check SSH logs
tail -50 /var/log/auth.log

# 5. Test SSH locally
ssh localhost

# 6. Check SSH configuration
sshd -t

# 7. Restart SSH
systemctl restart sshd

# 8. Check for failed login attempts (might be blocked)
grep "Failed" /var/log/auth.log | tail -20

# 9. Check fail2ban (if installed)
fail2ban-client status sshd

# 10. Unban IP if needed
fail2ban-client set sshd unbanip 192.168.1.100

Service Won't Start

Service Troubleshooting
# Check service status
systemctl status service-name

# View detailed logs
journalctl -u service-name -n 100

# Check service configuration
systemctl cat service-name

# Test configuration (if applicable)
nginx -t  # For Nginx
apachectl configtest  # For Apache

# Check dependencies
systemctl list-dependencies service-name

# Check for port conflicts
ss -tulpn | grep :80

# Check file permissions
ls -la /path/to/service/files

# Manually try to start service
/usr/bin/service-binary --config /path/to/config

# Reset failed state
systemctl reset-failed service-name

# Force kill and restart
pkill -9 service-name
systemctl start service-name

Disaster Recovery

⚠️ Emergency Contact Information

Always maintain an up-to-date contact list including:

  • Team lead and backup contacts
  • On-call engineer schedules
  • Vendor support contacts
  • Management escalation path
  • Datacenter/hosting provider support

Disaster Recovery Procedures

  • Assess the situation - what's down, what's working
  • Notify stakeholders and management
  • Activate incident response team
  • Document all actions taken with timestamps
  • Attempt immediate recovery from backups if needed
  • If corruption, preserve evidence before fixing
  • Follow documented DR procedures
  • Keep communication channels open
  • Once recovered, perform root cause analysis
  • Update DR documentation based on lessons learned
System Recovery Mode
# Boot into recovery mode (from GRUB):
# 1. Select "Advanced options"
# 2. Select "recovery mode"
# 3. Select "root - Drop to root shell prompt"

# Once in recovery mode:

# Remount filesystem as read-write
mount -o remount,rw /

# Check and repair filesystem
fsck -y /dev/sda1

# Restore from backup
tar -xzf /backup/latest.tar.gz -C /

# Reset root password if needed
passwd root

# Fix bootloader
grub-install /dev/sda
update-grub

# Reboot
reboot
✓ Incident Response Best Practices
  • Stay calm and methodical
  • Document everything you do
  • Don't make changes without understanding impact
  • Have rollback plan before making changes
  • Communicate proactively with stakeholders
  • Follow change control procedures even in emergencies
  • Preserve evidence if security incident
  • Conduct post-mortem after resolution

Quick Reference - Essential Commands

Category Command Description
System Info uname -a Kernel and system info
hostnamectl Hostname and OS info
uptime System uptime and load
Monitoring htop Interactive process viewer
free -h Memory usage
df -h Disk space usage
iostat -x 1 I/O statistics
Services systemctl status service Check service status
systemctl restart service Restart service
journalctl -u service -f Follow service logs
Network ip addr show Show IP addresses
ss -tulpn Show listening ports
ping -c 4 host Test connectivity
Logs tail -f /var/log/syslog Follow system log
journalctl -p err Show errors
grep -i error /var/log/* Search logs for errors
Files find / -name file Find file
du -sh directory Directory size
chmod 755 file Change permissions
Security sudo ufw status Check firewall
last Show login history
who Show logged in users

Remember: A Good Sysadmin is a Lazy Sysadmin

Automate everything you can.
Document everything you do.
Test everything before production.
Monitor everything that matters.
Backup everything important.

Linux Server Management Guide
Created: October 31, 2025
For System Administrators

Stay Calm and sudo !! 🐧

← Back to ServerMgmt Index ↑ Back to EXPANDED