📑 Table of Contents
Daily Operations
User Management
Performance & Monitoring
Services & Applications
Maintenance
Emergency Response
Morning Health Checks DAILY
Start each day with these essential checks to ensure your servers are healthy and running optimally.
Daily Morning Checklist
- Check system uptime and recent reboots
- Review disk space on all partitions
- Verify backup completion status
- Check for security updates
- Review system logs for errors/warnings
- Monitor CPU and memory usage
- Check service status (web, database, etc.)
- Review failed login attempts
- Verify network connectivity
- Check monitoring alerts
Quick Health Check Script
#!/bin/bash
# Save as: /usr/local/bin/morning-check.sh
echo "=========================================="
echo "SERVER HEALTH CHECK - $(date)"
echo "=========================================="
echo ""
# System Uptime
echo "=== UPTIME ==="
uptime
echo ""
# Disk Usage
echo "=== DISK USAGE ==="
df -h | grep -E '^/dev/'
echo ""
# Memory Usage
echo "=== MEMORY USAGE ==="
free -h
echo ""
# Load Average
echo "=== LOAD AVERAGE ==="
cat /proc/loadavg
echo ""
# Top 5 CPU Processes
echo "=== TOP 5 CPU PROCESSES ==="
ps aux --sort=-%cpu | head -6
echo ""
# Top 5 Memory Processes
echo "=== TOP 5 MEMORY PROCESSES ==="
ps aux --sort=-%mem | head -6
echo ""
# Failed Login Attempts (last 24 hours)
echo "=== FAILED LOGIN ATTEMPTS (Last 24h) ==="
grep "Failed password" /var/log/auth.log | tail -10
echo ""
# Service Status
echo "=== CRITICAL SERVICES ==="
for service in sshd nginx apache2 mysql postgresql docker; do
if systemctl list-unit-files | grep -q "^${service}.service"; then
systemctl is-active --quiet $service && echo "✓ $service: running" || echo "✗ $service: STOPPED"
fi
done
echo ""
# Last 10 System Errors
echo "=== RECENT SYSTEM ERRORS ==="
journalctl -p err -n 10 --no-pager
echo ""
# Check for Security Updates
echo "=== SECURITY UPDATES ==="
if command -v apt &> /dev/null; then
apt list --upgradable 2>/dev/null | grep -i security | head -5
elif command -v yum &> /dev/null; then
yum list updates --security | head -10
fi
echo ""
echo "=========================================="
echo "Health check completed at $(date)"
echo "=========================================="
# Make executable sudo chmod +x /usr/local/bin/morning-check.sh # Run the check sudo /usr/local/bin/morning-check.sh # Or add to cron for automated daily email 0 8 * * * /usr/local/bin/morning-check.sh | mail -s "Daily Server Health Report" admin@example.com
System Monitoring DAILY
Real-Time Monitoring Commands
# View current load average and uptime uptime # Continuous monitoring (updates every 2 seconds) watch -n 2 uptime # Detailed load info cat /proc/loadavg # Output: 0.52 0.58 0.59 1/395 12345 # Meaning: 1min, 5min, 15min load averages, running/total processes, last PID
# Interactive process viewer (better than top) htop # Traditional top top # Top commands within top: # P - Sort by CPU usage # M - Sort by memory usage # k - Kill a process # q - Quit # Show processes by CPU usage ps aux --sort=-%cpu | head -20 # Show processes by memory usage ps aux --sort=-%mem | head -20 # Monitor specific user's processes ps aux | grep username # Show process tree pstree -p # Find process by name pgrep -a nginx # Detailed process information ps -ef | grep process_name
# CPU information
lscpu
# Per-CPU statistics
mpstat -P ALL 1 5
# I/O wait time (high % = disk bottleneck)
iostat -x 1 5
# Real-time CPU usage per process
top -b -n 1 | head -20
# Check CPU temperature (if sensors available)
sensors
# CPU usage percentage
grep 'cpu ' /proc/stat | awk '{usage=($2+$4)*100/($2+$4+$5)} END {print usage "%"}'
# Memory usage summary
free -h
# Detailed memory info
cat /proc/meminfo
# Memory usage by process
ps aux --sort=-%mem | awk '{printf "%-10s %-10s %-10s %s\n", $2, $3, $4, $11}' | head -20
# Check for memory leaks
watch -n 1 'free -h'
# Show swap usage
swapon --show
# Memory usage breakdown
vmstat 1 5
# Page cache usage
sync; echo 3 > /proc/sys/vm/drop_caches # Clear cache (use carefully!)
Disk Management DAILY
# Disk usage by filesystem
df -h
# Human-readable with inodes
df -ih
# Show specific filesystem type
df -h -t ext4
# Disk usage sorted by size
df -h | sort -k 5 -r
# Monitor disk usage in real-time
watch -n 5 'df -h'
# Alert if disk usage > 90%
df -h | awk '0+$5 >= 90 {print $0}' | mail -s "ALERT: Disk Usage Critical" admin@example.com
# Find largest directories in current location
du -h --max-depth=1 | sort -hr | head -20
# Find largest files in /var
find /var -type f -exec du -h {} + | sort -hr | head -20
# Find files larger than 100MB
find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null
# Find files larger than 1GB
find / -type f -size +1G -exec ls -lh {} \; 2>/dev/null
# Disk usage of specific directory
du -sh /var/log/*
# Disk usage with depth limit
du -h -d 2 /var | sort -hr | head -20
# Find and list large log files
find /var/log -type f -size +50M -exec ls -lh {} \;
# Check what's using deleted files (still using disk space)
lsof | grep deleted
# I/O statistics iostat -x 1 5 # Disk I/O by process (requires iotop) sudo iotop # Simple I/O stats vmstat 1 5 # Detailed disk stats cat /proc/diskstats # Monitor specific disk iostat -x sda 1 5 # Check disk health (SMART) sudo smartctl -a /dev/sda # Test disk speed sudo hdparm -tT /dev/sda
# Clean package cache (Ubuntu/Debian) sudo apt clean sudo apt autoclean sudo apt autoremove # Clean package cache (RHEL/CentOS) sudo yum clean all # Clean journal logs (keep last 3 days) sudo journalctl --vacuum-time=3d # Clean old log files sudo find /var/log -type f -name "*.gz" -mtime +30 -delete sudo find /var/log -type f -name "*.log.*" -mtime +30 -delete # Clean temporary files sudo find /tmp -type f -atime +7 -delete sudo find /var/tmp -type f -atime +7 -delete # Clean old core dumps sudo find / -name "core.*" -mtime +7 -delete # Clean thumbnail cache rm -rf ~/.cache/thumbnails/* # Find and remove empty directories find /var/log -type d -empty -delete
Log Management DAILY
# System logs /var/log/syslog # General system log (Debian/Ubuntu) /var/log/messages # General system log (RHEL/CentOS) /var/log/auth.log # Authentication log (Debian/Ubuntu) /var/log/secure # Authentication log (RHEL/CentOS) /var/log/kern.log # Kernel log /var/log/dmesg # Boot messages /var/log/cron # Cron job log # Service logs /var/log/apache2/ # Apache web server /var/log/nginx/ # Nginx web server /var/log/mysql/ # MySQL database /var/log/postgresql/ # PostgreSQL database /var/log/docker/ # Docker logs # Application logs ~/.pm2/logs/ # PM2 process manager /var/log/application/ # Custom application logs
# Follow log in real-time (Ctrl+C to stop) tail -f /var/log/syslog # View last 100 lines tail -n 100 /var/log/syslog # View first 50 lines head -n 50 /var/log/syslog # Search for errors in syslog grep -i error /var/log/syslog # Search for errors in all logs (last hour) grep -i error /var/log/* | grep "$(date +'%b %d %H')" # Search with context (5 lines before and after) grep -C 5 "error" /var/log/syslog # Count errors by type grep -i error /var/log/syslog | cut -d' ' -f5- | sort | uniq -c | sort -rn # View logs from specific time range awk '/Oct 31 08:00/,/Oct 31 09:00/' /var/log/syslog # Case-insensitive search with line numbers grep -in "failed" /var/log/auth.log # Search multiple patterns grep -E "error|fail|critical" /var/log/syslog
# View all logs sudo journalctl # Follow logs in real-time sudo journalctl -f # Show only errors sudo journalctl -p err # Show only errors and critical sudo journalctl -p err -p crit # Logs since boot sudo journalctl -b # Logs from previous boot sudo journalctl -b -1 # Logs since specific time sudo journalctl --since "2025-10-31 08:00:00" sudo journalctl --since "1 hour ago" sudo journalctl --since today sudo journalctl --since yesterday # Logs until specific time sudo journalctl --until "2025-10-31 17:00:00" # Logs for specific service sudo journalctl -u nginx.service sudo journalctl -u ssh.service -f # Logs for specific user sudo journalctl _UID=1000 # Show kernel messages sudo journalctl -k # Output in JSON format sudo journalctl -o json # Show disk usage by journal sudo journalctl --disk-usage # Clean old journal entries (keep last 3 days) sudo journalctl --vacuum-time=3d # Clean old journal entries (keep max 500MB) sudo journalctl --vacuum-size=500M # Verify journal integrity sudo journalctl --verify
# Count failed SSH login attempts
grep "Failed password" /var/log/auth.log | wc -l
# Show unique IPs with failed SSH attempts
grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn
# Find most common errors
grep -i error /var/log/syslog | awk '{print $5}' | sort | uniq -c | sort -rn | head -10
# Monitor error rate over time
watch -n 5 'grep -c "error" /var/log/syslog'
# Find OOM (Out of Memory) kills
grep -i "killed process" /var/log/syslog
# Find segmentation faults
grep -i "segfault" /var/log/syslog
# Show disk full errors
grep -i "no space left" /var/log/syslog
# Analyze web server access log
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20
# Count HTTP status codes
awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn
# Find slow web requests (>5 seconds)
awk '$NF > 5 {print $0}' /var/log/nginx/access.log
- Review logs daily for errors, warnings, and security issues
- Implement log rotation to prevent disk space issues
- Archive old logs before deleting (compress and move to backup)
- Set up centralized logging for multiple servers (syslog-ng, ELK stack)
- Configure log retention policies based on compliance requirements
- Use log monitoring tools (fail2ban, logwatch, splunk)
- Create alerts for critical log events
- Regularly check for unusual patterns or anomalies
User Management AS NEEDED
# Create new user sudo useradd -m -s /bin/bash username # Create user with specific UID sudo useradd -m -u 1500 -s /bin/bash username # Set password sudo passwd username # Create user with home directory and set password in one go sudo useradd -m -s /bin/bash username && sudo passwd username # Delete user (keep home directory) sudo userdel username # Delete user and home directory sudo userdel -r username # Modify user's shell sudo usermod -s /bin/zsh username # Lock user account sudo usermod -L username # Unlock user account sudo usermod -U username # Change user's home directory sudo usermod -d /new/home/path username # Set account expiration date sudo usermod -e 2025-12-31 username # Add comment/description to user sudo usermod -c "John Doe - Developer" username
# Create new group sudo groupadd developers # Delete group sudo groupdel developers # Add user to group sudo usermod -aG developers username # Add user to multiple groups sudo usermod -aG developers,docker,sudo username # Remove user from group sudo gpasswd -d username groupname # View user's groups groups username # View all groups cat /etc/group # View members of a group getent group developers # Change user's primary group sudo usermod -g newgroup username
# List all users cat /etc/passwd # List only usernames cut -d: -f1 /etc/passwd # View specific user info id username # View last login information lastlog # View currently logged in users who w # View user's last logins last username # View failed login attempts sudo lastb # View password aging info sudo chage -l username # View sudo access sudo -l -U username # Find user's processes ps aux | grep ^username
File Permissions & Ownership
# View permissions ls -l filename ls -lh directory/ # Change file permissions (numeric) chmod 644 file.txt # rw-r--r-- (owner: rw, group: r, others: r) chmod 755 script.sh # rwxr-xr-x (owner: rwx, group: rx, others: rx) chmod 600 private.key # rw------- (owner: rw, group: none, others: none) # Change permissions (symbolic) chmod u+x script.sh # Add execute for user chmod g+w file.txt # Add write for group chmod o-r file.txt # Remove read for others chmod a+r file.txt # Add read for all # Recursive permission change chmod -R 755 /var/www/html/ # Change ownership sudo chown username:groupname file.txt # Change only owner sudo chown username file.txt # Change only group sudo chown :groupname file.txt # Recursive ownership change sudo chown -R username:groupname /home/username/ # Change ownership to match another file sudo chown --reference=file1 file2
# Set SUID (Set User ID) - file runs as owner chmod u+s /usr/bin/program chmod 4755 /usr/bin/program # Set SGID (Set Group ID) - file runs as group chmod g+s /shared/directory chmod 2755 /shared/directory # Set Sticky Bit - only owner can delete (useful for /tmp) chmod +t /shared/public/ chmod 1755 /shared/public/ # View special permissions ls -l filename # s = SUID/SGID # t = Sticky bit # Find files with SUID/SGID find / -perm /6000 -type f 2>/dev/null # Find files with Sticky bit find / -perm /1000 -type d 2>/dev/null
# View ACL getfacl filename # Set ACL for specific user setfacl -m u:username:rwx filename # Set ACL for specific group setfacl -m g:groupname:rx filename # Set default ACL for directory (inherited by new files) setfacl -d -m u:username:rwx directory/ # Remove ACL setfacl -x u:username filename # Remove all ACLs setfacl -b filename # Copy ACL from one file to another getfacl file1 | setfacl --set-file=- file2 # Recursive ACL setfacl -R -m u:username:rwx directory/
- Never use 777 permissions (rwxrwxrwx) - huge security risk!
- Be careful with SUID/SGID - can be security vulnerability
- Regularly audit SUID/SGID files
- Use ACLs for granular permission control
- Always use principle of least privilege
Sudo Management
# Edit sudoers file (ALWAYS use visudo!) sudo visudo # Add user to sudo group (Ubuntu/Debian) sudo usermod -aG sudo username # Add user to wheel group (RHEL/CentOS) sudo usermod -aG wheel username # Grant full sudo access to user # Add to /etc/sudoers: username ALL=(ALL:ALL) ALL # Grant sudo without password username ALL=(ALL) NOPASSWD: ALL # Grant specific command only username ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart nginx # Grant multiple specific commands username ALL=(ALL) NOPASSWD: /usr/bin/systemctl, /usr/sbin/reboot # Create sudo configuration file for user echo "username ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/username # View user's sudo privileges sudo -l -U username # Test sudo access as user sudo -u username sudo -l
# View sudo usage log sudo cat /var/log/auth.log | grep sudo # View recent sudo commands sudo journalctl -u sudo # View specific user's sudo usage sudo cat /var/log/auth.log | grep "sudo.*username" # Enable sudo logging (add to /etc/sudoers) Defaults logfile=/var/log/sudo.log Defaults log_input, log_output Defaults iolog_dir=/var/log/sudo-io # View sudo session recordings sudo sudoreplay -l # Replay sudo session sudo sudoreplay 00/00/01
Service Management DAILY
# View all services systemctl list-units --type=service # View only running services systemctl list-units --type=service --state=running # View all service status systemctl status # Check specific service status systemctl status nginx systemctl status apache2 systemctl status mysql # Start service sudo systemctl start nginx # Stop service sudo systemctl stop nginx # Restart service sudo systemctl restart nginx # Reload service configuration (without restart) sudo systemctl reload nginx # Enable service (start at boot) sudo systemctl enable nginx # Disable service (don't start at boot) sudo systemctl disable nginx # Enable and start service sudo systemctl enable --now nginx # Check if service is enabled systemctl is-enabled nginx # Check if service is active systemctl is-active nginx # View service dependencies systemctl list-dependencies nginx # View service configuration file location systemctl cat nginx # Edit service configuration sudo systemctl edit nginx # Reload systemd after service file changes sudo systemctl daemon-reload # View failed services systemctl --failed
# Create service file sudo nano /etc/systemd/system/myapp.service # Example service file: [Unit] Description=My Application Service After=network.target [Service] Type=simple User=myuser WorkingDirectory=/opt/myapp ExecStart=/usr/bin/python3 /opt/myapp/app.py Restart=always RestartSec=10 StandardOutput=syslog StandardError=syslog SyslogIdentifier=myapp [Install] WantedBy=multi-user.target # Reload systemd and start service sudo systemctl daemon-reload sudo systemctl enable myapp sudo systemctl start myapp sudo systemctl status myapp
# View service logs sudo journalctl -u nginx.service # Follow service logs in real-time sudo journalctl -u nginx.service -f # View last 100 lines of service log sudo journalctl -u nginx.service -n 100 # View service logs since specific time sudo journalctl -u nginx.service --since "1 hour ago" # Check service failures sudo journalctl -u nginx.service -p err # View service startup script systemctl cat nginx.service # Test service configuration sudo nginx -t # For nginx sudo apachectl configtest # For Apache # Kill service process (last resort) sudo killall nginx sudo pkill -9 nginx # Find service PID pidof nginx pgrep nginx # Restart all failed services sudo systemctl reset-failed
Daily Security Checks DAILY
Daily Security Checklist
- Review authentication logs for failed login attempts
- Check for suspicious user activity
- Verify no unauthorized users have been created
- Review sudo usage logs
- Check for rootkit infections
- Verify firewall rules are intact
- Check for security updates
- Review listening network services
- Check for suspicious processes
- Verify file integrity (critical system files)
# Check failed login attempts
sudo grep "Failed password" /var/log/auth.log | tail -20
# List unique IPs with failed attempts
sudo grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn
# Check for unauthorized users
awk -F: '$3 >= 1000 {print $1}' /etc/passwd
# View recently created users
sudo cat /var/log/auth.log | grep "new user"
# Check for new sudo users
grep -E '^sudo|^wheel' /etc/group
# View last logins
last | head -20
# Check for suspicious sudo usage
sudo cat /var/log/auth.log | grep "sudo" | tail -30
# List listening ports
sudo ss -tulpn
sudo netstat -tulpn
# Check for suspicious processes
ps aux | grep -E 'nc|ncat|socat|/tmp'
# Check for SUID files changed recently
sudo find / -perm /4000 -mtime -7 -ls 2>/dev/null
# List cron jobs for all users
sudo cat /etc/crontab
sudo ls -la /etc/cron.*
for user in $(cut -f1 -d: /etc/passwd); do echo "=== $user ==="; sudo crontab -u $user -l 2>/dev/null; done
# Check SSH authorized keys
find /home -name authorized_keys -exec ls -la {} \;
find /root -name authorized_keys -exec ls -la {} \;
# Install rootkit detection tools sudo apt install rkhunter chkrootkit # Debian/Ubuntu sudo yum install rkhunter chkrootkit # RHEL/CentOS # Run rootkit hunter sudo rkhunter --check sudo rkhunter --update # Run chkrootkit sudo chkrootkit # Check system binaries integrity sudo debsums -c # Debian/Ubuntu rpm -Va # RHEL/CentOS # Install and run Lynis security audit sudo apt install lynis sudo lynis audit system
Firewall Management
# Enable UFW sudo ufw enable # Disable UFW sudo ufw disable # View status sudo ufw status verbose sudo ufw status numbered # Allow service by name sudo ufw allow ssh sudo ufw allow http sudo ufw allow https # Allow specific port sudo ufw allow 8080/tcp sudo ufw allow 53/udp # Allow from specific IP sudo ufw allow from 192.168.1.100 # Allow from specific IP to specific port sudo ufw allow from 192.168.1.100 to any port 22 # Allow from subnet sudo ufw allow from 192.168.1.0/24 # Deny connection sudo ufw deny from 192.168.1.50 # Delete rule by number sudo ufw delete 3 # Delete rule by specification sudo ufw delete allow 80/tcp # Reset firewall (delete all rules) sudo ufw reset # Set default policies sudo ufw default deny incoming sudo ufw default allow outgoing # Enable logging sudo ufw logging on sudo ufw logging medium # View logs sudo tail -f /var/log/ufw.log
# Check firewalld status sudo firewall-cmd --state # Start/enable firewalld sudo systemctl start firewalld sudo systemctl enable firewalld # View active zones sudo firewall-cmd --get-active-zones # View all zones sudo firewall-cmd --get-zones # List all rules sudo firewall-cmd --list-all # Add service sudo firewall-cmd --permanent --add-service=http sudo firewall-cmd --permanent --add-service=https # Add port sudo firewall-cmd --permanent --add-port=8080/tcp sudo firewall-cmd --permanent --add-port=53/udp # Remove service sudo firewall-cmd --permanent --remove-service=http # Add rich rule sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.1.0/24" port port="22" protocol="tcp" accept' # Block IP address sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.1.50" reject' # Reload firewall (apply changes) sudo firewall-cmd --reload # View open ports sudo firewall-cmd --list-ports # View services sudo firewall-cmd --list-services
# View current rules sudo iptables -L -n -v # View rules with line numbers sudo iptables -L --line-numbers # Allow SSH sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT # Allow HTTP/HTTPS sudo iptables -A INPUT -p tcp --dport 80 -j ACCEPT sudo iptables -A INPUT -p tcp --dport 443 -j ACCEPT # Allow from specific IP sudo iptables -A INPUT -s 192.168.1.100 -j ACCEPT # Block specific IP sudo iptables -A INPUT -s 192.168.1.50 -j DROP # Delete rule by number sudo iptables -D INPUT 3 # Save rules (Ubuntu/Debian) sudo iptables-save > /etc/iptables/rules.v4 # Restore rules sudo iptables-restore < /etc/iptables/rules.v4 # Flush all rules (careful!) sudo iptables -F # Set default policies sudo iptables -P INPUT DROP sudo iptables -P FORWARD DROP sudo iptables -P OUTPUT ACCEPT # Allow established connections sudo iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT # Allow loopback sudo iptables -A INPUT -i lo -j ACCEPT
System Updates DAILY
# Update package list sudo apt update # Upgrade all packages sudo apt upgrade -y # Full upgrade (may remove packages) sudo apt full-upgrade -y # Dist upgrade (for version upgrades) sudo apt dist-upgrade -y # Security updates only sudo apt upgrade -y -o Dpkg::Options::="--force-confold" # List available updates apt list --upgradable # List security updates apt list --upgradable | grep -i security # Auto remove unused packages sudo apt autoremove -y # Clean package cache sudo apt clean sudo apt autoclean # Check for broken dependencies sudo apt check # Fix broken packages sudo apt --fix-broken install # Unattended upgrades (automatic security updates) sudo apt install unattended-upgrades sudo dpkg-reconfigure --priority=low unattended-upgrades
# Update all packages sudo yum update -y # Update specific package sudo yum update package-name # Security updates only sudo yum update --security # List available updates yum list updates # List security updates yum list updates --security # Check for updates without installing yum check-update # Clean cache sudo yum clean all # Auto remove unused packages sudo yum autoremove # Update with DNF (newer systems) sudo dnf update -y sudo dnf upgrade -y # Enable automatic updates sudo yum install yum-cron sudo systemctl enable yum-cron sudo systemctl start yum-cron
- Always test updates in staging environment first
- Take backup/snapshot before major updates
- Schedule updates during maintenance windows
- Enable automatic security updates for production servers
- Monitor for failed updates and dependency issues
- Keep kernel updated but plan reboots carefully
- Document all updates in change management system
- Subscribe to security mailing lists for your distribution
Network Configuration
# View all network interfaces ip addr show ip a # View specific interface ip addr show eth0 # View interface statistics ip -s link show eth0 # Traditional ifconfig (deprecated but still useful) ifconfig ifconfig eth0 # View routing table ip route show route -n # View ARP table ip neigh show arp -n # View DNS configuration cat /etc/resolv.conf # View hostname hostname hostnamectl # Set hostname sudo hostnamectl set-hostname newname
# Ubuntu/Debian - Netplan (newer versions)
# Config file: /etc/netplan/*.yaml
network:
version: 2
renderer: networkd
ethernets:
eth0:
dhcp4: true
eth1:
addresses:
- 192.168.1.10/24
gateway4: 192.168.1.1
nameservers:
addresses: [8.8.8.8, 8.8.4.4]
# Apply netplan configuration
sudo netplan apply
sudo netplan --debug apply
# Ubuntu/Debian - /etc/network/interfaces (older)
auto eth0
iface eth0 inet static
address 192.168.1.10
netmask 255.255.255.0
gateway 192.168.1.1
dns-nameservers 8.8.8.8 8.8.4.4
# RHEL/CentOS - /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
IPADDR=192.168.1.10
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
DNS1=8.8.8.8
DNS2=8.8.4.4
# Restart networking
sudo systemctl restart networking # Debian/Ubuntu
sudo systemctl restart network # RHEL/CentOS
sudo systemctl restart NetworkManager
# Add IP address to interface sudo ip addr add 192.168.1.10/24 dev eth0 # Remove IP address sudo ip addr del 192.168.1.10/24 dev eth0 # Bring interface up sudo ip link set eth0 up # Bring interface down sudo ip link set eth0 down # Add default gateway sudo ip route add default via 192.168.1.1 # Add static route sudo ip route add 10.0.0.0/8 via 192.168.1.254 # Delete route sudo ip route del 10.0.0.0/8 # Flush IP addresses sudo ip addr flush dev eth0 # Change MAC address sudo ip link set eth0 address aa:bb:cc:dd:ee:ff
Network Troubleshooting
# Ping test ping -c 4 8.8.8.8 ping -c 100 google.com # Test with specific packet size ping -s 1472 -c 5 8.8.8.8 # Flood ping (careful!) sudo ping -f 8.8.8.8 # Traceroute traceroute google.com traceroute -n 8.8.8.8 # Don't resolve hostnames # MTR (better than traceroute) mtr google.com mtr -n -c 100 8.8.8.8 # TCP traceroute sudo tcptraceroute google.com 443 # Test DNS resolution nslookup google.com nslookup google.com 8.8.8.8 # Dig (detailed DNS) dig google.com dig @8.8.8.8 google.com dig google.com +short # Host command host google.com
# Test TCP port connectivity nc -zv 192.168.1.100 22 nc -zv google.com 443 # Test UDP port nc -zvu 192.168.1.100 53 # Telnet to port telnet 192.168.1.100 80 # Test HTTP/HTTPS curl -I http://example.com curl -I https://example.com wget --spider http://example.com # Show listening ports sudo ss -tulpn sudo netstat -tulpn # Show established connections ss -tn netstat -tn # Show all connections ss -a netstat -a # Show connections for specific process sudo ss -tp | grep nginx sudo netstat -anp | grep :80 # Find which process is using a port sudo lsof -i :80 sudo fuser 80/tcp
# Capture packets on interface sudo tcpdump -i eth0 # Capture with more details sudo tcpdump -i eth0 -vv # Capture specific host sudo tcpdump -i eth0 host 192.168.1.100 # Capture specific port sudo tcpdump -i eth0 port 80 # Capture and save to file sudo tcpdump -i eth0 -w capture.pcap # Read capture file tcpdump -r capture.pcap # Capture HTTP traffic sudo tcpdump -i eth0 -A 'tcp port 80' # Capture SSH traffic sudo tcpdump -i eth0 port 22 # Capture DNS queries sudo tcpdump -i eth0 port 53 # Capture with filters sudo tcpdump -i eth0 'tcp[tcpflags] & (tcp-syn) != 0' # SYN packets sudo tcpdump -i eth0 icmp # ICMP only sudo tcpdump -i eth0 not port 22 # Exclude SSH # Limit capture size sudo tcpdump -i eth0 -c 100 -w capture.pcap # 100 packets sudo tcpdump -i eth0 -G 60 -w capture.pcap # 60 seconds # Monitor bandwidth usage sudo iftop -i eth0 sudo nethogs eth0
# Check if interface is up ip link show eth0 # Check if IP is assigned ip addr show eth0 # Check default gateway ip route | grep default # Verify DNS cat /etc/resolv.conf systemd-resolve --status # Test local network connectivity ping -c 4 192.168.1.1 # Gateway # Test internet connectivity ping -c 4 8.8.8.8 # Google DNS ping -c 4 google.com # Test DNS resolution # Check for IP conflicts sudo arping -I eth0 192.168.1.10 # View network errors/drops ip -s link show eth0 ethtool -S eth0 # Check MTU ip link show eth0 | grep mtu # Test with different MTU sudo ip link set eth0 mtu 1400 # Restart networking sudo systemctl restart networking sudo systemctl restart NetworkManager # Flush DNS cache sudo systemd-resolve --flush-caches sudo systemctl restart systemd-resolved
Performance Monitoring & Analysis
# Quick performance check vmstat 1 5 # CPU and memory summary free -h && uptime # I/O statistics iostat -x 1 5 # Full system monitoring dstat # All-in-one monitoring glances # System activity reporter sar -u 1 5 # CPU sar -r 1 5 # Memory sar -d 1 5 # Disk # Performance Co-Pilot pcp atop
# High CPU usage processes
ps aux --sort=-%cpu | head -10
# High memory usage processes
ps aux --sort=-%mem | head -10
# High I/O processes
sudo iotop -o
# High network usage processes
sudo nethogs
# Trace system calls (slow process)
sudo strace -p PID
# Profile CPU usage
sudo perf top
# Find what's causing high load
uptime
top -b -n 1 | head -20
# Check for zombie processes
ps aux | grep -w Z
# Find processes in uninterruptible sleep (D state)
ps aux | awk '$8=="D" {print $0}'
# Check context switches
vmstat 1 5
# Monitor interrupts
watch -n 1 'cat /proc/interrupts'
Backup Management DAILY
# Basic tar backup tar -czf backup-$(date +%Y%m%d).tar.gz /path/to/backup # Backup with exclusions tar -czf backup.tar.gz --exclude='/path/to/exclude' /path/to/backup # Backup multiple directories tar -czf backup.tar.gz /etc /var/www /home # Backup with verbose output tar -czvf backup.tar.gz /path/to/backup # Extract backup tar -xzf backup.tar.gz # Extract to specific directory tar -xzf backup.tar.gz -C /restore/path/ # List backup contents tar -tzf backup.tar.gz # Incremental backup using timestamp tar -czf backup-$(date +%Y%m%d-%H%M%S).tar.gz -N "2025-10-30" /path/to/backup
# Basic rsync backup rsync -av /source/ /destination/ # Rsync to remote server rsync -avz /source/ user@remote:/destination/ # Rsync with progress and delete rsync -avz --progress --delete /source/ /destination/ # Exclude files rsync -av --exclude='*.log' --exclude='cache/*' /source/ /destination/ # Dry run (test without changes) rsync -av --dry-run /source/ /destination/ # Backup with bandwidth limit (KB/s) rsync -av --bwlimit=1000 /source/ /destination/ # Incremental backup rsync -av --link-dest=/previous/backup /source/ /current/backup/ # Backup only files changed in last 24 hours rsync -av --update /source/ /destination/
Complete Backup Script
#!/bin/bash
# Comprehensive backup script
# Save as: /usr/local/bin/backup.sh
# Configuration
BACKUP_DIR="/backup"
BACKUP_NAME="server-backup-$(date +%Y%m%d-%H%M%S)"
LOG_FILE="/var/log/backup.log"
RETENTION_DAYS=30
REMOTE_HOST="backup-server"
REMOTE_USER="backup"
REMOTE_DIR="/backups"
# Directories to backup
BACKUP_PATHS=(
"/etc"
"/var/www"
"/home"
"/opt"
)
# Start logging
exec 1> >(tee -a "$LOG_FILE")
exec 2>&1
echo "=========================================="
echo "Backup started at $(date)"
echo "=========================================="
# Create backup directory
mkdir -p "$BACKUP_DIR"
# Create tar archive
echo "Creating backup archive..."
tar -czf "$BACKUP_DIR/$BACKUP_NAME.tar.gz" \
--exclude='*.log' \
--exclude='cache/*' \
--exclude='tmp/*' \
"${BACKUP_PATHS[@]}"
if [ $? -eq 0 ]; then
echo "✓ Backup archive created successfully"
# Calculate checksum
cd "$BACKUP_DIR"
sha256sum "$BACKUP_NAME.tar.gz" > "$BACKUP_NAME.sha256"
# Copy to remote server
echo "Copying backup to remote server..."
scp "$BACKUP_NAME.tar.gz" "$BACKUP_NAME.sha256" \
"$REMOTE_USER@$REMOTE_HOST:$REMOTE_DIR/"
if [ $? -eq 0 ]; then
echo "✓ Backup copied to remote server"
else
echo "✗ Failed to copy backup to remote server"
fi
# Clean old backups
echo "Cleaning old backups..."
find "$BACKUP_DIR" -name "server-backup-*.tar.gz" -mtime +$RETENTION_DAYS -delete
find "$BACKUP_DIR" -name "server-backup-*.sha256" -mtime +$RETENTION_DAYS -delete
echo "✓ Old backups cleaned"
else
echo "✗ Backup failed!"
exit 1
fi
echo "=========================================="
echo "Backup completed at $(date)"
echo "Backup size: $(du -h "$BACKUP_DIR/$BACKUP_NAME.tar.gz" | cut -f1)"
echo "=========================================="
# Optional: Send email notification
# echo "Backup completed successfully" | mail -s "Backup Success - $(hostname)" admin@example.com
# MySQL/MariaDB backup mysqldump -u root -p database_name > backup.sql mysqldump -u root -p --all-databases > all-databases.sql # MySQL with compression mysqldump -u root -p database_name | gzip > backup.sql.gz # PostgreSQL backup pg_dump -U postgres database_name > backup.sql pg_dumpall -U postgres > all-databases.sql # MongoDB backup mongodump --out /backup/mongodb/ # Restore MySQL mysql -u root -p database_name < backup.sql # Restore PostgreSQL psql -U postgres database_name < backup.sql # Restore MongoDB mongorestore /backup/mongodb/
- Follow 3-2-1 rule: 3 copies, 2 different media, 1 offsite
- Test restores regularly - backups are useless if you can't restore
- Automate backups with cron jobs
- Encrypt sensitive backups before storing
- Monitor backup success/failure with alerts
- Document backup and restore procedures
- Keep backup retention policy based on requirements
- Verify backup integrity with checksums
Weekly Maintenance Tasks WEEKLY
Weekly Maintenance Checklist
- Review weekly system performance reports
- Analyze disk space trends and plan for growth
- Review and rotate logs if necessary
- Update documentation for any changes made
- Test backup restore procedures (random sample)
- Review and update firewall rules if needed
- Check for and install non-critical updates
- Review user access and remove inactive accounts
- Verify SSL certificate expiration dates
- Review monitoring alerts and false positives
- Clean up old temporary files and caches
- Review cron jobs and scheduled tasks
Monthly Maintenance Tasks MONTHLY
Monthly Maintenance Checklist
- Full system audit and security review
- Review and update disaster recovery plan
- Performance capacity planning review
- Update system documentation
- Review and optimize database performance
- Test disaster recovery procedures
- Review SSL certificates for upcoming expirations (90 days)
- Audit user permissions and sudo access
- Review backup retention and storage costs
- Update emergency contact list
- Review and update runbooks
- Schedule and plan kernel updates/reboots
Automation Scripts
Disk Space Alert Script
#!/bin/bash
# Alert when disk usage exceeds threshold
# Save as: /usr/local/bin/disk-alert.sh
THRESHOLD=90
EMAIL="admin@example.com"
df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' | while read output;
do
usage=$(echo $output | awk '{ print $1}' | cut -d'%' -f1)
partition=$(echo $output | awk '{ print $2 }')
if [ $usage -ge $THRESHOLD ]; then
echo "ALERT: Disk usage on $partition is at ${usage}%" | \
mail -s "Disk Space Alert on $(hostname)" $EMAIL
fi
done
Service Monitoring Script
#!/bin/bash
# Monitor critical services and restart if down
# Save as: /usr/local/bin/service-monitor.sh
SERVICES=("nginx" "mysql" "ssh")
LOG="/var/log/service-monitor.log"
for service in "${SERVICES[@]}"; do
if ! systemctl is-active --quiet $service; then
echo "$(date): $service is down, attempting restart" >> $LOG
systemctl start $service
# Wait and check again
sleep 5
if systemctl is-active --quiet $service; then
echo "$(date): $service restarted successfully" >> $LOG
else
echo "$(date): FAILED to restart $service" >> $LOG
echo "$service restart failed on $(hostname)" | \
mail -s "CRITICAL: Service Restart Failed" admin@example.com
fi
fi
done
Log Cleanup Script
#!/bin/bash
# Clean up old logs and compress recent ones
# Save as: /usr/local/bin/log-cleanup.sh
LOG_DIR="/var/log"
DAYS_TO_KEEP=30
DAYS_TO_COMPRESS=7
# Delete old log files
find $LOG_DIR -name "*.log.*" -mtime +$DAYS_TO_KEEP -delete
find $LOG_DIR -name "*.gz" -mtime +$DAYS_TO_KEEP -delete
# Compress recent logs
find $LOG_DIR -name "*.log.*" -mtime +$DAYS_TO_COMPRESS ! -name "*.gz" -exec gzip {} \;
# Clean journal logs (keep 7 days)
journalctl --vacuum-time=7d
echo "$(date): Log cleanup completed" >> /var/log/log-cleanup.log
# Edit crontab crontab -e # View current crontab crontab -l # Example cron entries: # Daily backup at 2 AM 0 2 * * * /usr/local/bin/backup.sh # Hourly service check 0 * * * * /usr/local/bin/service-monitor.sh # Daily disk space alert at 8 AM 0 8 * * * /usr/local/bin/disk-alert.sh # Weekly log cleanup (Sunday at 3 AM) 0 3 * * 0 /usr/local/bin/log-cleanup.sh # Update and reboot monthly (first Sunday at 4 AM) 0 4 1-7 * 0 /usr/local/bin/monthly-update.sh # System health check every 5 minutes */5 * * * * /usr/local/bin/health-check.sh # Cron format: # ┌───────────── minute (0 - 59) # │ ┌───────────── hour (0 - 23) # │ │ ┌───────────── day of the month (1 - 31) # │ │ │ ┌───────────── month (1 - 12) # │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday) # │ │ │ │ │ # * * * * * command to execute
Emergency Troubleshooting Guide
Server is Slow/High Load
# 1. Check load average uptime # If load > number of CPU cores, system is overloaded # 2. Find CPU hogs ps aux --sort=-%cpu | head -10 # 3. Find memory hogs ps aux --sort=-%mem | head -10 # 4. Check disk I/O iostat -x 1 5 # 5. Check for high I/O processes sudo iotop -o # 6. Look for network issues netstat -s | grep -i error ip -s link # 7. Check for zombie processes ps aux | grep -w Z # 8. Review recent logs journalctl -p err -n 50 # 9. Check disk space df -h # 10. Look for runaway processes top -b -n 1 | head -20
Server Out of Disk Space
# Find what's using space
du -h / --max-depth=1 | sort -hr | head -10
# Find large files
find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null | head -20
# Clear common space hogs
sudo apt clean # Package cache
sudo journalctl --vacuum-size=100M # Journal logs
docker system prune -af # Docker (if installed)
# Remove old log files
sudo find /var/log -name "*.gz" -mtime +30 -delete
# Find and remove core dumps
sudo find / -name "core.*" -delete
# Clear user cache
rm -rf ~/.cache/*
# Check for deleted files still open
lsof | grep deleted | grep -v "/tmp"
Cannot SSH to Server
# From another terminal/console: # 1. Check if SSH service is running systemctl status sshd # 2. Check if SSH port is listening ss -tulpn | grep :22 # 3. Check firewall ufw status iptables -L -n | grep 22 # 4. Check SSH logs tail -50 /var/log/auth.log # 5. Test SSH locally ssh localhost # 6. Check SSH configuration sshd -t # 7. Restart SSH systemctl restart sshd # 8. Check for failed login attempts (might be blocked) grep "Failed" /var/log/auth.log | tail -20 # 9. Check fail2ban (if installed) fail2ban-client status sshd # 10. Unban IP if needed fail2ban-client set sshd unbanip 192.168.1.100
Service Won't Start
# Check service status systemctl status service-name # View detailed logs journalctl -u service-name -n 100 # Check service configuration systemctl cat service-name # Test configuration (if applicable) nginx -t # For Nginx apachectl configtest # For Apache # Check dependencies systemctl list-dependencies service-name # Check for port conflicts ss -tulpn | grep :80 # Check file permissions ls -la /path/to/service/files # Manually try to start service /usr/bin/service-binary --config /path/to/config # Reset failed state systemctl reset-failed service-name # Force kill and restart pkill -9 service-name systemctl start service-name
Disaster Recovery
Always maintain an up-to-date contact list including:
- Team lead and backup contacts
- On-call engineer schedules
- Vendor support contacts
- Management escalation path
- Datacenter/hosting provider support
Disaster Recovery Procedures
- Assess the situation - what's down, what's working
- Notify stakeholders and management
- Activate incident response team
- Document all actions taken with timestamps
- Attempt immediate recovery from backups if needed
- If corruption, preserve evidence before fixing
- Follow documented DR procedures
- Keep communication channels open
- Once recovered, perform root cause analysis
- Update DR documentation based on lessons learned
# Boot into recovery mode (from GRUB): # 1. Select "Advanced options" # 2. Select "recovery mode" # 3. Select "root - Drop to root shell prompt" # Once in recovery mode: # Remount filesystem as read-write mount -o remount,rw / # Check and repair filesystem fsck -y /dev/sda1 # Restore from backup tar -xzf /backup/latest.tar.gz -C / # Reset root password if needed passwd root # Fix bootloader grub-install /dev/sda update-grub # Reboot reboot
- Stay calm and methodical
- Document everything you do
- Don't make changes without understanding impact
- Have rollback plan before making changes
- Communicate proactively with stakeholders
- Follow change control procedures even in emergencies
- Preserve evidence if security incident
- Conduct post-mortem after resolution
Quick Reference - Essential Commands
| Category | Command | Description |
|---|---|---|
| System Info | uname -a |
Kernel and system info |
hostnamectl |
Hostname and OS info | |
uptime |
System uptime and load | |
| Monitoring | htop |
Interactive process viewer |
free -h |
Memory usage | |
df -h |
Disk space usage | |
iostat -x 1 |
I/O statistics | |
| Services | systemctl status service |
Check service status |
systemctl restart service |
Restart service | |
journalctl -u service -f |
Follow service logs | |
| Network | ip addr show |
Show IP addresses |
ss -tulpn |
Show listening ports | |
ping -c 4 host |
Test connectivity | |
| Logs | tail -f /var/log/syslog |
Follow system log |
journalctl -p err |
Show errors | |
grep -i error /var/log/* |
Search logs for errors | |
| Files | find / -name file |
Find file |
du -sh directory |
Directory size | |
chmod 755 file |
Change permissions | |
| Security | sudo ufw status |
Check firewall |
last |
Show login history | |
who |
Show logged in users |
Remember: A Good Sysadmin is a Lazy Sysadmin
Automate everything you can.
Document everything you do.
Test everything before production.
Monitor everything that matters.
Backup everything important.
Linux Server Management Guide
Created: October 31, 2025
For System Administrators
Stay Calm and sudo !! 🐧