Troubleshooting and Best Practices (Part 4 of 4)
1. NetworkManager Troubleshooting
1.1 Common NetworkManager Issues
Issue: Connection Fails to Activate
Symptoms:
- nmcli connection up fails
- Connection shows as inactive
- No IP address assigned
Diagnostic Steps:
# Check connection status
nmcli connection show "Connection-Name"
# Check device status
nmcli device status
# View detailed device info
nmcli device show eth0
# Check NetworkManager logs
journalctl -u NetworkManager -n 50 -f
# Check system logs
dmesg | tail -50
Common Fixes:
# Reload connection
nmcli connection reload
# Restart NetworkManager
sudo systemctl restart NetworkManager
# Bring device down and up
nmcli device disconnect eth0
nmcli device connect eth0
# Delete and recreate connection
nmcli connection delete "Connection-Name"
nmcli connection add type ethernet con-name "Connection-Name" ifname eth0
Issue: DNS Not Working
Symptoms:
- Cannot resolve domain names
- ping by IP works, but ping by hostname fails
- /etc/resolv.conf is empty or incorrect
Diagnostic Steps:
# Check DNS settings
nmcli device show eth0 | grep DNS
# Check resolv.conf
cat /etc/resolv.conf
# Test DNS resolution
nslookup google.com
dig google.com
# Check if systemd-resolved is running
systemctl status systemd-resolved
Fixes:
# Set DNS servers explicitly
nmcli connection modify "Connection-Name" \
ipv4.dns "8.8.8.8,8.8.4.4"
# Don't use automatic DNS
nmcli connection modify "Connection-Name" \
ipv4.ignore-auto-dns yes
# Restart NetworkManager
sudo systemctl restart NetworkManager
# Bring connection up
nmcli connection up "Connection-Name"
Issue: Changes Not Taking Effect
Solution Process:
# After modifying connection
nmcli connection reload "Connection-Name"
# Or reload all connections
nmcli connection reload
# Then bring connection down and up
nmcli connection down "Connection-Name"
nmcli connection up "Connection-Name"
# Or disconnect/reconnect device
nmcli device disconnect eth0
nmcli device connect eth0
Issue: Can't Modify System Connections
Symptoms:
- Permission denied errors
- Changes not saved
Fixes:
# Use sudo for system-wide connections
sudo nmcli connection modify "Connection-Name" ...
# Check file permissions
ls -l /etc/NetworkManager/system-connections/
# Fix permissions if needed
sudo chmod 600 /etc/NetworkManager/system-connections/*
sudo chown root:root /etc/NetworkManager/system-connections/*
1.2 Network Connectivity Issues
| Problem |
Check Command |
Likely Cause |
| No IP address |
nmcli device show eth0 |
DHCP failure or static IP misconfiguration |
| Can't reach gateway |
ping $(nmcli -g IP4.GATEWAY device show eth0) |
Wrong gateway or cable issue |
| Can ping IPs but not names |
cat /etc/resolv.conf |
DNS not configured |
| Intermittent connectivity |
nmcli device monitor eth0 |
Cable, driver, or hardware issue |
1.3 NetworkManager Debugging
Enable Debug Logging
# Edit NetworkManager.conf
sudo vi /etc/NetworkManager/NetworkManager.conf
# Add or modify:
[logging]
level=DEBUG
domains=ALL
# Restart NetworkManager
sudo systemctl restart NetworkManager
# View logs
journalctl -u NetworkManager -f
# Remember to disable debug logging when done!
[logging]
level=INFO
Connection Troubleshooting Script
#!/bin/bash
# Quick network diagnostics
echo "=== Network Diagnostics ==="
echo ""
echo "1. NetworkManager Status:"
systemctl status NetworkManager --no-pager | head -5
echo ""
echo "2. Devices:"
nmcli device status
echo ""
echo "3. Active Connections:"
nmcli connection show --active
echo ""
echo "4. IP Configuration:"
ip addr show
echo ""
echo "5. Default Route:"
ip route show default
echo ""
echo "6. DNS Servers:"
cat /etc/resolv.conf | grep nameserver
echo ""
echo "7. Gateway Test:"
GATEWAY=$(ip route | grep default | awk '{print $3}')
if [ -n "$GATEWAY" ]; then
if ping -c 3 -W 2 "$GATEWAY" &>/dev/null; then
echo "Gateway ($GATEWAY): OK"
else
echo "Gateway ($GATEWAY): FAILED"
fi
else
echo "No default gateway configured"
fi
echo ""
echo "8. Internet Test:"
if ping -c 3 -W 2 8.8.8.8 &>/dev/null; then
echo "Internet (8.8.8.8): OK"
else
echo "Internet (8.8.8.8): FAILED"
fi
echo ""
echo "=== End Diagnostics ==="
2. nm Symbol Table Troubleshooting
2.1 Common nm Issues
Issue: "No Symbols" Error
Error: nm: file: no symbols
Cause: Binary has been stripped of symbol information
Check:
# Verify if binary is stripped
file myprogram
# Output: "stripped" = no symbols, "not stripped" = has symbols
# Compare sizes
ls -lh myprogram
# Stripped binaries are smaller
Solutions:
- Use the non-stripped version if available
- Rebuild with debug symbols:
gcc -g program.c
- Install debug packages:
apt-get install package-dbgsym
- Use other tools:
objdump -t or readelf -s
Issue: C++ Name Mangling
Problem: C++ symbols are unreadable
_Z3addii
_ZN9MyClass10myFunctionEv
Solution: Always use
-C for C++ binaries
# Demangle C++ names
nm -C myprogram
# Compare:
nm myprogram | grep add
# Output: 0000000000001234 T _Z3addii
nm -C myprogram | grep add
# Output: 0000000000001234 T add(int, int)
Issue: Finding Missing Symbols
Linker Error:
undefined reference to `some_function'
Diagnostic Process:
# 1. Verify the symbol is actually undefined
nm -u myprogram.o | grep some_function
# 2. Search your object files
nm *.o | grep " T some_function"
# 3. Search system libraries
for lib in /usr/lib/x86_64-linux-gnu/*.so; do
if nm -D "$lib" 2>/dev/null | grep -q " T some_function"; then
echo "Found in: $lib"
fi
done
# 4. Use ldd to check dynamic dependencies
ldd myprogram
Solutions:
- Add the missing object file to link command
- Add library flag:
-lfoo
- Add library path:
-L/path/to/lib
- Check spelling and signature
2.2 Symbol Analysis Best Practices
For C++ Programs:
- Always use
nm -C to demangle names
- Check for name mangling issues with
c++filt
- Be aware of different mangling schemes (GCC vs Clang)
For Library Analysis:
- Use
nm -D for shared libraries
- Check both defined (
T) and undefined (U) symbols
- Use
--defined-only to see only exports
- Compare symbol tables between library versions
For Debugging:
- Use
-l to show line numbers when available
- Combine with
grep for targeted searches
- Sort by size to find large symbols:
--size-sort -S
- Use
-A when checking multiple files
3. Best Practices Summary
3.1 NetworkManager Best Practices
| Practice |
Reason |
Example |
| Use descriptive connection names |
Easy identification |
"Office-Static-192.168.1.100" |
| Set autoconnect priorities |
Control connection preference |
connection.autoconnect-priority 10 |
| Use -t flag for scripts |
Consistent, parseable output |
nmcli -t -f NAME connection show |
| Backup connection files |
Quick recovery |
cp /etc/NetworkManager/system-connections/* /backup/ |
| Test before permanent changes |
Avoid lockouts |
Test with temporary connection first |
3.2 nm Best Practices
| Practice |
Reason |
Command |
| Demangle C++ symbols |
Readability |
nm -C |
| Check undefined symbols first |
Find dependencies quickly |
nm -u |
| Use size analysis |
Identify optimization targets |
nm --size-sort -S |
| Prefix multi-file searches |
Track symbol locations |
nm -A *.o |
| Combine with other tools |
Complete analysis |
nm + ldd + objdump |
3.3 General Linux Command Best Practices
Documentation:
- Comment your scripts thoroughly
- Keep a log of network changes
- Document your connection profiles
- Create README files for complex setups
Testing:
- Test scripts in safe environments first
- Use dry-run options when available
- Verify changes before disconnecting
- Have a rollback plan ready
Security:
- Protect connection files (chmod 600)
- Don't hardcode passwords in scripts
- Use key-based authentication when possible
- Audit binaries for unexpected symbols
4. Quick Reference Cheat Sheet
4.1 NetworkManager Quick Commands
# Status
nmcli general status # Overall status
nmcli device status # Device list
nmcli connection show # Connection list
nmcli connection show --active # Active only
# Connect/Disconnect
nmcli connection up "Name" # Activate
nmcli connection down "Name" # Deactivate
nmcli device disconnect eth0 # Disconnect device
# Modify
nmcli connection modify "Name" \
ipv4.addresses "192.168.1.100/24" # Set IP
nmcli connection reload # Reload configs
# WiFi
nmcli device wifi list # List networks
nmcli device wifi connect SSID \
password "pass" # Connect to WiFi
4.2 nm Quick Commands
# Basic
nm file # List all symbols
nm -u file # Undefined symbols
nm -D file # Dynamic symbols
nm -g file # Global symbols only
# Analysis
nm -C program # Demangle C++
nm --size-sort -S file # Sort by size
nm -A *.o | grep symbol # Find in multiple files
# Filtering
nm file | grep " T " # Functions only
nm file | grep " [DdBb] " # Variables only
nm -D lib.so | grep " T func" # Specific function
5. Recovery Procedures
5.1 Network Connection Recovery
#!/bin/bash
# Emergency network recovery script
echo "=== Network Recovery ==="
# Stop NetworkManager
sudo systemctl stop NetworkManager
# Manual IP configuration (if needed for remote access)
sudo ip addr add 192.168.1.100/24 dev eth0
sudo ip link set eth0 up
sudo ip route add default via 192.168.1.1
# Test connectivity
ping -c 3 192.168.1.1
# Restart NetworkManager
sudo systemctl start NetworkManager
# Reactivate connection
nmcli connection up "Your-Connection"
5.2 Connection File Backup and Restore
# Backup all connections
sudo tar -czf nm-connections-backup-$(date +%Y%m%d).tar.gz \
/etc/NetworkManager/system-connections/
# Restore from backup
sudo tar -xzf nm-connections-backup-20241025.tar.gz -C /
# Fix permissions
sudo chmod 600 /etc/NetworkManager/system-connections/*
sudo chown root:root /etc/NetworkManager/system-connections/*
# Reload
sudo nmcli connection reload
6. Performance Tips
6.1 NetworkManager Performance
- Disable unused connection types (WiFi if not needed)
- Set appropriate autoconnect priorities
- Use static IPs when DHCP isn't required
- Disable IPv6 if not used:
ipv6.method ignore
- Reduce connection timeout values for faster failover
6.2 nm Performance
- Use
-g to filter global symbols for faster searches
- Process fewer files at once for large projects
- Use
-p (no sort) if order doesn't matter
- Pipe to grep early to reduce processing
- Consider using
objdump for complex analysis
7. Additional Resources
7.1 Man Pages
man nmcli
man nmcli-examples
man nm-settings
man NetworkManager.conf
man nm(1)
7.2 Useful Documentation Paths
/usr/share/doc/NetworkManager/
/etc/NetworkManager/
/var/lib/NetworkManager/
7.3 Related Tools to Explore
| Tool |
Purpose |
Relation to nm |
objdump |
Display object file information |
More detailed than nm |
readelf |
Display ELF file information |
Alternative to nm |
ldd |
Print shared library dependencies |
Complements nm -u |
c++filt |
Demangle C++ symbols |
Used by nm -C |
strip |
Remove symbols from files |
Opposite of nm |
8. Final Tips
System Administrator Wisdom:
- Always have a backup connection method
- Document your changes immediately
- Test in non-production first
- Keep rescue media available
- Monitor logs regularly
- Automate repetitive tasks
- Share knowledge with your team
- Stay current with updates
Remember:
- Network changes can lock you out of remote systems
- Always have out-of-band access (console, IPMI, etc.)
- Test major changes during maintenance windows
- Keep documentation of rollback procedures