📊 AI Log Analysis Basics

🎯 Who This Page Is For

You have been reading logs for years. You know grep, awk, journalctl, and tail -f the way a carpenter knows a hammer. You have written parsers, built dashboards, set up syslog forwarding, and at some point probably stared at 200,000 lines of Apache access logs looking for the one request that broke something.

This page is not going to explain what a log file is. It is going to explain what AI-assisted log analysis actually does — concretely, mechanically — and where it changes your workflow versus where your existing tools remain the right answer.

🚧 The Problem That Creates the Opportunity

Your existing tools work well until volume and variety overwhelm them. Consider what a busy production environment generates:

grep scales fine with volume — that is not the problem. The problem is variety and context. Your grep pattern finds what you told it to find. It does not notice that the successful login at 3 AM came from a new country, that the user who just ran sudo has never done so before, or that five separate low-frequency events across three log files add up to something worth investigating.

That is the gap AI fills. Not speed. Context across multiple signals.

🔧 What Your Existing Tools Do Well

Before adding anything new, be clear about what you already have. These tools remain the right answer for a large portion of log work:

grep / egrep

Fast, exact, scriptable. When you know what you are looking for, nothing beats it. Pipe it into a count, sort it, feed it to awk. A well-written grep one-liner catches known bad patterns faster than any AI tool will.

# Known pattern — grep wins every time
grep "Failed password" /var/log/auth.log | \
  awk '{print $11}' | sort | uniq -c | sort -rn | head -20

# Top 20 IPs hitting you with failed SSH — fast, reliable, no AI needed

journalctl

Structured, indexed, filterable by unit, priority, and time window. When you know which service you care about, journalctl with -u and --since gets you there instantly.

# Service-specific logs in a time window — journalctl wins
journalctl -u nginx --since "2 hours ago" -p err

# All errors across all units in the last hour
journalctl --since "1 hour ago" -p err..crit

awk / sed

When you need to extract fields, reformat output, or compute statistics from structured log lines, awk is still the right tool. Anything with a consistent field separator belongs to awk.

# Extract HTTP status codes and count them — awk wins
awk '{print $9}' /var/log/nginx/access.log | \
  sort | uniq -c | sort -rn

🤖 What AI-Assisted Analysis Actually Does

When you pipe log output to an AI tool, you are not running a smarter grep. You are handing a body of text to something that can:

What it does not do: it does not have persistent memory of your systems, it does not know what normal looks like for your specific environment, and it cannot run follow-up commands on its own. You are the operator. It is the analyst you hand output to.

🧪 Starting Simple: curl + the API

No new tools required. Every Linux system already has curl. This is the simplest possible AI-assisted log analysis workflow — pipe log output into a shell variable and send it to an AI API directly.

The example below uses the Anthropic API. Substitute your key and endpoint for OpenAI, a local ollama instance, or any OpenAI-compatible API.

# Step 1 — capture the log output you want analyzed
LOG_DATA=$(journalctl -u sshd --since "2 hours ago" --no-pager 2>&1 | tail -100)

# Step 2 — build the JSON payload and send it
echo "$LOG_DATA" | curl -s https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-haiku-4-5-20251001",
    "max_tokens": 512,
    "messages": [{
      "role": "user",
      "content": "You are a Linux sysadmin assistant. Analyze these SSH logs and summarize: any failed auth attempts, unusual patterns, or items worth investigating. Be concise.\n\n'"$LOG_DATA"'"
    }]
  }' | python3 -c "import sys,json; print(json.load(sys.stdin)['content'][0]['text'])"

For a more reusable version, wrap it in a function in your .bashrc:

# Add to ~/.bashrc
ask_claude() {
  local PROMPT="$1"
  local INPUT=$(cat)   # reads from stdin
  curl -s https://api.anthropic.com/v1/messages \
    -H "x-api-key: $ANTHROPIC_API_KEY" \
    -H "anthropic-version: 2023-06-01" \
    -H "content-type: application/json" \
    -d "{\"model\":\"claude-haiku-4-5-20251001\",\"max_tokens\":512,
       \"messages\":[{\"role\":\"user\",\"content\":\"$PROMPT\n\n$INPUT\"}]}" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['content'][0]['text'])"
}

# Then use it like any Unix filter:
journalctl -u nginx --since "1 hour ago" -p err | \
  ask_claude "Summarize these nginx errors. Group by error type. Flag anything unusual."

grep "sudo" /var/log/auth.log | tail -50 | \
  ask_claude "List all sudo commands run. Flag any that look unusual for a web server."

You now have AI log analysis with nothing installed beyond what shipped with your OS.

🔄 The Same Workflow With Better Tools

Once the concept clicks, the dedicated AI CLI tools make it cleaner. You do not have to manage JSON payloads by hand or parse the response with python3 inline.

With sgpt

Same pipeline, much less ceremony:

# Direct pipe — reads from stdin automatically
journalctl -u sshd --since "2 hours ago" | \
  sgpt "Summarize failed auth attempts. List source IPs and frequency."

# Combine grep extraction with AI summarization
grep "sudo" /var/log/auth.log | tail -100 | \
  sgpt "List all users who ran sudo commands and what they ran. Flag anything unusual."

# Multi-source: combine two logs before analysis
{ grep "Failed" /var/log/auth.log; \
  grep "404\|403" /var/log/nginx/access.log; } | tail -200 | \
  sgpt "These are SSH failures and web 403/404s from the same period. Any correlation?"

With fabric

When you are running the same analysis repeatedly — daily auth summaries, shift-change reports, scheduled log digests — fabric lets you define the prompt once as a reusable pattern and pipe to it like any Unix filter:

# One-time: create a custom pattern for your environment
# ~/.config/fabric/patterns/auth_summary/system.md
# "You are a Linux security analyst. Summarize auth log input: failed logins by IP,
#  successful logins, sudo usage, new SSH keys. Flag anomalies. Be concise."

# Then every day in cron or a shift-start alias:
journalctl -u sshd --since "24 hours ago" | fabric --pattern auth_summary

# Works across any log source with the right pattern
tail -500 /var/log/nginx/access.log | fabric --pattern web_anomaly

With ripgrep as the extraction layer

Before you hand anything to AI, you want to extract the right lines efficiently. ripgrep is significantly faster than grep on large log files and handles compressed logs and recursive directory search cleanly:

# ripgrep extracts, sgpt analyzes — clean separation of concerns
rg "Failed|Invalid|error" /var/log/auth.log | tail -200 | \
  sgpt "Summarize these auth failures. Any patterns worth investigating?"

# Search across all logs in a directory
rg "OOM\|killed process\|out of memory" /var/log/ | \
  sgpt "Which processes are being OOM killed and how often?"

📋 Quick Reference: Which Tool for Which Job

➡️ Where to Go Next

This page covered the foundations — what AI adds, where your existing tools stay superior, and how to wire the two together starting with tools you already have.