AI Home › Technical Deep-Dive › Ollama — Local AI Models on Linux

🦙 Ollama

Run large language models locally — no cloud, no API key, no data leaving your network

aider curl fabric fzf gh Copilot jq ollama ripgrep sgpt tmux

What Is Ollama?

Ollama is an open-source tool that makes it straightforward to download, run, and manage large language models on your own hardware. It handles the complexity of model quantization, GPU acceleration, and API serving — you interact with it through a simple CLI or a local REST API that speaks the same language as the OpenAI API.

For a Linux SysAdmin, Ollama is the answer to a specific class of problems: situations where you want AI assistance but cannot or should not send data to a cloud provider. FERPA-regulated data, internal documentation containing sensitive configuration details, security-sensitive scripts, or simply environments where any external network call is prohibited.

Ollama also integrates directly with the other tools in this series — sgpt, aider, and fabric can all be pointed at a local Ollama instance instead of cloud APIs.

"Cloud AI is faster and more capable. Local AI is private, free to run, and always available. Knowing when to use each is the skill."

💰 Cost & Licensing

Ollama is free and open source (MIT license). The models it runs are also free — most are released under open licenses by Meta, Mistral, Google, and others. The only cost is your hardware.

Hardware note: Ollama runs on CPU if no GPU is available, but performance is significantly better with a dedicated GPU. A modern NVIDIA GPU with 8GB+ VRAM runs 7B parameter models comfortably. 13B models need 16GB+ VRAM. CPU-only is usable for experimentation but slow for production workflows.

🔧 Installation on Linux

The official install script handles everything — dependencies, systemd service, and GPU detection:

curl -fsSL https://ollama.com/install.sh | sh

Ollama installs as a systemd service and starts automatically:

# Check service status systemctl status ollama # View logs journalctl -u ollama -f

Models are stored in /usr/share/ollama/.ollama/models/ by default. On systems with limited root partition space, change this before pulling models:

# Set custom model storage location mkdir -p /data/ollama/models echo 'OLLAMA_MODELS=/data/ollama/models' >> /etc/systemd/system/ollama.service.d/override.conf systemctl daemon-reload && systemctl restart ollama
Tip: Models range from 2GB to 40GB+ depending on size and quantization. Plan your storage accordingly before pulling several models.

📦 Models — What to Pull

Ollama hosts a library of models at ollama.com/library. For SysAdmin work:

Model Size Best For
llama3.2 2-3 GB Fast responses, general questions, light scripting tasks
llama3.1:8b 5 GB Good balance of speed and quality for most SysAdmin tasks
llama3.1:70b 40 GB Near cloud quality — needs substantial GPU VRAM or CPU RAM
mistral 4 GB Strong at code and structured output, good for script generation
codellama 4-34 GB Specialized for code — bash, Python, config file generation
gemma2:9b 5 GB Google's open model — solid general purpose performance
# Pull a model ollama pull llama3.1:8b # List downloaded models ollama list # Remove a model ollama rm modelname

💻 Basic Usage

Interactive chat

ollama run llama3.1:8b

This drops you into an interactive session. Type your prompt, get a response, continue the conversation. Type /bye to exit.

Single prompt (non-interactive)

ollama run llama3.1:8b "explain what this systemd unit file does" < myservice.service echo "write a bash one-liner to find the top 5 memory-consuming processes" | ollama run mistral

REST API

Ollama exposes a local REST API on port 11434 that is compatible with the OpenAI API format:

# Direct API call curl http://localhost:11434/api/generate \ -d '{"model": "llama3.1:8b", "prompt": "list 3 common causes of high load average on Linux", "stream": false}' \ | jq '.response' # OpenAI-compatible endpoint (for tools that speak OpenAI API) curl http://localhost:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "llama3.1:8b", "messages": [{"role": "user", "content": "hello"}]}'

🔗 Integration With Other CLI Tools

Because Ollama speaks the OpenAI API format, it slots directly into the tools covered in this series:

sgpt with Ollama

# Point sgpt at local Ollama instead of OpenAI sgpt --model ollama/llama3.1:8b "find all failed systemd services"

aider with Ollama

# Use a local model for file editing aider --model ollama/codellama myScript.sh

fabric with Ollama

# Run a fabric pattern against a local model cat logfile.txt | fabric --model ollama/llama3.1:8b --pattern summarize_logs

Direct API use in bash scripts

#!/bin/bash # Query local Ollama from a script ask_ai() { local prompt="$1" curl -s http://localhost:11434/api/generate \ -d "{\"model\": \"llama3.1:8b\", \"prompt\": \"$prompt\", \"stream\": false}" \ | jq -r '.response' } # Use it ask_ai "What is the correct syntax for adding a firewalld rich rule to allow port 8443?"
Tip: The bash API wrapper above is surprisingly useful for one-off automation tasks. Pipe log snippets into it, ask for explanations, get formatted output — all without leaving the script.

🔐 Network Access & Multi-User Setup

By default Ollama only listens on localhost. To make it available to other machines on your network (e.g. a dedicated inference server):

# Allow network access echo 'OLLAMA_HOST=0.0.0.0' >> /etc/systemd/system/ollama.service.d/override.conf systemctl daemon-reload && systemctl restart ollama
Security note: Ollama has no built-in authentication. If you expose it on the network, put it behind a firewall rule or reverse proxy with authentication. Do not expose port 11434 to the internet.

A dedicated Ollama server on your LAN lets multiple users and tools share the same local model without each machine needing the storage and GPU resources to run it independently.

✅ Strengths & Limitations

Works Well

  • 100% local — no data leaves your network
  • No API costs — run as many queries as you want
  • Works air-gapped
  • OpenAI-compatible API
  • Integrates with sgpt, aider, fabric
  • Simple model management (pull/list/rm)
  • systemd service — starts on boot
  • Free and open source

Watch Out For

  • Slower than cloud models on same tasks
  • Smaller local models less capable than Claude/GPT-4
  • GPU required for comfortable performance
  • Large model files consume significant storage
  • No authentication built in
  • Models improve more slowly than cloud
  • Context window smaller than cloud models

📊 Craig's Take

Ollama is the right answer to a specific question: "What do I do when I can't send this data to a cloud AI?" For a university SysAdmin dealing with FERPA considerations, internal infrastructure details, or simply working on a network segment with restricted egress — Ollama is the answer.

The quality gap between local models and Claude is real and significant for complex tasks. For simple tasks — syntax lookups, one-liner generation, explaining a config file — a local 8B model does the job well enough that the privacy benefit is worth it.

The most compelling use case is the dedicated inference server model: one machine with a decent GPU running Ollama, accessible to everyone on the LAN. sgpt, aider, and fabric all point at it. No one needs an API key. No queries leave the building. That's a meaningful capability for an institution with data handling obligations.

"Cloud AI for capability. Local AI for privacy. The good news is you don't have to choose — you can have both."

🔗 Resources