Monitoring, Audit and Observability

We have built an AI system with tools, permissions, autonomy, persistence and safety boundaries. Now suppose something important happens. The system reports success. A file changed. A service restarted. A tool failed. A human approved an action. An agent says another agent completed a task. How do we establish what actually happened?

Three Related Ideas

Monitoring, audit and observability overlap, but they are not identical.

Return to the HAAW Web Server

Suppose an agent reports:

That sentence contains several claims:

• the agent requested a restart,
• the request was authorized,
• a restart command was actually executed,
• the service state changed,
• the service became available afterward, and
• the recovery was verified.

Those are not the same event.

The Model's Output Is One Observation

The model's generated text is valuable evidence of what the model produced. But it is not automatically proof that an external action occurred.

If the model says:

we can distinguish:

What Can We Observe?

An AI system is a stack. Different layers expose different evidence.

No single layer necessarily tells the whole story.

Logs Are Evidence — But Not Reality Itself

Sysadmins know this one well.

A log entry might say:

That establishes something useful: the logging system recorded that event.

It does not by itself prove that:

• the service actually stopped,
• the service actually restarted,
• the new process remained healthy, or
• users could reach it afterward.

Instrumentation

Instrumentation is the mechanism deliberately added to expose useful information about a running system.

Examples include:

• application logs,
• metrics,
• distributed traces,
• event records,
• tool-call records,
• audit logs,
• performance counters,
• network telemetry,
• model instrumentation, and
• external verification probes.

Metrics, Logs and Traces

Traditional observability often uses three useful categories.

AI agent systems add more events to trace:

Correlation IDs and Event IDs

Once many events overlap, timestamps alone may not be enough.

A useful system can assign identifiers such as:

These are conceptual examples. The exact format does not matter.

The important idea is that later investigators can determine which events belong together.

Time Is Necessary — But Not Sufficient

Suppose:

Which result caused which later decision?

A timestamp helps, but causal structure needs more.

This is why our earlier multi-agent pages moved from a simple timeline toward an event/dependency graph.

The Event Field

A real AI system may have many things happening at once:

• one model inference running,
• another agent waiting for a tool,
• a database request in flight,
• a GPU processing another batch,
• a human reviewing an approval,
• a network response arriving,
• a scheduler waking another mission, and
• logs being written asynchronously.

There may be no single neat line of events.

Audit: Who Was Allowed to Do What?

Audit becomes especially important when an AI system can affect external state.

For an important action, an audit record might preserve:

• mission identity,
• requesting agent or process,
• requested action,
• target resource,
• active permissions,
• authorization decision,
• human approval if required,
• execution result,
• resulting state change, and
• verification outcome.

Audit Records Need Integrity

An audit trail is much less useful if the same process being investigated can silently rewrite or delete it.

Important evidence systems may therefore use controls such as:

• restricted write access,
• append-oriented records,
• separate logging services,
• checksums or cryptographic integrity mechanisms,
• replication,
• access controls, and
• retention policies.

No mechanism automatically makes evidence perfect. The goal is to make alteration detectable or difficult and provenance reconstructable.

Provenance

Provenance answers where information came from.

If an agent reports:

useful provenance might identify:

That allows the statement to be traced back toward its evidence.

Observation, Derived Fact, Interpretation

This is one of the most important WITNESS disciplines and belongs directly in HAAW.

Monitoring Can Trigger Action

Monitoring is not always passive.

A monitoring system may detect:

and trigger:

This connects directly to our persistent-agent page. The mission can sleep while ordinary monitoring watches the environment.

Monitoring the AI Itself

We can monitor not only the external system but the AI service.

Useful operational measurements may include:

• request volume,
• inference latency,
• tokens processed,
• GPU utilization,
• VRAM consumption,
• KV-cache pressure,
• tool-call frequency,
• tool failure rates,
• retry counts,
• permission denials,
• approval waits, and
• mission completion or failure states.

Those signals can reveal performance or control problems without telling us everything about why a particular model output occurred.

Observability Is Not Interpretability

We opened this distinction earlier.

Recording an activation tensor gives us an observation. Explaining what causal role that tensor played is an interpretability problem.

Self-Report Is Not Instrumentation

We should also preserve another earlier distinction.

If we ask a model:

the resulting explanation is another generated model output.

It may be useful. It may accurately summarize relevant factors. But it is not equivalent to an independently recorded hidden execution trace.

Instrumentation Can Change the System

Measurement itself has cost.

Heavy instrumentation can:

• consume CPU or GPU resources,
• increase memory use,
• add latency,
• increase network traffic,
• change timing,
• produce enormous data volumes, or
• alter the very behavior being studied.

This is why WITNESS has repeatedly returned to the idea of sterile capture: record what is needed while minimizing disturbance to the system under examination.

You Cannot Record Everything Forever

An AI system can generate an extraordinary amount of telemetry.

Imagine trying to preserve every:

• token,
• logit,
• activation,
• attention value,
• GPU kernel,
• memory transfer,
• tool event,
• network packet, and
• hardware counter

for every inference across a large production system.

The storage and performance cost can become enormous.

Known Gaps Are Part of the Record

If an event was not captured, a careful forensic system should not manufacture certainty afterward.

That statement is useful. It tells future investigators exactly where the evidence stops.

The WITNESS Connection

This entire page is very close to the reason WITNESS exists.

WITNESS is not merely a bigger application log. Its purpose is to help preserve enough trustworthy, structured evidence to reconstruct an AI mission after the fact.

A useful WITNESS event record might conceptually include:

WITNESS Records Before It Explains

The discipline can be stated simply:

Not:

The Boundary

Monitoring, audit and observability do not guarantee that an AI system will behave correctly.

They provide something different: the ability to detect conditions, preserve evidence, reconstruct events, verify claims and identify where the evidence runs out.