Goals, Constraints and Mission Success
Who Decides What “Done” Means?
Before Planning Comes the Mission
Page 37 showed how an agent can turn a goal into a plan and revise that plan when evidence changes. But that raises a deeper question:
What exactly is the system trying to accomplish—and who gets to define success?
An AI system can generate plans, choose tools and evaluate observations. None of those abilities automatically give it authority to invent the mission.
Five Things That Look Similar but Are Not
What We Want
The desired outcome.
What Must Remain True
Boundaries the system must respect while pursuing the goal.
How We Recognize Completion
Evidence or conditions that count as achieving the mission.
When Work Ends
Rules for completing, pausing, escalating or abandoning further work.
How We Check
The mechanism used to determine whether the success criteria actually hold.
A Concrete Mission
Constraint: Do not modify production.
Success criterion: Identify the likely cause and support it with sufficient evidence for a useful report.
Stop condition: Stop when evidence is sufficient, further access is unauthorized, or a human decision is required.
Verification: Compare service state, logs, configuration diagnostics and external behavior.
These statements do different jobs. Mixing them together makes system behavior harder to control and harder to reconstruct.
Goal ≠ Plan
↓
PLAN 1: CHECK SERVICE → LOGS → CONFIG TEST
NEW EVIDENCE
↓
PLAN 2: CHECK DNS → PROXY → EXTERNAL PATH
The plan changed. The goal did not.
A goal says where we are trying to get. A plan says how we currently propose to get there.
Constraint ≠ Inconvenience
Suppose the agent discovers that restarting the server would be a fast diagnostic test. The mission says:
The constraint is not merely another factor for the model to trade away because restart looks useful.
The correct response is to find another route, report the limitation, or request approval if the system supports that path.
Hard Constraints and Preferences
Not every instruction has the same strength.
| Type | Example | Behavior |
|---|---|---|
| Hard constraint | Do not modify production. | Must not be traded away for convenience. |
| Permission boundary | Read logs, but no write access. | Enforced by system authorization. |
| Preference | Prefer the fastest diagnostic route. | Can guide choices among allowed alternatives. |
| Resource limit | Use no more than ten tool calls. | Constrains planning and stop behavior. |
| Quality target | Support conclusions with evidence. | Shapes success and verification. |
A robust system should not treat a hard constraint as merely a low-weight preference.
Where Do Goals Come From?
Depending on the system, mission goals may come from several layers:
Human Request
The user states the desired outcome.
Application
The surrounding software defines a task or workflow.
System Instructions
Higher-level rules define behavioral boundaries and priorities.
Organization / Policy
Permissions, compliance rules and operational policy restrict what may be done.
The deployed assistant therefore operates inside a hierarchy of instructions and controls. The model is one component interpreting that information; it is not automatically the ultimate authority over it.
The Model Can Propose Subgoals
A model may decompose:
↓
SUBGOAL A: ESTABLISH HOST STATE
SUBGOAL B: ESTABLISH SERVICE STATE
SUBGOAL C: FIND FAILURE EVIDENCE
SUBGOAL D: VERIFY DIAGNOSIS
These subgoals are useful inventions of the planning process. But they should remain subordinate to the original mission and its constraints.
Goal Drift
Over a long agent loop, intermediate work can begin to dominate the original purpose.
Some later actions might be useful in another mission—but they are no longer obviously part of the requested diagnostic task.
This is goal drift: the working activity moves away from the original objective.
Mission State Should Preserve the Anchor
Constraints: Read-only production access.
Current subgoal: Determine whether configuration error explains service failure.
Current evidence: Log event E17, config test E22.
Remaining question: Does external behavior match the diagnosis?
Keeping the original goal and constraints alongside current state makes drift easier to detect.
What Does “Done” Mean?
An agent cannot reliably stop if completion is undefined.
Compare:
Weak
“Investigate the outage.”
Stronger
“Identify the likely cause, support it with evidence, preserve unresolved uncertainty, and produce a report without modifying production.”
The second statement gives the system much more information about success.
Success Is Not “The Agent Stopped”
An agent may stop because:
Those outcomes should not all be recorded as mission success.
Success Is Not “The Tool Returned 0”
Likewise:
Page 36 already gave us the distinction between execution success and outcome success. Mission success sits one level higher still.
Success Can Be Partial
Sometimes the system cannot satisfy the entire mission.
Verified: error reproduced with non-destructive config test.
Not established: who introduced the change.
Blocked: audit source unavailable under current permissions.
A useful final state may therefore be partial completion with explicit known gaps, not a fabricated complete answer.
Uncertainty Can Be a Valid Final Result
“Unknown from available evidence” can be a successful answer to an investigation.
If the mission is to determine what the evidence supports, refusing to invent certainty may satisfy the mission better than producing a confident unsupported conclusion.
Who Evaluates Success?
Different systems use different mechanisms.
Rule / Program
Machine-checkable condition: file exists, test passes, record matches.
External Verifier
Another tool or system checks the claimed result.
Model Evaluation
A model assesses whether criteria appear satisfied; useful but not infallible.
Human Judgment
A person decides whether the mission outcome is acceptable.
For high-consequence or ambiguous tasks, human authority may remain the final decision point.
Self-Declared Success Is Weak Evidence
A model saying:
is generated text. It is not, by itself, verification.
Claimed success and verified success are different evidence categories.
Success Criteria Should Be Observable
“Do a good job” is difficult to verify. Better criteria connect to observable outcomes.
| Vague | More Observable |
|---|---|
| Fix the problem well. | Service starts, health check passes, expected page returns successfully. |
| Research thoroughly. | Required sources examined; conflicts documented; claims traceable to evidence. |
| Finish the report. | Required sections present; evidence links preserved; unresolved gaps identified. |
Conflicting Objectives
A mission may contain competing preferences:
These cannot always be maximized simultaneously. The system needs priorities, policies or human guidance.
But this is different from violating a hard constraint. A preference tradeoff is not permission to cross an authorization boundary.
Optimization Is Not Authority
A model may identify a faster or more efficient route. That does not give it authority to change the goal or discard constraints.
Multi-Agent Mission Success
In a multi-agent system, individual agents can succeed while the overall mission fails.
Agent B: completed log analysis ✓
Agent C: external verification failed ✗
Mission: insufficient evidence for final diagnosis — not yet complete.
Local task success and global mission success are separate levels.
Delegation Must Preserve Constraints
If a supervisor delegates work, the receiving agent needs the relevant mission boundaries.
A dangerous handoff would send only:
while silently dropping the parent mission's “do not modify production” constraint.
Constraint propagation is therefore part of multi-agent coordination.
Replanning Must Preserve Mission Identity
Page 37 taught us that plans should change when evidence changes. But replanning should not silently rewrite the mission.
↓
PLAN P1
↓ new evidence
PLAN P2
↓ new evidence
PLAN P3
G1 + C1 REMAIN THE ANCHOR UNLESS AUTHORIZED INPUT CHANGES THEM
Humans Can Change the Mission
Suppose the user says:
That is not agentic goal drift. It is an authorized change to the mission constraints.
The system should update mission state and record who changed what and when.
WITNESS: Mission Definition Is Evidence
To evaluate an AI system after the fact, WITNESS needs more than the final output. It needs the mission against which behavior should be interpreted.
Without the original mission, investigators can observe actions but may not be able to judge whether those actions were relevant, permitted or successful.
Mission Status Should Be More Than PASS / FAIL
| Status | Meaning |
|---|---|
| Completed — verified | Defined success criteria were met and checked. |
| Completed — with known gaps | Requested deliverable produced while explicit uncertainties remain. |
| Blocked | Required permission, resource or dependency unavailable. |
| Stopped by policy / human | Work intentionally terminated by authority. |
| Failed | System could not achieve required outcome. |
| Cancelled | Mission withdrawn before completion. |
The Deep Question: Who Decides “Done”?
At the lowest level, software may test a condition. At another level, an orchestrator may decide enough evidence has been collected. A model may recommend stopping. A human may accept or reject the result.
But the authority to define the mission comes from the system's legitimate instruction and control structure—not merely from whichever component can generate the most persuasive sentence.
The AI may help determine whether the success criteria have been met. It should not quietly appoint itself the author of those criteria.
The Boundary
Goal: desired outcome.
Constraint: boundary that remains in force while pursuing the outcome.
Success criterion: condition used to recognize mission completion.
Stop condition: condition under which work ends, pauses or escalates.
Verification: evidence-producing check of the claimed outcome.
A model can propose subgoals and plans, but those remain subordinate to the mission.
Replanning changes the route; it does not automatically change the destination or the rules.
Mission success is not “the AI stopped.” Mission success means the defined outcome was achieved to the degree the evidence supports, within the constraints that governed the work.
HAAW Checkpoint
Goal: The desired mission outcome.
Constraint: A boundary that restricts how the goal may be pursued.
Hard Constraint: A requirement that must not be traded away as a mere preference.
Preference: A desired property used to choose among otherwise acceptable options.
Success Criterion: A condition indicating that the requested outcome has been achieved.
Stop Condition: A rule indicating when work should finish, pause or escalate.
Goal Drift: Movement of activity away from the original mission objective.
Constraint Propagation: Carrying relevant parent-mission boundaries into delegated work.
Local Success: Completion of an individual task or subgoal.
Mission Success: Satisfaction of the overall mission criteria within applicable constraints.
Claimed vs Verified Success: Saying the mission succeeded is not the same as independently establishing that its success criteria hold.
Central lesson: The system may plan the route, but the mission defines the destination and the boundaries of legitimate travel.
Authority, Escalation and Human Control — When Does the AI Have to Stop and Ask?