AI-assisted visual disclosure: the featured image for this post was generated with Google’s Gemini image model on Vertex AI.
Most agent failures do not look dramatic from the outside. The user asks for a report, a ticket update, a refund check, or a code change. The agent returns something plausible. Everyone moves on.
Then the incident review starts. A tool call silently failed. A retry loop burned through tokens. The agent used stale memory. The final answer looked fine, but the database state was wrong. Nobody can explain the path from the user’s goal to the output because the trace stops at “LLM response generated.”
That is why agentic harness observability matters. The model is not the production boundary anymore. The agent run is.
A serious AI agent is not only a prompt wrapped around an LLM. It has tools, memory, policies, retries, context management, evaluation, cost controls, and human approval points. That surrounding system is the harness. If you cannot observe the harness, you cannot run agents with confidence.
Why the harness matters more than the model in production
Early AI demos usually worship the model. Production systems punish that thinking fast.
In a real project, the answer depends on much more than model quality. It depends on which tools the agent was allowed to call, what context was loaded, how the plan was decomposed, whether the retrieval step found the right source, whether the tool result was parsed correctly, and whether a guardrail blocked a risky action.
Anthropic’s agent-eval guidance makes this point clearly: when you evaluate an agent, you are evaluating the model and the harness together. The harness provides the tools, runs the task, records the steps, grades the result, and aggregates what happened across trials.
That matches what teams learn the hard way. The LLM might be smart, but the harness decides whether that intelligence becomes a reliable workflow or a very confident mess.
What to trace in an agentic harness
The first mistake teams make is logging only the final prompt and final response. That is useful for demos. It is not enough for operations.
An agent run should have one stable run_id that ties together the whole story:
- User goal and input context
- Planner decisions and task decomposition
- Model calls, model names, latency, token usage, and finish reasons
- Retrieval queries and source IDs
- Tool calls, arguments, results, errors, and retries
- Memory reads, writes, summaries, and compaction events
- Guardrail decisions and policy blocks
- Eval scores and human review outcomes
- Total cost and cost per successful task
That last line matters. Cost without outcome is just a bill. Outcome without cost is just wishful reporting. A useful harness connects both.
OpenTelemetry gives agent runs a common language
OpenTelemetry is becoming the practical foundation for GenAI observability because it gives teams a shared vocabulary for traces, logs, and metrics.
The OpenTelemetry GenAI conventions describe agent activity as a trace tree. A top-level agent invocation can contain child spans for model calls, tool execution, retrieval, and other work. In practice, that means an engineer can open a trace and see whether latency came from the model, a slow API, a retrieval step, or a retry loop.
A good trace for an agentic system usually includes:
invoke_agent customer_refund_agent
chat gpt-4.1
execute_tool lookup_order
execute_tool check_refund_policy
chat gpt-4.1
execute_tool create_refund_ticket
eval refund_policy_compliance
Do not capture every prompt and tool payload by default. OpenTelemetry’s own GenAI material calls out why: prompts, tool arguments, and tool results can contain sensitive data. Start with metadata such as model, duration, token counts, status, and tool names. Add content capture only where there is a clear debugging need, a retention policy, and access control.
Evals should grade the outcome, not just the sentence
LLM-as-a-judge is useful, but it is not magic. It can grade tone, instruction following, completeness, and policy fit. It should not be the only judge of whether an agent did the work.
For agentic systems, the real question is often: did the environment end in the right state?
If a support agent says, “I have refunded the customer,” the eval should check the order system. If a coding agent says, “I fixed the bug,” the eval should run the tests. If a research agent cites a source, the eval should check whether the source supports the claim.
Use LLM judges where human language needs judgment. Use deterministic checks wherever the system has a real state to inspect.
A practical eval setup has three layers:
- Regression evals: stable tests that catch known failures before release.
- Capability evals: harder tasks that measure whether the agent is getting better.
- Production audits: sampled real runs reviewed by humans, especially for high-risk workflows.
The eval result should land back on the same trace. When a run fails, the team should be able to jump from “policy compliance score dropped” to the exact context, tool call, and decision that caused it.
FinOps for agents means cost per successful task
Traditional cloud FinOps asks who spent what, why, and whether the spend created value. AI does not change that discipline. It makes it more urgent.
The FinOps Foundation frames AI spend around allocation, forecasting, cost optimization, and business value. For agents, the useful unit is not only monthly token spend. It is cost per successful task.
Track these numbers per agent, workflow, team, and customer segment:
- Input, output, and cached tokens
- Model cost by step
- Tool/API cost
- Retry and loop cost
- Latency by step
- Eval pass rate
- Cost per successful run
This is where observability and FinOps meet. A cheap run that fails is expensive. An expensive run that prevents a human escalation might be a bargain. You only know when cost and outcome sit in the same dataset.
Guardrails are part of observability, not a separate checklist
Guardrails should not be invisible. Every blocked prompt injection, risky tool request, policy refusal, and memory-write rejection should appear in the run record.
For agentic systems, the risky parts are not limited to the final answer. A malicious instruction can arrive inside a web page, a document, a tool result, or a memory item. A good harness watches the path, not just the output.
At minimum, log these decisions:
- Why a tool call was allowed or blocked
- Whether a human approval gate was required
- Which memory writes were accepted, summarized, rejected, or quarantined
- Which external content was treated as untrusted
- Which policy rule fired
This creates a security audit trail and a learning loop. If the agent keeps hitting the same guardrail, the team can decide whether the policy is right, the prompt is weak, or the workflow needs a safer tool.
A rollout plan for technical teams
You do not need a giant platform team to start. You need a disciplined first version.
- Define the run boundary. Decide what counts as one agent run and generate a stable
run_id. - Instrument the main loop. Trace planning, model calls, tool calls, retrieval, memory, and retries.
- Add token and latency metrics. Break them down by model, agent, tool, and workflow.
- Attach evals to traces. Store the result beside the run, not in a separate spreadsheet.
- Track cost per successful task. This becomes your north-star FinOps metric.
- Log guardrail decisions. Treat blocked actions as signal, not noise.
- Review failures weekly. Pick the top cost, quality, latency, and safety issues. Fix the harness before blaming the model.
The real lesson
The best agent teams are not the ones with the fanciest prompts. They are the ones that can explain what happened during a run.
OpenTelemetry shows the path. Evals tell you whether the path produced the right outcome. FinOps tells you whether the outcome was worth the cost. Guardrails tell you where the system refused to take a risk.
Put those together and the agentic harness stops being a buzzword. It becomes the operating layer that lets technical teams ship agents they can debug, improve, and defend in production.
