Agentic systems fail across several layers at once. A slow answer may come from retrieval, a model provider, a tool, a human approval queue, or repeated planning turns. A wrong answer may begin with the wrong context rather than the wrong model. Traditional application logs remain necessary, but they need a run-centered data model that follows the workflow across these components.

Observability for a Brain API combines logs, traces, metrics, and evaluations. Logs describe discrete events. Traces connect work across services and agent steps. Metrics reveal trends and service health. Evaluations measure whether the workflow achieved the intended outcome. The system is diagnosable when these signals share stable identifiers and version metadata, not when each team has a separate dashboard with unrelated names.

Choose a run as the primary unit

Assign a run identifier when the Brain API accepts a task. Propagate it through model calls, retrieval, tool execution, memory access, handoffs, approvals, and final delivery. A session can contain several runs, and one run can contain several model turns. Keep those identifiers separate. This hierarchy lets operators inspect one failed task without loading an entire user history.

Add a trace identifier for distributed work and a correlation identifier for client-visible support. If the workflow creates child jobs, preserve the parent relationship. Every event should identify the workflow version, prompt version, route, and environment. Without version metadata, two visually identical failures may come from different deployed behavior and cannot be compared reliably.

Write structured events, not narrative logfiles

A structured event has a stable name and typed fields. Useful examples include context_built, route_selected, model_started, model_completed, tool_proposed, tool_authorized, tool_completed, approval_requested, memory_written, guardrail_blocked, and run_finished. Each event can include duration, status, error class, token usage, bytes, and references to redacted payloads. This is more searchable than a sentence assembled by each developer.

Do not place full prompts, raw documents, credentials, or private tool results in general logs. Store sensitive replay artifacts in a restricted system with separate access and retention. The operational event can record a content hash, artifact identifier, size, and redaction status. This keeps standard analytics useful without turning the logging platform into an uncontrolled copy of user data.

Trace model and tool spans consistently

A run trace should show the sequence and overlap of context retrieval, model inference, tool calls, retries, and validations. Give each model call a span with provider, model route, input and output size, cache status, and finish reason. Give each tool a span with tool name, authorization result, idempotency key, target class, and side-effect status. Nested spans make repeated loops visible immediately.

Record planned and actual timing. Queue delay, provider latency, tool execution, and post-processing should be separate. When a run times out, the trace should reveal which budget was consumed. Sampling can reduce cost for high-volume successful traffic, but retain complete traces for errors, policy blocks, canaries, and evaluation runs. Keep the sampling decision in the trace metadata.

Use metrics for service objectives

Core operational metrics include request rate, success status, latency percentiles, error rate, timeouts, retries, queue depth, and resource saturation. AI-specific metrics can include tokens or compute per successful task, tool-call count, schema-invalid rate, guardrail blocks, approval wait time, and fallback frequency. Segment by workflow and route so one expensive use case does not distort the entire service.

Define service-level objectives that reflect user outcomes. A run can be technically successful but operationally poor if it takes too long, requires repeated correction, or never completes an approved action. Pair availability and latency with task-completion and correction metrics. Avoid vanity metrics such as total tokens generated, which can grow while product value declines.

Attach evaluations to production telemetry

Offline evaluations compare workflow versions on a fixed case set. Online evaluations sample real runs after redaction and score properties such as completeness, evidence coverage, action correctness, and policy compliance. Store evaluation results with the run and evaluator version. A number without its rubric and source case is not reproducible.

Use deterministic checks wherever possible. JSON schema validity, expected tool selection, prohibited tool usage, citation presence, and exact calculations do not require a model judge. Human review remains important for nuanced quality and high-impact changes. Model-based grading can expand coverage, but calibrate it against human decisions and monitor drift. The release dashboard should show both aggregate scores and the concrete cases that changed.

Adopt a common telemetry model

The OpenTelemetry documentation describes a vendor-neutral framework for traces, metrics, and logs. A Brain API can use that common foundation while adding domain-specific attributes for models, tools, prompts, evaluations, and policy outcomes. Standard propagation and export reduce the need to build a separate transport for every observability vendor.

Create a semantic convention for the organization. Name fields consistently, define which values are safe to export, and document cardinality limits. User IDs, prompt text, and document titles should not become metric labels. High-cardinality details belong in traces or restricted events. A small governance group can review new attributes and prevent teams from creating incompatible representations of the same concept.

Turn analytics into operating decisions

Analytics should answer questions that lead to action. Which workflow versions produce the highest correction rate? Which tools cause repeated retries? Where does context size increase without improving success? Which model route is cheapest per completed task? How often do approvals expire? These analyses require clean event semantics and a definition of completion that matches the product.

Build cohorts around releases, tenants, task types, and service classes. Look at distributions rather than only averages. A small group of extremely long runs can create cost and reliability problems even when the median looks healthy. Use anomaly alerts for sudden shifts, then link the alert directly to example traces. Operators should move from a chart to a representative run without manually joining identifiers.

Prepare for incidents and replay

An incident runbook should identify how to disable a model route, tool, workflow version, or memory write path independently. Preserve the minimum records needed to understand the incident, and restrict access to sensitive artifacts. A kill switch that requires deploying new code is too slow. Configuration changes should be authenticated, audited, and reversible.

Replay can run a stored case against a new workflow in a safe environment. Replace side-effecting tools with recorded or simulated results unless the test is explicitly authorized. Compare step sequences, costs, latency, and final outcomes. The developer tools page outlines how traces, QA cases, Git review, and command-line workflows can form one release process.

Implementation checklist

Before the first production release

  • Use separate run, session, trace, and client correlation identifiers.
  • Emit typed events with workflow, prompt, and model-route versions.
  • Keep private payloads in restricted replay storage, not general logs.
  • Trace model, retrieval, tool, approval, and validation spans.
  • Define service objectives around completed tasks and corrections.
  • Attach reproducible evaluation results to runs and releases.
  • Control telemetry cardinality and sensitive attributes.
  • Maintain audited kill switches and safe replay procedures.

Conclusion

Agentic observability is the connective tissue between product quality and system operations. Logs explain events, traces expose the workflow path, metrics show population-level behavior, and evaluations test whether outcomes are useful. Shared identifiers and versions make those signals part of one story.

Begin by defining the run record and a small semantic convention. Instrument the critical path, protect sensitive artifacts, and link evaluation cases to production failures. A Brain API that can explain its own operations is easier to debug, safer to change, and less dependent on intuition when models or providers evolve.