Observe
Metrics, traces, alerts; exposing what your AI apps and agents are actually doing, to the people who need to see it.
Metrics
golden signals plus tokens, cost, tool calls
Traces
reconstruct what an agent did, with what, on what data
Alerts
routed to the builder who owns the app
What matters
signals that predict incidents, not dashboard decoration
Model behavior
drift, silent provider updates, quality regressions
You cannot govern what you cannot see. Observability for AI covers the classic golden signals plus a new set: tokens, tool calls, model behavior.
The surface:
- Metrics: usage, latency, error rates, token consumption, cost per app / per user / per team.
- Traces: what did the agent do, in what order, with which tools, touching which data.
- Alerts: exposing alerts to the users who own the app, not just a central NOC.
- User-facing observability: builders see their own app’s health; leadership sees the portfolio.
- What matters: separating the signals that predict incidents from dashboard decoration.
- Model behavior: drift, refusal rates, quality regressions when a provider silently updates a model.
Questions to ask: If an agent misbehaved yesterday, could you reconstruct what it did? Who gets paged? Can the person who built the app see its metrics without filing a ticket?
O → C
Observe hands off to Control. Keep a hand on the wheel.