How to Monitor AI Costs, Latency, and Quality

An AI feature needs more than a monthly invoice and a demonstration that sounds convincing. Its operators need to know what each task costs, how long users wait, and whether the response meets the task's standard. These three dimensions can move in different directions: a cheaper model may need more retries, while a faster answer may omit required evidence. Monitoring them together makes tradeoffs visible without pretending that one score captures the whole product.

Define the unit of work

Choose a task boundary that users recognize, such as one support summary or one document classification. Count every model call, retrieval step, retry, tool call, and human correction needed to complete that task. Per-request cost is useful for debugging but can hide the cost of a workflow that makes several requests. Separate test traffic from production, and record the model, configuration, prompt version, and application revision. Avoid putting prompt text or personal data in metric labels.

Agree on a quality outcome before adding charts. A classification task might require a valid label and supporting span; an answer task might require a citation that actually supports its sentence. For open-ended content, use an explicit review rubric and a sample of real tasks with permission to inspect them. Human reviewers should be able to mark an answer wrong even when its format is valid. Record uncertainty where no automatic check can establish factual correctness.

Measure cost at the right boundary

Collect provider usage fields when available, apply the relevant price schedule for the recorded model and date, and distinguish estimated from invoiced cost. Include input, output, cache, and tool charges where the provider bills for them. Track task completion rate and cost per completed task, because a lower per-call price can be offset by failures and repeated calls. If a user abandons the task, account for the calls already made instead of excluding them from the denominator.

Set budgets at several levels: a request guardrail to stop accidental loops, a workflow budget for retries, and an aggregate alert for unusual spend. An alert should point to a diagnostic breakdown: task type, version, model, and retry reason. Do not use a unique user ID as a metric dimension without a privacy review; high-cardinality labels can also make dashboards expensive and hard to read.

OpenTelemetry's metrics documentation describes measurements and attributes used to aggregate them. Its concepts help structure counters and duration distributions, but a metric name alone does not define the business outcome being measured.

Separate waiting time from processing time

Record end-to-end time from user action to usable result, then record component times for retrieval, model response, tools, validation, and presentation. For streaming interfaces, time to first visible token can matter, but completion time still matters when a user needs the whole answer. Show distributions, especially slower cases, rather than only an average. Segment by task type and input size so a new batch of long documents does not masquerade as a service regression.

Add a trace identifier that connects application stages without storing sensitive content in telemetry. When latency rises, the trace should reveal whether requests queued, a dependency slowed, or retries increased. A faster model call does not fix a slow retrieval layer. Conversely, caching retrieval may not help if output generation dominates. Investigate the observed critical path before selecting an optimization.

Evaluate quality with examples and review

Build a small versioned evaluation set containing representative ordinary cases and carefully chosen edge cases. For each example, write the desired behavior and the reason it matters. Run candidate prompt or model changes against the same set, inspect disagreements, and then sample live outcomes under appropriate privacy controls. An evaluation set can become stale or overfit; periodically add cases from observed failure modes without treating the set as a complete picture of users.

OpenAI's evaluation guidance explains how to create and run evals against a target use case. The useful editorial principle is to test specific expected behavior rather than accepting a fluent answer as proof of quality.

Use multiple checks when their failure modes differ. Schema validation catches malformed output; a reference check catches unsupported citations; expert review can catch a plausible but misleading conclusion. None guarantees correctness on unseen inputs. Track review disagreement and retain a path for users to flag errors. If a quality threshold is crossed, define a response: disable a new prompt, route to review, or return a narrower answer.

A hypothetical change review

Imagine a hypothetical assistant that drafts incident summaries. Version B reduces median response time but omits the incident timeline in several difficult evaluation cases. The team records cost per complete summary, completion-time distribution, and rubric results for the same test set. It does not declare B better because it is faster. Instead it examines whether the omission came from a shorter prompt, a retrieval limit, or an output truncation, and repeats the relevant test after a targeted fix.

The example gives a decision method, not a claim about a deployed system. If reviewers disagree about whether the timeline is required, the rubric needs refinement before a release choice. A dashboard cannot resolve an undefined task standard. Qualitative review and operational metrics should inform each other rather than compete for a single headline number.

1. Define completed work and its quality criteria.
2. Record all calls and stages under a versioned task identifier.
3. Aggregate cost and latency by relevant task type.
4. Run versioned evaluations and inspect sampled failures.
5. Choose a change, set rollback criteria, and compare again.

An actionable monitoring checklist

Before launching a change, verify the task boundary, price assumptions, usage fields, retry cap, latency timestamps, and redaction rules. Confirm that dashboards display missing data rather than converting it into zero. Check that an evaluation case includes difficult inputs, that reviewers can explain their judgments, and that a bad answer has a visible reporting path. Write down the decision owner and rollback trigger. Revisit the dashboard after a model or provider change because billing fields and behavior can change.

Keep the interpretation modest. A stable cost chart does not prove answers are reliable, and a high evaluation score does not prove production costs are acceptable. Monitor the actual task and document what each measure leaves out. The result is a workflow that can make informed tradeoffs as conditions change, without pretending its measurements are universal guarantees.