LLM Observability in Production: What to Actually Log and Measure

When I started building the LLM experimentation platform at Intura, our biggest blind spot was not model quality. It was that nobody could answer simple operational questions: how much did that feature cost us yesterday, why did p95 latency double after a prompt change, and how often were we shipping confidently wrong answers. Observability for LLMs is mostly classic observability with three twists: cost is now a first-class metric, latency is wildly variable, and "correctness" is fuzzy and has to be measured continuously rather than asserted once.

This post is the checklist I wish I had on day one: what to log, how to trace a pipeline, how to run evals as monitoring, and what to actually alert on.

What to log on every LLM call

Treat every model call as a span with structured fields. The set below has covered ~95% of the production questions I get asked.

Cost per request: input tokens, output tokens, and the derived dollar cost using a per-model price table. Do not trust a single hardcoded rate; prices and models change.
Token usage: prompt vs. completion tokens separately. A latency spike is often just a long completion, and you can only see that if the two are split.
Latency: time to first token (TTFT) for streaming, and total wall-clock. Record raw values, not pre-aggregated averages, so you can compute p50/p95/p99 later.
Failure and timeout rates: distinguish provider 429/5xx, your own timeouts, schema/parse failures, and content-filter refusals. "It failed" is useless; "it failed because the JSON didn't parse" is actionable.
Output quality and groundedness: at least a sampled signal — did the answer cite retrieved context, did a judge or rule flag it, did the user thumbs-down it.
Prompt and version tracking: a hash or semver of the prompt template, the model id, the temperature, and the retrieval config. Without this, every other metric is uninterpretable across deploys.

Never log raw prompts and completions to your general logging pipeline without a redaction pass. User PII, secrets, and pasted credentials end up in LLM inputs constantly. Hash or truncate by default and gate full payloads behind a separate, access-controlled store.

Why percentiles, not averages

LLM latency distributions are long-tailed. A mean of 1.2s can hide a p95 of 9s caused by a few long generations or a retry storm against a rate-limited provider. We alert on p95 and p99, not the average, and we always slice by model and prompt version because a "global" latency number blends three workloads that have nothing to do with each other.

A decorator that records call metrics

Here is a trimmed version of the instrumentation we wrap around model calls. It is a decorator so it stays out of business logic, it computes cost from a price table, and it emits one structured record per call. The same shape works as FastAPI middleware if you prefer per-endpoint capture.

import time
import uuid
import functools
import structlog
 
log = structlog.get_logger()
 
# USD per 1K tokens; keep this in config, not code, and version it.
PRICE_PER_1K = {
    "model-a": {"in": 0.003, "out": 0.015},
    "model-b": {"in": 0.0005, "out": 0.0015},
}
 
def cost_usd(model: str, in_tok: int, out_tok: int) -> float:
    p = PRICE_PER_1K.get(model, {"in": 0.0, "out": 0.0})
    return (in_tok / 1000) * p["in"] + (out_tok / 1000) * p["out"]
 
def observe_llm(prompt_version: str):
    def deco(fn):
        @functools.wraps(fn)
        def wrapper(*args, model: str = "model-a", **kwargs):
            req_id = str(uuid.uuid4())
            t0 = time.perf_counter()
            status, in_tok, out_tok, err_kind = "ok", 0, 0, None
            try:
                resp = fn(*args, model=model, **kwargs)
                in_tok = resp.usage.input_tokens
                out_tok = resp.usage.output_tokens
                return resp
            except TimeoutError:
                status, err_kind = "error", "timeout"
                raise
            except Exception as e:
                status, err_kind = "error", type(e).__name__
                raise
            finally:
                latency_ms = (time.perf_counter() - t0) * 1000
                log.info(
                    "llm_call",
                    request_id=req_id,
                    model=model,
                    prompt_version=prompt_version,
                    status=status,
                    error_kind=err_kind,
                    input_tokens=in_tok,
                    output_tokens=out_tok,
                    latency_ms=round(latency_ms, 1),
                    cost_usd=round(cost_usd(model, in_tok, out_tok), 6),
                )
        return wrapper
    return deco
 
@observe_llm(prompt_version="summarize@v3")
def summarize(text: str, *, model: str = "model-a"):
    return client.responses.create(model=model, input=text)

The important details: the finally block guarantees a record even on exceptions and timeouts (so your failure rate is real), cost is derived not stored as a magic number, and prompt_version rides along on every record so dashboards can group by it.

Structured logging is the whole game

That structlog call above is doing the heavy lifting. If you log a human sentence like "LLM call took 1.2s", you cannot aggregate it. Emit one flat JSON object per call with stable field names, ship it to your warehouse or metrics backend, and every dashboard becomes a query. At Intura these records land in BigQuery, which means a p95-by-prompt-version question is a GROUP BY, not a code change.

SELECT prompt_version, model,
       APPROX_QUANTILES(latency_ms, 100)[OFFSET(95)] AS p95_ms,
       SUM(cost_usd) AS cost,
       COUNTIF(status = 'error') / COUNT(*) AS error_rate
FROM llm_calls
WHERE _PARTITIONTIME >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 DAY)
GROUP BY prompt_version, model
ORDER BY cost DESC;

Tracing across the pipeline

A single user request is rarely one model call. A typical RAG path is: retrieve, rerank, build prompt, call model, validate output, maybe a repair call. If you only instrument the final generation, you will misattribute a retrieval timeout to "the LLM being slow." Propagate one trace_id from the entry point and make every stage a child span carrying its own latency and token counts. The questions a trace must answer:

Where did the time go — retrieval, the model, or your own post-processing?
How many model calls did one user request actually trigger? Retry and self-repair loops quietly multiply cost.
Which retrieved chunks were in context when the answer went wrong? You cannot debug groundedness without this.

OpenTelemetry semantics map cleanly here. Use span attributes (gen_ai.request.model, token counts, prompt version) so the same trace data feeds both a tracing UI and your cost dashboards.

Evals as monitoring

This is the part teams skip and regret. A test suite you run once in CI tells you the prompt was fine on the day you merged it. Production drifts — inputs change, the provider silently updates a model, your retrieval index goes stale. So I run evals continuously against a sample of live traffic.

Two layers work well together:

Cheap online checks on every (or sampled) request: schema validation, refusal detection, and a groundedness heuristic such as the fraction of claims supported by retrieved context. These are fast and run inline.
A periodic batch eval: an LLM-as-judge or labeled golden set scoring a sample for correctness and helpfulness, written back as a quality metric over time.

def groundedness_score(answer: str, contexts: list[str]) -> float:
    # Sampled, cheap heuristic; the batch judge is the source of truth.
    sentences = [s for s in answer.split(".") if s.strip()]
    if not sentences:
        return 1.0
    supported = sum(
        any(_overlaps(s, c) for c in contexts) for s in sentences
    )
    return supported / len(sentences)

The point is to turn quality into a time series you can chart next to latency and cost, so a regression after a prompt edit shows up as a line going down, not a customer complaint a week later.

Tag every eval result with the same prompt_version you log on calls. When quality drops, you want to bisect to the exact prompt or model change in one query, the same way you would bisect a code regression to a commit.

Alerting that respects on-call sanity

Alert on symptoms users feel, not on every wobble. The rules that earned their keep for us:

Error/timeout rate over a short window crossing a threshold (for example, sustained above a few percent for 5 minutes), split by error kind so a provider outage and a parse-failure bug page differently.
p95 latency exceeding the SLO for a given prompt version.
Cost burn rate — a sudden jump in spend per minute usually means a retry loop or a prompt that ballooned the context, and catching it in minutes instead of at the monthly invoice is worth a lot.
A quality-metric drop from the batch eval, routed to the owning team rather than to pager.

Borrow circuit-breaker discipline from regular service work: when a provider degrades, trip the breaker, fail fast or fall back to a cheaper model, and emit a distinct metric for it instead of letting timeouts pile up and inflate latency for everyone.

Takeaways

Log every call as one structured JSON record: tokens, derived cost, latency, status, error kind, and prompt version.
Measure p50/p95/p99 from raw values and always slice by model and prompt version.
Trace the whole pipeline with one trace_id so you can attribute time and cost to retrieval vs. model vs. your own code.
Run evals continuously as monitoring, not just once in CI, and chart quality next to cost and latency.
Alert on user-facing symptoms — error rate, p95, cost burn — and protect the system with circuit breakers and fallbacks.
Redact by default; gate full prompts and completions behind a separate, access-controlled store.