telemetry.dev

Teams building AI agents and LLM-powered applications struggle to see what happens inside a multi-step agent run: which model calls were made, which tool steps executed, where retrievals happened, and where time and money were spent. Without this visibility, debugging a slow or failed step, a tool timeout, or an unexpected spike in token cost becomes guesswork. MCP-based tool traffic adds another layer of opacity, since failures and latency by method or tool are hard to see without dedicated tracing. Teams also need to balance this visibility against the risk of storing sensitive prompts and completions.

telemetry.dev groups model calls, tool steps, and retrievals into a single trace per agent run, showing span-level duration, token usage, and cost. It offers MCP analytics to show spans, failures, and p95 duration by tool and method, email alerts for cost, budgets, agent loops, and model or tool errors, and prompt management for versions, labels, and rollouts without a redeploy. It also supports evals that trace each run with its cost and errors. Integration is via first-party TypeScript and Python SDKs, provider-specific packages (OpenAI, Vercel AI SDK), or any OpenTelemetry OTLP/HTTP exporter, and the product includes a read-only MCP server so an AI agent can investigate errors and cost directly. Per-environment prompt capture can be disabled before storage, with built-in and project-specific redaction patterns.