Kimpton

Teams building frontier models and agents need to know how those systems behave in realistic, extended tasks involving planning, memory, multi-agent coordination, and decisions under uncertainty, not just static benchmark scores. Standard evaluation methods may not capture how a model recovers from mistakes, adapts over long action sequences, or performs in domains like finance, life sciences, cybersecurity, or physical systems. Without this, it is hard to compare checkpoints or validate whether a model's predictions and actions hold up in conditions resembling real deployment.

Kimpton runs models inside live environments and records the resulting trajectories, which can be used for evaluation and for training new model checkpoints. Its EvalRouter product lets users run benchmarks and environments across model providers via a CLI or API, select a model route or connect an OpenAI-compatible endpoint, review compatibility and cost for a versioned evaluation, and export scores as JSON, CSV, or HTML. Its Koliseum offering builds private environments and benchmarks where customers define tasks and scoring, then compare checkpoints using outcomes and recorded trajectories, covering long-horizon tasks, multi-agent evaluation, and world-model research. Featured public environments include a Trading Floor (frontier models trading the S&P 500) and a Poker Room (six-max poker decisions and hand histories), and the company also offers adaptive, generative environments with persistent world state scored by independent graders.