Tokenless

Teams training models with reinforcement learning or running agent evals rely on environments and graders that become more complex as models get more capable. This complexity creates more opportunities for models to find unexpected behaviors and exploits in tasks and graders, a problem known as reward hacking. When these flaws go undetected, training can amplify them as models learn to exploit small weaknesses, and it becomes hard to tell whether performance gains reflect real capability or just better exploitation of the environment.

Tokenless provides tooling and expertise to audit RL environments and evals for reward hacks and grading bugs. Their work has uncovered 110 reward hacks and grading bugs across known benchmarks including Terminal-Bench, DeepSWE, and BFCL. They have introduced EnvCheck, a tool related to this auditing work, and Eval³, described as a benchmark for environment auditors that evaluates how well auditors catch these flaws. The company works with teams training models with RL, building environments, or running agent evals who want more confidence that their environments are free of exploitable flaws.

As models become more capable, training and evaluating them requires increasingly complex tasks and longer interactions, creating more opportunities for unexpected behavior and exploits in tasks and graders.