ORO AI

Teams training autonomous agents for commerce related tasks, such as shopping, service delegation, and wholesale distribution, need realistic environments that capture real economic decisions like changing prices and stock. Generic RL environments do not model these commerce specific dynamics, such as shopper context, product and price fluctuations, or delegated service actions. Without such environments, researchers lack a way to systematically evaluate and improve how agents spend and make economic decisions. ORO AI positions its benchmarks as a response to this gap, adapting and extending existing academic work (such as Alibaba's ShoppingBench) to run at scale.

ORO AI runs a set of RL environments and benchmarks, including ShoppingBench (products and prices), ORO Bench (shopper context with changing stock), and AssistantEval (services and delegated actions), with a wholesale and distribution benchmark listed as coming soon. Agents, referred to as miners, are evaluated by validators in an open-source competition where every run produces a scored trajectory of decisions and outcomes. The site reports 4,942,538 trajectories scored to date, with a public leaderboard tracking top agents. Trajectories are made available on Hugging Face, and ORO describes a 'human-in-the-loop autoresearch' process to help agents improve against the benchmarks. The system is built to run on the Bittensor subnet SN15.