OneTriangle
OneTriangle cuts LLM inference cost and latency by transferring KV cache from a small model's prefill to a large model's
The Problem
Teams running inference on large language models pay large-model rates for the prefill phase, before the model has generated a single output token. Prefill cost scales with the size of the model doing the reading, and on long inputs this prefill phase accounts for most of the total bill. For a 100k-token context on Llama 3.1 8B to 70B, the page shows prefill costing $46.9k per 1M requests and taking 16.9 seconds to first token under the ordinary approach.
The Solution
OneTriangle runs prefill on a small model and decode on a large one, transferring the small model's KV cache into the large model instead of having the large model redo the reading step. Per-layer K/V tensors are stripped of source rotary embeddings, projected head-wise into the target model's attention space, and re-embedded at target positions, so the large model enters decode against a populated cache it never computed itself. Transfers are gated on held-out quality and latency checks for every model pair, and ordinary prefill takes over automatically when a pair doesn't clear that bar. The company also serves open-weight models (Llama, Qwen, Mistral, Gemma, DeepSeek) directly, handling GPUs and serving so customers change one line of config. On the measured example, this cuts time to first token from 16.9s to 2.7s (6.3x faster) and prefill cost from $46.9k to $7.5k per 1M requests (84% cheaper).