Moss

Teams building voice agents, copilots, and other real-time AI systems rely on external vector databases like Pinecone, Qdrant, or Chroma for retrieval. These require a network round trip for every query, adding 100-500 ms of latency before inference can even begin, which the page says breaks the illusion of a natural, real-time conversation. Benchmarks on the page show Moss returning results in 3-5 ms compared to 350-950 ms for ChromaDB, Qdrant, and Pinecone on the same 100K-document test. This retrieval delay, not the LLM itself, is framed as the actual bottleneck in production voice and agent systems.

Moss is a retrieval layer that runs directly inside the same process as the AI agent, compiled to WebAssembly so it can execute in the browser, on edge runtimes, on device, or in the cloud without a required network connection. By eliminating the network hop to an external vector store and embedding API call on the critical path, it claims to cut end-to-end retrieval latency to under 10 ms, with indexes syncing automatically when connectivity is available. It installs via an SDK (npm install @moss-dev/moss or pip install moss) and integrates with existing stacks including LangChain, DSPy, Vercel AI SDK, LiveKit, Pipecat, and ElevenLabs, so teams keep their LLM and voice stack and only change where retrieval happens. The FAQ also describes a privacy-by-default architecture, keeping data on-device or within the customer's own infrastructure, which it positions as supporting SOC 2 and HIPAA compliance paths.