fabraix

Companies deploying AI agents (chatbots, coding agents, browser agents, voice assistants, RL systems) face failure modes traditional software doesn't have, and because agents are non-deterministic the same input can pass one run and fail the next. Manual red-teaming engagements take weeks, cost six figures, and go stale the moment the audit finishes, since every agent update, prompt change or tool integration can reopen old vulnerabilities. The page cites real examples found in Fortune 500 agents: a coding agent whose auto-approve guardian was tricked into escaping its read-only sandbox, a browser agent that leaked a password-manager credential via indirect prompt injection, an LLM-as-judge bypassed to exfiltrate API keys over network egress, and a real-estate agent jailbroken into discriminatory steering. Without continuous adversarial testing, these gaps surface only after attackers find them.

Nyx is an autonomous, pure blackbox testing harness that probes AI agents across security, logic and alignment without requiring integration, source code, model weights or network access. It runs multi-turn, adaptive attacks (the page claims 10,000+ distinct offensive strategies) that reason across conversation turns rather than firing canned single-shot prompts, and it tests multiple modalities including voice, text, images, and custom files or websites for browser agents. It is described as massively parallel, running thousands of concurrent simulations so coverage scales with compute rather than headcount, and can plug into CI/CD to re-test every agent update. The page also states Nyx's attack success rate on the AgentHarm benchmark beats a named frontier model (78% vs 67% for GPT-5.6 Sol), and describes a self-improving loop where each attack feeds back into the system.