Agents on Rails is the Rails Foundation’s LLM benchmark. Evil Martians designed its evaluation tasks and the tooling to run trials at scale. On our blog, we’re going behind the scenes. First up: how we wrote black-box checks for apps whose code we’d never seen, to keep trials fair and results reliable. Bonus: the same evals run against Rust.

The Agents on Rails Files: designing black-box evals for coding agents