Introducing custom Benchmarks by runloop

Use Custom Benchmarks from test cases you select, or create new ones from scratch. Discover and use test cases from other benchmarks or make something new. With Runloop you can easily create test cases from your git history

Developing agents is an iterative process.
Runloop accelerates the test loop by running enormous test suites at scale so that you can discover the perfect prompt that optimizes agent performance on the problems you care about.

The rate of change for models and agents is nothing short of remarkable. However, not every change is beneficial. Custom benchmark runs integrate into your release workflow to ensure rollouts meet quality thresholds.

Create test cases from your git repo in just a few lines of code, or take a working repository and introduce changes. Runloop accelerates sophisticated training algorithms like SFT and RFT by helping you create test cases from your own data.

Take existing test cases and redefine what success looks like. Measure & improve your agent's performance along different dimensions, from well established metrics like cost & security, to subjective measures like word choice and tone.

Train your agent to adhere to relevant rules and regulations. Use Custom Benchmarks teach your agent how to write code that adheres to your industry's laws while running your code in a private, isolated sandbox. Your data is secure and protected by default.