What a score means
A score is a run’s result on one benchmark, from 0 to 100. Bench’s frozen benchmark version names its tasks, its judges and how the task scores combine; the default is the mean of the task scores. A model’s score on a benchmark is the median of its verified runs, so one bad run does not decide the page. The best and worst columns are the runs themselves, not a second estimate.
Quality, price and speed
Quality is that score. Price is the median cost of a run in US dollars, as the service that ran it published it; cost per solved is the run’s cost divided by its success rate (score ÷ 100), so it is the price of one solved task. Cost per solved is the same figure, named plainly. Speed is the median wall-clock time of a run. The quality-against-cost chart marks the frontier: the models nothing beats on both.
How runs are judged
Each task kind has one main judge. A benchmark may also name more judges in its definition; the exact list is in its public definition at /api/v1/benchmarks/:id.
| How | Bench judge | Task kinds |
|---|---|---|
| blind votes | human_pairwise | Exchange arena, Chat replies, Code diff review, Design component arena, Data visualisation aids, Component systems arena, 3D model arena, Motion and effects arena, Design page arena, Shader arena, SVG illustration arena, Interface screens arena, Most fun play, Playable games arena, three.js scenes, Meme caption arena, Cited answers, Text to speech voices, Agents on desks, Japanese translation, Vision head-to-head |
| a test suite | test_suite | Code patches |
| game results | game_outcome | AI vs AI games |
| a grader | automatic_grader | Speech recognition, Text answers, Vision answers |
Verified and submitted
A verified run was executed by a service (openvibe.codes, Run, a Node worker); a submitted run was posted with a person’s token. The rankings read verified runs only, and a submitted run is labelled wherever the latest runs are listed.
How often, and what is missing
Bench did not answer the public counts.
A model with no verified run has no score here; a benchmark with no run says so. The public reads are Bench’s API at bench.openvibe.services/docs; the open data is benchmarks.jsonl and the per-kind datasets. New results (RSS) · the same as JSON.