New benchmark run

Benchmark a model on an OpenAI-compatible endpoint.

Endpoint

Base URL

Usually ends in /v1.

API key (if the endpoint needs one)

Stored encrypted, never shown again.

Custom headers

Optional. One per line, Name: value — the same syntax as the CLI's --header. A custom Authorization overrides the API key above. Header values are kept with the run in clear — unlike the API key, which is encrypted. Put a real secret in the API key field.

Model

Discovered from the endpoint when it offers a list. Typing your own always works.

Thinking

automatic (per problem) always on always off

Reasoning level

Sent as reasoning_effort, and only on problems that are allowed to think. What a level means is the model’s own business — several chat templates collapse them into fewer buckets.

Suite

What to run

Complete run Custom run

The whole suite, one pass — the only kind that can go on the leaderboard.

Types

Execution

Concurrency

Auto-tune

Timeout (s)

Server backends

How many inference backends sit behind the endpoint. The limiter expects per-request rates to divide by this, so it is not read as congestion.

Minimum parallelism

Floor for the auto-tuned allowance, which moves between this and Concurrency. Set it to your backend count to keep every backend fed.

Time is measured and reported; it does not affect the score.

Scoring

Price extra interaction turns (thinking cost is always reported, never deducted)

Efficiency thresholds

Reasoning budget / turn

Turn floor

Safety net: hard reasoning cap

Cut the stream after

A guillotine, not a tuning knob: past the zero threshold the score is already nil and the generation is only holding a server slot. To ask for less deliberation, use the reasoning level above.

Run name

Complete run

Problems: —
Queue: —

Top models

full board →

Best score per model on the suite you have selected — what you are about to measure against.

—