How do I compare models and release one for a task?

Open the task’s Changes tab, compare candidates on the same examples and rubric, inspect their returned answers, and release an eligible model. The release pins the task version and records the comparison when linked, so later requests can be traced to the configuration you chose.

Who can do this: Admins here. Task observers can read every comparison and release.

Start a comparison

Open a task’s Changes tab and pick candidates: up to twelve models with a provider key. The same examples, the same rubric, and the same judge are used for every candidate. Set a budget if you want one: a cap on tokens, cost, or minutes. A run that hits its budget keeps the results it has.

Before queuing work, Orlo checks the saved provider-access information, model status, judge eligibility, and rubric. Known unavailable choices are refused. This preflight does not make a fresh provider call; unverified access can still be tried, and provider failures or charges can occur once the run starts.

How scoring works

Three layers, in order:

  1. Rules. Each answer runs through the task’s rules. Rule results contribute to scoring; inspect failed checks as well as the combined score.
  2. Match. The answer is compared with the expected answer: exact for classification, semantic for free text.
  3. Judge. A model that is never a candidate scores the answer against the rubric. It receives the task rubric, expected answer, and reference passages where configured. With two judges and an agreement threshold, examples where they disagree are flagged for a person.

Each candidate’s score is an average across examples with a 95% confidence interval estimated from the scored examples. A candidate that is clearly behind after enough examples is dropped early to save budget.

Reading the result

The result lists candidates with their score and interval. When two intervals overlap, Orlo says they are too close to call and does not recommend one. That is the honest answer: more examples would narrow the interval; a bigger model would not change the fact.

Open any example to read the answers side by side, the rule results, the judge’s reasoning, and the passages used when the task has sources.

Release

Release pins the selected model and task version and snapshots validation and routing settings, then makes the release live. Sources and retrieval settings are saved with the release. The release records who made it and which comparison chose the model. A release linked to a comparison must use a matching task version and strategy, with an eligible model that returned usable results. Review the answers and your acceptance criteria; eligibility alone is not a guarantee of suitability.

On an advisory task, a release can also be made without a comparison, from first-run setup, from “release with sources,” or through the API. The record then shows no comparison, so a reviewer can tell the two apart.

A release that points at a draft skill or a retired skill version is refused, with the version named.

Undo

Undo an eligible release. Earlier releases remain in history, but an unavailable model or configuration must be replaced.

When a provider retires a model

Replace a deprecated or retired model. The Inbox links to affected releases. A replacement needs a deliberate comparison and release decision.

Require evidence before release

A ranking comparison and a release test answer different questions. Require a passing release test to tie eligibility to a fixed paired test, declared tolerance, and exact candidate configuration. Required tasks refuse releases without qualifying evidence.

Inspect regression regions or test sources on/off when you need to understand the difference.