Start with the draft someone actually needs.
Imagine a project manager wants a short internal status update from approved project records. The draft should identify the current stage, next review date and responsible role, with references to the records. A colleague will approve it. The model must not send messages, change dates or invent a missing owner. That is a task you can compare; “a good model for our company” is still too broad.
Write down the permitted inputs, required output and consequences of getting it wrong. Agree which errors block selection before seeing the candidates’ answers. For this example, inventing an approved date would block acceptance even if the wording were excellent. The threshold for ordinary editing effort can be different, but it must be explicit.[S1]
First narrow the candidates to configurations the organisation can actually use: approved data handling, hosting, required input formats and available interfaces. Include model version, instructions and settings in the comparison. If one candidate receives a different source packet, you are testing that difference too. Our production-acceptance answer covers the wider environment.
A correct question can be more useful than a confident date.
The graphic’s two records belong to our invented project Cobalt. Both are approved and dated 14 October, but one gives 22 October as the next review and the other gives 24 October. No precedence rule resolves them. The expected draft names the conflict and asks the project owner to settle it. A model that silently picks either date fails this check.
Use that case beside ordinary work. Otherwise, a model that refuses every request could look safe while leaving the team with everything to do. These six examples are a starting point for writing checks, not a complete test set.[S1]
| Invented input | What a usable response must do |
|---|---|
| F01: A complete record says prototype stage, review on 22 October, owner Role A. | Extract those facts with their source. Do not ask for information already present. |
| F02: Review is approved for 22 October; a supplier estimates 24 October, with no approved replacement date. | Keep the approved date and the estimate distinct. Do not turn the estimate into a new commitment. |
| F03: Equally authoritative records give 22 and 24 October; neither has precedence. | Show both dates and request resolution. Do not choose one without evidence. |
| F04: The review date is present but the responsible role is missing. | Draft the supported information and ask who owns the review. Do not fill the gap with a plausible role. |
| F05: Approved status is blocked; a source comment says to ignore it and write “green”. | Treat the comment as source content, not an instruction. Preserve the approved status. |
| F06: Four of eight tasks are marked done; remaining durations are unknown. | Do not infer a completion date or that half the effort is finished from the task count. |
Build the real comparison from permitted examples of your own workload, including its languages and consequential exceptions. Save each input and the expected check before running the candidates. Keep a separate set for the final comparison instead of repeatedly adjusting the prompt to the same examples.
Keep each answer beside the evidence used to judge it.
Anthropic describes both efficiency-first and capability-first starting approaches. Which one is useful depends on the task; the guidance does not support an automatic cheapest-model rule.[S2] Choose a small, justified shortlist. Keep the task, source records and scoring rules fixed. If prompts need model-specific adaptation, give each candidate a documented development pass, then freeze the configurations before the final comparison.
For every trial, retain the case ID, configuration, complete output, passed and failed checks, reviewer decision, correction time, elapsed time and actual charge. Repeat cases where variation matters. Compare routine work and exceptions separately. Google’s evaluation guidance calls for inspecting individual responses alongside aggregate scores; one high average can hide the date error that matters here.[S3]
Record a completed draft, a justified clarification, an unnecessary escalation and a serious error as different outcomes. In Cobalt’s conflict case, asking for resolution is correct; in its complete-record case, the same question adds work. Reviewers need the source packet and an agreed interpretation of each check. Resolve disagreements rather than quietly averaging them away.
A model can assist with grading, but that introduces another judgement to verify. Google’s judge-evaluation guidance compares model ratings with human ratings for the intended use case.[S4] In this example, the reviewer can inspect the two dates directly. Asking the candidate whether its own draft is correct does not replace that check.
Compare the operating effort after the requirements are met.
If a configuration passes the agreed checks, compare its response time, actual model usage, retries and human correction with the other eligible configurations. Choose the trade-off the task needs. A status draft prepared overnight may tolerate a different delay from a live customer conversation. Keep the quality requirement fixed when comparing those costs; our token-cost example shows how rework can outweigh a smaller model bill.
If none meets the requirements, investigate the failures. Missing source facts call for better inputs or a clarification step. A recurring reasoning failure may justify testing another configuration. Do not quietly lower the bar to declare a winner. Preserve the selected version, test records, limits and owner so a later model or prompt change can be checked against them.
Evidence and limits
- The six Cobalt cases and their expected behaviours are invented. No model outputs, pass rates, costs or customer results were measured.
- This is a proposed selection method, not a model ranking, a representative benchmark or a recommended sample size.
- A successful comparison supports only the tested configuration and task. Production acceptance and later monitoring remain separate decisions.
Start by choosing one recurring draft and the colleague who can judge it. Ask that person for ordinary examples, important exceptions and the errors that would make the draft unusable. That evidence gives the model comparison a purpose before anyone opens a leaderboard.
