IRIS8 Talk to us

Platform · 02 Benchmark

Scored on your cases before it goes live

You set the pass mark, and we report the scores against it whether the model passes or not.

The pass mark

Move the pass mark and see

A version below your pass mark is held, whatever it scored yesterday.

Benchmark · loan file QASample data
Test set
500 held-out cases
Frontier model94
Open base model71
Your model v395

PassedAt this pass mark, v3 passes and waits for your sign-off.

What happens

Five things written down first

Before the first week ends, these are agreed in writing.

  • The task
  • The test set
  • The metric
  • The pass mark
  • The model we compare against
IRIS8 console · loan-file-qa / benchmarkSample data

Benchmark report · loan-file-qa v3

Task
Loan file QA
Test set
500 held-out cases, never used in training
Metric
Answers a reviewer marks correct
Compared with
The frontier model in use today
Result
95 against a pass mark of 92
Signed off by
Head of Credit Operations
Filed in
Model record

Your decisions

You decideIRIS8 does
The pass markBuilds the test set from your cases, with your experts
Which cases go in the test setRuns every candidate and your current model on it
Who signs offWrites the report, pass or fail
Whether a passing version goes liveHolds any version that misses

Your decisions, our work

Take this with you

Benchmark sign-off sheet

One page per model version. Use it with us or with any other supplier.

IRIS8 · Platform · Benchmark1 page

Benchmark sign-off sheet

  • Model and version
  • Task, in one sentence
  • Test set: source and date frozen
  • Test set: number of cases
  • Used in training?NoYes (stop here)
  • Metric, and how it is scored
  • Pass mark, and who set it
  • Comparison model, and its score
  • This version's score
  • Weak spots found in testing
  • DecisionReleaseHold
  • Previous version kept for rollbackYesNo
  • Signed, role, date

iris8.ai/platform/benchmark.html

Questions

Questions

Talk to us
Q1What happens if the model misses the pass mark?

It is not released. You get the report with the scores and the reasons it fell short.

Q2Who builds the test set?

We build it from your real cases with your subject experts, and it is kept out of training.

Q3Can we change the pass mark later?

Yes. The change is recorded, and every later version is scored against the mark in force.

Q4Is the benchmark run again after go-live?

Every new version runs the same benchmark before release.

Q5Do you test for misuse as well as accuracy?

Tell us the risks your reviewers care about and we add test cases for them.

Next step

Bring one workflow and your current baseline