Platform · 02 BenchmarkIRIS8
Scored on your cases before it goes live
You set the pass mark, and we report the scores against it whether the model passes or not.
The pass markIRIS8
Move the pass mark and see
A version below your pass mark is held, whatever it scored yesterday.
- Test set
- 500 held-out cases
PassedAt this pass mark, v3 passes and waits for your sign-off.
What happensIRIS8
Five things written down first
Before the first week ends, these are agreed in writing.
- The task
- The test set
- The metric
- The pass mark
- The model we compare against
Benchmark report · loan-file-qa v3
- Task
- Loan file QA
- Test set
- 500 held-out cases, never used in training
- Metric
- Answers a reviewer marks correct
- Compared with
- The frontier model in use today
- Result
- 95 against a pass mark of 92
- Signed off by
- Head of Credit Operations
- Filed in
- Model record
Your decisionsIRIS8
Your decisions, our work
Take this with youIRIS8
Benchmark sign-off sheet
One page per model version. Use it with us or with any other supplier.
Benchmark sign-off sheet
- Model and version
- Task, in one sentence
- Test set: source and date frozen
- Test set: number of cases
- Used in training?NoYes (stop here)
- Metric, and how it is scored
- Pass mark, and who set it
- Comparison model, and its score
- This version's score
- Weak spots found in testing
- DecisionReleaseHold
- Previous version kept for rollbackYesNo
- Signed, role, date
iris8.ai/platform/benchmark.html
QuestionsIRIS8
Questions
Talk to usQ1What happens if the model misses the pass mark?
It is not released. You get the report with the scores and the reasons it fell short.
Q2Who builds the test set?
We build it from your real cases with your subject experts, and it is kept out of training.
Q3Can we change the pass mark later?
Yes. The change is recorded, and every later version is scored against the mark in force.
Q4Is the benchmark run again after go-live?
Every new version runs the same benchmark before release.
Q5Do you test for misuse as well as accuracy?
Tell us the risks your reviewers care about and we add test cases for them.
Next stepIRIS8