For the Head of AIIRIS8
How do we know it is good enough?
The model is scored on your own cases against the model you use now, at a pass mark you set, before it goes live.
Your briefIRIS8
What you answer for
- 01Quality you can show with numbers from your own work
- 02A model that improves without quietly regressing
- 03A method your risk team will accept
QuestionsIRIS8
Five questions Heads of AI ask us
The answers we give on the first call.
- 01
Will a smaller fine-tuned model match the frontier model we use now?
On some narrow, repeated tasks it can. We do not ask you to take that on trust. The benchmark compares both models on a held-out set of your own cases, and you see the scores before anything goes live. If it misses the pass mark, you get the report and the reasons.
- 02
How is the benchmark built, and who sets the pass mark?
Before the first week ends we write down the task, the test set, the metric, the pass mark and the comparison model. The test set comes from your real cases and is never shown to the model in training. You set the pass mark. Your reviewer signs off the result.
- 03
How do you train it?
We fine-tune a licensed open-weight model inside your environment: low-rank adapters first, then preference tuning on reviewed examples. Each run is versioned and scored. Personal data the task does not need is masked before training. You receive the weights and the record of every run.
- 04
How does it improve without getting worse?
Reviewed feedback from production is curated into new training data. Each new version runs the same benchmark and must pass before you approve it. The previous version is kept for rollback, so a version that scores worse never reaches your users.
- 05
Fine-tuning or retrieval?
Often both. Retrieval supplies facts that change. Fine-tuning teaches the format, the judgement and the vocabulary of a task. We test the options against your benchmark and keep what scores best. The pilot is scoped around one workflow, so the answer is specific to it.
Benchmark and retrain loopIRIS8
Benchmark · loan file QA
- Test set
- 500 held-out cases
- Signed off by
- Head of Credit Operations
Retrain loop
- Reviewed feedback
- New version
- Same benchmark
- Your approval
- Live
Previous version kept
The gate a first version passes is the gate every later version passes.
For your teamIRIS8
Take back to your team
Next stepIRIS8