How to Train an LLM on Your Own Data: Fine-Tuning, RAG, or Both?
Train an LLM on your own data without sending it out. When to use fine-tuning vs RAG, the six steps, where your data goes, and what to ask before production.
When a team asks how to train an LLM on your own data, they usually mean something narrower than it sounds. They don't want to build a model from scratch. They want a model that knows their policies, documents and tone, and they want the data to stay inside their own systems. This guide explains the three ways to do that, how to choose between them, and what to check before anything reaches production.
Three ways to use your own data
1. Put it in the prompt. You paste relevant text into each request. It needs no training and works today. It does send that text to whoever runs the model, and it is limited by the model's context window.
2. Retrieval-augmented generation (RAG). You store your documents in a search index. For each question, the system finds the most relevant passages and gives them to the model with the question. The model's weights don't change. The method was described by Lewis et al. in 2020 (Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks).
3. Fine-tuning. You continue training an existing open-weights model (such as Llama, Mistral or Qwen) on examples from your own work, such as past claims decisions or answered support tickets. The model's weights change, so the behavior is built in. Methods like LoRA train a small set of extra weights instead of the whole model, which cuts the compute needed (Hu et al., LoRA: Low-Rank Adaptation of Large Language Models). Preference methods like DPO teach the model which of two answers your reviewers prefer (Rafailov et al., Direct Preference Optimization).
Fine-tuning vs RAG: how to choose
| Your situation | Start with | Why |
|---|---|---|
| Answers must quote current documents (policies, circulars, contracts) | RAG | Update the index, not the model. Answers can cite the source passage. |
| Documents change weekly | RAG | Retraining every week is slow and costly. |
| The task is a repeated format: classify, extract, summarize, draft in a house style | Fine-tuning | The behavior is learned once; prompts get shorter. |
| You want a smaller, cheaper model to do one job a large model does now | Fine-tuning (often distillation) | On narrow tasks, small fine-tuned models can match large general ones (see below). |
| You need both current facts and a fixed format | Both | Fine-tune for the task, retrieve for the facts. |
In our work, most production systems end up using both.
Can a small model really match a frontier model?
On narrow tasks, public studies say it can:
- Predibase fine-tuned 310 small models (2–8 billion parameters) with LoRA across 31 tasks. On average, the fine-tuned models scored 10 points higher than GPT-4, and 224 of the 310 beat GPT-4 on their task. GPT-4 still won on broad tasks such as coding and general knowledge (MMLU) (Zhao et al. (Predibase), LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4).
- Google researchers trained a 770-million-parameter T5 model with "distilling step-by-step" that outperformed the 540-billion-parameter PaLM model on a benchmark, using 80% of the available training data (Hsieh et al., Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes).
In our work, specialised models perform on par with frontier models on narrow, domain-specific tasks. Results depend on the task, so we benchmark every model on your own cases before it goes live.
The six steps
- Pick a base model. Choose an open-weights model whose license allows your commercial use, sized to your hardware and latency needs. Check the license terms yourself; they differ by model.
- Prepare the data. Collect real examples of the task: inputs and the outputs a skilled employee would give. Remove or mask personal data you don't need. Plan for this to take the most time: in Anaconda's 2020 survey of 2,360 data professionals, respondents said about 45% of their time went to loading (19%) and cleaning (26%) data, more than model selection, training and deployment combined (about 34%) (Anaconda, 2020 State of Data Science).
- Fine-tune. Run training (for example LoRA, then DPO) inside your own environment. Version every run, so you can say exactly which data produced which model.
- Benchmark on your tasks. Score the model on a held-out set of your own real cases, side by side with the frontier model you use today. Agree the pass mark before you start, not after.
- Deploy where your data lives. Serve the model in your VPC (AWS, Azure, GCP), IRIS8-hosted, or on-prem as a scoped data-centre install in a managed engagement. Pin versions and keep a rollback path.
- Keep improving. Capture reviewed production outputs, retrain on a schedule, and promote a new version only if it passes the same benchmark.
How many examples do you need?
Fewer than most teams expect, if they are good ones:
- OpenAI's fine-tuning guide sets a minimum of 10 examples, says it sees improvements from 50–100, and recommends starting with 50 well-made examples. If 50 examples change nothing, it advises rethinking the task before adding more data (OpenAI, Supervised fine-tuning guide (Right number of examples)).
- The LIMA study fine-tuned a 65-billion-parameter LLaMA model on 1,000 carefully chosen examples. In human comparisons, its answers were equal to or preferred over GPT-4's in 43% of cases (Zhou et al., LIMA: Less Is More for Alignment).
- In the Predibase study, the number of training examples did not predict how much fine-tuning helped, which suggests quality and task fit matter more than volume once you have enough (Zhao et al. (Predibase), LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4).
So start with 50 to a few hundred real, reviewed examples and a benchmark, then add data only where the scores show gaps.
Where your data goes at each step
This is the question your compliance team will ask, so answer it step by step:
- Data prep and training: if training runs in your cloud account or data center, the training data does not need to leave it.
- Inference: a self-hosted model processes prompts on your own servers. With a third-party API, each prompt goes to the provider.
- Logs and feedback: check where prompts and outputs are stored, who can read them, and for how long.
Rules such as GDPR (GDPR, Regulation (EU) 2016/679) and HIPAA (HHS, HIPAA Privacy Rule) set requirements on how personal and health data is processed and shared. Self-hosting does not meet those requirements by itself; it makes the data flow easier to document and control.
What it costs, and when it doesn't pay
A self-hosted model has fixed costs: GPUs, storage, and people to run it. An API charges per use. At high, steady volume, a small self-hosted model can cost less than API calls. At low volume, an idle GPU costs money anyway.
So the honest answer is: it depends on your volume. Work it out with your own numbers: monthly tokens, current API bill, and the GPU size the model needs.
What to ask any vendor
- Who owns the trained weights at the end of the contract?
- Can training and inference run entirely in our account or data center?
- Which base model, and what does its license allow?
- How will you benchmark it against what we use today, on our cases?
- What records do we get for audit: data used, training runs, eval results, sign-offs?
- What happens when the base model is updated or deprecated?
Start with one workflow
IRIS8 runs an 8-week pilot on one workflow: we pick the base model, fine-tune it on your data inside your environment, benchmark it against the model you use now, and deploy it in your VPC or on-prem. You keep the weights. Success criteria are agreed before we start.
Book a pilot scoping call · See how the 8-week pilot works
Related: Private LLM for enterprise · Deploy an LLM in your own cloud or on-prem · Custom fine-tuning
Frequently asked questions
Can I train an LLM on my own data without sending it to OpenAI or another provider?
Yes. Fine-tune an open-weights model and run it in your own cloud account (customer VPC), or on-prem as a scoped data-centre install in a managed engagement, where the data stays in your environment.
Is fine-tuning better than RAG?
Neither is better in general. RAG suits facts that change; fine-tuning suits repeated tasks and formats. Many systems use both.
How much data do I need to fine-tune?
Often less than you think. OpenAI recommends starting with 50 good examples (OpenAI, Supervised fine-tuning guide (Right number of examples)), and the LIMA model was tuned on 1,000 (Zhou et al., LIMA: Less Is More for Alignment). Quality matters more than volume.
How long does it take?
IRIS8's pilot target is 8 weeks from first workload to an owned specialist model in production (IRIS8). That is a target, not a guarantee.
Will a fine-tuned model make things up?
It can. Benchmark it on your own cases, and use retrieval with citations where answers must match a source document.