The required example count is a stop rule, not a target
A custom LLM needs enough training examples to meet an acceptance threshold in each operational failure slice, at a marginal labeling cost that still beats the alternatives. There is no reusable answer to how many examples to fine-tune an LLM.
The popular answers are vendor starting points, not completion criteria. OpenAI accepts fine-tuning datasets with as few as 10 examples and frames 50 good examples as a checkpoint: add more only after observing improvement, and reconsider the task or prompt if they do not help.1 Google Cloud recommends at least 100 to 500 examples for supervised tuning of Gemini models.2 Unsloth calls 100 rows a bare minimum and describes more than 1,000 as preferable for optimal performance.3
Those ranges differ because the model, adaptation method, task, evaluation, and data quality differ. Planning for "1,000 examples" converts a qualified recommendation from one setting into a budget commitment for another.
The useful question is narrower: which failures still justify another batch of labels.
A raw example count hides the failures that carry operating cost
An aggregate score can improve while the model remains unsafe or unusable on the slice that causes escalations, incorrect tool calls, or policy violations. A systematic review of imbalanced classification found that raw accuracy can create a false impression of strong performance because it discounts minority-class contribution.4
That result is from classification, not LLM fine-tuning. The evaluation problem transfers directly: an average pass rate says little about a rare request type when that type has a distinct failure mode and operational consequence.
As an illustration: a support-triage assistant's evaluation can report a pass rate that looks strong in aggregate while the model still mishandles every ticket that arrives with two conflicting shipping addresses on the same order, which is the pattern that routes a ticket to a returns specialist. That combination is rare enough in the evaluation set that the aggregate score barely moves, so the model can look finished while the one failure mode that actually generates escalations goes unmeasured.
Build the training and evaluation plan around a failure taxonomy before deciding dataset size. The taxonomy should describe the work the model must do, the evidence used to judge it, and the consequence of failure. It should not be a collection of prompt topics.
| Failure slice | What an evaluation case must establish | What more labels are meant to improve | Stop decision |
|---|---|---|---|
| Policy boundary | The response applies the correct rule, refusal, or escalation path | Coverage of ambiguous wording and conflicting instructions | Stop when the slice meets its threshold with uncertainty small enough for the owner to accept |
| Tool-use path | The selected tool, arguments, and state transition are valid | Reliable selection among similar actions and recovery after tool errors | Stop when residual failures are cheaper to handle through workflow controls than through labeling |
| Grounded knowledge request | The answer is supported by the retrieved source and handles missing evidence correctly | Use of retrieved context and citation behavior | Compare fine-tuning against retrieval before buying more labels |
| Long or malformed input | The model follows the required behavior despite noisy, incomplete, or conflicting input | Resilience to production-shaped inputs | Stop only after cases resemble the inputs that reach production |
| High-value rare workflow | The output satisfies the task-specific acceptance rubric | Performance on the failure mode that motivates the project | Keep labeling if the slice remains below threshold, regardless of aggregate performance |
A slice belongs in the taxonomy when a different remediation would follow from its failure. If a tool-call error needs schema validation while a policy error needs a human escalation, they need separate evaluation rows and separate learning curves.
This also prevents a common data mistake: collecting hundreds of variations of the easy path because those examples are plentiful, then calling the resulting corpus representative.
An annotator working through a backlog of chat transcripts will recognize the pull. Transcripts of the assistant successfully rebooking a flight after a routine cancellation are abundant, while transcripts of a passenger disputing a fare rule buried in a fine-print fare notice are almost never captured. The former is what fills a labeling queue fast. The latter is the failure slice that motivated the project.
Stratified learning curves turn labeling into an engineering decision
A learning curve is useful only when it is measured per failure slice against a frozen evaluation set. Train successive versions on cumulative, versioned batches of labeled examples. Evaluate every version on the same held-out cases and record the result by slice, not only as one total score.
For each slice, capture the acceptance metric, the observed result, and its uncertainty. A binary pass or fail rubric can use a confidence interval around the pass rate. A judged generative rubric needs adjudication records and a resampling-based uncertainty estimate. The exact method matters less than the rule: do not call a curve flat when the apparent gain is within the measurement uncertainty.
Representative sampling makes rare slices expensive to evaluate. Assume, as a modelled example, that a critical failure slice represents 5 percent of requests and that the evaluation plan requires 100 adjudicated cases from that slice. A representative set would need 100 divided by 0.05, or 2,000 cases, to supply those 100 cases. This is arithmetic from the stated assumptions, not a measured dataset requirement. In practice, deliberately oversample the critical slice and retain its production prevalence separately for cost decisions.
The curve should answer four questions after every labeled batch:
- Did the critical slice improve beyond its uncertainty interval?
- Did a different slice regress?
- Is the remaining failure reducible through more demonstrations, or does it require retrieval, validation, routing, or a changed prompt?
- Is the expected value of that improvement greater than the next batch's labeling, training, and evaluation cost?
The last question is the stopping point. Labeling cost includes annotator time, domain review, disagreement resolution, and the evaluation work needed to trust the result. It also includes the time engineers spend checking outputs. That work should be budgeted explicitly, using the same discipline described in How much of my team's time will checking the AI take?.
A team that cannot state the cost of a failed output should avoid pretending that a generic corpus size is a business case. It can still use learning curves to identify technical improvement, but it cannot rationally decide whether that improvement is worth purchasing.
Curated examples can outperform large undifferentiated corpora
Data quality changes the curve because an example is an instruction about desired behavior, not merely a token contribution. AWS guidance for fine-tuning Anthropic Claude 3 Haiku states that quality outweighs quantity and calls a well-prepared dataset key.5
LIMA demonstrated the boundary of count-based reasoning. Its authors fine-tuned a 65B LLaMa model with standard supervised loss on 1,000 carefully curated prompt-response pairs, without reinforcement learning or human preference modeling.6 In its paired human evaluation, LIMA responses were equivalent to or preferred over GPT-4 responses in 43 percent of cases.6
That result does not make 1,000 a universal target. It shows that a curated example can carry far more decision-relevant signal than a loosely collected one. A later study attributes disagreement over fine-tuning best practices partly to diverging evaluation paradigms.7 Change the evaluation, and the dataset size that appears sufficient can change with it.
For a custom system, label examples that resolve actual ambiguity in the failure taxonomy. Preserve the production context needed to interpret them: source material, tool state, prompt version, expected action, reviewer rationale, and any policy version that governed the decision. A polished answer without that context often becomes a training example that teaches style while leaving the underlying failure untouched.
Compare fine-tuning with retrieval and workflow changes before labeling another batch
Fine-tuning is a poor default for facts that change frequently, must be attributable to a current source, or can be retrieved at runtime. A controlled comparison of knowledge injection methods found that retrieval-augmented generation consistently outperformed unsupervised fine-tuning for both knowledge already seen during training and entirely new knowledge.8
That finding does not prove retrieval will win for every custom system. It does establish that a knowledge task should enter the same evaluation harness with retrieval as a peer candidate. Evaluate it against the same grounded-knowledge slice, including source selection, unsupported-answer handling, and failure recovery.
Prompt and workflow changes also belong in the comparison. If the model fails because the task definition is underspecified, a better rubric, structured output schema, deterministic validator, or router can remove the failure without adding labels. If the model fails because an operator expects hidden business knowledge, retrieval may be the correct mechanism. More supervised examples only make sense when examples contain the missing behavioral signal.
Adaptation method affects the compute side of this decision. In the GPT-3 experiments described by the LoRA paper, low-rank adaptation reduced trainable parameters by 10,000 times and GPU memory requirements by 3 times relative to full fine-tuning.9 That reduction can make another training run cheap enough to test. It does not make expert labels cheaper, nor does it repair an evaluation set that fails to expose the important errors.
Keep the comparison reproducible. Record the base model identifier, training method, adapter configuration, prompt version, dataset version, evaluation-set version, slice definitions, and reviewer rubric. A buyer should be able to ask for evidence that a custom model can be rebuilt, as described in Require vendors to prove every custom model can be rebuilt.
Learning curves do not discover missing failure modes
A stratified curve only measures the strata it contains. It cannot reveal a production workflow that was absent from the taxonomy, a policy revision that changes the expected answer, or a tool integration whose state was not represented in the data.
As an illustration: a curve built before a vendor adds a refund-approval step to its order-management tool will look flat or improving right up to the week that step ships, then miss every case where the model must now decide whether to invoke it. The taxonomy did not lie. It described a system that no longer exists.
It also does not make extrapolation safe. A meta-analysis of downstream scaling laws found predictable scaling in only 39 percent of studied cases.10 Do not fit an early curve, project its plateau, and authorize a full labeling program from the projection. Run the next batch, measure it, and update the decision from observed slice-level behavior.
Evaluation itself can fail. Reviewer criteria drift, automated judges inherit blind spots, and agent traces can omit the context that explains why an action was wrong. Where evaluation records originate in agent execution, preserve the trace, tool inputs, retrieved evidence, and final outcome using the controls in How to govern AI agent traces used for evaluation.
The durable stopping rule is therefore conditional: stop adding examples when every critical failure slice meets its agreed threshold with acceptable uncertainty, recent labeled batches no longer deliver value above their full marginal cost, and retrieval or workflow alternatives do not produce a better result on the same evaluation set.