Choose Between Prompting, Retrieval, Fine-Tuning, and Training

Diagnose the failure before changing the model.

The learning center's assistant gives the wrong Saturday hours. Someone proposes fine-tuning it on the handbook. Another person suggests a larger model. A third wants to build a new model from scratch.

Then the team checks the retrieved passage. The system was using the retired policy.

The immediate problem was source selection. A more expensive model could repeat the same wrong rule with greater confidence. Good customization begins by locating the failure, then changing the layer responsible for it.

Describe the failure in observable terms

Collect examples with the input, expected behavior, actual output, retrieved evidence, tool results, and relevant configuration. Separate failures that merely look similar.

A missing current fact differs from a failure to follow a clear output format. A poor search result differs from a correct passage being misinterpreted. A model may also lack the capability needed for the task even when instructions and evidence are adequate.

Use a diagnosis table:

Fine-tuning is not a repair for missing server-side authorization. A prompt is not a substitute for a correct database query. Keep the boundary between model behavior and application guarantees clear.

Improve task framing and examples first

A prompt can clarify the objective, relevant context, output contract, and handling of uncertainty. Examples can show distinctions that prose leaves ambiguous.

For the center's topic classifier, “classify this” is underspecified. A better task defines HOURS, LOANS, and OTHER, with examples and a rule for questions outside the categories. The evaluation should include cases that distinguish those labels.

More instructions are not always better. Remove contradictions and irrelevant material. Add an instruction because it addresses an observed failure, then test whether it helps without causing regressions.

A strong prompting baseline is inexpensive to revise and easy to compare. It also tells you whether later training is solving a real limitation or merely compensating for an unclear task.

Use tools for exact operations

A model does not need to perform every calculation or lookup internally. A validated calculator, database query, or constrained transformation can provide exact results for well-defined operations.

The model may interpret a request and select an allowed tool, while ordinary code checks arguments and performs the operation. Evaluate the integration as well as the model's choice. A correct calculator is useless if the wrong numbers are passed to it.

Keep authoritative state outside generated text. The system of record should establish whether a booking exists, what a balance is, or which policy is current. A model's confident answer is not that record.

Use retrieval when evidence must remain current and traceable

Retrieval is useful when answers depend on changing documents, source citations, or user-specific access. It lets the application update the source collection without necessarily changing model weights.

It also introduces parsing, indexing, ranking, and access-control work. Article 123 showed how a retriever can miss an exception or expose a conflict. Adding retrieval does not automatically make answers grounded.

Compare the answer with and without the appropriate evidence. If the model succeeds when given the right passage but the system fails end to end, improving retrieval is a more direct experiment than retraining the generator.

The original RAG paper provides research context for this combination. Your production choice still depends on the current implementation and task evidence.

Consider fine-tuning for recurring behavior

Fine-tuning adjusts an existing model using additional training examples. It may help with a repeated format, domain-specific pattern, style, or task behavior when the dataset and evaluation are appropriate.

It is not a reliable database update mechanism. Training a model on a new policy does not guarantee that it forgets the old policy, cites the right version, or respects per-user access. Keep those requirements in the source and application layers.

Fine-tuning also creates a maintenance obligation: data preparation, experiment tracking, compute, evaluation, artifact management, and future compatibility. Hugging Face's fine-tuning guide explains the general training workflow for adapting pretrained models.

Before tuning, ask whether a better prompt, a small set of examples, constrained output, or a different existing model meets the need. Training should earn its place through measured improvement.

Distinguish continued training from starting from scratch

Continued pretraining exposes an existing model to additional data under a training objective. Supervised fine-tuning uses examples of desired behavior. Training from scratch begins without the pretrained weights and requires a suitable architecture, data pipeline, optimization process, and evaluation program.

A small teaching model trained from scratch can demonstrate learning mechanics. It does not establish that a beginner-scale experiment can reproduce a broadly capable language model. Data diversity, compute, engineering, and evaluation requirements grow substantially with the ambition of the task.

For most small applications, starting from a suitable existing model is the practical baseline. A scratch-trained model is justified by a specific research or product requirement, not by the assumption that owning every step automatically improves results.

Run a controlled comparison

Use the same task set and acceptance criteria across approaches. Keep a held-out set untouched while developing prompts, retrieval settings, or training data. Record model identity, source version, inference settings, and total cost.

For the center, a comparison might include a clear prompt, the same prompt with approved retrieval, and a small tuned model with the same necessary retrieval. That design helps separate behavioral adaptation from access to evidence.

Include an ablation: remove one component and observe the effect. If removing the tuned adapter does not reduce held-out performance, the adapter may not justify its maintenance cost. If removing retrieval destroys current-policy accuracy, retrieval is carrying an essential part of the task.

Do not compare a carefully optimized candidate against a deliberately weak baseline. Use realistic alternatives and report the effort spent developing each.

Work through four choices

A new opening-hours policy: Update and verify the authoritative source and retrieval path. Test current and historical questions separately.

A classifier that alternates between labels and long explanations: Clarify the label contract and try constrained output. Consider tuning if enough representative examples exist and the simpler approach still fails.

A research assistant that misses specialized relationships despite correct sources: Test task decomposition, a stronger suitable model, and domain examples. Consider adaptation only after identifying the remaining capability gap.

A narrow research project studying how a model learns a synthetic grammar: Training a tiny model from scratch may fit the educational objective. Its results should be interpreted at that scale.

These choices address different causes. There is no single ladder on which every project should climb from prompting to frontier training.

Decide using total useful performance

Compare factual quality, required abstention, access behavior, latency, review effort, and full operating cost. A tuned model that saves tokens but requires frequent corrections may be a worse system.

State what would change the decision. More volume, a new language, changing data, or a stricter response-time requirement may justify reevaluation. Preserve the baseline and evaluation set so the next comparison starts from evidence.

A reusable prompt

Diagnose these failures before recommending customization. Separate missing evidence, poor retrieval, unclear instructions, exact-tool needs, output inconsistency, and capability limits. Compare prompting, tools, retrieval, fine-tuning, and training using the same acceptance criteria and total-cost accounting. Include a strong simpler baseline and an ablation that tests each added component's contribution.

For students: choose the experiment that teaches the intended idea

Students can compare prompts without training, study retrieval with a small approved corpus, or design a tuning experiment using synthetic labels. The choice should follow the learning objective and available resources.

If the goal is understanding gradient descent, a tiny model is enough. If the goal is evaluating a policy assistant, source quality and held-out questions may matter more than training. Follow course rules and document the work actually performed.

Practice: match the method to the cause

Take four failure examples and write a diagnosis, a first intervention, and a test that could disprove your diagnosis. Identify which approaches would leave the underlying problem untouched.

Completion check: The chosen method addresses an observed cause and is compared with a simpler credible alternative.

Stretch: Remove one component from a working prototype, rerun the same held-out cases, and report its actual contribution rather than its intended purpose.