Fine-Tune a Small Model and Check Whether It Improved

Treat adaptation as a controlled experiment with a baseline, clean splits, and a result that can be rejected.

A small model learns to classify workshop questions and performs beautifully on examples used during training. The team celebrates an improvement. Then it encounters unfamiliar wording and starts assigning nearly every question to the most common category.

The model learned something. The demonstration did not establish useful generalization.

Fine-tuning should begin with a precise hypothesis and a held-out evaluation plan. The purpose is to find out whether the adapted model improves the intended task without introducing unacceptable regressions.

Define a narrow hypothesis

Our fictional task classifies questions as HOURS, LOANS, or OTHER. HOURS concerns opening or closing times. LOANS concerns borrowing and returning eligible equipment. OTHER covers questions outside those categories or ambiguous requests that cannot be assigned under the defined rules.

The hypothesis is: “A small supervised adaptation will improve exact label accuracy on unseen question families compared with the same base model using a carefully written prompt.”

The output contract permits exactly one label and no explanation. That makes part of the evaluation deterministic. The semantic correctness of each label still depends on a reviewed answer key.

Use a credible baseline: the chosen model, its correct chat format, clear instructions, and a few representative examples if those are part of the intended deployment. Do not compare a carefully tuned candidate with an intentionally vague baseline prompt.

Choose a model and record its provenance

One small educational candidate is Qwen/Qwen2.5-0.5B-Instruct. Its official model card identifies approximately 0.49 billion parameters and lists the model's architecture and license metadata. This is a concrete candidate for the experiment, not a claim that it is the best current model for every device or task.

Before execution, resolve and record an immutable model revision, tokenizer revision, license file, runtime versions, and hardware details. Check the actual distribution you will use, including any conversion or quantization.

The authoring environment did not contain model weights or the training frameworks needed for this experiment. This article therefore provides the outline's alternative: a complete experiment design with tested data-validation code. No fine-tuning run or model-quality gain is claimed.

Build authorized examples around meaning

For the full proposed experiment, prepare 900 reviewed examples grouped into 180 question families, with five variants per family. Use 60 families per label. A family groups paraphrases and closely related versions of the same underlying request.

Allocate whole families by label: 40 families per label to training, eight to validation, and twelve to final testing. That produces 600 training examples, 120 validation examples, and 180 test examples.

These are proposed dataset sizes, not an existing collected dataset or a universal minimum. Pilot the labeling rules before creating all examples. If reviewers disagree about a question, clarify the rule or mark the case ambiguous rather than forcing a convenient label.

Use original synthetic questions or material you are authorized to adapt. Remove unnecessary personal information. Keep the source, authoring method, review status, and family identifier with each example.

Prevent leakage before training begins

Do not place one paraphrase in training and a near-identical version in testing. Grouping helps, but it depends on honest family definitions. Also check normalized duplicates, repeated source records, and templates that make the answer obvious for the wrong reason.

Save the test set before iterating. Use validation for prompt and training choices. If you repeatedly inspect the test failures and revise the model around them, that set becomes part of development and a fresh final test is needed.

The following complete program validates a tiny nine-example teaching dataset and demonstrates two leakage checks. Save it as split_lab.py and run it with Python 3.9 or later. It is not the proposed 900-example training corpus.

import re

LABELS = {"HOURS", "LOANS", "OTHER"}
ROWS = [
    {"split": "train", "family": "opening", "text": "When do you open?", "label": "HOURS"},
    {"split": "train", "family": "borrow-count", "text": "How many items may I borrow?", "label": "LOANS"},
    {"split": "train", "family": "parking", "text": "Where can I park?", "label": "OTHER"},
    {"split": "validation", "family": "closing", "text": "What is the closing time?", "label": "HOURS"},
    {"split": "validation", "family": "extension", "text": "May I extend my equipment loan?", "label": "LOANS"},
    {"split": "validation", "family": "refund", "text": "Can I get a workshop refund?", "label": "OTHER"},
    {"split": "test", "family": "weekday-access", "text": "Is the center open on Thursday?", "label": "HOURS"},
    {"split": "test", "family": "return-deadline", "text": "When must this borrowed item be returned?", "label": "LOANS"},
    {"split": "test", "family": "transport", "text": "Does a bus stop nearby?", "label": "OTHER"},
]


def validate(rows):
    families, texts = {}, {}
    counts = {name: 0 for name in ("train", "validation", "test")}
    for row in rows:
        if set(row) != {"split", "family", "text", "label"}:
            raise ValueError("Unexpected fields")
        split = row["split"]
        if split not in counts or row["label"] not in LABELS:
            raise ValueError("Unknown split or label")
        if not isinstance(row["text"], str) or not row["text"].strip():
            raise ValueError("Missing text")
        family = row["family"]
        if not isinstance(family, str) or not family:
            raise ValueError("Missing family")
        normalized = re.sub(r"\s+", " ", row["text"].strip().casefold())
        if family in families and families[family] != split:
            raise ValueError("Family crosses split boundary")
        if normalized in texts:
            raise ValueError("Duplicate normalized text")
        families[family], texts[normalized] = split, split
        counts[split] += 1
    return counts


def main():
    assert validate(ROWS) == {"train": 3, "validation": 3, "test": 3}
    leaks = [
        {**ROWS[0], "split": "test", "family": "different-name"},
        {**ROWS[0], "split": "test", "text": "Tell me when the doors open."},
    ]
    for leak in leaks:
        try:
            validate(ROWS + [leak])
        except ValueError:
            pass
        else:
            raise AssertionError("Leakage fixture accepted")
    print("PASS: valid splits accepted; duplicate and family leakage rejected")


if __name__ == "__main__":
    main()

The validator passed the valid split and rejected a duplicate under a different family name and a family crossing the split boundary. It does not detect every semantic paraphrase or incorrect label. Human review and task-specific similarity checks remain necessary.

Specify a bounded training run

For a first proposed adapter experiment, use the following starting configuration after verifying support in the selected software and hardware:

These settings define a small hypothesis test; they are not claimed optimal. With 600 training examples and an effective batch of 16, one epoch is roughly 38 optimizer updates, depending on batching details. Verify how the framework handles the final partial batch and stopping controls.

Use a precision supported by the hardware and implementation. Do not enable an unsupported precision merely because it appears in an example. Check trainable parameter names and counts before the first update.

Verify which tokens contribute to the loss

For an instruction model, preserve the appropriate conversation template. The training target should be the desired label response, not accidental metadata or a duplicated prompt.

Hugging Face's SFT Trainer documentation distinguishes prompt-completion and conversational training formats, and documents completion-only and assistant-only loss behavior. Some masking options depend on the chat template. Inspect a tokenized sample and its loss mask rather than assuming the configuration name guarantees the intended target.

The LoRA paper provides the method's original formulation. Freezing the base and training smaller updates can reduce trainable parameters, but the base model still needs to be loaded and used during training.

Evaluate paired outcomes and regressions

Run the baseline and selected candidate on the same untouched test examples with recorded inference settings. Report exact label accuracy, per-label performance, malformed outputs, ambiguous-case behavior, and relevant regression tasks outside the narrow training target.

Consider these fictional results on 180 cases, 60 per label:

Overall accuracy rises from 75% to 85%, a ten-percentage-point increase. OTHER falls from about 66.7% to 63.3%. If the preregistered requirement is at least 75% for every label, the candidate fails despite its overall gain.

These numbers illustrate interpretation; they are not training results. Preserve paired case outcomes so you can see which cases improved and which regressed. Account for the small sample and correlated paraphrases when estimating uncertainty.

Save an experiment packet

Keep dataset lineage and split identifiers, model and tokenizer revisions, code and dependency versions, training settings, checkpoints, logs, selection criteria, and evaluation results. Include failures and abandoned runs where they affect the conclusion.

An adapter also depends on the correct base model. Record that relationship so it cannot be accidentally loaded onto an incompatible revision. Retain the previous verified deployment and its configuration as a recovery option.

Do not publish a tuned artifact with claims stronger than the evidence. “Improved this held-out classification task under these settings” is a useful result. “Understands all student requests” is not established by the experiment.

A reusable prompt

Plan a small supervised adaptation for this narrow behavior. Define a strong baseline, authorized examples, family-based splits, tokenization and loss checks, bounded settings, checkpoint selection, and held-out evaluation. Report per-group gains and regressions, model provenance, and recovery options. Separate executed training results from proposed settings and illustrative numbers.

For students: a well-designed experiment can be the deliverable

Students without suitable hardware can prepare the dataset rules, leakage checks, configuration review, and evaluation packet. That is meaningful experimental work when clearly labeled as a design rather than a completed training run.

Where training is available, use approved resources and small budgets. Explain why the test examples are unseen in a meaningful sense, not merely stored in a different file. Follow the course's disclosure rules for generated training examples and AI assistance.

Practice: test the data boundary first

Run the validator. Create a paraphrase that shares an existing family and verify it cannot enter another split. Then write the full experiment record, including the conditions under which you would reject a candidate with higher overall accuracy.

Completion check: Any claimed improvement comes from unseen cases, and regressions, data lineage, and unexecuted portions are reported explicitly.

Stretch: Compare the adapter with a stronger prompting baseline at similar total development and inference cost, retaining a fresh final evaluation set.