Build Your First Prediction or Classification Model

Train a small model, compare it with a trivial baseline, and keep the final test out of development.

A classifier announces 95% accuracy. Then you discover that 95% of the examples belong to one class and the model almost always predicts that class. The impressive headline hides the fact that it misses the cases you care about.

Your first machine-learning project should teach you to see beyond that headline. You need a defined target, suitable data, a development process, an untouched final comparison, and metrics that reveal the actual mistakes.

This article provides a complete small lab using the public Iris dataset. The code was executed during manuscript preparation. Its results are educational measurements from one fixed split, not evidence of a field-ready botanical identification system.

Define the target and intended use

The model predicts one of three Iris classes from four numerical flower measurements: sepal length, sepal width, petal length, and petal width. Inputs use centimeters. It does not accept photographs, identify arbitrary plants, or determine whether a plant is suitable for any particular use.

The UCI repository describes 150 examples, four features, and three classes with fifty examples each. It provides dataset attribution and identifies a CC BY 4.0 license. See the UCI Iris dataset record. Preserve that attribution when reusing the dataset.

Our actual input comes from scikit-learn 1.8.0’s bundled load_iris data. Use the versioned loader documentation to identify that distribution. Do not assume every file called “Iris” is byte-identical; source versions can differ. The lab records a hash of the numerical arrays it used.

This is a compact historical teaching dataset. Its balance and limited scope do not represent every real-world identification setting.

Separate features, labels, and learned parameters

Features are the measurements supplied to the model. Labels are the class identities used as training targets. Learned parameters are values adjusted during training to connect the features to those labels.

A trained classifier uses that learned relationship to produce class predictions for inputs it did not use for fitting. It can be wrong even when the measurements are well formed.

The lab uses logistic regression, which despite its name is used here for classification. Its regularization setting C controls the strength of a constraint on the fitted model; smaller positive C means stronger regularization in this implementation. See scikit-learn 1.8’s LogisticRegression documentation.

The goal is understanding the workflow. A large neural network would add complexity without improving this introductory lesson.

Reserve training, validation, and test roles

The bundled data contains 150 rows but only 149 distinct feature vectors: rows 101 and 142 have identical measurements. Matching measurements do not prove these are the same physical flower. As a conservative teaching boundary, the code keeps identical feature vectors together in one subset. It uses grouped, stratified folds and obtains ninety training examples, thirty validation examples, and thirty test examples in the verified run. Each subset is balanced by class in this run; grouped stratification does not guarantee exact balance for every dataset. See the versioned StratifiedGroupKFold documentation.

Training data fits the candidate models. Validation data selects between the two predeclared values of C: 0.1 and 1.0. Ties select 0.1 under a rule written before evaluation.

After selection, the chosen configuration is refitted on the combined 120 training-and-validation examples. The final thirty test examples remain excluded from fitting and model selection. The predeclared baseline and selected model are then evaluated on that same test set.

This grouped split serves the small teaching task. In an applied project, use the actual unit of independence—such as a person, site, or device—and respect time order when the task requires it. Grouping identical measurements does not detect every related observation or fix a mismatch between the sample and the intended population.

Fit preprocessing only on development inputs

The pipeline standardizes the numerical features and then fits the classifier. During candidate comparison, the standardizer learns its values only from the ninety training rows. It applies those same learned transformations to validation inputs.

During the final refit, preprocessing is learned from the combined development set, still excluding the test set. The pipeline then applies the fitted transformation to test inputs.

This distinction prevents information from the test data from entering preprocessing. Scikit-learn’s common-pitfalls guide explains why transformations must be learned on the fitting data and why pipelines help preserve the separation.

A pipeline does not solve every leakage problem. If a feature itself contains information recorded after the outcome, putting it in a pipeline does not make it legitimate.

Run the complete lab

The verified environment used Python 3.12.13, NumPy 2.3.5, SciPy 1.17.0, and scikit-learn 1.8.0. Use a separate learning environment with those dependencies when reproducing the reported run. The script loads bundled data and does not require an API key or download a model.

Save the complete code below as first_model.py and run python3 first_model.py. Its JSON output includes versions, split indices, candidate validation scores, final metrics, and the data hash. Retain the output with your script for review.

import hashlib
import json
import platform

import numpy as np
import scipy
import sklearn
from sklearn.datasets import load_iris
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, confusion_matrix, precision_recall_fscore_support
from sklearn.model_selection import StratifiedGroupKFold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

iris = load_iris()
X, y = iris.data, iris.target
# Conservatively keep identical feature rows together. Matching measurements
# do not prove that two rows describe the same physical flower.
_, groups = np.unique(X, axis=0, return_inverse=True)
outer = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=42)
train_val, test = next(outer.split(X, y, groups))
inner = StratifiedGroupKFold(n_splits=4, shuffle=True, random_state=43)
train_local, validation_local = next(
    inner.split(X[train_val], y[train_val], groups[train_val])
)
train, validation = train_val[train_local], train_val[validation_local]
for left, right in [(train, validation), (train, test), (validation, test)]:
    assert not set(left) & set(right)
    assert not set(groups[left]) & set(groups[right])
assert len(train) + len(validation) + len(test) == len(y)

# Freeze these candidates and the tie rule before inspecting results.
candidates = [0.1, 1.0]
validation_scores = []
for C in candidates:
    model = make_pipeline(
        StandardScaler(), LogisticRegression(C=C, max_iter=1000)
    )
    model.fit(X[train], y[train])
    predicted = model.predict(X[validation])
    validation_scores.append(float(accuracy_score(y[validation], predicted)))
# np.argmax returns the first maximum: ties select C=0.1.
best_C = candidates[int(np.argmax(validation_scores))]

# The selection is now fixed. Refit using training + validation only.
final_model = make_pipeline(
    StandardScaler(), LogisticRegression(C=best_C, max_iter=1000)
)
final_model.fit(X[train_val], y[train_val])
baseline = DummyClassifier(strategy="most_frequent")
baseline.fit(X[train_val], y[train_val])

# One final evaluation of the selected model and the predeclared baseline.
reports = {}
for name, model in [("baseline", baseline), ("logistic", final_model)]:
    predicted = model.predict(X[test])
    precision, recall, f1, support = precision_recall_fscore_support(
        y[test], predicted, labels=[0, 1, 2], zero_division=0
    )
    reports[name] = {
        "accuracy": float(accuracy_score(y[test], predicted)),
        "confusion_matrix": confusion_matrix(
            y[test], predicted, labels=[0, 1, 2]
        ).tolist(),
        "precision": precision.tolist(), "recall": recall.tolist(),
        "f1": f1.tolist(), "support": support.tolist()
    }

payload = X.astype("<f8").tobytes() + y.astype("<i8").tobytes()
report = {
    "versions": {"python": platform.python_version(),
                 "numpy": np.__version__, "scipy": scipy.__version__,
                 "sklearn": sklearn.__version__},
    "data_sha256": hashlib.sha256(payload).hexdigest(),
    "group_count": int(len(np.unique(groups))),
    "grouping": "identical feature vectors kept in one split",
    "class_order": iris.target_names.tolist(),
    "split_sizes": [len(train), len(validation), len(test)],
    "split_indices": {"train": train.tolist(),
                      "validation": validation.tolist(), "test": test.tolist()},
    "candidate_C": candidates, "validation_accuracy": validation_scores,
    "selected_C": best_C, "test_reports": reports
}
print(json.dumps(report, indent=2))

Read the actual result carefully

The two candidates scored 29/30 and 30/30 on validation, respectively. The predeclared rule selected C=1.0. A perfect validation score in thirty cases does not mean perfect generalization.

On the final test set, the selected model classified 30 of 30 examples correctly: 100% observed accuracy. The baseline always predicted the most frequent development class. The development classes were tied in count, and this run selected setosa. That baseline classified 10 of 30 test examples correctly, or approximately 33.3%.

The selected model’s confusion matrix uses actual classes as rows and predicted classes as columns:

No candidate-model errors were observed in these thirty test cases. Each class contains ten examples, and observed precision and recall are 1.00 for every class. This small result cannot establish that errors will not occur on new data. The baseline still supplies twenty errors to inspect: it misses every versicolor and virginica.

Precision asks how many predictions of a class were correct. Recall asks how many actual examples of a class were found. They happen to match here; they need not match in general.

The baseline has no predictions for two classes. Its precision for those classes is undefined mathematically; the code’s explicit zero_division=0 convention reports zero. Record that convention rather than interpreting the displayed zero as an ordinary estimated proportion.

Audit correction: The earlier row-based split put identical measurements into training and validation and reported 28/30 on its test set. This edition repairs that boundary and reruns the lab. The revised test contains different examples, so 30/30 versus the earlier 28/30 is not evidence of a model improvement. The split rule was changed to repair separation, not selected by searching for a higher test score.

Preserve the run’s identity

The numerical data hash from the verified run is:

aa06b8008ceba42efc654be0f83fdafc786239c9e8f13146044d078f5aab8f23

It is the SHA-256 of the feature array serialized as little-endian 64-bit floats followed by the labels as little-endian 64-bit integers, as shown in the code. It is not the hash of every possible Iris source file.

The output also records row indices for all three splits. This makes the result more inspectable than a seed alone. Versions and numerical settings still matter when reproducing a run in another environment.

If your result differs, first compare data identity, package versions, split indices, and configuration. Do not search for a seed that produces a better score and then present it as the original planned evaluation.

Know what the final test can and cannot establish

The held-out result estimates performance for the defined sampling setup. It does not establish performance on photographs, unseen species, measurements taken differently, or other populations.

Thirty examples also provide limited precision. One changed classification moves the accuracy by approximately 3.33 percentage points. Writing 100.000000% would not create more evidence.

Once the test result guides a new model choice, it is no longer serving as an untouched final check for that new choice. You can continue learning from errors, but clearly mark the work as further development and arrange an appropriate independent evaluation before making a fresh final claim.

The baseline comparison shows that the fitted model learned useful class distinctions in this experiment. It does not show that the model is the best possible method or that deployment is justified.

Explore thresholds without quietly changing the evaluation

This model normally chooses the class with the largest predicted probability. A more cautious workflow could abstain when the largest value falls below a preselected threshold.

That creates a trade-off between coverage and the errors among accepted predictions. A system can improve accepted-case accuracy by declining more difficult cases while becoming less useful overall.

Choose any threshold using development data and an explicit objective. Report the number accepted, the number referred for review, and performance within the relevant groups. Do not assume the model’s probability estimates are perfectly calibrated.

A binary decision has related trade-offs between false positives and false negatives. Article 085 introduced those counts in maintenance alerts; here they become part of the model-development process rather than just an audit of supplied predictions.

Write a small model report

A useful report records the target, intended users, dataset and license, feature units, split strategy, preprocessing, baseline, model-selection rule, environment, final metrics, failure cases, and limits.

For this lab, the status is “educational model trained and evaluated locally.” It is not “botanical identification service approved for use.” No monitoring system, user interface, or external action has been deployed.

An operational project would also need input validation, appropriate review or fallback, monitoring, and a process for changes in data or use. Start by identifying the concrete gap between the teaching experiment and the intended application rather than attaching a generic deployment checklist to a high score.

A reusable prompt

Plan a first model for this dataset and intended use. Identify provenance, feature and label definitions, units, split strategy, baseline, leakage risks, preprocessing, and metrics matched to mistakes. Predeclare candidate settings and selection rules. Keep test data out of fitting and model choice. Provide complete reproducible code and a report with actual counts, versions, failure cases, and scope limits. Distinguish an executed educational experiment from an operational system.

For students: explain each dataset role

Before running the code, explain why there are three subsets and why the final refit can use training plus validation while still excluding test data.

Beginning students can read the output tables and trace the confusion matrix. Students with Python experience can run the full script and inspect the saved indices. Advanced students can propose a separate development experiment while preserving a new independent evaluation plan.

Follow the assignment’s AI-use rules and disclose assistance as required. A strong submission includes a choice you can explain, an error you can interpret, and a claim you deliberately limit. Copying the final accuracy does not demonstrate those skills.

Practice: reproduce and document the first model

Run the script in the stated environment where available and save its JSON output. Compare the data hash, split sizes, validation scores, selected C, confusion matrix, and baseline with the manuscript.

Write a short model report explaining the baseline’s class failures, the limited evidence from a perfect thirty-case model result, and why this test does not establish accuracy on arbitrary plants. If you cannot execute the code, complete a clearly labeled interpretation exercise using the provided outputs; do not claim a personal training run.

Completion check: The verified reference run uses 90/30/30 development-and-test roles, selects C=1.0 through validation, and yields 30/30 correct test classifications versus 10/30 for the baseline, with identical feature vectors kept in the same subset. The report includes per-class results, preprocessing boundaries, data identity, and a clear limit on generalization.

Stretch: Using development data only, compare two predeclared abstention thresholds. Report coverage and accepted-case errors separately. Keep the already viewed test result out of threshold selection, and explain what independent evidence a final claim about the new policy would require.