Run and Evaluate Models Locally

Test the model, runtime, hardware, and data flow as one system.

A small model starts on a laptop and answers a question. The owner concludes that the project is private, inexpensive, and ready for daily use.

Those are three different claims. Starting establishes that the software can load and generate something. Privacy depends on the full data flow. Cost includes hardware and maintenance. Usefulness depends on the actual task, speed, and quality.

Local models can support offline work, experimentation, and control over deployment. The right evaluation asks whether those advantages hold for your intended workload.

Define the reason for local use

Write the requirement in concrete terms. “Prepare drafts while disconnected from the internet” is testable. “Keep specified documents within an approved device and its authorized storage” is also testable, though it requires inspecting more than the model.

Local use may reduce dependence on a hosted generation service, but software updates, model downloads, web-search tools, telemetry, synchronization, and backups can still use the network. A local interface can also send requests to a cloud model.

Map what leaves the device and why. If offline operation is required, test it after the needed software and weights are available, with external connections disabled in the intended operating setup.

Estimate memory without confusing weights with the whole system

A rough raw-weight estimate is parameter count multiplied by bits per parameter, divided by eight. Seven billion parameters at four bits require about 3.5 billion bytes, or roughly 3.26 GiB, for that simplified weight representation.

Actual storage and runtime memory are larger because of quantization metadata, unquantized components, working buffers, the key/value cache, and the serving process. Training requires additional memory for gradients, optimizer state, and activations.

For illustration, a transformer cache with 32 layers, eight key/value heads, head dimension 128, 4,096 tokens, and two-byte cache elements would use:

2 × 32 × 8 × 128 × 4,096 × 2 bytes = 536,870,912 bytes, or 512 MiB, for one sequence under those assumptions.

The first factor of two represents keys and values. Different architectures, cache formats, batch sizes, and implementation choices change the result. This is a memory calculation, not a measurement of a particular model.

Leave room for the operating system and other applications. A device with 32 GiB of system RAM does not automatically have 32 GiB of fast dedicated GPU memory.

Check hardware support and the model card

Verify the runtime's support for the exact CPU, GPU, operating system, and driver combination. A model that can run on the CPU may have a very different latency from one fully offloaded to a supported GPU. Ollama's hardware-support documentation lists its supported arrangements and relevant requirements.

Read the model card and license for the exact model and distribution. Identify its architecture, intended use, limitations, and any terms affecting your planned use or redistribution. Downloadable weights do not by themselves establish unrestricted use.

Hugging Face's model-card documentation explains the role of these records. Treat benchmark claims in a card as evidence about the reported conditions, then evaluate your own task separately.

Understand quantization as a trade-off

Quantization stores or computes values at reduced precision. It can lower memory use and change performance, but its effects vary by model, method, hardware, and task.

Compare two variants on the same prompts and acceptance criteria. Include the tasks most likely to expose small errors: exact extraction, numbers, negation, structured outputs, and the languages you actually use.

A compressed model that writes fluent prose may still regress on a narrow classification rule. Conversely, a smaller quantized model may be entirely adequate for a constrained job. Measure the relevant behavior instead of treating parameter count or bit width as a complete quality score.

Configure a local serving boundary

The following benchmark targets an already installed Ollama server at 127.0.0.1:11434. It does not install software or download a model. Set OLLAMA_MODEL to the exact local model name shown by your installation.

For local-only operation, configure the server accordingly before running. Ollama's FAQ documents OLLAMA_NO_CLOUD=1, the need to restart after configuration changes, and the default loopback binding. Those controls concern Ollama; inspect other connected components separately.

Choose a modest context setting appropriate to the task. Increasing context can increase memory requirements, and runtime defaults are not necessarily the model's theoretical maximum. See the runtime's context-length guidance.

Run a small task benchmark

Save this complete program as local_benchmark.py and run it with Python 3.9 or later after configuring the local server and installed model. It uses five synthetic classification cases twice, records the model digest reported by the server, and prints one JSON record per result.

import json
import os
import time
from urllib.error import HTTPError, URLError
from urllib.request import HTTPRedirectHandler, ProxyHandler, Request, build_opener


class NoRedirects(HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newurl):
        return None


def main():
    model = os.environ.get("OLLAMA_MODEL", "").strip()
    if not model:
        raise SystemExit("Set OLLAMA_MODEL to the exact name of an installed local model")
    opener = build_opener(ProxyHandler({}), NoRedirects())
    def request(path, payload=None):
        data = None if payload is None else json.dumps(payload).encode("utf-8")
        req = Request("http://127.0.0.1:11434" + path, data=data,
                      headers={"Content-Type": "application/json"})
        with opener.open(req, timeout=60) as response:
            raw = response.read(1048577)
        if len(raw) > 1048576:
            raise ValueError("Response exceeds lab size limit")
        value = json.loads(raw)
        if not isinstance(value, dict) or "error" in value:
            raise ValueError("Server returned an unusable response")
        return value
    installed = request("/api/tags").get("models", [])
    matches = [entry for entry in installed if entry.get("name") == model]
    if len(matches) != 1:
        raise SystemExit("The configured name does not identify exactly one installed model")
    print(json.dumps({"model": model, "digest": matches[0].get("digest")}))
    tasks = [
        ("When do you open on Saturday?", "HOURS"),
        ("How many items may I borrow?", "LOANS"),
        ("Where is the parking area?", "OTHER"),
        ("What is the refund policy?", "OTHER"),
        ("What time do you close?", "HOURS"),
    ]
    accepted = 0
    for repetition in range(2):
        for number, (question, expected) in enumerate(tasks, 1):
            started = time.perf_counter()
            prompt = ("Classify the question. Return only HOURS for opening/closing times, "
                      "LOANS for borrowing, or OTHER for anything else. Question: " + question)
            result = request("/api/generate", {"model": model, "prompt": prompt, "stream": False,
                             "options": {"temperature": 0, "num_predict": 64, "num_ctx": 2048}})
            if result.get("done") is not True or not isinstance(result.get("response"), str):
                raise ValueError("Incomplete or missing model response")
            answer = result["response"].strip()
            passed = answer == expected
            accepted += int(passed)
            duration = result.get("eval_duration", 0)
            count = result.get("eval_count", 0)
            rate = count * 1e9 / duration if duration > 0 else None
            print(json.dumps({"repetition": repetition + 1, "case": number, "answer": answer,
                  "accepted": passed, "wall_seconds": time.perf_counter() - started,
                  "load_ns": result.get("load_duration"), "output_tokens_per_second": rate}))
    print(json.dumps({"accepted": accepted, "attempted": 10}))


if __name__ == "__main__":
    try:
        main()
    except (HTTPError, URLError, OSError, ValueError) as exc:
        raise SystemExit(f"Benchmark stopped: {type(exc).__name__}; no retry attempted") from None

The request and timing fields follow Ollama's generation API, and the model inventory comes from its list-models API. Generation duration is reported in nanoseconds, so the script converts it when calculating output tokens per second.

During preparation, the program's accounting and rate conversion were checked with synthetic API responses. No Ollama model was installed in the authoring environment, so no actual model throughput or quality result is reported here.

Interpret the measurements at the right level

Wall time includes the work observed by the client. Token-generation rate measures a narrower portion of the response. Loading, prompt processing, queueing, and other overhead can make a request slow even when token generation is fast.

The first measured request is not necessarily a cold start if the model was already loaded. Use the recorded load duration and a deliberate test procedure when comparing cold and warm behavior. The script uses non-streaming responses, so it does not measure time to first token.

A 60-second socket timeout is not a universal overall deadline for every stage. A production benchmark should define request deadlines and record failures explicitly. The demonstration stops at a request error and does not silently exclude it from a completed success report.

Five repeated cases are a smoke test, not a representative quality benchmark. Expand the task set, include held-out examples, and record resource use with the operating system's appropriate tools. Compare accepted outcomes, not just tokens per second.

Account for local operating effort

A local deployment still needs updates, storage, backups where appropriate, model provenance, and recovery. Electricity and hardware time may matter, especially for continuous workloads, but do not assume a universal cost advantage without measuring the actual utilization.

Keep the working model artifact and configuration identifiable. A mutable model name can point to different weights after an update. Preserve the digest, runtime version, settings, and test set with each result.

If a model update improves speed but fails the classification rules, keep the previous verified configuration available. Local control is useful partly because you can make that decision deliberately.

A reusable prompt

Evaluate local model use for this hardware and task. Separate raw weight memory from runtime and cache requirements. Check hardware support, exact model provenance and terms, quantization, context settings, and all external data flows. Benchmark accepted task outcomes, wall latency, loading, throughput, and operating effort. Label synthetic harness checks separately from real model measurements.

For students: use local work to understand the system

Students can inspect the benchmark without downloading weights. The memory calculations, response fixtures, and evaluation design are useful exercises on their own.

Where a school provides approved hardware and models, run a small non-sensitive task set and compare predictions with an answer key. Do not claim a device is private merely because the chat window runs locally. Verify the server and connected tools actually used.

Practice: measure one suitable configuration

Choose an installed model appropriate to your device, record its identity, and run the smoke test. Expand the cases before drawing a quality conclusion. If you cannot run a model, complete the hardware and data-flow assessment and label performance unmeasured.

Completion check: Resource and quality claims come from the actual device and task, and privacy claims match the complete observed data flow.

Stretch: Compare two quantization levels with the same held-out cases and context settings, including any differences in errors and review effort.