Manage Cost, Latency, Scaling, and Model Changes

The useful unit of cost is a completed result that meets the task's requirements.

The learning center replaces one model configuration with a cheaper option. The model bill falls. Staff members then spend longer correcting answers, and more requests end without an acceptable result.

Was the change an improvement?

A token invoice cannot answer that question. You need the cost of the complete workflow, the number of acceptable outcomes, and the time users experienced. Efficiency means improving those measures while preserving the requirements that make the service useful.

Set service objectives before optimizing

Define the task boundary first. In this lesson, one request asks a public question and ends with an evaluated answer or an explicit failure. An acceptable result meets the same correctness, evidence, and permission requirements across every configuration.

Choose a quality threshold, a latency objective, and a cost measure. The synthetic exercise below requires at least 90% acceptable results and a reported 95th-percentile latency no greater than ten seconds. These are teaching assumptions, not universal targets.

The 95th percentile describes the upper part of the latency distribution: under the chosen measurement convention, roughly 95% of measured requests finish at or below that duration. State whether you include failed and timed-out requests, and how you account for them. Excluding the slowest failures can make a broken service look fast.

Throughput measures how much work finishes in a period. Concurrency measures how many requests are active at once. They are related but not interchangeable. A configuration that handles short questions well may struggle when many long-document requests arrive together.

Count the whole workflow

For each configuration, record attempted requests, acceptable outcomes, provider and tool charges, retries, review time, and allocated operating costs. Avoid counting one request twice merely because it made a second model call.

Review time should include time spent on rejected results. It was still consumed. Likewise, failed model calls may still contribute charges or infrastructure costs, depending on the provider and architecture.

Allocate shared costs consistently over a stated volume and period. Hosting, maintenance, and setup effort do not vanish because they are absent from an API receipt. For a small pilot, show both variable cost and the effect of allocating setup costs; the allocation may change substantially as usage grows.

Use this equation:

Cost per acceptable result =model and tool cost + review cost + allocated operating costnumber of acceptable results

When no results are acceptable, this ratio has no finite value. Do not report zero cost per acceptable result simply because the denominator is zero.

Compare three configurations with the same standard

The following records are fictional. They are not provider prices, measured model benchmarks, or product recommendations. Each configuration processes the same hypothetical set of 100 requests. Review labor is valued at $30 per hour, and each receives a $5 allocation for infrastructure and maintenance that is separate from answer review.

Configuration B has the lowest model and tool bill, but the greatest total cost and an unacceptable success rate. Configuration C meets the exercise thresholds and has the lowest cost per acceptable result among the eligible choices.

That conclusion is limited to the supplied records. A real selection also needs the evaluation's severe-failure checks, representative workload coverage, and uncertainty assessment. A ten-second latency threshold does not compensate for a permission failure.

Execute the accounting instead of eyeballing it

Save the following complete standard-library program as cost_lab.py and run python3 cost_lab.py. It uses decimal arithmetic for the currency calculations, checks the totals and eligibility decisions, and verifies behavior when there are no acceptable outcomes.

from decimal import Decimal as D

CONFIGS = [
    {'name': 'A', 'attempted': 100, 'accepted': 90, 'model_tools': '5',
     'review_minutes': 60, 'allocated_cost': '5', 'p95_seconds': 8},
    {'name': 'B', 'attempted': 100, 'accepted': 80, 'model_tools': '2',
     'review_minutes': 100, 'allocated_cost': '5', 'p95_seconds': 5},
    {'name': 'C', 'attempted': 100, 'accepted': 95, 'model_tools': '4',
     'review_minutes': 30, 'allocated_cost': '5', 'p95_seconds': 7},
]


def score(row):
    attempted, accepted = row['attempted'], row['accepted']
    if not 0 <= accepted <= attempted or attempted <= 0:
        raise ValueError('Invalid outcome counts')
    review = D(row['review_minutes']) * D(30) / D(60)
    total = D(row['model_tools']) + review + D(row['allocated_cost'])
    if min(review, D(row['model_tools']), D(row['allocated_cost'])) < 0:
        raise ValueError('Negative cost')
    rate = D(accepted) / D(attempted)
    return {'name': row['name'], 'total': total,
            'cost_per_accepted': total / D(accepted) if accepted else None,
            'eligible': rate >= D('0.90') and row['p95_seconds'] <= 10}


def main():
    results = [score(row) for row in CONFIGS]
    assert [r['total'] for r in results] == [D(40), D(57), D(24)]
    assert [r['eligible'] for r in results] == [True, False, True]
    winner = min((r for r in results if r['eligible']),
                 key=lambda r: r['cost_per_accepted'])
    assert winner['name'] == 'C'
    zero = dict(CONFIGS[0], accepted=0)
    assert score(zero)['cost_per_accepted'] is None
    assert not score(zero)['eligible']
    for result in results:
        print(result['name'], f"${result['cost_per_accepted']:.3f}",
              'eligible=' + str(result['eligible']))
    print('PASS: full-cost totals, quality gate, and zero-output case checked')


if __name__ == '__main__':
    main()

The program was executed for this manuscript and printed:

A $0.444 eligible=True
B $0.712 eligible=False
C $0.253 eligible=True
PASS: full-cost totals, quality gate, and zero-output case checked

These results validate the arithmetic and gate for the authored records. The code does not measure latency, calculate a percentile from request traces, or run a live service. The p95 figures are supplied assumptions.

For an actual comparison, collect individual request records and calculate latency percentiles consistently. Keep the raw measurements so you can examine slow cases, retry patterns, and differences between task groups.

Apply efficiency techniques where they fit

Shorter context can reduce unnecessary processing, but deleting the paragraph containing a policy exception can lower answer quality. Compare a focused evidence selection process against the baseline before accepting the savings.

Batching can improve throughput when the serving system supports it. It may also add waiting time while a batch forms. A nightly classification job and an interactive help desk have different latency needs.

Routing assigns requests to different configurations. Use information available before the answer is known: task type, input length, required tools, supported language, or a validated difficulty signal. Do not design a retrospective router that selects the correct answer using ground-truth labels unavailable in deployment.

A fallback can be another evaluated configuration, a narrower supported response, or a human handoff. It must preserve the original permissions and evidence requirements. If no configuration can satisfy those conditions, an explicit inability to complete the task is preferable to presenting an unsupported answer as successful.

Parallel work can reduce elapsed time when tasks are sufficiently independent, as Article 122 showed. Account for extra calls, coordination, and review. More simultaneous workers do not automatically lower the cost of useful work.

Cache only results that remain valid for the requester

An application answer cache reuses a previous output. A provider's prompt-processing cache may reuse internal computation for repeated input. Those are different mechanisms with different semantics. Verify the current provider documentation before relying on a particular pricing rule, retention period, or cache behavior.

For an application cache, a normalized question alone is often an inadequate key. The result may depend on the authorized data scope, source version, language, task settings, and model or prompt configuration. Two people asking identical words may be entitled to different information.

Recheck authorization when serving a cached result. Permission changes must invalidate or make inaccessible affected cached entries. Source updates also require an explicit invalidation or expiry policy. A short expiration interval limits some staleness but does not guarantee immediate removal after access is revoked.

Be careful with cached failures. A temporary source outage should not cause an “information unavailable” response to persist long after the source recovers. Record why the result was cached and how that condition will be reconsidered.

Handle load without multiplying the problem

A queue absorbs short bursts; an unbounded queue can conceal overload while users wait indefinitely. Set capacity and waiting limits, propagate deadlines, and provide a clear response when the service cannot accept more work.

Measure the resource requirements of requests, not only their count. Long contexts and multiple tool calls can make two requests very different workloads. Google's SRE discussion of overload explains why requests per second alone can be a poor proxy for resource demand and why services need explicit behavior when capacity is exhausted. Google SRE: Handling Overload

Bound retries and coordinate them across layers. If the user interface, workflow runner, and provider client all retry independently, one request can expand into many attempts. Respect documented rate-limit signals, add appropriate delay, and stop when the overall request budget is exhausted.

For operations with external effects, preserve the operation identity and reconcile unknown outcomes as Article 121 described. A user clicking “try again” should not silently create a second intended reservation when the first result is merely unknown.

Test overload using synthetic work in an environment intended for such tests. Record when admission stops, how queued requests expire, and whether the system recovers when load falls. A successful quiet-period benchmark does not establish burst capacity.

Treat model changes as system changes

Record the model identifier or revision, inference settings, prompt version, tool contracts, source collection, retrieval configuration, and relevant dependencies. Some services expose immutable revisions; others expose moving aliases. Document the actual guarantee available to you.

Before a change, rerun the release evaluation and compare the baseline on the same cases. Check formatting, tool arguments, refusal behavior, citations, latency, and cost as well as task accuracy. A more capable model can still behave differently in a way that breaks your workflow.

Use a controlled rollout with explicit stop criteria where the application warrants it. Inspect a limited portion of eligible traffic before expanding. Retain an evaluated fallback configuration and confirm that it can still run with current inputs and dependencies.

Rolling back a model does not undo messages, purchases, database writes, or cached outputs already produced. Recovery for those effects needs its own procedure. Record enough information to identify the affected work without retaining unnecessary sensitive content.

For students: optimize your project budget without hiding your time

Use a small, authorized project such as classifying synthetic library questions. Compare a simple baseline, an AI-assisted configuration, and one proposed improvement using the same held-out cases.

Record your review and correction time. If your study has no paid labor, report minutes separately and, if useful, show a clearly labeled hypothetical monetary rate. Do not present that rate as an actual expense.

Keep a fixed spending or compute limit. For a local model, note hardware and energy assumptions instead of calling the run free. For a hosted model, use current official prices when estimating charges and verify actual usage afterward.

Explain why the preferred option meets the task requirements. A configuration that is cheapest because it skips difficult cases needs its completion rate and supported scope stated plainly. Follow course rules for assistance and disclose how AI contributed to the project.

A reusable prompt

Compare these configurations using the same acceptance criteria. Calculate total cost and cost per acceptable outcome, including failed attempts, review time, and stated allocations. Report latency and its measurement convention. Apply the quality and severe-failure gates before recommending an eligible option. Distinguish measured records from assumptions. Propose one efficiency experiment, specify its held-out evaluation, and identify the conditions under which the change should be rolled back.

Practice: make the cheapest acceptable choice

Run the synthetic accounting example. Then change one cost or outcome count and predict the effect before rerunning it. Explain why Configuration B fails the exercise even though its model and tool charge is smallest.

Create a measurement plan for a real bounded task. Specify the request boundary, acceptance rubric, review-time capture, allocation period, latency records, and change procedure.

Completion check: Your preferred configuration meets the stated quality and latency requirements, and its accounting includes unsuccessful work and human review.

Stretch: Evaluate a routing policy on held-out tasks using only signals available at routing time. Compare its complete cost and subgroup results with a fixed-configuration baseline.