Costing an AI feature before you build it

Token pricing is the visible cost of an AI feature and rarely the largest one. A team that budgets only for inference will be roughly right about their API bill and badly wrong about what the feature actually costs to own — because the recurring expense sits in evaluation, retries, and the humans who stay in the loop.

Here is how to build an estimate before the feature exists.

Cost per successful outcome, not per API call

The unit that matters is one user-visible outcome the user accepted — a summary they kept, an extraction that passed validation, a draft they sent. Everything spent getting there belongs in the numerator:

• The initial call
• Retries after a malformed or refused response
• Additional calls in a tool-use or multi-step loop
• Generations the user rejected and regenerated
• Any validation or grading call you run on the output

Teams that measure cost per API call routinely discover a 2–4× gap between their model and their invoice, and the gap is almost always here. A retry factor of 1.2 to 1.5 is realistic for anything involving tool use or structured output; higher if you enforce a strict schema.

Build the per-request model from four numbers

Prototyping ten real requests and measuring their actual token counts takes an afternoon and beats any amount of estimating from intuition. Token counts are one of the few inputs in software estimation you can simply measure before you commit, and it is strange how rarely anyone does.

Which lever moves cost the most?

Model choice, by a wide margin. The price gap between tiers within a single model family is typically an order of magnitude, and most production workloads contain a majority of requests that a smaller model handles correctly.

The pattern that works is tiered routing: send everything to the small model, detect failure, escalate only those to the large one. Detection can be as simple as schema validation, a confidence threshold, or a cheap grader call. Teams that adopt this routinely cut spend by more than half without a measurable quality change — because the large model was being used on easy requests it never needed to see.

Two smaller levers, both usually available:

• Prompt caching. If a long system prompt or a fixed document is resent on every request, caching it converts the dominant cost line into a fraction of itself. The cheapest token is the one you do not resend.
• Context discipline. Retrieving twelve documents when three would do multiplies input cost by four and frequently makes output quality worse. Trimming retrieval is one of the rare changes that improves cost and quality simultaneously.

What are the costs that are not inference?

These are recurring line items, not one-off project costs, and they are where most AI feature budgets go wrong.

• Evaluation. A test set, the harness to run it, and someone who looks at regressions. Without this you cannot change a prompt or upgrade a model without gambling.
• Human review. On features where a wrong answer has real consequences — anything customer-facing, financial, clinical or legal — a reviewer is part of the system. Review cost frequently exceeds inference cost by an order of magnitude, and unlike inference it does not fall as models get cheaper.
• Monitoring and tracing. Per-request logging of prompts, responses, latency and cost. Observability platforms for LLM workloads price per trace and the bill scales with your success.
• Guardrails. Input filtering, output validation, PII handling. Often another model call, and therefore another line in the per-outcome cost.
• Prompt maintenance. Every model version change is a regression risk against prompts tuned for the previous one. Budget engineering time for this on a recurring basis; it does not go away.

Inference cost falls every year. Evaluation and human review do not. A three-year model that assumes today's cost mix will hold is usually wrong in a direction that flatters the business case.

A worked shape

For a feature handling 50,000 outcomes a month, with a tiered routing setup:

1. Per-outcome inference. 85% of requests on the small model, 15% escalated, retry factor 1.3. Compute both tiers separately and add — a blended average hides which tier you should be optimising.
2. Monthly inference. Per-outcome cost × 50,000.
3. Evaluation. One engineer-day a month maintaining the test set, plus the inference cost of running it. Both real.
4. Review. Fraction of outcomes reviewed × minutes per review × loaded hourly cost. Model this honestly even if the answer is uncomfortable — especially then.
5. Platform. Tracing, storage, vector database if you retrieve.
6. Sensitivity. Recompute at 3× volume. Features that look cheap at pilot scale sometimes stop working economically at production scale, and it is much better to find out in the model.

Present the result as a cost per successful outcome alongside the monthly total. The per-outcome figure is the one that lets a business decide whether the feature is worth having, because it can be compared directly against what the outcome is worth.

Cost is not the only constraint

Two others belong in the same model and are routinely omitted.

Latency. A cheaper model that takes eight seconds may be unusable in a flow where the user is waiting. Cost per outcome should be read alongside a latency budget, and sometimes the expensive fast model is the only viable option — which is a legitimate finding, not a failure of cost control.

Rate limits and capacity. Provider throughput ceilings are a real design constraint at volume. A model that is cheapest per token but rate-limited below your peak load has a hidden cost: the engineering to queue, shard or fall back.

Before you commit

1. Have I measured token counts on real requests rather than estimated them?
2. Is my denominator successful outcomes, with retries in the numerator?
3. Have I modelled a tiered routing option, not just one model?
4. Are evaluation, review and monitoring in the recurring costs?
5. Does the model still work at 3× volume?
6. Have I stated a latency budget alongside the cost?

Answer those six and you have a number that survives contact with a finance team — and, more usefully, one that tells you where to spend the engineering effort. In most AI features the cost structure points at exactly one change worth making, and the point of the model is to find it before you have built the expensive version.

This is the analysis that feeds directly into a build-versus-buy decision: without a defensible per-outcome cost, the two sides of that comparison cannot be put on the same axis.