The true cost of enterprise AI: price the workflow, not the token

A cheap model can still power an expensive system

Token prices are visible, so they dominate early cost conversations. Production cost is broader: retrieval, vector storage, tool calls, orchestration, observability, evaluation, security controls, human review, and the rework created by weak outputs.

The right unit is not cost per token. It is cost per successfully completed business outcome: a reconciled payment, resolved support case, reviewed document, or qualified lead.

Build a cost model around demand

Start with request volume and shape. Separate short routine tasks from long-context investigations, identify daily peaks, and measure input, cached input, output, retrieval, and tool use for each class.

AWS guidance recommends treating the preproduction cost model as a living document rather than a one-time estimate. That turns cost from a launch surprise into an architectural input.

• Volume: requests, concurrency, seasonality, and retry rates.
• Compute: input, output, reasoning, embeddings, and hosted inference.
• System: storage, retrieval, gateways, guardrails, and monitoring.
• Operations: evaluation, human review, corrections, and incident response.

Four levers change the economics

First, route each task to the smallest model that meets its quality threshold. Second, reuse stable prompt prefixes so provider-side caching can reduce repeated input processing. Third, batch work that does not need an immediate response. Fourth, cap agent loops and tool calls before an edge case becomes an open-ended bill.

Current provider features reinforce this approach: prompt caching rewards repeated context, while asynchronous batch processing can materially reduce model cost for delay-tolerant workloads. Exact pricing changes, but the architecture principle is stable—design around latency needs instead of paying real-time rates for every job.

Connect spend to operating value

Cost control without an outcome metric encourages teams to optimize the wrong thing. A smaller bill is not progress if adoption falls or manual corrections rise.

Track cost per accepted output, cycle time, exception rate, downstream rework, and the financial value of the completed workflow. This reveals whether AI removed a constraint or merely moved it to another team.

The executive question

Ask: at expected volume, what does one reliable outcome cost, and how does that compare with today’s process? If the team cannot answer, the system is still a technical experiment—not yet an operating investment.