Ilse led a small machine learning team at a mid-sized logistics firm, and every month she sat through the same meeting. Finance put the cloud total on a slide. The room went quiet. Someone asked whether the models were getting too expensive, and no one could answer, because nobody knew what too expensive meant. What follows is how that team stopped debating the total and started pricing a single prediction. It is a composite sketch, but the pattern will be familiar to anyone who runs AI workloads.
The meeting that went nowhere
The team ran a demand forecasting model that fed purchasing decisions. It trained on a schedule, served predictions through an internal service, and fed a set of reports. Finance saw one line item that mixed all of this together with experiments, notebooks and storage. Engineers saw dashboards full of utilisation graphs. Neither picture matched the other, so each side defended its own.
The turning point came when a finance analyst, Tomas, asked a plain question: what does one forecast cost us? Nobody had an answer. They had plenty of metrics, but none of them was expressed in a unit that a purchasing manager would recognise. That gap is where this whole practice begins.
Choosing a unit the business recognises
Ilse resisted the first suggestion, which was cost per GPU hour. It was precise, but it described the supplier’s pricing, not the team’s output. A purchasing manager does not care about hours. She cares about forecasts delivered for the warehouses she manages. After some back and forth the team settled on two related units.
- Cost per prediction served. The recurring price of answering a request, covering compute, memory and the data lookups that surround it.
- Cost per retraining cycle. The price of one full refresh of the model, spread across the predictions it then supports.
Having two units mattered. A model can be cheap to serve and expensive to refresh, or the other way round, and a single blended number hides which. If you want a broader view of why decisions need such framing, the piece on how AI supports better business decisions covers the demand side; this story is about the cost side of the same coin.
Building the first rough number
The first version of cost per prediction was ugly. Tomas pulled the month’s charges for the serving environment, Ilse’s team counted requests from their logs, and they divided one by the other. Shared items, such as the feature store used by several models, were split by a simple rule they wrote down: proportional to the number of requests each model made. It was not perfect. It was good enough to start a conversation, and the team agreed to revisit the rule after a quarter.
Something happened as soon as the number existed. People started asking sharper questions. Why did predictions made late at night cost the same as busy-hour ones? Why was a particular batch of requests so much heavier than the rest? The team realised that a few warehouse regions asked for forecasts at a far finer granularity than the purchasing process ever used. Those requests were expensive, and almost nobody read the results.
What drove the cost, and what did not
The surprise was how little of the unit cost came from the model itself. The neural network that everyone worried about was a modest slice. Most of the price sat in the data preparation before each request and in keeping redundant copies of the serving stack warm in case of traffic spikes that rarely came. Had the team stared at the monthly total forever, they would never have seen this, because the total blended everything.
It reminded Ilse of something she had read about optimizing production with AI in a factory setting: you cannot improve a line you only see as one lump. You have to see the stations. A unit cost is how you see the stations of a cloud workload.
The trade-offs they could finally discuss
With cost per prediction on the table, the argument changed character. Instead of “the bill is too high”, the team debated concrete choices.
- Should fine-grained regional forecasts be produced on demand, or computed once a day and cached?
- Is it worth keeping extra serving capacity warm for rare peaks, or should the system accept a slower answer during them?
- How often does the model genuinely need retraining, given how little the underlying patterns shifted between cycles?
Each question had an owner and an effect that could be stated in the unit. The purchasing manager, who had never joined the cloud conversation, ended up deciding the first one herself. Her answer, that daily was plenty for the finest level of detail, shifted the unit cost without anyone touching the model. Engineering felt heard rather than blamed, and finance felt informed rather than ignored.
Where the team learned to run this as a practice
As the habit spread, Ilse wanted a shared language rather than a set of private spreadsheets. She and Tomas looked at FinOpsSchool, a provider of FinOps training and certification, as a place where people from engineering and from finance study side by side, learning to put cloud costs in front of everyone who shapes them. The appeal was not a tool but a common vocabulary that both sides could use in the same meeting.
What stayed true afterwards
A few months later the cloud total had not become small, and the team never claimed it should. What changed was that the total stopped being frightening. When it rose, they could say which unit moved and why. When a new model was proposed, the first question was what one prediction from it would cost and who would pay. Sometimes the answer was that the value justified a higher price. Sometimes it was a reason to simplify the design before building it.
The lesson is portable. A big number invites suspicion; a small, recognisable number invites curiosity. If your AI workloads feel like a black box on the invoice, pick the unit your business already talks about, divide, and keep the first attempt rough. The precision will come with use, and the arguments will get a great deal more useful long before it does.