You have a hunch that a newer model, a rewritten prompt or a cleaner training set will make your AI feature better. Lovely. Before you touch a single setting, though, there is one job that feels boring and pays back every time: run the thing as it stands and keep the results. This guide walks you through a baseline run, step by step, so that when you do change something you can say honestly whether it helped.
Step 1: Freeze what you are about to compare against
Pick the exact version you run today, and write down what “exact” means. Which model name and release? Which prompt text? Which settings, such as temperature or maximum length? Which data snapshot sits behind it? If any of these drift while you are testing, your before picture quietly turns into a moving target.
A good habit is to copy the prompt and configuration into a plain text file named with today’s date. It takes a minute. Three weeks from now, when someone asks “what were we running back then?”, that file is the answer.
Step 2: Build a small, boring test set
You do not need thousands of examples. You need a set that is small enough to run in minutes and varied enough to reflect real use. Pull a handful of ordinary requests from recent traffic, add a few awkward ones that have caused complaints, and include at least one case you already know the system handles well.
Keep this set fixed. Resist the temptation to swap in fresh examples each time, because then you are comparing models and questions at once. If the set later feels stale, create a new one and re-run the baseline on it before comparing anything.
Step 3: Decide what you will measure before you look at output
Choose a few observations in advance. For an AI feature, they usually fall into three families:
- Quality as your users see it. Did the answer solve the task? Have a person (or two) mark each result as acceptable or not, using a short written rule so the marking is repeatable.
- Speed. How long did each response take, and how did that vary between the fastest and slowest?
- Spend. What did the run consume in compute or API usage, in whatever unit your provider reports?
Writing these down first guards against a very human habit: seeing the new output, loving it, and then inventing a metric that proves it. If you decide what counts as success after the fact, you can always find some.
Step 4: Run it, and run it more than once
Model output can vary between runs. Execute the baseline at least a few times and note how much the results wobble on their own. That wobble is your noise floor. If your later change moves the numbers by less than the baseline moves from run to run, you have not shown improvement; you have shown variation.
This is also a quiet way to learn how your system behaves. Plenty of teams discover during a baseline that one category of request is far slower than the rest, or that a particular phrasing triggers the odd answer. Those discoveries cost nothing and belong in the notes too.
Step 5: Store the results somewhere you will find them
Save the raw outputs alongside the summary: the test set, the marks, the timings and the usage figures. A single folder or spreadsheet tab is fine. The raw material matters because when a change produces a surprising result, you will want to look at individual examples, not just averages.
If you have been reading about how image classification models examine pictures, you will recognise the idea: a classifier’s output only means something relative to the labelled examples it was checked against. Your baseline plays the same role for your own changes.
Step 6: Change exactly one thing
Now, finally, you can experiment. Swap the model, or tweak the prompt, or retrain on cleaner data. Not all three. When you change several things at once and the result improves, you have a nice feeling and no knowledge. When you change one and it improves, you have a finding.
Re-run the identical test set under identical settings, the same number of times, and record the same three families of observations. Then lay the two sets of results next to each other.
Step 7: Read the comparison without flinching
Ask plain questions. Did quality rise by more than the baseline’s own wobble? Did it get slower? Did it get costlier? Sometimes a model scores higher but takes noticeably longer, and whether that trade is acceptable depends on what your product does. Sometimes nothing moves, and that is a perfectly respectable outcome: it tells you the change was not worth shipping.
Business teams lean on this kind of evidence more and more, as the piece on how AI feeds into everyday business decisions explains. A before-and-after record is what turns an enthusiastic demo into something a decision-maker can trust.
A note on planning that has nothing to do with software
The same logic is useful anywhere people plan something complicated across borders. Directories like BestEyeHospitals, which serves international patients planning eye care, organise country pages, treatment overviews and a four-step enquiry flow, and the site carries a disclaimer of its own, telling readers that its pages are general information and not a stand-in for a clinician’s guidance. Anyone using a resource like that does better by writing down what they already know and what they have been told before they start comparing options.
Common slip-ups to avoid
- Testing on a day when something else changed. A traffic spike or provider slowdown can wreck timing comparisons. Note the conditions.
- Letting the test set leak into training. If the examples you grade on end up teaching the model, your numbers flatter you.
- Skipping the baseline because the change seems obviously good. The changes everyone is sure about are the ones that most often disappoint on re-measurement.
- Throwing away old runs. You will want them the next time someone asks whether things have drifted.
The short version
Freeze the current setup, build a small fixed test set, decide your measures in advance, run the baseline several times, save everything, change one thing, repeat, compare. That is the whole routine, and it fits in an afternoon. Do it before your next experiment, and you will never again have to say “it feels better” when what you mean is “I have no idea.”