Teams adopting AIOps keep circling a few recurring questions, usually after a rough week. Someone retyped the same commands for the fortieth time, or someone wired up an automated fix on day two and it made an outage worse. This page answers the questions plainly. The short version of AIOps automation: do the step by hand a few times, see what never varies, script the fixed portion, and hand every judgment call to a human.
Should we automate a task the first time we see it?
No. The first run teaches you what the task really is. You discover the odd precondition, the flag everyone forgets, the permission that is missing on one server. If you script it before you know those things, you encode a guess, and guesses become outages when they run unattended.
Run it manually three or four times. Write each step in a scratch note as you go. Only the steps that repeat exactly are ready.
How do we tell the repeatable part from the judgment part?
Look at your notes from those runs and mark each step with one of two labels. “Same every time” means the inputs and the order did not change. “Depends” means somebody looked at something and chose. Collecting logs from a failing service, restarting a worker, opening a ticket with the usual fields: same every time. Deciding whether the logs point to a bad deploy or a downstream outage: depends.
Script the first group. Present the output of that group to a person for the second.
What does a good first automation look like?
Small, boring and reversible. Think of a script that gathers the standard diagnostics into one folder, or a job that pre-fills an incident ticket with the host, the time window and the last few deploys. Nobody argues about whether that script should exist, and if it breaks you lose a few minutes, not a customer.
The pipeline question is worth reading about in more depth on this site, for instance in a comparison of automated and manual data pipelines for analytics delivery, which shows the same trade-off at larger scale.
Is AI needed for this?
Often not. A shell script, a scheduled job or a small runbook-in-code covers most repeated chores, and it does so predictably. Machine learning earns its place when the input is messy, such as grouping thousands of similar alerts or reading free-text tickets. Even then, the sound sequence holds: handle a sample manually, learn the pattern, then let the model carry the volume while a human reviews the edge cases.
For a gentler introduction to the vocabulary, the piece on getting started with smart IT automation and modern systems is a decent warm-up.
What should never be automated away?
A short list, kept deliberately blunt:
- Approving changes to production when the blast radius is unclear.
- Deleting data. If a script must delete, make it list first and wait for a yes.
- Declaring an incident over.
- Anything that spends money or sends messages to customers.
- Reading the weird result that does not match the pattern.
A rule of thumb that holds up: if a wrong answer is cheap and obvious, automate. If a wrong answer is quiet and costly, keep a human on it.
How do we stop an automation from quietly rotting?
Give each script a plain header: what it does, who last touched it, what it assumes. Make it log what it did in a way a sleepy on-call engineer can follow during a night page. Put a date in the calendar to look at it again. Systems change underneath scripts, and the script that saved an hour a week can start failing silently after a library upgrade.
Add a tiny self-check where you can. If the script expects three servers and finds two, it should stop and say so, not carry on.
Our team is split. Half want to script everything, half want nothing touched. What now?
Run an experiment on one chore that both sides agree is annoying. Time it by hand across a week. Then automate only the part that was identical in every run, and compare the time spent and the errors made. Facts end arguments faster than opinions do. A written count of how many times a step was repeated, and how often it went wrong, is more persuasive than a conference talk.
Does this apply outside engineering?
Very much. Any team that fields repeated requests meets the same pattern. A medical tourism website like HipHospitals arranges its content around countries, hospitals and doctors, with a free-quote form and an online appointment request form on the site. Those forms exist because the same opening questions get asked of every enquirer, so the site collects them once, up front. That is administration of an enquiry, nothing clinical, and a good picture of a repeated step handled by a form.
How long until automation pays for itself?
Do a crude sum. Minutes per run, times runs per month, times months you expect to keep the task. Then add the time to build it and the time to maintain it. If the answer is close, do not bother yet. Hand it to a checklist and revisit when volume grows. If the answer is lopsided, build it this week.
Two traps appear regularly. The first is automating a task that is about to disappear because a platform change will replace it. The second is automating a process that is itself broken, which simply produces mistakes faster. Fix the process by hand first, even if that feels slower for a fortnight. A short stretch of manual repair is cheaper than months of fast, wrong output.
What is the smallest step we can take on Monday?
Pick one chore your team repeats. Open a shared note, and for the next few occurrences record every step and every decision. Circle what stayed the same. Turn those circled steps into a short script, a template or a checklist. Review it with the person who does the chore most often, because they know where it hides surprises.
The goal is not an automated shop floor. It is a team that spends its attention on diagnosis, design and the odd strange failure, while the dull, identical steps run by themselves and leave a clear trail behind them.
Quick recap
- Do it by hand first, several times.
- Separate “same every time” from “depends”.
- Automate the first group, show results for the second to a person.
- Keep dangerous steps behind a human yes.
- Write down what the automation does, and schedule a look at it.
Follow that order and the machine handles the monotony while your people keep the decisions that matter.