Priya (not her real name, and not a real person) had been training models for three years when her pager went off at a quarter to three in the morning. A recommendation service had started returning nonsense. She knew how to retrain a model, tune a feature pipeline and read a loss curve. She did not know how to run a room full of worried people. That gap, the ability to steer a team through an unplanned disruption, is what the field calls IncidentOps, and for anyone shipping AI-powered products it is turning into a surprisingly valuable thing to be good at.
What follows is a career-minded tour of that skill set, covering the order to study things and a modest way to show them on a résumé. The scenario above is a generic illustration, not a record of an actual event.
First, what the skill actually is
The term, per the Ops Atlas page cataloguing IncidentOps, is an informal, fairly rare label for incident management: declaring a problem, taking command, communicating, and learning from it. The Atlas admits there is no formal discipline behind it, and no certification ladder to climb. For you, that is good news. You cannot buy the skill in a weekend course, so people who can demonstrate it stand out.
It also matters for AI work in particular. Models fail in odd ways: a data feed changes shape, a dependency slows down, a score drifts. Whether the cause is clear or murky, somebody has to decide who is in charge, what readers or customers hear, and which quick fix to try first.
Skill one: learning when to call it
Priya’s first mistake was waiting. She spent forty minutes poking at logs alone because she did not want to wake anyone for something that might be minor. The Atlas recommends written criteria for declaring an incident. The example it cites from Google’s practice is any of: another team is required, customers can see the problem, or it remains unsolved after about an hour. Under that rule she would have raised the flag at a clear moment.
Career angle: ask your team whether such criteria exist. If not, offering to draft them is an easy, visible contribution that shows judgement rather than coding speed.
Skill two: wearing a role instead of fixing everything
When Priya finally declared, three colleagues joined and all of them started typing commands. The Atlas describes the healthier pattern. One person coordinates and explicitly does not fix things. The group that touches production stays small. Someone else posts updates to everybody else. Roles are hats, not job titles, so a junior engineer can be coordinator on the call if they hold the rank that matters there.
Practise this on low-stakes problems. Volunteer to coordinate a small outage and notice how different it feels to ask questions instead of answering them. It is a muscle, and it grows with repetition.
Skill three: mitigating before understanding
The recommendation service had been updated that evening. A coordinator who has absorbed the Atlas’s advice does not demand a theory; she asks whether rolling back is possible. The entry lists generic mitigations such as rollback, traffic draining, switching a feature off, failover and load shedding, and points out that they need the fault located, not explained. For people from a research background, this is the habit that is hardest to build, because curiosity pushes you toward the why.
- Learn your own system’s quickest safe reversal: a model version pin, a feature flag, a cached fallback.
- Write the steps down so a tired colleague can follow them.
- Test the reversal once when nothing is wrong.
Skill four: writing for people who are anxious
Communication is the skill most often missing from technical résumés. The Atlas suggests a first public note roughly five minutes after the response begins, another shortly after to describe scope, and then a refresh no less often than every twenty minutes through the opening two hours, ending with a closing note that says plainly whether any data was lost. One popular hosted status page keeps things simple with four labels: investigating, identified, monitoring, resolved.
A good exercise is rewriting a technical status line for a non-technical reader. “Inference latency elevated on region B” becomes “Some customers see slow recommendations; we know where the fault is and are reversing a recent change.” Do this ten times and your updates improve noticeably.
Skill five: reading numbers sceptically
Interviewers in this area like to ask about metrics. The Atlas supplies a good answer: mean time to repair has at least four meanings, and incident durations are lopsided, so averages jump around even when nothing has improved. In its lab example, one severe incident moved a mean by more than three-quarters of an hour. The stronger practice is to report distributions, how much reliability budget an incident consumed, how heavy the pager load is, and how reliably follow-up actions are finished.
If you can say this crisply, you signal that you have moved beyond the “make the line go down” mindset. Pair it with the observation that the person who owns the number should know its limits.
Skill six: running a learning review
Two days later, Priya’s team met. The facilitator had not been on the call, which the Atlas, quoting an e-commerce company’s practice, recommends. They named an owner before the call ended, kept the list of actions short, and gave each a single owner and date. They also resisted writing “human error”. The Atlas cites a safety scholar’s remark that hindsight distorts how we judge people after the fact, and suggests stopping at system conditions: an unclear release note, a missing check in the pipeline.
Career angle: write a review of an incident you were involved in, redact it, and keep it in your portfolio. It shows maturity better than another model benchmark.
A learning path you can follow
- Weeks one to two. Read the Atlas entry and your own company’s severity scale. Shadow the on-call person.
- Weeks three to six. Join an exercise or tabletop session. Volunteer as scribe, the person who records the timeline, which is the gentlest first role.
- Weeks seven to ten. Try the communications seat for a small incident.
- Afterwards. Take the coordinator seat on a rehearsal, then a live low-severity event.
Still choosing a direction? The starter list in our beginner roadmap for DevOps engineers pairs nicely with the sequence above. And for the reasoning side of the job, the article about AI-assisted decisions in companies shows why calm decisions under pressure matter to the people who rely on your systems.
Presenting the skill without overselling it
On a résumé, avoid grand titles. Instead of “incident management expert”, write something concrete: “coordinated the response to a customer-facing model outage; introduced written declaration criteria; shortened the gap between detection and first customer update.” Skip invented credentials, and never quote a figure you could not defend in an interview.
Priya’s night ended at five. The rollback took ten minutes once somebody finally suggested it. What changed her career was not the model, it was the morning after, when she wrote the review, offered to draft the declaration rules and agreed to rotate through the coordinator role. Six months on, she was the colleague people wanted on the call. That is the quiet promise of IncidentOps for AI practitioners: calm, structured judgement is rare, learnable and visible.