The best SOC analyst I ever worked with could look at a handful of events and tell you within a minute whether they were worth pulling the thread on. She was not merely following rules; she was applying judgment shaped by three years of triage experience. Her calls were fast, and they rarely sent the team down an empty path.
A year after she left, the analyst who replaced her had the same tools, the same queue, and the same SOP. The team’s true-positive rate dropped. The replacement was not less capable, but one part of the operating system around the role was gone: the queue no longer told anyone whether they had been right.
That sequence did not prove a single cause. Alert mix, detection changes, and incident volume could all have moved the number. What I could observe was that one informal mechanism had disappeared: the senior analyst had reviewed borderline calls, challenged weak closures, and carried incident outcomes back to the people who first triaged them. The tooling preserved the queue and the SOP. It did not preserve that return path.
A senior analyst’s departure is a revealing case study in what feedback does for judgment. Duhigg and a stack of professional poker players both have something to say about fixing it.
The SOC runs on priors
An analyst triaging an alert is doing applied Bayesian reasoning, whether they have ever heard the word or not. They take a prior (“alerts from this detection are almost always false positives”), update it with evidence (“but this host is in the PCI zone and the process is certutil.exe”), and arrive at a posterior (“worth a deeper look”). The whole thing happens in seconds, mostly below conscious thought.
When the priors are well-calibrated, triage is fast and accurate. When the priors are stale or wrong, two failure modes appear. Either the analyst escalates too much (wasting IR capacity on nothing) or escalates too little (missing the one alert that mattered). Both failures look the same from the outside: a queue getting worked. Neither shows up in a metrics dashboard that only counts alerts closed.
The base rate problem in ML-based detection is the same problem one level up the stack. A 99%/99% classifier generates roughly 1,000 false positives for every true positive when the base rate of malice is one in 100,000. A human analyst without feedback is making repeated classifications on the same distribution without seeing enough labels to recalibrate.
How priors decay
Priors decay when the feedback loop breaks. In most SOCs, the loop is broken by default. Here is what the typical flow looks like:
- Alert fires.
- Analyst triages, closes as false positive or escalates to IR.
- IR investigates, writes a report, closes the incident.
- The report lives in a ticketing system that the original analyst does not read.
Step 4 is where the learning dies. The analyst who made the call in step 2 never finds out what the actual ground truth was. If they closed it as false positive, nothing contradicts them. If they escalated it, they see “incident closed: benign” or “incident closed: true positive” only for the minority of escalations they personally followed up on.
The result is a one-sided learning signal. Analysts get copious feedback on their over-escalations (IR pushes back, the ticket gets reclassified, the Slack thread goes quiet) and almost no feedback on their under-escalations. The prior drifts in exactly one direction: “probably nothing.”
Insight
Asymmetric feedback builds asymmetric priors
If the only feedback you get is on the alerts you escalated, your prior on “this is worth escalating” gets sharper over time while your prior on “this is safe to close” drifts unexamined. The failure mode is silent: your false-negative rate rises, but you never see the evidence, because the evidence is in incidents that never got opened.
Duhigg makes a version of this point in Smarter, Faster, Better, through Beth Crandall’s research in neonatal intensive care units around Dayton. The nurses who outperformed their peers were not the ones with better pattern recognition in the passive sense. They were the ones running an explicit “what would I expect to see if this baby were healthy?” narrative, so that a mottled skin tone and a slightly distended belly registered as a departure from a prediction rather than as unremarkable details. The skill is holding an expectation and noticing the gap.
Prediction is only half of it, though. An expectation you never score is a guess you keep making. Kahneman and Klein’s synthesis on intuitive expertise draws the important boundary: experience improves judgment when the environment contains learnable regularities and the decision-maker receives adequate feedback. A SOC can offer the first condition while doing a poor job of the second.
Most SOC analysts never get scored.
The poker version of the same lesson
Annie Duke, the former professional poker player, makes the same point more sharply in Thinking in Bets (2018). A bad decision that yields a good outcome is indistinguishable, from the outside, from a good decision, unless you force yourself to separate process from result. Duke calls the conflation “resulting,” and argues that professional poker players outperform amateurs in part because they have trained themselves to evaluate decisions on process quality alone, independent of the hand’s outcome. In security triage, the same failure shows up as the analyst who closes a wave of alerts in bulk on Friday afternoon, suffers no visible consequence over the weekend, and reinforces the habit. The process was bad. The outcome was fine. No learning occurs, because the outcome was the only thing anyone measured.
A SOC that takes calibration seriously has to build a system that scores decisions against ground truth independently of whether the outcome was good or bad. That is harder than it sounds, because ground truth in security is expensive to produce. You only find out the alert was a real attack if someone investigates deeply enough to prove it, which, for alerts that were closed as false positives, basically never happens.
What a working feedback loop looks like
There is no clever trick here. A SOC that maintains calibrated priors invests in three things that most SOCs skip:
Deeper review of closed alerts. A sampled subset of alerts closed as false positive should receive an independent review, ideally with additional endpoint, identity, or network evidence. A second opinion is not automatically ground truth, but a documented adjudication process can produce a more useful label and expose recurring disagreements. Deception-based detections can provide unusually strong labels, although authorized testing, accidental discovery, and misrouted automation still need to be excluded before treating every hit as hostile.
Post-incident attribution to the original triager. When an incident resolves, the ticket should route back to the analyst who first saw the alert, with the ground-truth label and the delta between their call and the outcome. Not as a blame artifact. As a calibration signal. This is the piece the NICU analogy does not supply on its own: a nurse who flags a baby learns within hours whether she was right, because the labs come back and the child either has sepsis or does not. A SOC has to build that return path deliberately, because nothing about the ticketing system produces it by default.
Deliberate calibration exercises. Periodically, run a set of labeled alerts past the team (a mix of known true positives and known false positives from the historical queue) and ask each analyst to make a call. Score them. Discuss the misses. This is cheap, uncomfortable the first time, and one of the highest-leverage training investments a SOC can make.
Insight
Calibration has to be a managed investment
You cannot will calibration into existence by telling analysts to “pay more attention.” The feedback has to be produced by the system. If nobody’s job is to close the loop between triage decisions and ground truth, the loop stays broken regardless of how capable the analysts are. This is an engineering problem wearing a people-problem costume.
Why this is hard to sell
The cost of broken feedback is invisible in exactly the same way that false negatives are invisible. Nobody writes a postmortem titled “the SOC’s collective prior drifted 3% over the last year and we missed an event we would have caught in 2024.” It just looks like the team is less good than it used to be, without any single person being measurably worse.
When a SOC loses a senior analyst and gets quietly worse afterward, a missing feedback loop should be investigated alongside staffing, alert volume, and detection drift. In the case above, the senior analyst had supplied that loop herself. The team kept the tools, SOPs, and headcount after she left. It did not keep the loop, because nobody had written it into the system.
You can choose to build this as infrastructure instead of hoping it lives in a person. Sampling closed alerts, routing incident outcomes back to original triagers, running calibration sessions: none of it is exotic. It is just work that has to be funded and staffed, the same way detection engineering is funded and staffed.
The operating lesson
Good analysts run on priors that decay without feedback, and feedback has to be built as a system, because it will not produce itself. The SOCs that retain their edge over time are the ones that treat analyst calibration the way a good poker player treats their decision journal: as something you actively maintain, because the alternative is drifting in a direction you cannot see.
If your SOC has not measured whether analysts’ confidence tracks outcomes, then calibration is unknown. That is a useful finding in itself: it tells you which feedback mechanism to build next.
Sources
- Daniel Kahneman and Gary Klein (2009): Conditions for Intuitive Expertise: A Failure to Disagree
- Annie Duke (2018): Thinking in Bets
- Charles Duhigg (2016): Smarter Faster Better