Nobody talks about this part.
The vendor demo showed the AI triaging a phishing alert in 30 seconds. Clean verdict. MITRE mapping. Suggested containment. The room was impressed. Procurement moved forward.
Six weeks later, at 3:14 AM on a Tuesday, the AI flagged a legitimate email from your CFO’s travel agent as a credential harvester, auto-quarantined the CFO’s laptop, and locked her out ofthe corporate VPN while she was presenting quarterly results to the board from a hotel in Singapore.
The overnight SOC analyst saw the containment action logged in the war room. He didn’t question it. Why would he? The AI said 87% confidence. The reasoning chain said “suspicious domain, unusual login time, geographic anomaly.” All three factors checked out on paper. The AI was technically right about each individual signal. It was dead wrong about the conclusion.
Nobody called the CFO to verify because the system didn’t surface the one piece ofcontext that would have changed everything: she’d filed a travel request three days earlier, and her assistant had forwarded the hotel booking confirmation through the exact email address the AI flagged.
This didn’t happen at one company. Versions ofit happen every week at organizations running AI triage without governance controls. And when it does happen, the fallout goes in directions nobody planned for.
The analyst stops trusting the system
The overnight analyst who watched the CFO get locked out? He stopped trusting AI verdicts entirely. Not because the AI was wrong once, but because he couldn’t tell the difference between a wrong call and a right one. The confidence score said 87% both times.
The reasoning chain looked the same. Ifthe AI can be this confident and this wrong, what’s the point ofthe confidence score?
This is the real cost ofa bad call. Not the incident itself, which gets fixed in an hour. The trust damage, which takes months to repair.
Researchers who studied AI SOC deployments found that analysts may see AI-generated summaries and risk scores on every alert, but that doesn’t mean they trust them. Once an analyst gets burned by a high-confidence wrong call, they start manually reviewing everything the AI touches. You’re back to square one, except now you’re paying for an AI platform AND doing manual triage.
A practitioner on Reddit described an experiment where he ran an LLM against 348 known false positives plus one real threat. The model scored 71% accuracy, flagged obvious false positives as malicious, and missed the actual test incident entirely. The practitioner’s conclusion wasn’t “AI doesn’t work.” It was “I can’t deploy this without knowing when it’s going to be wrong.”
That’s the question nobody in the vendor demo addresses.
The board asks a question you can‘t answer
The Monday after the CFO incident, the CISO gets a call from the board chair. Not “what happened” but “how did the AI decide to do that, and who approved it?”
Ifyour AI SOC doesn’t have an immutable audit trail, you are standing in that meeting with nothing. You can show the alert. You can show the action. You cannot show the decision path. You cannot show why the AI weighted “geographic anomaly” higher than “known travel schedule.” You cannot show what data the model considered and what it ignored. You cannot show who, ifanyone, had a chance to intervene before the containment action fired.
This is where the conversation shifts from “AI in the SOC” to “ungoverned automation in the SOC.” And that’s a very different conversation to have with a board.
At a Fortune 500 food manufacturer that ran a six-month AI SOC pilot, the security team deliberately maintained strict guardrails including enforced citations, human approval gates, tool allow lists, and full audit logging. That’s not because the AI was bad. It performed well on metrics. It’s because the team understood that when something goes wrong, the ability to explain what happened is as important as the ability to prevent it.
In a manufacturing environment, this matters even more than in a typical enterprise. The security leader noted that operational downtime directly impacts revenues, production lines, and worker safety, and that reality shaped every architectural and governance decision. A false positive that isolates a workstation in an office is annoying. A false positive that shuts down a production line is a six-figure loss per hour.
The vendor‘s explanation makes it worse
After the incident, you call the vendor. Their response follows a pattern I’ve seen play out multiple times now.
First, they’ll tell you the AI “performed within expected parameters.” Which is true. An 87% confidence score implies a 13% error rate. You just happened to be on the wrong side ofthat 13% at the wrong time.
Second, they’ll suggest you “tune the model” by adjusting thresholds. Which means: reduce sensitivity to reduce false positives, which increases the chance ofmissing a real threat.
You’re being asked to trade one failure mode for another.
Third, they’ll mention that “AI-ready organizations” would have integrated their travel booking system with the SOC data pipeline, giving the AI the context it needed. In other words, the wrong call was your fault for not connecting enough data sources. Researchers describe this pattern as displacing accountability from product immaturity onto buyer psychology, and call it a structural failure of feedback between engineering reality and go-to-market messaging.
None ofthese responses address the actual problem: the AI made a decision with irreversible consequences, and nobody had a chance to say “wait.”
What a wrong call actually costs you
The CFO scenario is dramatic but the damage from everyday wrong calls is quieter and probably worse.
The analyst who now manually re-checks everyAI verdict adds 20 minutes per case. Multiply that across 300 cases a day and you’ve lost most ofthe efficiency the AI was supposed to create. The SOC manager who has to explain to leadership why the AI locked out the wrong person now has a credibility problem that makes every future automation initiative harder to approve. And the overnight shift, the one that’s supposed to be where AI covers for thin staffing, is the exact shift where wrong calls cause the most damage because there are fewer humans to catch them.
SANS data from 2025 found that false positives are the number one analyst pain point, with stale model data as the leading cause. Reduction is real. Elimination is not. And a system tuned to hit zero false positives will start missing actual threats. There’s a floor below which you can’t push false positives without sacrificing detection.
So what do you actually do about this?
What was actually missing at 3 AM
I keep coming back to the same questions whenever I hear about incidents like this. Not abstract questions about AI philosophy. Practical ones about what was missing when the wrong call happened.
CouldtheanalystseewhytheAImadethatdecision?
Not the confidence score. The actual decision path. “I checked cluster membership, found no cluster, identified 3 IOCs, queried threat intelligence, checked asset criticality, checked user travel history (no data available), and concluded: suspicious.” Ifthe analyst could see “no data available” next to “user travel history,” he would have paused. He would have called. The missing context would have been visible instead ofinvisible.
Didthesystemaskforpermissionbeforeacting?
An AI that quarantines a laptop at 87% confidence without human approval is not governed. It’s autonomous in the worst sense, the sense where speed matters more than accuracy and nobody can intervene. A tiered approval system, where low-risk actions execute automatically but high-impact actions require human confirmation, would have caught this. Block a known malicious IP? Auto-approve. Quarantine the CFO’s production endpoint? Ask first.
Couldyouprovetoanauditorexactlywhathappened?
Not reconstruct. Not explain after the fact. Prove, with an immutable record that was written at decision time, not created after someone complained. EveryAI decision logged: what data was examined, what was weighted, what was missing, who approved (or didn’t), and what action was taken. This isn’t optional. Under NIS2, DORA, and increasingly under SOC 2 as well, you need to demonstrate that automated decisions are governed. “The AI did it” is not an acceptable answer to a regulator.
The uncomfortable question for everyAI SOC buyer
Here’s what I’d ask any vendor, including my own, before putting AI in production with containment authority:
“Show me the last time your AI made a wrong call in a customer environment. Show me what happened, how it was caught, and what changed as a result.”
Ifthey can’t answer that question, they’re either lying about the maturity oftheir deployments or they don’t track wrong calls, which is worse.
Google Cloud’s Anton Chuvakin has argued that the market needs case studies with falsifiable claims, something specific enough that a buyer can hold the vendor to it. He’s right. “50% faster investigations” is meaningless without knowing: faster than what, measured how, and what happened when it was wrong.
The AI SOC that actually works in production isn’t the one that never makes mistakes. It’s the one that makes the mistake visible, makes the reasoning reviewable, and gives a human the chance to say “wait” before something irreversible happens.
That’s not a feature request. It’s the minimum viable deployment for anyAI that’s going to act on your behalfat 3 AM when nobody’s watching.
