AI SOC analysts: what they do and how to evaluate one

Why the SOC needs a different model

The modern SOC has a volume problem and a time problem. Mid-size enterprises routinely generate thousands of alerts a day across SIEM, EDR, identity, cloud and email security. Most are benign; a few matter enormously. Analysts triage what they can, tune what they can't, and suppress the rest — which is how real intrusions end up sitting in a queue marked "low".

Meanwhile, adversaries have become faster. Breakout times — from initial access to lateral movement — are now commonly measured in tens of minutes, and AI-assisted attackers compress them further. A process that takes four hours to look at an alert is structurally too slow.

SOAR playbooks were the last answer. They work well for well-defined, repeatable cases, but they're brittle: every new alert type needs a new playbook, and they cannot reason about something they weren't scripted for.

What an AI SOC analyst actually does

An AI SOC analyst (or AI SOC agent) is software that performs the investigative work of a human analyst: given an alert, it decides what evidence it needs, gathers it by querying your tools, reasons over the results and writes a conclusion — benign, suspicious or malicious — with the evidence attached.

A good investigation looks like a good analyst's: Who is the user? Is this normal for them? What else happened on this device in the last hour? Has this file, domain or IP been seen anywhere else in the environment? Did anyone else receive the same email? The difference is that the agent does this for every alert, in about a minute, at 3 a.m. on a Sunday.

More capable systems extend beyond triage:

• Response: taking containment actions — isolating a host, revoking sessions, disabling an account — under policy.
• Hunting: pivoting from a confirmed incident to find related activity across the estate.
• Incident management: maintaining a timeline, coordinating responders, producing status updates and executive briefs.
• Reporting: tracking regulatory deadlines and drafting notifications.

Nightwatch does all five, and is unusual in one respect: it is trained on the same offensive models as our attack platform, so it reasons about alerts from an attacker's perspective — what would I do next if this were me?

How much should you automate?

The right answer is: more than you think for investigation, less than vendors suggest for response — at first. We recommend a tiered model.

Two weeks in shadow mode — where the agent investigates in parallel with your team and changes nothing — gives you the data to set those thresholds honestly.

Measuring accuracy (and avoiding hallucinated verdicts)

The core risk with AI-driven triage is a confident, wrong answer. The mitigation is to demand grounding and to measure it.

• Every claim must cite evidence. A verdict like "the user logged in from their usual location" should link to the exact query and result. If the agent can't ground a conclusion, it should escalate, not close.
• Measure false negatives, not just false positives. The cost asymmetry is huge: closing a real incident as benign is far worse than escalating a benign one. Seed known-malicious test activity — or let an offensive platform like Rogue generate it — and see whether the agent catches it.
• Track agreement with senior analysts on a random sample of closed alerts every week, especially in the first months.
• Make investigations replayable so you can audit how a conclusion was reached after the fact.

An evaluation plan you can run in two weeks

1. Pick representative sources: your SIEM, primary EDR, identity provider and email security. Read-only access is enough for a shadow trial.
2. Define success criteria up front: agreement rate with your analysts, median time to verdict, false-negative rate on seeded attacks, analyst hours saved per week.
3. Run in shadow for two weeks. The agent investigates every alert; your team works as normal.
4. Inject known-bad activity. Run a controlled attack simulation or autonomous pentest during the trial and measure what the agent catches and how fast.
5. Review disagreements together. Where the agent and your analysts disagree, who was right? This is the most useful hour of the trial.
6. Decide tier-1 thresholds from the data, not from the demo.

How an AI SOC agent connects to your stack

Deployment architecture determines both how useful an agent is and how much risk it introduces. A sensible design looks like this:

• Read paths into your SIEM, EDR, identity provider, cloud audit logs, email security and asset inventory, using scoped API credentials. The agent should query in place rather than requiring you to ship all your data to a new lake.
• Write paths for response that are separate, narrowly scoped and individually revocable — one credential to isolate hosts, another to revoke sessions — so you can grant autonomy action by action.
• Human channels in Slack or Microsoft Teams for approvals and questions, and in your case-management system for the system of record.
• An audit trail of every query, inference and action, exported to your own SIEM so the agent is itself monitored.

Be wary of designs that require broad admin credentials "for simplicity". An AI SOC agent is a privileged identity in your environment and should be governed like one.

Building the business case

The ROI model for an AI SOC agent is straightforward to build with your own numbers:

1. Analyst time recovered: daily alert volume × average minutes per manual investigation × share of alerts the agent closes with evidence. At 2,000 alerts a day, 12 minutes each and 90% autonomous closure, that is roughly 360 analyst-hours a day of work you are either doing today or — more likely — not doing at all.
2. Coverage gained: the share of alerts that currently receive no investigation. For many teams this is the bigger number, and the one that matters for risk.
3. Time to contain: the difference between hours and minutes, multiplied against your incident history. Containment speed is the single strongest lever on breach cost.
4. MDR and tooling consolidation: reduced escalation volume often allows renegotiation of per-escalation or per-seat services.

Present the first two as efficiency and the third as risk reduction. Boards respond to the third.

Questions to ask AI SOC vendors

• Does it investigate every alert, or only those matching predefined types?
• Can I see the evidence and queries behind every verdict?
• How are containment actions governed, and can I set blast-radius limits per action?
• What happens when a data source is unavailable mid-investigation?
• Is our telemetry used to train shared models?
• How do you measure and report false negatives?
• Does it work alongside our existing MDR, or replace it?
• Can it run the incident — timeline, roles, comms — or only triage?

Where this is going

The direction is clear: the SOC becomes a small team of senior responders supervising agents that do the investigative heavy lifting. The organisations that adapt fastest will be the ones that measure their agents as rigorously as they'd measure a new hire — and that pair defensive agents with offensive ones, so the defense is continuously tested against the kind of attacker it will actually face.

Frequently asked questions