Why most alerts are false: the base-rate problem in alert triage

#Precision is a property of the environment, not just the detector

Let $\pi$ be the prevalence of malicious events among those a detector evaluates, $s$ its true-positive rate and $f$ its false-positive rate. Bayes' theorem gives the probability that an alert is real — its precision or positive predictive value:

When $\pi$ is small, the denominator is dominated by $f(1-\pi )\approx f$, so $\mathrm{PPV}\approx s\pi /f$. Precision scales with the ratio of prevalence to false-positive rate. Detector quality alone cannot fix it.

A "99% accurate" detector watching events where one in ten thousand is malicious produces alerts that are real about 1% of the time. That isn't a tuning failure. It's arithmetic.

#An AI triage layer is a second detector — with the same problem

Put a triage model after detection. It sees alerts, not raw events, so its prevalence is higher, but still small. In our scenario: 1,200 alerts per day of which 0.2% relate to real incidents (about 2.4 a day). The triage model assigns each alert a score; we use a binormal model where benign scores are $\mathcal{N}(0,1)$ and malicious scores $\mathcal{N}(d^{\prime },1)$. Separability $d^{\prime }$ summarizes how good the model is; the ROC curve follows from the threshold $t$:

The cost-optimal threshold

Let a missed true alert cost $C_{FN}$ and an unnecessary human review cost $C_{FP}$. Expected cost per alert is

minimized where the ROC slope equals $\frac{1-\pi }{\pi }\cdot \frac{C_{FP}}{C_{FN}}$. For the binormal model the slope at $t$ is $\phi (t-d^{\prime })/\phi (t)=e^{d^{\prime }t-d^{\prime 2}/2}$, so

We set $C_{FN}/C_{FP}=2000$: missing one real alert is as bad as 2,000 wasted reviews.

#Results: what separability buys

A weak model ($d^{\prime }=1.5$) at its cost-optimal threshold still escalates 685 of 1,200 alerts and misses 3.4 true alerts a month. Raising separability to $d^{\prime }=3.5$ escalates 108 a day and misses 1.1. Even then, escalation precision is only 2.2%: most escalations are still benign, because the base rate is still low.

And if policy demands 99.5% recall regardless of cost, the $d^{\prime }=3.5$ model must escalate 215 alerts a day — double the cost-optimal number — while the weak model must escalate 1,031, i.e. nearly everything.

#What this implies for AI SOC products

• "Closes 90% of alerts" means nothing without the miss rate. Any model can close 90% of alerts; the question is how many of the ~2 real ones per day it closed along with them. Demand false-negative rates measured on seeded attacks.
• Separability comes from evidence, not prose. A triage verdict that cites queries across identity, endpoint and network data separates classes better than one that reasons from the alert text. That is the practical meaning of grounding.
• Raise the prevalence the model sees. Correlating alerts into incidents before triage increases $\pi$ and therefore precision at every threshold. Deduplicating 40 alerts from one incident into a single case is worth more than a better classifier.
• Write the cost ratio down. $C_{FN}/C_{FP}$ is a business decision. Making it explicit is the only way to choose a threshold you can defend after an incident.

#Worked example: correlation beats a better model

Precision depends on prevalence, and prevalence depends on the unit you triage. Suppose the 1,200 daily alerts are first grouped into cases: benign alerts collapse roughly 5:1 into 240 benign cases, and the day's 2.4 true alerts belong to a single incident case. Case-level prevalence is $1/241\approx 0.41\%$, about twice the alert-level 0.2%.

Holding the $d^{\prime }=2.5$ model and its operating point fixed (TPR 96.4%, FPR 24.4%), precision of escalations rises from 0.79% to 1.62%, and humans review about 59 cases a day instead of 294 alerts. Doubling precision this way required no model improvement at all; reaching the same precision by improving the model alone would require a substantially higher $d^{\prime }$.

Choosing the threshold in code

from math import log
from scipy.stats import norm

def optimal_threshold(d_prime, prevalence, c_fn, c_fp):
    t = d_prime / 2 + log((1 - prevalence) / prevalence * c_fp / c_fn) / d_prime
    return t, 1 - norm.cdf(t - d_prime), 1 - norm.cdf(t)   # threshold, TPR, FPR

print(optimal_threshold(2.5, 0.002, 2000, 1))   # (0.695, 0.964, 0.244)

In production, estimate $d^{\prime }$ from labeled history (or seeded attacks), set $C_{FN}/C_{FP}$ per alert class, and recompute thresholds when prevalence shifts — for example during an active campaign.

#One threshold per alert class

The cost ratio is not the same for every alert. Applying the optimal-threshold formula with $d^{\prime }=2.5$ to three illustrative classes:

For ransomware staging, the rational policy is to escalate nearly half of all such alerts to a human, because a miss is catastrophic. For policy violations, the same model should close almost everything automatically and accept missing one in six. A single global threshold is wrong for both. This is also the most useful question to ask any AI triage vendor: can I set the cost ratio per alert class, and can I see the resulting operating point?

#Limitations

The binormal ROC is a standard idealization; real score distributions are skewed and multi-modal. Costs are not constant across alerts — missing ransomware staging is worse than missing a policy violation — so a production system should apply different thresholds by alert class. Prevalence varies by source and by day. Scenario numbers are illustrative; the formulas are general.

#Frequently asked questions

#Sources

1. [1] S. Axelsson, “The Base-Rate Fallacy and the Difficulty of Intrusion Detection,” ACM Transactions on Information and System Security 3(3):186–205, 2000