Alert Reduction

Synthetic, deterministic, reproducible in under a second. This page documents the AiSOC alert-reduction benchmark — what it measures, how a noisy alert stream collapses into actionable incidents, and how to reproduce the numbers locally.

Two numbers are published, and only one of them describes this product. 33.3 % is measured against RawAlert.correlation_key(), the grouping Correlator actually calls. 75.3 % comes from a four-tier scheme implemented inside a test; it is retained for continuity and does not describe this product. See Why there are two numbers before quoting either.

Read this first

Both numbers are synthetic. The 1,000-alert stream is generated with a fixed RNG seed and is not a capture from a real SIEM.

The legacy suite in services/agents/tests/test_alert_reduction.py was long described here as a faithful in-harness re-implementation of the production grouping rules. That description was wrong, and the gap is wider than "minus the DB-backed deduplicator and the ML scorer": the test does not merely reimplement fusion's grouping, it implements different grouping. Its four tiers key on (rule_id, host, user) with 10/30/5-minute windows, while RawAlert.correlation_key() keys on {tenant}:{entity}:{tactic} over a one-hour window.

services/fusion/tests/test_alert_reduction_real.py measures the real key. Neither suite is an end-to-end measurement of the production fusion service against real customer telemetry.

TL;DR

Metric	Value	Target	Describes AiSOC?
Reduction ratio (product logic)	33.3 % (1,000 alerts → 667 incidents)	20–95 %	Yes — groups with RawAlert.correlation_key()
Repeat folding (deduplication)	28.4 % (1,000 inputs → 716 alerts, 284 folded)	not floored — see below	Yes — folds with RawAlert.fingerprint(), the function behind alerts.dedup_hash
Cross-vendor root cause	16.8 % (1,000 inputs → 832 records; 51 findings carry more than one vendor)	not floored — see below	Yes — groups with alert_sink.root_cause_key, the function the sink uses
Reduction ratio (legacy suite)	75.3 % (1,000 alerts → 247 incidents)	≥ 70 %	No — four-tier scheme implemented inside the test
Critical alerts preserved (legacy suite)	93 / 93 (zero drop)	100 %	Invariant of the legacy suite
Storm incidents collapsed (legacy Tier 3)	16 (avg ~8.9 alerts each, 3–8 distinct hosts)	≥ 1	Legacy suite only
Determinism	Same RNG seed → same byte-for-byte output	Required	Both
Wall time	~8 ms on a laptop	< 100 ms	Both

The real-key gate is bounded on both sides rather than floored. A floor alone is satisfied by a key that collapses everything into one incident — 99.9 % reduction and a useless SOC — so "more reduction is better" is only true up to a point, and the gate says where.

Why cross-vendor root cause is a third row

Repeat folding above handles the same finding arriving again. It cannot reach the other half of the same problem: Security Hub, Defender for Cloud, a CNAPP connector and GuardDuty each raising their own finding about one public bucket produce four different fingerprints — different titles, different vendors, different severities — so every dedup layer in the pipeline correctly leaves all four standing. They are four reports of one root cause, and the analyst closes the same misconfiguration four times.

A finding is keyed on the triple (resource, technique, principal), and the conjunction is the control. Resource alone over-collapses: two unrelated findings on one busy host are two findings, which is the lesson v21 paid for when a missing tactic collapsed an exfiltration alert into a port scan. Technique alone is a category. Principal alone is an account's whole day. A NULL in any component does not match another NULL — "we do not know the principal" is not evidence that two findings share one.

Every original is kept and stays queryable. Promoting one vendor's alert to "the" alert would make the survivor's severity, title and vendor an accident of arrival order, and would discard the fact that four independent products agreed, which is itself the strongest evidence in the group. The finding's severity is the worst any member carried, never an average: one critical report among three makes a critical finding.

The same accounting applies, with resolvability alongside balance:

[eval] cross-vendor accounting: 1000 inputs = 832 findings
       + 0 unkeyed + 168 folded into a finding

Like repeat folding, this row carries no floor. How many products report the same misconfiguration is a property of how many cloud-security tools a tenant has bought, not of how good AiSOC is.

The three figures are never added

Correlation, repeat folding and cross-vendor root cause reduce different things over overlapping inputs. A reader who summed them would count some alerts three times. That is exactly the defect the retracted 75.3 % figure carried — a headline whose denominator nobody could name — and scripts/check_alert_reduction_claims.py refuses a page that quotes any of their sums.

Why repeat folding is a separate row, and why it did not go up

Deduplication and correlation reduce different things and are published as two numbers that must never be added together. Correlation groups related alerts under correlation_key(). Deduplication folds the same finding arriving again under fingerprint(), which is the function behind alerts.dedup_hash and uq_alerts_tenant_dedup. A reader who summed them would be double-counting, which is precisely the defect the retracted 75.3 % figure carried: a headline whose denominator nobody could name.

The folding itself is not new — three layers have collapsed repeats since migration 120. What wave 2's Phase 3.3 changed is that the folded repeat is now kept rather than discarded, as a linked occurrence on the alert it folded onto (alert_occurrences, migration 151): a count that includes the surviving alert, the real first- and last-seen window, and a bounded window of the original payload ids. A vendor that re-reports one finding 79 times is one alert that says 79.

So the measured reduction is identical before and after 3.3, and that is the honest reading of this row. The number worth publishing is not a larger reduction; it is the same reduction with all 284 folded inputs accounted for instead of absent. The suite asserts that accounting directly — every input lands in exactly one of retained alone, attached to a group, folded onto a retained alert, or pending/failed, and the four sum to the denominator:

[eval] full-cohort accounting: 1000 inputs = 658 retained alone
       + 58 attached to a group + 284 folded occurrences + 0 pending/failed

A second assertion requires each folded repeat to resolve back to the id of the alert that absorbed it. Balance alone says nothing was lost in aggregate; resolvability says the evidence can still be followed. A count that balances while pointing nowhere cannot be audited.

This row carries no floor. A deduplication rate is a property of how noisy the connected vendors are, not of how good the product is, so a target would be measuring the customer's estate rather than AiSOC.

The legacy snapshot lives at eval-results/eval/results/latest.json under suites.alert_reduction. Every successful build on main / develop appends a new commit-keyed entry alongside it (see CI artefact location).

What this measures

A SOC's first-order pain isn't "did we catch the threat" — it's "did we drown in the catch." Modern detection stacks routinely emit thousands of alerts per hour, and the same underlying event (a single misconfigured agent, one phish campaign, one credential-stuffing burst) can light up dozens of rules at once. The job of fusion is to take that raw rule output and emit a small, deduplicated, human-reviewable list of incidents.

This benchmark answers exactly one question:

Given a controlled, realistic distribution of noise, what fraction of incoming alerts collapse into incidents — and does the substrate drop any of the alerts that an analyst genuinely needed to see?

Answered twice, because the harness has answered it twice. The other three substrate suites on the public eval page are tautological by design (dataset and judge written together to gate template breakage); neither alert-reduction suite is. But only one of them groups the way the product groups.

Why there are two numbers

The original suite was honest about being synthetic and always carried a PARTIAL row saying it gated "an in-test fusion re-impl". Reading it closely, the gap was wider than that wording admitted: the test did not merely reimplement fusion's grouping, it implemented different grouping.

Its four tiers key on (rule_id, host, user) with 10/30/5-minute windows. RawAlert.correlation_key() — what Correlator actually calls — keys on {tenant}:{entity}:{tactic} over a one-hour window. Different dimensions, different windows, different answer. So the 75.3 % described an algorithm the product does not run, and a reimplementation can drift from the thing it stands for without any test failing.

services/fusion/tests/test_alert_reduction_real.py measures the real key. It reports 33.3 % on a comparable stream. That is less flattering and it is ours.

Correlator itself needs Redis, but Redis is where it stores incidents — the grouping decision is entirely correlation_key() plus the configured window, so the ratio is computed from the real method without the storage layer.

Both numbers stay published. Deleting the old one would make the history unreadable; presenting it without this note would be the thing this page exists to prevent.

The sections below describe the legacy suite's dataset and tier structure, which is where the 75.3 % comes from. Read them as documentation of a retained regression gate, not of the shipping correlation logic.

The synthetic dataset

The 1,000-alert stream is generated by generate_noisy_alert_stream with a fixed seed so every run is byte-for-byte identical. The mix is deliberately shaped to look like the worst hour of a real SIEM:

Bucket	Share	What it represents
Pure duplicates	~25 %	Same (rule, host, user) within the same minute — the agent re-firing on the same telemetry
Near-duplicates	~30 %	Same (rule, host), different user or ±5-minute window — same root event, slight metadata drift
Storms	~15 %	One rule firing across many hosts within a 5-minute window — campaign / worm / misconfigured push
Benign noise	~10 %	Low-score rules (score < 0.35) — the chatter every SOC silences but the SIEM still emits
Unique high-signal events	~20 %	Distinct (rule, host, user) events with no friends — the things you actually want to look at

The full distribution from the latest run:

Severity in	Alerts	Incidents out (post-fusion)
critical	93	38
high	417	130
medium	303	79
low	187	0 (collapsed below the noise floor)
Total	1,000	247

The low-severity bucket disappears entirely: those alerts carry score < 0.35 by construction and get dropped by the noise threshold. This is intentional — the benign-noise bucket exists to verify the noise-floor cut still fires, not to be shown to a human.

The legacy suite's 3-tier grouping

The legacy algorithm has three rules, applied in order. The exact code lives in fuse_alerts inside the test file. It is not the production logic in services/fusion, which keys on {tenant}:{entity}:{tactic} over a one-hour window — see Why there are two numbers.

Tier 1 — strict identity collapse

Same (rule_id, host, user) within a 10-minute window → 1 incident. This is the "the agent re-fired on the same telemetry" case. Most pure duplicates and same-source loops collapse here.

Tier 2 — host-scoped merge

Same (rule_id, host) within a 30-minute window, but spanning multiple users (or the same user with ±5-minute drift) → merged into one Tier-1 incident. This catches "same root event, slightly different metadata" duplicates, including a single attacker pivoting between local accounts on the same host.

Tier 3 — rule-storm collapse

Same rule_id firing within a 5-minute window across ≥ 3 distinct hosts → one "storm" incident with host = <storm:N-hosts>. This is the "campaign / worm / misconfigured deployment push" case — the kind of event that historically generates a hundred near-identical tickets.

Noise-floor drop

Any incident with aggregate score < 0.35 is dropped before being emitted. This is the configurable knob a tenant uses to silence chronically chatty rules without de-listing them; in the benchmark it means the 187 low alerts vanish.

Latest results

The numbers below are produced by python3 scripts/run_evals.py --json on the current main and rendered into eval/results/latest.json on the eval-results branch by the CI gate. run_evals.py emits the legacy suite under the alert_reduction key; the real-key figure is produced by pytest in services/fusion and is shown alongside it.

Headline

Field	Product logic (real correlation key)	Legacy suite
metric	reduction_ratio	reduction_ratio
value	0.333	0.753
target	0.20 – 0.95 (bounded both sides)	≥ 0.70 (floor only)
passed	true	true
duration_ms	~30	7.9
Describes this product	yes	no

Collapse breakdown

Bucket	Count	Avg alerts collapsed	Notes
Tier 1/2 multi-member incidents (non-storm)	56	~8.8 alerts each	Same root event, deduplicated
Tier 3 storm incidents	16	~8.9 alerts each	3 – 8 distinct hosts per storm
Single-member incidents	175	1 each	Genuine unique signal — exactly what an analyst needs to triage
Total emitted incidents	247	—	from 1,000 raw alerts

The 175 single-member incidents are the floor: those are the alerts that should survive fusion, because they represent distinct events. Anything beyond that floor is a deduplication win.

Critical-alert preservation

Severity tier	Alerts in	Preserved in incidents	Dropped
critical	93	93 (100 %)	0

This is the non-negotiable invariant: fusion never silently drops a critical alert. The substrate may merge a critical alert into a multi-member incident (it does — 93 critical alerts are represented across the 38 critical incidents emitted), but it will never delete one. The test_critical_alerts_never_dropped test asserts this directly and is part of the same CI gate.

Determinism

Two consecutive runs on the same commit produce byte-for-byte identical output. The benchmark uses a fixed RNG seed inside generate_noisy_alert_stream, and the fusion algorithm is order-stable on its inputs. This matters for two reasons:

1. CI flakiness has a single root cause. If two CI runs disagree on the reduction ratio, it's a code change, not a coincidence.
2. Diff-friendly regression review. When a PR moves the number, you can inspect which incidents changed shape (single-member ↔ multi-member ↔ storm) instead of being told "the noise moved."

The test_run_is_deterministic test enforces this — it runs fuse_alerts twice and compares the incident lists element-for-element.

Reproduce locally

From a fresh clone:

git clone https://github.com/beenuar/AiSOC && cd AiSOC
python3 scripts/run_evals.py

Expected output (excerpt):

[PASS] alert_reduction               reduction_ratio        0.753  (target >= 0.70)

For machine-readable output:

python3 scripts/run_evals.py --json --out /tmp/eval.json
python3 -c "import json; print(json.dumps(json.load(open('/tmp/eval.json'))['suites']['alert_reduction'], indent=2))"

To run the figure that describes this product — the real correlation key:

cd services/fusion
python3 -m pytest tests/test_alert_reduction_real.py -v -s
# [eval] alert reduction (real correlation_key): 1000 alerts -> 667 incidents = 33.3%

To run only the legacy alert-reduction tests under pytest:

cd services/agents
python3 -m pytest tests/test_alert_reduction.py -v

That covers four assertions:

• test_reduction_ratio_meets_target — the headline ≥ 70 % gate
• test_critical_alerts_never_dropped — the zero-loss invariant
• test_run_is_deterministic — same seed → same output
• test_storms_collapse — Tier 3 fires when ≥ 3 hosts share a rule in a 5-minute window

CI artefact location

Every push to main or develop triggers .github/workflows/ci.yml, which runs scripts/run_evals.py --ci --out report.json. On success, the report is committed to the orphan eval-results branch:

eval/results/<commit_sha>.json   # one snapshot per commit
eval/results/latest.json         # most recent passing build on main
eval/results/badge-reduction.json # shields.io endpoint backing the badge above

Reading the latest snapshot from CI:

curl -s https://raw.githubusercontent.com/beenuar/AiSOC/eval-results/eval/results/latest.json \
  | python3 -c "import json,sys; d=json.load(sys.stdin); print(json.dumps(d['suites']['alert_reduction'], indent=2))"

Drop that into a dashboard, a Slack notifier, or a tenant-facing report — it's the same JSON shape run_evals.py writes locally.

What this benchmark does not tell you

In the spirit of being painfully honest about synthetic vs. real:

• The legacy fuse_alerts does not implement the production grouping at all. It keys on (rule_id, host, user); the product keys on {tenant}:{entity}:{tactic}. This was described here as a faithful re-implementation for several releases and it was not one.
• Neither suite measures the production fusion service end-to-end. The real-key measurement drives RawAlert.correlation_key() directly, but the production service additionally runs a Postgres-backed deduplicator (cross-window persistence) and an ML scorer (lifts and depresses scores based on historical FP rate). Those components have their own unit tests; they are out of scope here because this gate has to run in milliseconds in CI.
• It does not exercise real telemetry. The 1,000-alert stream is shaped to look like the worst hour of a SIEM, but it isn't a capture from one. Real-customer evaluation is on the v1.5 roadmap and will be opt-in / federated; see apps/docs/docs/benchmark.md.
• It does not compute per-rule false-positive rates. That is the next stage of this benchmark, tracked under issue #5 — per-rule cross-fire FP eval gate (PR #89, in flight). Once that ships, this page will gain a "per-rule FP rate" section that surfaces which rules contribute most to the noise floor and how their merge rate trends over time.
• It does not score "alerts an analyst would have wanted." That requires labeled human-judgement data, which we don't have synthetically. The closest proxy here is the critical-alert preservation invariant — zero alerts above the noise floor are silently dropped.

Roadmap

The pieces that turn this from a substrate gate into a buyer-grade benchmark:

• Per-rule FP rate breakdown — depends on issue #5 / PR #89. Will surface per-rule contribution to the noise floor and gate net-new noisy detections in CI.
• Per-tenant fusion config replay — let an operator paste their fusion knobs (window sizes, noise threshold, severity overrides) and replay the stream against them. Lets a buyer answer "what would my number look like" before deploying.
• Live-fusion online benchmark — once the production services/fusion service has a stable replay API, run the same 1,000-alert stream through the real DB-backed deduplicator + ML scorer in a nightly job. The delta vs. this in-harness number is the "what does the substrate gate miss" signal.
• Federated, opt-in real-tenant evaluation — collapsed against tenant telemetry under DP-style aggregation so the headline number reflects a population, not a single deployment.

PRs against any of those are welcome — the Help us harden the harness section on the umbrella benchmark page lists the contribution path.

See also

• Public Eval Harness — the umbrella page covering all five CI-gated suites (MITRE accuracy, alert reduction, investigation completeness, response quality, playbook completion).
• Synthetic telemetry corpus — the 14-source backing event corpus that connector and Sigma rule PRs wire against.
• scripts/run_evals.py — the unified runner that produces the JSON snapshot consumed by this page.
• services/fusion/tests/test_alert_reduction_real.py — the measurement against RawAlert.correlation_key(). This is the one that describes AiSOC.
• services/agents/tests/test_alert_reduction.py — the legacy test file containing both the synthetic stream generator and the four-tier grouping under test.