Evaluate AiSOC on your own history
The published benchmark measures AiSOC against a synthetic corpus, and three of its four axes are substrate self-consistency rather than agent accuracy. That is a useful regression gate and a poor basis for deciding whether to trust the product on your estate.
Replay evaluation answers the question that actually matters: how does AiSOC triage compare with what your analysts already decided, on your data?
Everything on this page is in the tree and reachable: the history readers, the
replay runner, the scoring report, the aisoc replay command, the API job and
the Evaluate on your history console page. Nothing here describes a
capability that does not exist.
The one rule worth reading first
A label AiSOC cannot name is excluded from accuracy. It is never guessed.
This is the difference between an evaluation and a sales sheet, and it costs sample size on purpose.
Every vendor in this list ships a way for an analyst to say "I do not know".
Splunk Enterprise Security has dispositions named Other and Undetermined.
Microsoft Sentinel has an Undetermined classification. Defender XDR has
Unknown. An analyst who chose one of those made no claim, and folding it into
true_positive because the finding happened to be closed would manufacture
agreement out of an admission of uncertainty.
Those rows are read, counted and reported as unlabeled. They never enter the
confusion matrix. If your history is mostly unlabeled, the report tells you
that instead of printing a confident number derived from a handful of rows.
The canonical taxonomy
Vendor labels map onto the same taxonomy AiSOC already uses when it writes a disposition back into your SIEM, so a verdict means the same thing in both directions.
| Canonical | Means |
|---|---|
true_positive | Valid detection of malicious or unauthorised activity. |
benign_true_positive | Valid detection of authorised or expected activity. The rule was correct, so this is not a false positive and never counts toward a rule's false-positive rate. |
false_positive | Invalid detection. The intended condition was not present. |
benign | Real but non-threatening activity, making no claim about whether the rule was right. |
needs_review | Insufficient evidence to decide safely. |
escalate | Analyst-forced escalation. |
unlabeled | Not a verdict. The analyst recorded nothing this platform can name. Excluded from accuracy. |
What each vendor needs
Splunk Enterprise Security
Reads notables at status 5 (Resolved) and 6 (Closed) through the notable
macro, which is what resolves the correct index on a customised install.
The six stock dispositions map as follows. Dispositions 5 and 6 are absent on
purpose, and a site that has added custom dispositions from 7 upward will see
them as unlabeled.
| Splunk ES | Canonical |
|---|---|
disposition:1 True Positive, Suspicious Activity | true_positive |
disposition:2 Benign Positive, Suspicious But Expected | benign_true_positive |
disposition:3 False Positive, Incorrect Analytic Logic | false_positive |
disposition:4 False Positive, Inaccurate Data | false_positive |
disposition:5 Other | unlabeled |
disposition:6 Undetermined | unlabeled |
Enterprise Security is routinely customised, so the reader accepts a
search_override. Supply your own SPL if your site renames statuses or keeps
review state in its own lookup. It must return these fields: event_id,
rule_id, rule_name, urgency, disposition, review_time, reviewer,
comment.
It should also return the notable's asset and identity fields — src, dest,
dvc, orig_host, user and src_user — which is what the default search
does. | fields keeps only what it names, so a search that omits them
produces findings with no host and no user, and triage is then graded on an
alert that says who raised it and nothing about who it happened to.
Microsoft Sentinel
Reads incidents with properties/status eq 'Closed', following nextLink so a
window larger than one page is read whole rather than truncated to whatever
sorted first.
TruePositive maps to true_positive, BenignPositive to
benign_true_positive, FalsePositive to false_positive, and
Undetermined to unlabeled. classificationReason and
classificationComment are carried as the analyst's reason.
An incident's properties carry no account and no host, so the reader also
calls each incident's entities endpoint and attaches the result. The app
registration therefore needs to be able to read entities on the workspace; if
it cannot, the incident still contributes its analyst's classification and
reaches triage without a host or a user rather than failing the window.
Elastic Security
Elastic ships no disposition field. Closing a signal sets
kibana.alert.workflow_status to closed and records no reason at all.
So an untagged Elastic deployment yields no labels, and the reader says so
rather than inferring that a closed signal was a true positive. If you want
your Elastic history graded, adopt one of these values in
kibana.alert.workflow_tags:
| Workflow tag | Canonical |
|---|---|
true_positive | true_positive |
benign_positive or benign_true_positive | benign_true_positive |
false_positive | false_positive |
benign | benign |
A signal carrying two conflicting tags is unlabeled, because picking one
would be a guess.
IBM QRadar
Reads offenses with status = CLOSED. An offense carries only a numeric
closing_reason_id, so the id-to-text table is read from your appliance at
/api/siem/offense_closing_reasons rather than hardcoded, because closing
reasons are site-configurable.
The three stock reasons map as follows. A custom reason, or one the appliance
declines to resolve, is unlabeled.
| QRadar closing reason | Canonical |
|---|---|
| False-Positive, Tuned | false_positive |
| Non-Issue | benign |
| Policy Violation | true_positive |
"Non-Issue" is deliberately benign and not benign_true_positive. It makes
no claim about whether the rule was right, and crediting the detection with
being correct on the strength of an analyst saying only that nothing happened
would inflate the rule's apparent quality.
If the closing-reason lookup is refused, the offenses are still read and every
row lands unlabeled. A partial answer beats no answer.
Microsoft Defender XDR
Reads alerts with status eq 'Resolved', following @odata.nextLink.
Defender separates two fields that must not be conflated. classification is
the verdict and is what maps to a disposition. determination is the reason
(Malware, SecurityTesting, Phishing and so on) and is carried as the analyst's
reason, never scored. Putting a reason code into a confusion matrix would be a
category error.
| Defender classification | Canonical |
|---|---|
TruePositive | true_positive |
InformationalExpectedActivity | benign_true_positive |
FalsePositive | false_positive |
Unknown | unlabeled |
How a replay runs
Shadow mode writes nothing
A replay runs the same triage path a live alert takes. Not a copy of it: the
same FusedAlertTriageWorker.triage the Kafka consumer calls on every fused
alert, constructed with two different sinks.
Persistence is injected. In production the worker holds a writer that records
the verdict to the Investigation Ledger and the alerts row, writes the
outcome prior, queues approvals, caches the verdict for deduplication and
pushes the disposition back to your SIEM. In a replay it holds one that counts
each of those and performs none of them. The count is reported, because a
replay that quietly stopped replaying and a replay whose writes were suppressed
both write nothing, and only the count tells them apart.
The full investigation graph is not run during a replay, because every node it executes records itself to the ledger. That does not change the measurement: the verdict is fixed before escalation is reached, and a test drives the real worker with a graph runner that tries to rewrite the verdict to prove it.
Point-in-time context, so the test window cannot answer itself
History is ordered by close time and cut by time, 70/30 by default. The later period is the test window. A finding closed at the exact split instant stays on the train side.
Organisation memory and outcome priors are captured once, at the split, and served from that snapshot for the whole run. Without this, a replay feeds itself: production writes every verdict back as a per-signature prior and reads that prior before triage, so the verdict on one alert auto-closes the next alert with the same evidence, and the evaluation grades an answer it supplied three seconds earlier. The deduplication cache does the same thing one layer earlier and leaves no trace in either store.
There is one honest gap. Organisation-memory statements, as the API serves
them, carry no creation time, so a statement cannot be tested against the split
and is kept. The report publishes how many statements were in that position, as
statements_without_timestamp, rather than claiming a tighter freeze than the
data supports.
What the report says
Recall on malicious leads, because it is the number you are deciding on and the one an imbalanced queue hides. Beside it: per-class precision and recall, a confusion matrix, the abstention rate, reliability bins with an expected calibration error, the hallucination rate with the indicators behind it, per-rule and per-source breakdowns, and bootstrap confidence intervals.
Two rules govern what it prints.
Below 30 malicious cases in the test window, no headline accuracy is printed. The count and the reason are printed instead. On a queue where almost everything is a false positive, an agent that calls everything benign scores well, and that figure would describe your queue rather than the product.
A rate with no denominator reads "not measured", never 0. A zero in a recall column says the agent missed every case of that class; having never been asked is a different fact.
What a replay is not
A replayed finding is normalized by the same connector normalize() your live
pipeline uses, reached over HTTP in the connectors service so there is no
second copy of any vendor's field mapping to drift. What it does not carry is
everything fusion adds to a live alert: correlation across related events, the
fused confidence score, the deterministic narrative and entity resolution.
Every report states this in its method section. Replay measures triage on a
single normalized finding, not the whole pipeline.
Running one
There are three ways in, and all three drive the same job. The API
orchestrates it, because no single process can hold the three services
involved: services/actions owns the SIEM credentials and the readers,
services/agents owns triage, and services/connectors owns normalize().
All three package their code as a top-level app, so a process that imported
two of them would get one of them.
The console
Evaluate on your history, in the left-hand navigation. Pick a connected source and a window, and the page shows the report when the run finishes.
Two things about how it presents a result are deliberate.
No number appears without the sample size behind it. Recall on malicious renders with the count of malicious cases it was computed over, the headline accuracy with the count of answered decisions, and the history read with how many of those findings carried an analyst label at all. A precision of 1.00 over two predictions is not a precision of 1.00 over two hundred, and a panel that prints only the ratio has thrown that away.
When the window is too thin for a headline, the page prints the reason rather than a number. Below the floor of malicious cases the headline card reads "withheld" and the sentence explaining why is rendered underneath: how many malicious cases the window held, how many are required, and why a figure computed over fewer would describe the queue rather than the agent. It is not a dash, and it is not a zero. Zero would say the agent got every answer wrong, which is a different fact with a different remedy.
The method block sits beside the report, not behind a tab: the split point, the frozen-context counts, the bootstrap seed and resample count, how many writes shadow mode intercepted, and the list of enrichments a replayed finding does not carry.
The CLI
aisoc replay --connector-id <uuid> --output report.md
The tenant comes from the credential (--api-key, or AISOC_API_KEY). There
is no tenant flag, because a flag would be a value you chose that nothing
checked.
Useful options:
| Option | What it does |
|---|---|
--since / --until | Pin the window. Omitting them dates it from now, so a second run covers a different window and produces a different report. |
--format markdown|json|pdf | Which export to write. All three come from the artefact stored when the run completed. |
--output PATH | Where to write it. Without it, markdown and JSON go to stdout and every progress line goes to stderr, so aisoc replay ... > report.md produces the report and nothing else. |
--exclude-latency | Replace the two wall-clock latency figures with a note. See below. |
--no-wait / --collect <id> | Queue a run and collect it later. |
--seed / --resamples | Change the bootstrap. Both travel into the report either way. |
The API
POST /api/v1/evaluations/replay -> 202 with an id
GET /api/v1/evaluations/replay/{id} -> poll to `completed` or `failed`
GET /api/v1/evaluations/replay/{id}/export?format=markdown|json|pdf
GET /api/v1/evaluations/replay/{id}/decisions
Starting a run needs connectors:write, the same bar as testing a connector,
because it does the same kind of thing: an outbound call to your SIEM with
stored credentials. Reading a report needs reports:read.
Results live in two tenant-scoped tables with row-level-security policies. The report is stored as the renderer produced it and served back unchanged, rather than re-rendered on each request: a report re-rendered later by a newer renderer is a different artefact from the one you read. The per-decision rows are kept separately, and carry the evidence triage was given, so a number you disagree with can be re-derived rather than re-argued.
The design-partner kit
The three paths above give you a report. The kit wraps the same run in the two things you need before a report can leave your building: a signature that says it came from your deployment unaltered, and a redaction pass that means you can hand it to us without handing over your estate.
make replay-eval ARGS="--connector-id <uuid>"
It needs two values in the environment — AISOC_API_KEY, and
AISOC_REPLAY_SIGNING_KEY, which is any secret you choose and keep. The
signing key never leaves your deployment and is not recoverable from a
bundle, so a bundle you send us proves it was produced by a holder of that
key and has not been edited since, and nothing more.
The output is a JSON evidence bundle:
{
"payload": { "schema_version", "generated_at", "method", "scores", "privacy", "items"? },
"integrity": { "payload_sha256", "algorithm": "HMAC-SHA256", "signature" }
}
Verify one — yours, or one we send back to you — with:
make replay-eval-verify ARGS="path/to/bundle.json"
It exits non-zero and says so when the signature does not match the key in your environment, which is the same answer for an edited bundle and for one produced by a different deployment, because from the outside those are the same event.
Aggregates by default; items only if you ask. A bundle carries the
scores, the method and the privacy block. Per-finding rows are withheld
unless you pass --export-items, and the hallucinated-indicator examples are
withheld with them, because an example is a verbatim quotation of your
analyst's or the model's prose about one of your machines.
What redaction does and does not do. Identifiers — finding id, rule,
source, tenant, connector — are replaced whole with stable tokens (RULE_1,
HOST_2), so two rows about one rule can still be seen to concern one rule
while the rule is not named. The agent's free text goes through the same
tokens plus the platform's pattern redactor plus a pass for host-shaped words
the identifier pass never saw, which is what catches a machine the model
quoted in its reasoning and nothing else mentioned. Public taxonomy is held
back deliberately: SHA-256, CVE-2026-1234, RFC-1918 and x86-64 all
match the host shape and all stay readable, because reasoning rewritten into
HOST_4 hashing protects nothing and cannot be reviewed. The raw evidence
triage was given is dropped outright rather than redacted.
The bundle states its own limits in a privacy block rather than leaving you
to infer them: what the free-text pass does not catch, that the token map is
never exported, and that evidence was withheld. Read it before you send one.
Egress is classified, not assumed. Every bundle records whether anything
left your estate, derived from the models the run actually resolved rather
than from the flags you passed: none when every call went to a local model
or no call was placed at all, hosted when a third-party provider was used —
which the kit refuses to do unless you pass --hosted — and unverified
when calls were placed that the gateway did not attribute, which is reported
as unknown rather than quietly as none.
Published outcomes: not measured
Everything above describes a capability. This section records what has actually been measured with it, and the honest answer is nothing.
| Outcome | Value | Denominator |
|---|---|---|
| False-positive reduction | not measured | — |
| Escalation rate | not measured | — |
| Time to verdict | not measured | — |
Not 0. A zero here would say AiSOC was replayed against a real queue
and reduced nothing, which is a claim nobody is entitled to make. The three
rows are absent because the run has never happened.
Why
make replay-eval reads a window of closed findings from a SIEM that
somebody's analysts have already labelled. That is the whole point of it:
the grader compares AiSOC's verdict with a human decision, so the input
cannot be synthesised without the measurement becoming a comparison between
the product and a fixture somebody wrote to be compared against.
No design partner has offered a window of their closed alerts. Until one does, there is no denominator, and a published figure would be fabricated. The synthetic corpus on the benchmark page measures something different and says so.
The exact reproduction, for whoever runs it first
Against a deployment with a SIEM connector configured and at least one labelled, closed finding in the window:
# The connector id is the row in Settings → Connectors, or:
# curl -s -H "Authorization: Bearer $TOKEN" \
# "$AISOC_URL/api/v1/connectors" | jq -r '.items[] | "\(.id)\t\(.connector_type)"'
make replay-eval ARGS="--connector-id <uuid> \
--since 2026-07-01T00:00:00Z \
--until 2026-10-01T00:00:00Z"
# Verify the signed bundle it wrote, before reading a number out of it:
make replay-eval-verify ARGS="aisoc-replay-<id>.bundle.json"
Pin --since and --until. An omitted window dates from now, which is a
different window on a second run and therefore not reproducible — the caveat
stated under Reproducibility.
Three things to read before quoting any figure the run produces:
- the
unlabeledcount. Rows an analyst closed as Undetermined, Unknown or Other are excluded from accuracy rather than folded intotrue_positive. A history that is mostly unlabeled yields a report that says so instead of a confident number over a handful of rows; - the sample size beside every rate. A precision of 1.00 over two predictions is not a precision of 1.00 over two hundred;
- the attribution block. A run whose calls no party attributed is not published as that model's performance.
When this has been run, replace the table above with the measured figures, their denominators, the hardware and the commit — and leave this paragraph in place, so the next reader can tell a measured row from an absent one.
Reproducibility, stated precisely
A report reproduces byte for byte between two runs over the same pinned window, apart from the two wall-clock latency figures.
Everything else is a property of the input and the code: the bootstrap is
seeded and records its seed, the split is a total order over (closed_at, finding_id), and the renderer emits no timestamp and iterates nothing
unsorted. Mean and p95 latency measure the machine the replay ran on and will
differ between two runs on one host, let alone two.
--exclude-latency on the CLI, and exclude_latency=true on the export
route, replace those two figures with a note. That is a filter over the stored
artefact, not a second rendering, and the function that removes the line lives
beside the one that emits it so the two cannot drift.
Two caveats worth stating plainly:
- Pin the window. An omitted
--sinceor--untildates the window from now, which is a different window on a second run and therefore a different report for a reason that has nothing to do with determinism. - Reproducibility is not accuracy. A deterministic model path reproduces because it is deterministic. Pointed at a hosted model, two runs may differ, and that is a property of the model rather than of this pipeline.
tests/e2e/test_replay_cli_end_to_end.py is what holds this claim up. It runs
the CLI twice against a mocked Splunk ES holding 200 recorded closed notables,
through the API, the actions service, the connectors service and the agents
service, and compares the bytes. It also fetches both reports without the
exclusion and asserts that latency is the only line that differs, so the
comparison cannot pass on a stripped artefact that hid something else.
Privacy: where your data goes
Reading your history happens inside your deployment, against credentials you already configured for that SIEM. The findings are processed by the services you are already running.
Data leaves your deployment only when the model you configured is hosted by a third party. If AiSOC is pointed at a hosted provider, the alert content that triage reasons over is sent to that provider in the ordinary way, exactly as it is during live triage. If you run a local model, through the bundled gateway or your own, nothing leaves.
This is a property of your model configuration and not of replay evaluation. Replay adds no new egress: it reads findings you already own and runs them through the same triage path production uses.
Limits
- Reachability, not completeness. The readers parse each vendor's documented response shape. They cannot know whether your retention window holds the period you asked for, or whether your analysts labelled consistently.
- An unparseable close time is an error, not a default. A silently wrong close time would put a finding on the wrong side of a train/test split and leak the answer into its own evaluation, so it raises.
- Elastic yields nothing without tags, as above.
- The vendor's own label is kept verbatim alongside the mapped one, so you can audit a mapping you disagree with rather than having to trust it.
- Fusion's enrichments are absent, as above. A replay grades triage on one normalized finding at a time.
- Tool calls are structurally zero. Shadow mode declines escalation, and escalation is the only stage of this path that calls tools. The field is recorded rather than omitted so a future change that gives triage a tool shows up as a number moving off zero.
- PDF export needs a native stack. WeasyPrint's Pango, Cairo and GLib libraries are installed in the API container image and are routinely absent on a local checkout. When they are, the PDF route answers 503 naming them, rather than serving an empty file; JSON and Markdown carry the same report and need nothing extra.
- Redaction is a pattern pass, and patterns have edges. A single-word
machine name with no digit and no separator is indistinguishable from an
ordinary word, and a host whose first segment collides with public
taxonomy — one literally named
sha-1— is kept for the same reasonSHA-256is. The bundle names both limits in its own privacy block. Read a bundle before sending it; the kit makes a leak unlikely, not impossible. - A signature proves integrity and origin, not honesty. It says a holder of your key produced those bytes and nobody changed them afterwards. It says nothing about whether the window was chosen fairly, which is why the method block travels inside the signed payload.
- Hallucination counting errs high. Indicators are extracted from the agent's own reasoning with the same pattern set the synthetic-corpus grader checks against, so a phrase shaped like a domain is checked and, if absent from the evidence, counted. The published rate is a ceiling, and the indicators behind it travel with the report so you can re-derive it.