Repository overview

This page is the canonical map of the AiSOC monorepo. It used to live in README.md; it was extracted here so the front page stays focused on "what is AiSOC and how do I run it" while this page stays useful for anyone navigating the code.

For the data-flow architecture (Kafka spine, fusion, agents, storage tier, surface) see Architecture. For the deeper system-design write-up (ML fusion, Neo4j-at-ingest schema, threat-intel pipeline) see docs/architecture/SYSTEM_DESIGN.md.

Monorepo layout

AiSOC/
├── apps/
│   ├── web/              # Next.js console + marketing landing + Responder PWA route
│   └── docs/             # Docusaurus documentation site (this site)
├── services/
│   ├── api/              # Core REST API + Neo4j graph + rule engine + auth + RBAC + compliance + Investigation Ledger
│   ├── ingest/           # Go · OCSF normalization · Shodan + CVE
│   ├── enrichment/       # Go · IOC enrichment
│   ├── fusion/           # Python · dedup + ML scoring + per-alert confidence + correlation narrative
│   ├── agents/           # Python · LangGraph + Qdrant RAG + investigation ledger writer + hunt engine
│   ├── actions/          # Python · SOAR + blast-radius gating + ChatOps verification
│   ├── threatintel/      # Python · CISA KEV by default; MISP / OTX / TAXII when an
│   │                     # operator configures one · threat-actor attribution
│   ├── realtime/         # Node.js · per-channel WebSocket fan-out + VAPID Web Push
│   ├── ueba/             # Python · User & Entity Behavior Analytics (Welford online baseline)
│   ├── honeytokens/      # Python · deceptive credential traps (HMAC-signed)
│   ├── purple-team/      # Python · Atomic Red Team + Caldera + ATT&CK + detection drift snapshots
│   ├── osquery-tls/      # Python · native osquery TLS server + FIM + pack distribution
│   ├── osquery-extensions/ # Go · AI threat-intel table + ML anomaly score table + 3 more
│   ├── slack-bot/        # Python · ChatOps surface (Slack Bolt + HMAC signature verification)
│   └── mcp/              # TypeScript · Model Context Protocol server (npm: @aisoc/mcp — built and packed on every tag, upload blocked on registry credentials)
├── packages/
│   ├── types/            # Shared TS types
│   ├── ui/               # Shared React primitives
│   ├── ocsf/             # OCSF normalization helpers
│   ├── sdk-ts/           # TypeScript client SDK for AiSOC API (npm: @aisoc/sdk — unpublished, see below)
│   ├── sdk-py/           # Async Python client SDK (PyPI: aisoc-sdk — published)
│   ├── sdk-go/           # Go client SDK + models (module: github.com/beenuar/AiSOC/packages/sdk-go)
│   ├── plugin-sdk-ts/    # TypeScript plugin development SDK
│   ├── plugin-sdk-py/    # Python plugin development SDK (PyPI: aisoc-plugin-sdk — published)
│   ├── plugin-sdk-go/    # Go plugin development SDK (module: github.com/beenuar/AiSOC/packages/plugin-sdk-go)
│   ├── aisoc-cli/        # CLI: scaffold / validate / publish plugins & detections (PyPI: aisoc-cli — published)
│   ├── aisoc-sandbox/    # Offline 30-second simulator (PyPI: aisoc-sandbox — published)
│   └── aisoc-detections/ # Python detection-authoring framework (PyPI: aisoc-detections — published)
├── detections/           # Detection rules as YAML. A *projection* of the compiled
│                         # ruleset, not the engine's input: 6,991 rules on disk,
│                         # 2,511 executable. (More YAML files than that live here —
│                         # the surplus are response playbooks, not rules.) Recount with
│                         # scripts/generate_corpus_stats.py rather than trusting this line.
├── hunts/                # Hunt-as-Code YAML definitions (hypothesis + indicators + schedule)
├── playbooks/            # SOAR playbooks (YAML) — 62 packs
├── plugins/              # First-party plugins (Go + Python) + marketplace manifests
├── marketplace/          # Marketplace index (JSON, generated by scripts/build_marketplace.py)
├── infra/
│   ├── coolify/          # Coolify (self-hosted Heroku-style PaaS) quickstart
│   ├── fly/              # Fly.io machines + deploy script (one recipe among several;
│   │                     # no hosted demo is operated from here)
│   ├── railway/          # Railway template (railway.toml)
│   ├── terraform/        # AWS (VPC, EKS, RDS, ElastiCache, MSK) + GCP + Azure skeletons
│   └── helm/             # Kubernetes Helm chart (HPA, PDB, Ingress per service)
├── docs/
│   ├── openapi.yaml      # OpenAPI 3.1 spec for the Core API
│   ├── architecture/     # System-design docs (SYSTEM_DESIGN.md, threat-actor attribution, …)
│   ├── decisions/        # Architecture Decision Records (ADR-0001 onwards)
│   ├── demo/             # Screencast brief + shot list
│   ├── operations/       # Runbooks, managed-mode notes, audit health probes
│   └── runbooks/         # Per-service operational runbooks
└── scripts/
    ├── aisoc-demo.ts     # One-shot demo orchestrator (powers `pnpm aisoc:demo`)
    ├── aisoc-doctor.ts   # Local health check (ports, containers, demo data, API + WS)
    ├── aisoc-acceptance.ts # Cold-start acceptance test
    ├── run_evals.py      # Public eval harness (per-case + per-template macros, telemetry coverage)
    ├── generate_eval_incidents.py  # 200-incident synthetic generator (55 templates) + synthetic_telemetry.jsonl
    ├── build_marketplace.py        # Build marketplace/index.json from detections+playbooks+plugins
    ├── validate_detections.py      # YAML schema validation for Sigma detections
    ├── validate_playbooks.py       # YAML schema validation for playbooks
    ├── backup.sh         # Postgres + ClickHouse + plugins → S3/R2
    └── restore.sh        # Point-in-time restore

Service map

Two port columns, because conflating them has cost real time. Container is what the process listens on inside the Docker network, and is what another service must address. Host is what docker-compose.yml publishes on 127.0.0.1, and is what your browser and curl use. They differ for most services.

litellm and ollama are CORE — see Architecture for why the gateway without a model behind it could not do AI triage at all. connectors and actions are CORE too: a default install has to be able to pull from a vendor as well as receive a push, and the investigation agent's vendor tools have to reach something. CORE is 16 long-running services and full is 22 long-running services; both are enforced against docker-compose.yml by scripts/check_profile_service_counts.py, so this table cannot drift from the compose file without failing CI.

Service	Lang	Container	Host	Profile	Role
`web`	Next.js	3000	3000	core	SOC console, Investigation Rail, marketing landing
`api`	Python · FastAPI	8000	8000	core	Alerts, cases, RBAC, rules, audit, compliance, federated-search fan-out, ledger reads
`ingest`	Go	8080 · 9090	8081 · 9090	core	OCSF normalization, Kafka publish, inbox webhooks, graph-at-ingest writers
`fusion`	Python	8003	8003	core	Detection engine, promotion, correlation, confidence, narrative, lake writer
`agents`	Python · LangGraph	8084	8001	core	Auto-triage worker, investigation agents, playbook engine, ledger writer
`realtime`	Node.js · `ws`	4000	8086	core	Per-channel WebSocket fan-out, VAPID Web Push
`threatintel`	Python	8005	8005	core	CISA KEV polling with no credentials; MISP, OTX and TAXII feeds register only when an operator supplies a URL and credentials. Threat-actor attribution
`litellm`	— (image)	4000	4000	core	LLM gateway; every `aisoc-<role>` alias resolves here and nowhere else
`ollama`	— (image)	11434	11434	core	The local model behind the gateway, pinned at `llama3.2:3b-instruct-q4_K_M`
`actions`	Python	8085	8002	core	Governed capability execution and post-action verification
`connectors`	Python	8003	8088	core	Scheduled vendor polling, credential vault, federated query translators
`enrichment`	Go	8082	8080	full	IOC enrichment fan-out (VT, AbuseIPDB, GreyNoise)
`ueba`	Python	8004	8007	full	User and entity behaviour analytics
`honeytokens`	Python	8005	8008	extras	Honeytoken lifecycle and webhook alerting
`purple-team`	Python	8006	8006	extras	Atomic Red Team, Caldera, ATT&CK heatmap, detection drift
`osquery-tls`	Python	9001	8091	osquery	osquery TLS server — enrol nodes, distribute packs, stream telemetry
`slack-bot`	Python	8089	8009	chatops	ChatOps approvals and the `/aisoc` slash command
`osquery-extensions`	Go	—	—	—	Custom osquery tables and decorators
`mcp`	TypeScript	— (stdio)	—	—	Model Context Protocol server exposing AiSOC tools to IDE-side agents

Storage tier

Store	Purpose
PostgreSQL	Tenants, users, cases, detection rules, RBAC, audit log, compliance · Row-level security
ClickHouse	High-cardinality event analytics + alert metrics
OpenSearch	Full-text IOC + actor + report search · Sigma backend
Qdrant	Vector RAG for agents, semantic ATT&CK lookup
Neo4j	Knowledge graph: entities, attack paths, blast radius (graph-at-ingest writers in `services/ingest/internal/graph/`)
Redis	Cache, pub/sub, IOC bloom filter, enrichment TTL, short-lived realtime tickets
Kafka	Event streaming spine (raw, fused, vulnerability, anomaly, action, graph-updates)

API surface

The full OpenAPI 3.1 spec lives at docs/openapi.yaml, and that committed file is the one to read.

The interactive views — /api/docs (Swagger) and /api/redoc (ReDoc) — are mounted only when the API is not in production mode, and make up starts a production-class stack, so on a default install both answer 404 along with /api/openapi.json. That is deliberate rather than a bug: a public deployment should not serve a live schema explorer. Read the committed spec, or run the API with a non-production ENVIRONMENT if you want the explorer.

Package publication status

Eight packages are declared. Five are published and resolve today — the PyPI side, uploaded by trusted publishing from a tag. The remaining three are npm and are not published, because the first upload needs an NPM_TOKEN this repository does not hold; that is an account action, not an engineering one. Build them from source in the meantime.

release.yml builds and packs all eight on every tag regardless, so the pipeline cannot rot while half of it is idle, and scripts/check_published_packages.py checks the tree against the live registries in both directions, so neither a package that quietly stopped resolving nor one described as unpublished after it shipped can survive a CI run.

The PyPI name aisoc belongs to an unrelated project, which is why the Python sandbox package ships as aisoc-sandbox; the npm name aisoc is free.

Next stops

Architecture — data-flow + service interaction diagrams.
docs/architecture/SYSTEM_DESIGN.md — deeper write-up.
apps/docs/docs/intro.md — feature catalogue index.
Plugin overview — how to extend.

Monorepo layout​

Service map​

Storage tier​

API surface​

Package publication status​

Next stops​