Repository overview

This page is the canonical map of the AiSOC monorepo. It used to live in README.md; it was extracted here so the front page stays focused on "what is AiSOC and how do I run it" while this page stays useful for anyone navigating the code.

For the data-flow architecture (Kafka spine, fusion, agents, storage tier, surface) see Architecture. For the deeper system-design write-up (ML fusion, Neo4j-at-ingest schema, threat-intel pipeline) see docs/architecture/SYSTEM_DESIGN.md.

Monorepo layout

AiSOC/
├── apps/
│   ├── web/              # Next.js console + marketing landing + Responder PWA route
│   └── docs/             # Docusaurus documentation site (this site)
├── services/
│   ├── api/              # Core REST API + Neo4j graph + rule engine + auth + RBAC + compliance + Investigation Ledger
│   ├── ingest/           # Go · OCSF normalization · Shodan + CVE
│   ├── enrichment/       # Go · IOC enrichment
│   ├── fusion/           # Python · dedup + ML scoring + per-alert confidence + correlation narrative
│   ├── agents/           # Python · LangGraph + Qdrant RAG + investigation ledger writer + hunt engine
│   ├── actions/          # Python · SOAR + blast-radius gating + ChatOps verification
│   ├── threatintel/      # Python · TAXII / MISP / OTX / KEV + threat-actor attribution
│   ├── realtime/         # Node.js · per-channel WebSocket fan-out + VAPID Web Push
│   ├── ueba/             # Python · User & Entity Behavior Analytics (Welford online baseline)
│   ├── honeytokens/      # Python · deceptive credential traps (HMAC-signed)
│   ├── purple-team/      # Python · Atomic Red Team + Caldera + ATT&CK + detection drift snapshots
│   ├── osquery-tls/      # Python · native osquery TLS server + FIM + pack distribution
│   ├── osquery-extensions/ # Go · AI threat-intel table + ML anomaly score table + 3 more
│   ├── slack-bot/        # Python · ChatOps surface (Slack Bolt + HMAC signature verification)
│   └── mcp/              # TypeScript · Model Context Protocol server (npm: @aisoc/mcp, lands in v8.0)
├── packages/
│   ├── types/            # Shared TS types
│   ├── ui/               # Shared React primitives
│   ├── ocsf/             # OCSF normalization helpers
│   ├── sdk-ts/           # TypeScript client SDK for AiSOC API (npm: @aisoc/sdk, lands in v8.0)
│   ├── sdk-py/           # Async Python client SDK (PyPI: aisoc-sdk, lands in v8.0)
│   ├── sdk-go/           # Go client SDK + models (module: github.com/beenuar/aisoc/sdk-go)
│   ├── plugin-sdk-ts/    # TypeScript plugin development SDK
│   ├── plugin-sdk-py/    # Python plugin development SDK (PyPI: aisoc-plugin-sdk, lands in v8.0)
│   ├── plugin-sdk-go/    # Go plugin development SDK (module: github.com/beenuar/aisoc/plugin-sdk-go)
│   └── aisoc-cli/        # CLI: scaffold / validate / publish plugins & detections (PyPI: aisoc-cli, lands in v8.0)
├── detections/           # Community Sigma detection rules (YAML) — 800+ native
├── hunts/                # Hunt-as-Code YAML definitions (hypothesis + indicators + schedule)
├── playbooks/            # Community SOAR playbooks (YAML)
├── plugins/              # First-party plugins (Go + Python) + marketplace manifests
├── marketplace/          # Marketplace index (JSON, generated by scripts/build_marketplace.py)
├── infra/
│   ├── coolify/          # Coolify (self-hosted Heroku-style PaaS) quickstart
│   ├── fly/              # Fly.io machines + deploy script (canonical hosted demo target)
│   ├── railway/          # Railway template (railway.toml)
│   ├── render/           # Render blueprint (render.yaml at repo root)
│   ├── terraform/        # AWS (VPC, EKS, RDS, ElastiCache, MSK) + GCP + Azure skeletons
│   └── helm/             # Kubernetes Helm chart (HPA, PDB, Ingress per service)
├── docs/
│   ├── openapi.yaml      # OpenAPI 3.1 spec for the Core API
│   ├── architecture/     # System-design docs (SYSTEM_DESIGN.md, threat-actor attribution, …)
│   ├── decisions/        # Architecture Decision Records (ADR-0001 onwards)
│   ├── demo/             # Screencast brief + shot list
│   ├── operations/       # Runbooks, managed-mode notes, audit health probes
│   └── runbooks/         # Per-service operational runbooks
└── scripts/
    ├── aisoc-demo.ts     # One-shot demo orchestrator (powers `pnpm aisoc:demo`)
    ├── aisoc-doctor.ts   # Local health check (ports, containers, demo data, API + WS)
    ├── aisoc-acceptance.ts # Cold-start acceptance test
    ├── run_evals.py      # Public eval harness (per-case + per-template macros, telemetry coverage)
    ├── generate_eval_incidents.py  # 200-incident synthetic generator (55 templates) + synthetic_telemetry.jsonl
    ├── build_marketplace.py        # Build marketplace/index.json from detections+playbooks+plugins
    ├── validate_detections.py      # YAML schema validation for Sigma detections
    ├── validate_playbooks.py       # YAML schema validation for playbooks
    ├── backup.sh         # Postgres + ClickHouse + plugins → S3/R2
    └── restore.sh        # Point-in-time restore

Service map

Service	Lang	Port	Role
web	Next.js + React	3000	SOC console (alerts detail + Investigation Rail), benchmark scoreboard, marketing landing
api	Python · FastAPI	8000	Alerts (detail envelope + correlation narrative), cases, RBAC, graph, rules, audit, compliance, detection proposals (DAC), federated search fan-out, SLA tracking
realtime	Node.js · ws	8086	Per-channel WebSocket fan-out + VAPID Web Push (short-lived signed tickets)
agents	Python · LangGraph	8001	Multi-agent reasoning + Qdrant RAG + Hunt-as-Code engine & scheduler + Investigation Ledger writer
fusion	Python	8003	Dedup + ML scoring (LightGBM, IsoForest), alert confidence, entity risk / RBA, correlation narrative projection for API
actions	Python	8002	SOAR with blast-radius gating + ChatOps verification + signed approval URLs
connectors	Python	—	Connector polling (APScheduler), credential vault, federated query translators
threatintel	Python	8005	TAXII / MISP / OTX / KEV polling + threat-actor attribution engine
ueba	Python	8007	User & Entity Behavior Analytics
honeytokens	Python	8008	Honeytoken lifecycle + webhook alerting
purple-team	Python	8006	Atomic Red Team + Caldera + ATT&CK heatmap + detection drift snapshots
osquery-tls	Python	8091	Native osquery TLS server — enroll nodes, distribute packs, stream FIM/process/network telemetry
osquery-extensions	Go	—	Custom osquery extensions (AI-powered threat intel table, ML anomaly score table, + 3 more)
slack-bot	Python	8009	ChatOps surface — interactive approvals for high-blast-radius actions, /aisoc slash command
mcp	TypeScript	— (stdio)	Model Context Protocol server exposing 13 AiSOC tools — discovery, deep-dive, action/replay, and warm-tier lake query
ingest	Go	8081	OCSF normalization + Shodan/CVE + Neo4j graph-at-ingest writers
enrichment	Go	8080	IOC enrichment (VT, AbuseIPDB, GreyNoise)

Storage tier

Store	Purpose
PostgreSQL	Tenants, users, cases, detection rules, RBAC, audit log, compliance · Row-level security
ClickHouse	High-cardinality event analytics + alert metrics
OpenSearch	Full-text IOC + actor + report search · Sigma backend
Qdrant	Vector RAG for agents, semantic ATT&CK lookup
Neo4j	Knowledge graph: entities, attack paths, blast radius (graph-at-ingest writers in services/ingest/internal/graph/)
Redis	Cache, pub/sub, IOC bloom filter, enrichment TTL, short-lived realtime tickets
Kafka	Event streaming spine (raw, fused, vulnerability, anomaly, action, graph-updates)

API surface

The full OpenAPI 3.1 spec lives at docs/openapi.yaml. Endpoint groups are summarised in Architecture and the interactive view is at http://localhost:8000/docs (Swagger) or /redoc (ReDoc) when you run the local stack.

Next stops

• Architecture — data-flow + service interaction diagrams.
• docs/architecture/SYSTEM_DESIGN.md — deeper write-up.
• apps/docs/docs/intro.md — feature catalogue index.
• Plugin overview — how to extend.