# Private AI in Your SOC: How to Run LLMs Locally

> Operating LLMs locally can help analysts summarize and triage sensitive telemetry without shipping data to third-party services.

- Published: 2025-08-11
- Updated: 2026-07-24
- Tags: Security Operations, AI, Privacy, Systems
- Source: https://stevenfoerster.com/notes/private-ai-in-your-soc-how-to-run-llms-locally/

SOC teams want LLMs for summarization and triage, but they cannot send raw logs and alerts to a public API.

A private model changes the risk posture: you keep the data local, control retention, and decide what leaves the boundary.

## Install Ollama on your SOC host

Use a dedicated host or VM in the same private segment as your Wazuh manager.

On Linux:

```bash
curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama
```

On macOS:

```bash
brew install ollama
brew services start ollama
```

Quick health check:

```bash
ollama --version
curl http://127.0.0.1:11434/api/tags
```

If this API is reachable from the network, put it behind a firewall and only allow trusted SOC automation hosts.

## Pull and run a real self-hosted model

For alert summarization and first-pass triage, start with `llama3.1:8b`.

```bash
ollama pull llama3.1:8b
ollama run llama3.1:8b "Summarize this alert: multiple failed SSH logins from 203.0.113.44 in 2 minutes."
```

For automation pipelines, call the local API directly:

```bash
curl -sS http://127.0.0.1:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Summarize this alert: SSH brute force from 203.0.113.44.",
  "stream": false,
  "format": {
    "type": "object",
    "properties": {
      "summary": {"type": "string"},
      "confidence": {"type": "string", "enum": ["low", "medium", "high"]},
      "next_step": {"type": "string"}
    },
    "required": ["summary", "confidence", "next_step"],
    "additionalProperties": false
  },
  "options": { "temperature": 0.1 }
}'
```

Ollama's `format` field constrains generation to JSON or a supplied JSON schema. Validate the parsed object anyway: structured generation improves reliability, but it does not make the content true or safe. Low temperature reduces variation; it does not make inference deterministic across model or runtime changes.

## Decide what the model is allowed to see

Start with scope. Most SOC use cases do not require full packet captures or raw identity data.

Define the minimum context the model needs and redact everything else. Think in tiers:

> - **Tier 1:** Alert metadata, signatures, severity, and timestamps.
> - **Tier 2:** Sanitized log excerpts with IDs and IPs masked.
> - **Tier 3:** Full raw events only when a human explicitly requests them.

Your redaction layer is the security control; the model only consumes what survives it.

## Choose a model and runtime with operational constraints

Running locally means you own latency, cost, and performance.

Make the tradeoffs explicit:

> - Smaller models are easier to run on CPU and are often sufficient for summarization.
> - Larger models improve reasoning but require GPU capacity and careful scheduling.
> - Quantization reduces memory but can change behavior.

Pick a runtime that supports offline operation, audit logging, and explicit model versioning.

Treat models like dependencies you version and audit, not a SaaS subscription.

## Wire Ollama into Wazuh alert enrichment

Use Wazuh Integrator on the manager to invoke a custom script for selected alerts.

Add a custom integration block in `/var/ossec/etc/ossec.conf`:

```xml
<integration>
  <name>custom-ollama-enrich</name>
  <level>10</level>
  <alert_format>json</alert_format>
</integration>
```

Create `/var/ossec/integrations/custom-ollama-enrich`:

```python
#!/usr/bin/env python3
import datetime
import fcntl
import json
import os
import sys
import urllib.error
import urllib.request
from typing import Any

OLLAMA_URL = "http://127.0.0.1:11434/api/generate"
MODEL = "llama3.1:8b"
OUTFILE = "/var/ossec/logs/llm-enrichment.json"
SCHEMA = {
  "type": "object",
  "properties": {
    "summary": {"type": "string"},
    "confidence": {"type": "string", "enum": ["low", "medium", "high"]},
    "next_step": {"type": "string"},
  },
  "required": ["summary", "confidence", "next_step"],
  "additionalProperties": False,
}

def validate_enrichment(value: Any) -> dict[str, str]:
  if not isinstance(value, dict) or set(value) != {"summary", "confidence", "next_step"}:
    raise ValueError("response does not match the enrichment schema")
  if not isinstance(value["confidence"], str) or value["confidence"] not in {"low", "medium", "high"}:
    raise ValueError("invalid confidence")
  for key in ("summary", "next_step"):
    if not isinstance(value[key], str) or not value[key].strip() or len(value[key]) > 2000:
      raise ValueError(f"invalid {key}")
  return value

def call_ollama(prompt: str) -> tuple[dict[str, str], dict[str, Any]]:
  payload = {
    "model": MODEL,
    "prompt": prompt,
    "stream": False,
    "format": SCHEMA,
    "options": {"temperature": 0.1}
  }
  req = urllib.request.Request(
    OLLAMA_URL,
    data=json.dumps(payload).encode("utf-8"),
    headers={"Content-Type": "application/json"}
  )
  with urllib.request.urlopen(req, timeout=45) as resp:
    raw = json.loads(resp.read().decode("utf-8"))
  result = validate_enrichment(json.loads(raw.get("response", "")))
  provenance = {
    "requested_model": MODEL,
    "reported_model": raw.get("model"),
    "created_at": raw.get("created_at"),
  }
  return result, provenance

def append_json_line(path: str, value: dict[str, Any]) -> None:
  # Open for each record so external rotation does not leave us writing to an old inode.
  line = (json.dumps(value, separators=(",", ":")) + "\n").encode("utf-8")
  fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o640)
  with os.fdopen(fd, "ab") as out:
    fcntl.flock(out, fcntl.LOCK_EX)
    out.write(line)
    out.flush()

def main() -> int:
  alert_path = sys.argv[1]
  with open(alert_path, "r", encoding="utf-8") as f:
    alert = json.load(f)

  rule = alert.get("rule", {})
  # Deliberately omit raw event text, identity fields, and source addresses.
  # Add fields only after documenting why the model needs them and how they are redacted.
  prompt = (
    "You are a SOC assistant. Summarize the alert and propose one analyst check. "
    "Do not recommend automated containment. "
    f"rule_id={rule.get('id')} rule_level={rule.get('level')} "
    f"description={rule.get('description')}"
  )

  llm, provenance = call_ollama(prompt)
  enriched = {
    "integration": "ollama-enrich",
    "timestamp": datetime.datetime.now(datetime.timezone.utc).isoformat(),
    "wazuh_rule_id": rule.get("id"),
    "wazuh_level": rule.get("level"),
    "llm": llm,
    "provenance": provenance,
  }

  append_json_line(OUTFILE, enriched)
  return 0

if __name__ == "__main__":
  try:
    raise SystemExit(main())
  except (OSError, ValueError, json.JSONDecodeError, urllib.error.URLError) as exc:
    print(f"ollama enrichment failed: {exc}", file=sys.stderr)
    raise SystemExit(1)
```

Set permissions:

```bash
sudo chown root:wazuh /var/ossec/integrations/custom-ollama-enrich
sudo chmod 750 /var/ossec/integrations/custom-ollama-enrich
```

Now ingest the enrichment file back into Wazuh:

```xml
<localfile>
  <location>/var/ossec/logs/llm-enrichment.json</location>
  <log_format>json</log_format>
</localfile>
```

This creates an enrichment path: Wazuh alert -> bounded local prompt -> schema-constrained response -> validated JSON line -> searchable context. It is still a reference implementation. A production deployment also needs rate limiting, back-pressure, a dead-letter path, health metrics, and a tested log-rotation policy.

## Validate the flow end to end

1. Trigger a known alert (for example, repeated SSH login failures).
2. Confirm Wazuh runs the integration script.
3. Check `/var/ossec/logs/llm-enrichment.json` for appended JSON.
4. Verify the enrichment event appears in Wazuh with `llm.summary`, `llm.confidence`, and `llm.next_step`.

Keep the model assistive. Use enrichment to prioritize and summarize, then let analysts and rules decide containment.

## Build an auditable pipeline

A durable pattern is:

> 1. Normalize alerts into a consistent schema.
> 2. Redact and classify sensitive fields.
> 3. Run local inference for summarization, prioritization, or enrichment.
> 4. Store output with provenance and version metadata.

The SOC should be able to answer, "Which model produced this summary?" without guesswork.

To stand up the retrieval-and-generation half of this pattern from scratch, [Build a Local RAG Pipeline with Ollama and ChromaDB](https://stevenfoerster.com/tutorials/build-a-local-rag-pipeline-with-ollama-and-chromadb/) walks through chunking, embedding, and grounded generation step by step, all running inside the boundary.

## Keep the model inside the security boundary

Local LLMs are still software that can be exploited.

Treat the inference service as a sensitive system:

> - Isolate it on a private network segment.
> - Disable outbound network access by default.
> - Log the minimum metadata needed for auditability. Redact sensitive content, restrict access, set retention, and sample full prompts only when the investigation justifies it.
> - Apply the same hardening standards you use for other production services.

## Accept the limits

> Local models can assist with triage and narrative building, but they do not replace detection logic, rules engines, or human-led incident response.
>
> Use them to reduce analyst toil, not to make the final call. Keep the model in an assistive role until you have strong validation.

If the model is not allowed to see plaintext, this design is the wrong design. Change the data flow or the trust boundary rather than treating advanced cryptography as a drop-in wrapper for a local inference service.

For a hands-on implementation that builds a RAG pipeline over Wazuh alerts, see [Run a Private AI Assistant in Your SOC](https://stevenfoerster.com/tutorials/run-a-private-ai-assistant-in-your-soc/).

## Sources

- [Ollama API: Generate a response](https://docs.ollama.com/api/generate)
- [Ollama: Structured outputs](https://docs.ollama.com/capabilities/structured-outputs)
- [Wazuh: External API integration](https://documentation.wazuh.com/current/user-manual/manager/integration-with-external-apis.html)