LLM-IN-THE-LOOP · SELF-HOSTED · TWO-PHASE ALERTING

The alert goes out in seconds.
The investigation follows.

Infra-Sentinel polls Zabbix and Prometheus with cheap, constant checks and calls a self-hosted model only on real news. v1 refused to alert unless the model had reasoned about it — argued for in writing, as rigour. Then one call took 9 minutes, and a three-minute outage got reported half an hour late. v2 splits the cycle in two: the alert never waits for the model, the investigation follows.

9m00s
one model call on CPU, from the production incident log — not reproducible in this repo
2 phases
alert without the model, report after it
~0.1s evidence
four bounded queries, no model, before the prompt
0 calls/hr
steady state: the model is never touched
The middle path

Alerting stacks are usually dumb or expensive.

Forward every raw trigger and you drown the operator. Pipe everything to a cloud LLM and you pay per token to reason about noise. Infra-Sentinel splits the job: local polling does detection, a local model does reasoning — only on the delta.

DUMB

Raw-trigger firehose

Every flap, every port, every recovery becomes a page. Operators mute the channel — and miss the one that mattered.

EXPENSIVE

Everything → cloud LLM

Reasoning quality, but you pay per token to summarize noise, leak infra telemetry off-site, and depend on someone else's uptime.

INFRA-SENTINEL

Cheap detect · local reason

Constant free polling finds the delta. A self-hosted LLM reasons only on real news — then tells you exactly what to do. Nothing leaves the network.

Live pipeline

Follow a signal through the watchdog.

Two monitoring sources are diffed against persisted state; the delta runs a gauntlet of guardrails; only survivors reach the LLM and the operator. Pick a scenario and watch the path light up.

Steady state — the diff is empty, the model is never called.
Zabbix problems ≥ High Prometheus up == 0 diff vs baseline + dedup · cooldown silence no-op · model idle circuit breaker flood? threshold 8 self-hosted LLM Ollama · self-hosted terse alert LLM skipped Notifier → Telegram
🛰️
Infra-Sentinel
on watch
The parts worth reading

Guardrails, not glue.

The value isn't the plumbing — it's the pure, unit-tested functions that keep the watchdog useful instead of noisy, wasteful or silent.

01

Baseline seeding

The first cycle records everything already broken as the baseline and raises nothing. You're never paged for problems that predate the watchdog.

02

Dedup by objectid + cooldown

A flapping trigger emits a new event id every cycle. Sentinel keys "already alerted?" on the stable trigger id and suppresses re-alerts within DEDUP_COOLDOWN_SEC. One flapping port = one alert, not one per cycle.

03

Mass-outage circuit breaker

A flood of new problems in one cycle (MASS_OUTAGE_THRESHOLD, default 8) is the signature of a core/power failure. Sentinel short-circuits to one terse alert — no per-event LLM calls, no pager storm.

04

Event window, exactly once

The active-problem query answers "what is broken now", so an incident that opens and closes between two rounds is invisible to it — recorded perfectly by the monitoring system, never seen by the watchdog. A second query asks for events in a time range; the ranges overlap on purpose so nothing falls between them, and a ledger of alerted ids is what stops the overlap from alerting twice. Dropping the overlap would trade a duplicate for a miss, and for a watchdog a duplicate gets noticed while a silence does not.

05

Degrade the enrichment, never the delivery

The alert is built from raw facts and sent before the model is contacted, and the baseline advances with it — if the send fails, it does not advance and the next round retries. Everything after that point is best-effort by construction. This reverses what v1 did, and the reason it reversed is a measurement.

Two brains · capability, not release

From what to watchwhat to do.

Rung 1 is in production: it turns raw triggers into one impact-aware alert. Rung 2 is the pilot leap — a model with the infra runbook baked in, reasoning over the trigger's real metrics (not guesses), that returns the concrete fix.

rung 1
Observe & explain
production · Qwen 27B (base)
▲ Core switch uplink unreachable.
Distribution path for the wireless VLAN is at risk. Check the switch and its power feed before APs drop.
Turns a pile of triggers into one human-readable, impact-aware message. Great triage — but the operator still has to know the fix.
rung 2
Recommend the fix
pilot · sentinel-infra (runbook-baked)
▲ Gi0/2 negotiated 10 Mbps — expected 1000 (Gi class).
Cause: speed/duplex mismatch after a link flap. interface Gi0/2 speed 1000 duplex full Run from the distribution switch CLI. No recovery in 2 cycles → escalate to network on-call.
A dedicated model with the runbook in its system prompt — name↔device maps, expected link speeds by interface class, vendor CLI, escalation. Each event is enriched with the trigger's real metrics (current speed, state, counters), so it reasons on hard data, not a hunch.
The reversal

I argued for it in writing. Then a number took it apart.

v1 ran detect → reason → notify and only advanced its baseline once the whole pipeline had succeeded. The README defended the consequence as rigour, in these words: "an operator alert is either LLM-reasoned or it doesn't go out." That argument is coherent. It also put the model on the critical path of the notification — and nobody priced that, because on the hardware in the design document the model answers in seconds.

On the hardware that exists, one call took 9 minutes. With 15-minute rounds, a three-minute outage was being reported up to half an hour late. The decision was right about what it optimised for and wrong about what mattered.

v1Refuse outright
detect → reason → notify
Buys: every message that arrives has been reasoned about. Nothing half-finished reaches a human, or a downstream action layer.

Spends: the notification itself. A slow model delays it; an unreachable one withholds it entirely. The failure is invisible from the outside — a silent channel is also what a healthy day looks like.
v2Split the guarantees
alert → advance baseline → investigate → report
Phase 1 is load-bearing. Raw facts, no model, seconds. The baseline advances here: if delivery fails it does not advance, and the next round retries.

Phase 2 is best-effort. If the investigation breaks, a short message says so. Never silent, never holding the alert back.
The rule was already written down. Degrade the enrichment, never the delivery was in the design notes from the start. v2 is not a new principle — it is the point where the code started obeying one it had been stating. That gap is the ordinary way this fails: the policy is in the prose, the ordering is in the code, and nothing checks that they agree.
What a fallback is still not allowed to do: degrade and stay quiet about it. A raw alert sent with no marker is indistinguishable from a reasoned one, and the channel silently gets worse. Phase 2 failing produces a message that says the investigation failed — because "it broke" and "it found nothing" call for different responses from the person reading.
The vision

The road to an autonomous infra agent.

Three rungs from a smart watchdog to an agent that works the infrastructure on its own. Two are shipped or in pilot. The third is the honest, careful part.

1

Observe & explain

Detect the delta cheaply and constantly; turn raw triggers into one impact-aware alert. Guardrails keep it quiet and safe.

✓ SHIPPED · v1
2

Recommend the fix

A runbook-grounded model reasons over real metrics and returns the exact command, where to run it, and when to escalate. Human executes.

◆ PILOT · v2
3

Act — with a human in the loop

Execute safe, idempotent remediations from a closed catalog, approved from chat, then verify the problem is gone before closing. From advisor to operator.

→ NEXT

What rung 3 still needs (the honest list)

  • Action allowlist — a closed catalog of safe, reversible ops (unlock, restart, ticket), each an idempotent, validated function.
  • Approval loop — inline approve/reject in Telegram; nothing runs unattended at first.
  • Verify return-to-BAU — re-poll after acting; if the trigger persists, escalate, don't retry-storm.
  • Audit trail — every proposed and executed action logged and reversible.
  • Statistics & trends — a daily digest of flapping/noise, historical baselines, SLO-style reporting.
  • Change-management fit — actions that respect windows, approvals and security (e.g. never auto-unlock a possibly-compromised account).
Where it honestly stands: the reasoning half is largely solved — v2 already recommends the exact fix, grounded in a runbook and live metrics. The acting half — a safe action layer with approvals and verification — is mostly unbuilt, and that's where the real engineering (and the real risk) lives. Close on the brain; deliberate on the hands.
Why the model is rarely called

The model is a scalpel, not a firehose.

watchdog.py — the hot path
# cheap + constant: runs every cycle, no model
delta = diff(current, baseline)
delta = dedup_with_cooldown(delta, state)

if not delta:
    return                 # steady state → 0 model calls

if len(delta) >= MASS_OUTAGE_THRESHOLD:
    notify(terse_outage_alert(delta))
    return                 # flood → LLM skipped

# rare + local: only genuine, bounded news
event  = enrich(delta, zabbix_metrics)   # real numbers
alert  = llm_reason(event)              # runbook-grounded
notify(alert)

The poll loop is local and free. The LLM is only ever asked to reason about a small, deduped, enriched set of genuinely new events — and it answers with a concrete fix.

Python · stdlib-only Ollama · self-hosted LLM Zabbix + Prometheus Telegram notifier systemd · 15 min Docker Compose demo ruff + pytest CI

The trade-off, stated plainly

The scarcity that keeps the model idle is also the scarcity of what the system sees. The LLM never watches raw metrics or the full monitoring state — it only ever reasons over the small diff the pollers already flagged as new. If a problem doesn't surface as a change in Zabbix or Prometheus, it never reaches the model at all.

When the model does not answer, the alert has already gone out and the follow-up says the investigation failed. That distinction is load-bearing: a model outage and an uneventful night must not look the same to the operator, and under an ordering where the notification waits for the model, both are silence.

Which is also why rung 3 of the roadmap — acting, not just recommending — stays a human-in-the-loop design on purpose. A model that reasons well over a diff is not the model you hand the keys to production infrastructure, and this repo doesn't pretend otherwise.

Grounded in sentinel/analyzer.py, sentinel/cli.py and sentinel/watchdog.py — no-fallback retry semantics and diff-only snapshots, as shipped.