Infra-Sentinel polls Zabbix and Prometheus with cheap, constant checks and calls a self-hosted model only on real news. v1 refused to alert unless the model had reasoned about it — argued for in writing, as rigour. Then one call took 9 minutes, and a three-minute outage got reported half an hour late. v2 splits the cycle in two: the alert never waits for the model, the investigation follows.
Forward every raw trigger and you drown the operator. Pipe everything to a cloud LLM and you pay per token to reason about noise. Infra-Sentinel splits the job: local polling does detection, a local model does reasoning — only on the delta.
Every flap, every port, every recovery becomes a page. Operators mute the channel — and miss the one that mattered.
Reasoning quality, but you pay per token to summarize noise, leak infra telemetry off-site, and depend on someone else's uptime.
Constant free polling finds the delta. A self-hosted LLM reasons only on real news — then tells you exactly what to do. Nothing leaves the network.
Two monitoring sources are diffed against persisted state; the delta runs a gauntlet of guardrails; only survivors reach the LLM and the operator. Pick a scenario and watch the path light up.
The value isn't the plumbing — it's the pure, unit-tested functions that keep the watchdog useful instead of noisy, wasteful or silent.
The first cycle records everything already broken as the baseline and raises nothing. You're never paged for problems that predate the watchdog.
objectid + cooldownA flapping trigger emits a new event id every cycle. Sentinel keys "already alerted?" on the stable trigger id and suppresses re-alerts within DEDUP_COOLDOWN_SEC. One flapping port = one alert, not one per cycle.
A flood of new problems in one cycle (MASS_OUTAGE_THRESHOLD, default 8) is the signature of a core/power failure. Sentinel short-circuits to one terse alert — no per-event LLM calls, no pager storm.
The active-problem query answers "what is broken now", so an incident that opens and closes between two rounds is invisible to it — recorded perfectly by the monitoring system, never seen by the watchdog. A second query asks for events in a time range; the ranges overlap on purpose so nothing falls between them, and a ledger of alerted ids is what stops the overlap from alerting twice. Dropping the overlap would trade a duplicate for a miss, and for a watchdog a duplicate gets noticed while a silence does not.
The alert is built from raw facts and sent before the model is contacted, and the baseline advances with it — if the send fails, it does not advance and the next round retries. Everything after that point is best-effort by construction. This reverses what v1 did, and the reason it reversed is a measurement.
Rung 1 is in production: it turns raw triggers into one impact-aware alert. Rung 2 is the pilot leap — a model with the infra runbook baked in, reasoning over the trigger's real metrics (not guesses), that returns the concrete fix.
v1 ran detect → reason → notify and only advanced its baseline once the whole pipeline had succeeded. The README defended the consequence as rigour, in these words: "an operator alert is either LLM-reasoned or it doesn't go out." That argument is coherent. It also put the model on the critical path of the notification — and nobody priced that, because on the hardware in the design document the model answers in seconds.
On the hardware that exists, one call took 9 minutes. With 15-minute rounds, a three-minute outage was being reported up to half an hour late. The decision was right about what it optimised for and wrong about what mattered.
Three rungs from a smart watchdog to an agent that works the infrastructure on its own. Two are shipped or in pilot. The third is the honest, careful part.
Detect the delta cheaply and constantly; turn raw triggers into one impact-aware alert. Guardrails keep it quiet and safe.
A runbook-grounded model reasons over real metrics and returns the exact command, where to run it, and when to escalate. Human executes.
Execute safe, idempotent remediations from a closed catalog, approved from chat, then verify the problem is gone before closing. From advisor to operator.
# cheap + constant: runs every cycle, no model delta = diff(current, baseline) delta = dedup_with_cooldown(delta, state) if not delta: return # steady state → 0 model calls if len(delta) >= MASS_OUTAGE_THRESHOLD: notify(terse_outage_alert(delta)) return # flood → LLM skipped # rare + local: only genuine, bounded news event = enrich(delta, zabbix_metrics) # real numbers alert = llm_reason(event) # runbook-grounded notify(alert)
The poll loop is local and free. The LLM is only ever asked to reason about a small, deduped, enriched set of genuinely new events — and it answers with a concrete fix.
The scarcity that keeps the model idle is also the scarcity of what the system sees. The LLM never watches raw metrics or the full monitoring state — it only ever reasons over the small diff the pollers already flagged as new. If a problem doesn't surface as a change in Zabbix or Prometheus, it never reaches the model at all.
When the model does not answer, the alert has already gone out and the follow-up says the investigation failed. That distinction is load-bearing: a model outage and an uneventful night must not look the same to the operator, and under an ordering where the notification waits for the model, both are silence.
Which is also why rung 3 of the roadmap — acting, not just recommending — stays a human-in-the-loop design on purpose. A model that reasons well over a diff is not the model you hand the keys to production infrastructure, and this repo doesn't pretend otherwise.
Grounded in sentinel/analyzer.py, sentinel/cli.py and sentinel/watchdog.py — no-fallback retry semantics and diff-only snapshots, as shipped.