Infra-Sentinel polls Zabbix and Prometheus with cheap, constant checks — and only when something is genuinely new does a self-hosted LLM reason about it. v1 tells an operator what to look at. v2 hands them the exact command. The road ahead: an agent that acts.
Forward every raw trigger and you drown the operator. Pipe everything to a cloud LLM and you pay per token to reason about noise. Infra-Sentinel splits the job: local polling does detection, a local model does reasoning — only on the delta.
Every flap, every port, every recovery becomes a page. Operators mute the channel — and miss the one that mattered.
Reasoning quality, but you pay per token to summarize noise, leak infra telemetry off-site, and depend on someone else's uptime.
Constant free polling finds the delta. A self-hosted LLM reasons only on real news — then tells you exactly what to do. Nothing leaves the network.
Two monitoring sources are diffed against persisted state; the delta runs a gauntlet of guardrails; only survivors reach the LLM and the operator. Pick a scenario and watch the path light up.
The value isn't the plumbing — it's the pure, unit-tested functions that keep the watchdog useful instead of noisy, wasteful or silent.
The first cycle records everything already broken as the baseline and raises nothing. You're never paged for problems that predate the watchdog.
objectid + cooldownA flapping trigger emits a new event id every cycle. Sentinel keys "already alerted?" on the stable trigger id and suppresses re-alerts within DEDUP_COOLDOWN_SEC. One flapping port = one alert, not one per cycle.
A flood of new problems in one cycle (MASS_OUTAGE_THRESHOLD, default 8) is the signature of a core/power failure. Sentinel short-circuits to one terse alert — no per-event LLM calls, no pager storm.
If the self-hosted LLM is unreachable, Sentinel does not degrade to a raw alert. It raises, aborts the cycle, and leaves state unadvanced — the same event is retried next cycle. An alert is either LLM-reasoned or it doesn't go out at all.
v1 is in production: it turns raw triggers into one impact-aware alert. v2 is the pilot leap — a model with the infra runbook baked in, reasoning over the trigger's real metrics (not guesses), that returns the concrete fix.
Three rungs from a smart watchdog to an agent that works the infrastructure on its own. Two are shipped or in pilot. The third is the honest, careful part.
Detect the delta cheaply and constantly; turn raw triggers into one impact-aware alert. Guardrails keep it quiet and safe.
A runbook-grounded model reasons over real metrics and returns the exact command, where to run it, and when to escalate. Human executes.
Execute safe, idempotent remediations from a closed catalog, approved from chat, then verify the problem is gone before closing. From advisor to operator.
# cheap + constant: runs every cycle, no GPU delta = diff(current, baseline) delta = dedup_with_cooldown(delta, state) if not delta: return # steady state → 0 GPU calls if len(delta) >= MASS_OUTAGE_THRESHOLD: notify(terse_outage_alert(delta)) return # flood → LLM skipped # rare + local: only genuine, bounded news event = enrich(delta, zabbix_metrics) # real numbers alert = llm_reason(event) # runbook-grounded notify(alert)
The poll loop is local and free. The LLM is only ever asked to reason about a small, deduped, enriched set of genuinely new events — and it answers with a concrete fix.
The scarcity that keeps the GPU cold is also the scarcity of what the system sees. The LLM never watches raw metrics or the full monitoring state — it only ever reasons over the small diff the pollers already flagged as new. If a problem doesn't surface as a change in Zabbix or Prometheus, it never reaches the model at all. Detection is the ceiling on reasoning, not the other way around.
And when the model doesn't answer, Sentinel doesn't degrade — it goes quiet. There is deliberately no fallback: if the self-hosted LLM is unreachable, the cycle raises, aborts, and leaves state unadvanced, so the same event is retried next cycle instead of shipped un-reasoned. That is the right call for correctness — no confidently wrong alert ever goes out — but it means a GPU outage and an uneventful night look identical to the operator: both are silence.
Which is also why rung 3 of the roadmap — acting, not just recommending — stays a human-in-the-loop design on purpose. A model that reasons well over a diff is not the model you hand the keys to production infrastructure, and this repo doesn't pretend otherwise.
Grounded in sentinel/analyzer.py, sentinel/cli.py and sentinel/watchdog.py — no-fallback retry semantics and diff-only snapshots, as shipped.