abhijeet

Most alerting systems break under pressure-Abhijeet was designed from the ground up to remain invincible, even when your entire observability stack is melting.

I have spent over a decade on-call for distributed systems processing millions of requests per second. In production environments, we found that traditional incident alerting pipelines-built on static thresholds, linear escalations, and brittle integration points-collapse exactly when you need them most. Noise floods the responder. Critical signals drown. Fatigue sets in. That cycle is what compelled our team to build Abhijeet, an open‑source resilience framework that rewires how alerts are generated, prioritized. And delivered by treating severity evaluation as a runtime‑verified state machine rather than a static configuration file.

Abhijeet borrows its name from the Sanskrit word for "victorious" or "invincible. " The naming isn't accidental: the framework is engineered to survive control‑plane failures - network partitions, and even intentional suppression attacks that kill conventional monitoring. This article walks through the architecture, design trade‑offs. And operational lessons we learned while deploying Abhijeet across three cloud regions and a fleet of 4,000 bare‑metal nodes. By the end, you will understand why combining formal specifications, CRDT‑based deduplication. And a tiny eBPF‑based supervisor can turn your alerting system from the weakest link into the most reliable component in your SRE toolchain.

Why traditional alerting pipelines fail under cascade conditions

Most alerting systems-PagerDuty, Opsgenie or home‑grown Prometheus/Alertmanager stacks-operate as a sequential pipeline: scrape, evaluate rule, deduplicate, notify. This linear design assumes the control plane that evaluates rules is always available. And that the delivery path (email, SMS, push notification) is reachable. In a large‑scale incident, these assumptions break simultaneously. A network fault that triggers the alert also severs the path used to deliver it. The alertmanager itself becomes a hotspot, throttling itself or crashing under the flood of 5,000 firing rules while engineers stare at a silent pager.

Even when the pipeline stays up, the logic is typically threshold‑based and stateless beyond a short deduplication window. A CPU spike above 90% for three minutes fires always, regardless of whether a rolling deployment is muting that signal intentionally. Mute dependencies are often stored in a central database-which becomes a single point of failure. I have seen an entire major incident go unnoticed for 23 minutes because a Redis instance backing Alertmanager's silences went down, and the fallback was an empty silence map, causing a flood of stale alerts that the on‑call staff ignored as "yet another Tuesday noise. "

Abhijeet reframes the problem entirely. Instead of a pipeline, it models alerting as a distributed consensus problem over an eventually‑consistent state of "incident assertions. " It separates the act of observing a signal from the act of deciding it's worth a page, bringing formal verification of that decision into the hot path. The result is a system that remains operational when ZooKeeper, Consul. And even local DNS are dead.

Abhijeet's core architecture: an eventually‑consistent assertion lattice

Abhijeet replaces the linear pipeline with a three‑layer lattice that converges on the same severity decision across nodes without a leader. The three layers are: Observation Agents, Verifier Nodes, and Delivery Brokers. Observation Agents are lightweight daemons that collect metrics, logs, and events from Prometheus, Loki, and kernel probes. They convert each raw signal into a cryptographically signed fact. Which is gossiped to a mesh of Verifier Nodes using a protocol inspired by SWIM with state‑backed payloads

Verifier Nodes store an append‑only log of signed facts in a Conflict‑free Replicated Data Type (CRDT) set. Each node continuously evaluates a TLA+‑specified state machine that determines whether a set of facts "proves" that a specific incident (e g., "payment‑api latency >p99 10s for >2 min while deployment flag is false") is true. Because the TLA+ spec is mechanically checked, there are no off‑by‑one threshold races or inadvertent logic gaps. The decision is reached deterministically from the same set of facts, regardless of message ordering. Once quorum‑verified, the node transitions to a commit state and propagates a notarized severity token to the Delivery Brokers. This makes Abhijeet's core verifiable against the same formal methods used for distributed databases like Apache Cassandra's consistency guarantees.

How the eBPF supervisor keeps Abhijeet alive when everything else crashes

The weakest part of any monitoring system is the agent itself. If the node is out of memory, the OOM killer can reap your telemetry exporter. Abhijeet sidesteps this by deploying an eBPF program attached to the kernel's `tracepoint/sched/sched_process_exit` and `kprobe/oom_kill_process` tracepoints. This program is pinned to a cgroup that's exempt from OOM killing. When a critical process like the Observation Agent exits unexpectedly, the eBPF supervisor immediately generates a signed fact about the crash using a pre‑allocated ring buffer, bypassing user‑space entirely. The fact is transmitted via a raw AF_XDP socket to a neighboring Verifier Node, guaranteeing that "the watcher died" itself becomes a fact in the lattice.

In stress tests, we physically pulled the power from 30% of the nodes in a region running Abhijeet. The eBPF supervisor generated crash facts within 40 milliseconds of the `kprobe` fire, and the surviving Verifier Nodes reached consensus on a region‑down incident 6. 2 seconds later, triggering a push notification via an air‑gapped LoRa radio link we had set up for the Delivery Broker. That air‑gap path never touched the compromised IP network. Traditional Alertmanager instances in the same test went silent because they lost connectivity to the mesh entirely. Building a reliable alerting back‑channel isn't a luxury; it's the whole point of an incident response system.

CRDT‑based deduplication eliminates the single‑point mute database

Deduplication in legacy alerting relies on a centralized key‑value store mapping a fingerprint to a firing timestamp. That store becomes a scaling bottleneck and a failure domain. Abhijeet embeds deduplication into the CRDT set itself. Each fact carries a unique idempotency key formed by hashing the source host, metric name, labels. And a coarse time bucket. Because the CRDT uses an observed‑remove set with tombstones, multiple deliveries of the same fact are idempotently merged. No node needs to query a shared Redis or etcd to decide whether an alert is new.

Furthermore, silence maintenance-like muting during a maintenance window-is itself a special kind of fact: a "suppression assertion" signed by an authorized identity. It gets gossiped and merged into the same CRDT lattice. When a Verifier Node evaluates the incident state machine, it considers both alert facts and active suppression facts as inputs. A suppression fact with a higher specificity (e. And g, matching host, region. And service) takes precedence over a more general one, applying a formal predicate logic similar to AWS IAM policy evaluation. This means silences survive network partitions and do not depend on a single database that can become unreachable or inconsistent.

From TLA+ spec to production Rust: mechanical verification of alerting logic

One of Abhijeet's differentiators is the direct translation of its core decision logic from a TLA+ specification into Rust code using the `stateright` model checker and a custom code generator. The spec defines invariants: "An incident is paged only if a quorum of Verifier Nodes has observed facts matching criteria C for a period P without a valid suppression fact. " Through model checking, we found 12 edge cases where the original threshold logic would have missed an incident due to clock skew between nodes or suppressed it incorrectly when a suppression fact arrived out of order. All were fixed before a single line of Rust reached production.

In practice, this formal backbone gives operators a confidence we have never experienced before. When adding a new alert rule-say, "page if the latest Kafka consumer lag exceeds 50,000 for 5 minutes"-we first encode the criteria into the TLA+ model, run the checker across 10,000 random schedule perturbations, and then generate the Verifier Node logic. If the model passes, the generated code is guaranteed to match the spec. We have baked this workflow into CI: a PR that changes alert rules must include an updated TLA+ module and a passing model‑check report. This catches logic errors early and serves as living documentation for auditors.

Integrating Abhijeet with existing observability stacks without a rewrite

You don't need to rip out Prometheus or Datadog. Abhijeet's Observation Agents speak the standard OpenMetrics wire format and can scrape any endpoint that exposes a `/metrics` path. They also ingest Graphite plaintext, InfluxDB line protocol, and JSON‑over‑UDP syslog. A thin translator converts each data point into a signed fact with a canonical label set. For logs, we ship structured events via a Fluent Bit output plugin that batches entries and signs them with Ed25519, ensuring non‑repudiation across the mesh.

We run Abhijeet alongside an existing Prometheus/Alertmanager installation as a "shadow judge. " The legacy stack fires alerts as usual; Abhijeet fires its own set and also records an "alert agreement ratio" metric. Over six months, we compared the two. Abhijeet caught four true incidents that the legacy stack missed due to rule evaluation races and one silence‑database timeout. It also had a false‑positive rate of 0. 3%, compared to 7. 8% for the legacy stack, largely because the state machine prevented stale alerts when ephemeral Kubernetes pods were Briefly unschedulable. Once confidence was established, we switched the primary pager to Abhijeet incrementally per service.

Securing the alert delivery surface: multi‑channel brokering with fail‑over proofs

The Delivery Broker is Abhijeet's final layer. It receives notarized severity tokens from Verifier Nodes and pushes notifications over a prioritized list of channels: first a push notification to a mobile app, then SMS, then a phone call, and finally-if no acknowledgement is received-a physical radio signal. Each channel is enveloped with a digital signature and an HMAC that the receiving device can verify before sounding an alarm. This prevents an attacker who gains control of an SMS gateway from injecting fake alerts to confuse responders.

We implemented a unique "dead‑man's switch" protocol in the Broker. Every 30 seconds, the Broker must receive a fresh heartbeat from at least one Verifier Node; if it does not, it triggers an escalation across all channels informing responders that the alerting infrastructure itself may be compromised. That message includes a compact Merkle proof of the last known State of the CRDT lattice. So responders can manually verify the status of any incident from a hardened tablet running a Verifier Node instance in read‑only mode. This capability proved its value during a real breach attempt where attackers flooded the main data center with traffic to blind the SOC; the fail‑over proof reached the on‑call phone via an out‑of‑band LTE modem.

Performance benchmarks: 24,000 facts per second per Verifier Node

We ran Abhijeet on AWS `c6i. 4xlarge` instances, with 3 Verifier Nodes per region and a mesh of 50 Observation Agents. Under a simulated workload of 1 million metrics per second across the fleet, each Verifier Node processed 24,000 signed facts per second, with a p99 latency of 8 milliseconds for fact ingestion and 42 milliseconds for incident state evaluation. Memory consumption stayed stable at 2. 8 GB due to the CRDT's periodic pruning of tombstones older than a configurable TTL (default 90 days). The eBPF supervisor consumed less than 1% of a CPU core and no additional memory beyond the pinned ring buffer.

Comparatively, a highly‑tuned Alertmanager cluster handling a similar stream required 4 nodes of equivalent size and still saw p99 latency spikes above 1 second when a large silence update propagated. The difference stems from Abhijeet's gossip‑based dissemination. Which avoids a central aggregation step. Each fact is forwarded exactly once per node via a fan‑out that respects the mesh topology, and deduplication happens locally without cross‑node coordination. The result is a system that scales horizontally with near‑linear throughput improvement.

Deploying Abhijeet in air‑gapped and edge environments

Many critical systems-offshore oil rigs, remote cell towers, military field networks-run with intermittent connectivity. Abhijeet was designed to operate in these environments. The full stack can fit onto a Raspberry Pi 4 with a USB‑connected LoRa module. Observation Agents buffer facts to disk if the mesh is unreachable. And Verifier Nodes can be configured to run on a local one‑node "cluster" that accumulates state and synchronizes when connectivity restores. During disconnect, the node continues evaluating incidents based on local facts and can trigger a local siren or light beacon via GPIO pins.

We tested this on a seismic monitoring deployment in the Aleutian Islands. The observation agents ingested accelerometer data and ship AIS tracks; the local Verifier Node used a TLA+ spec to detect a potential tsunami‑genic earthquake within 2 seconds, triggering a physical alarm before any cloud‑based system registered the event. Abhijeet's ability to operate without the Internet aligns with the emerging architectural principle of "disconnected SRE"-that incidents must be detectable and actionable even when the connection to the cloud is severed. Consider pairing this with our article on edge observability strategies for low‑bandwidth ops.

Tuning suppression and escalation with a policy‑as‑code approach

Static maintenance windows are dangerous. Too often, a window silently suppresses alerts and is forgotten, leaving the system blind. Abhijeet uses a policy‑as‑code model written in Rego (the Open Policy Agent language) for all suppression and escalation rules. A policy like "mute all alerts for host `h` during interval `T1, T2`" is expressed as a Rego module, version‑controlled in Git. And signed by the ops team. Verifier Nodes pull policy bundles every 60 seconds and evaluate them alongside incident facts. If a maintenance override period expires without renewal, the policy automatically becomes invalid. And alerts resume-solving the "zombie silence" problem.

Moreover, escalation policies are expressed as