rafiki saïd

When an SRE's pager buzzes at 3 a m, and, it rarely carries wisdom-just raw noiseBut what if you had a companion that, like the wise mandrill Rafiki, could remind you: "Ah, yes… the past can hurt. But the way I see it, you can either run from it or learn from it. " That's the promise of rafiki saïd, an AI-driven observability agent we built to transform chaotic incident streams into actionable narratives. Learn how to build a self-healing alerting system that channels the wisdom of Rafiki-Saïd to slash MTTR and turn incident chaos into a learning loop.

The idea came during a postmortem for a cascading failure that took down our payment pipeline for 47 minutes. We had all the telemetry-metrics, traces, logs-but the alert storm overwhelmed on‑call engineers. The real root cause was hidden in plain sight: a database connection pool exhaustion that three previous incidents had foreshadowed, just dressed in slightly different clothes. Our retrospective boiled down to what rafiki saïd in that Disney classic: the past was trying to teach us. But we were running from it. That moment seeded the design for an intelligent agent that wouldn't just fire alerts; it would correlate, narrate. And suggest mitigations grounded in incident history.

This article deconstructs our production-grade implementation of rafiki saïd-a system that combines large language models, OpenTelemetry and chaos engineering to create an observability assistant that talks like a staff engineer who has read every postmortem your team has ever written. We'll walk through the architecture, the data pipelines, the correlation engine. And the metrics that prove this isn't just a toy. Wherever possible, I'll reference real specifications, like the W3C Trace Context and RFC 5424, so you can reproduce the stack in your own environment.

1. Decoding the Legacy: Why Rafiki Saïd Matters for Site Reliability Engineering

Most alerting systems are glorified threshold monitors. When CPU exceeds 85%, they scream via PagerDuty. But a screaming mandrill is still just noise. rafiki saïd takes a different posture: it treats every alert as a fragment of a story that has happened before, perhaps in a slightly different service. By ingesting years of incident retrospectives, runbook executions, and chat transcripts, the agent builds a semantic map of failure modes. That map lets it say, "This latch contention on the payment-db connection pool mirrors incident #1423 from last October-here's the runbook that resolved it in 4 minutes. "

In our production cluster, the difference was measurable. Before deploying the rafiki saïd agent, the mean time to acknowledge (MTTA) for non-trivial alerts was 9. 2 minutes. After the agent started enriching alerts with contextual "Rafiki narratives" and a confidence score, MTTA dropped to 1. 8 minutes. Engineers stopped chasing phantom CPU spikes from batch jobs and started trusting the system to surface the signal. The underlying technology isn't magic; it's a well-orchestrated pipeline of event streaming, vector embeddings. And a chat interface that acts as a force multiplier for every on-call rotation.

But why tie such a critical piece of infrastructure to a cartoon mandrill, and because cultural resonance accelerates adoptionWhen a new engineer asks, "What would Rafiki say about this. And " they're invoking the agent's knowledge baseWe found that naming the platform rafiki saïd lowered the psychological barrier to trusting an automated suggestion. It's the same pattern that gave us Chaos Monkey; anthropomorphism builds engagement. And when the agent occasionally hallucinates a useless runbook, the team laughs and corrects it-making the system smarter for the next incident.

2. Engineering a Rafiki-Saïd Agent with Large Language Models

At the heart of rafiki saïd is an LLM‑powered decision engine, but we deliberately avoided a monolithic "ask‑ChatGPT" wrapper. Instead, we implemented a retrieval-augmented generation (RAG) pipeline that constrains the model's output to verified internal sources. The agent uses LangChain to orchestrate a sequence of actions: classify alert type (infrastructure, application, security), retrieve similar past incidents from a vector store, fetch relevant runbooks from a Git repository, then compose a structured advisory message. The LLM we chose for initial deployment was Llama 3 8B running on an on‑prem GPU cluster to keep latency under 300 ms.

One critical design decision was to make the agent fail-safe and transparent. Before any suggestion is sent to Slack or the incident channel, the rafiki saïd engine attaches a citation trail-links to the exact runbook lines, past incident tickets. And the bits of telemetry it used. That trail not only builds trust but also provides an audit log for compliance. If the model suggests restarting a service, the citation shows the last three times that action resolved a similar fingerprint, with timestamps and mean time to recovery. Production users quickly learned to glance at the confidence badge: green (correlation > 0, and 85), yellow (0, and 5‑085), red (guess)