amy hunt

When a pager goes off at 3 a m., the first thing a senior SRE reaches for isn't coffee - it's a trace. That's exactly how amy hunt - a fictionalized but eerily representative staff engineer - confronted a production outage that nearly took down a payment processing pipeline. Her investigation didn't rely on guesswork. It relied on distributed tracing, log correlation, and a healthy skepticism of "it works on my machine. " This article unpacks the technical decisions, tooling choices. And architectural lessons from a single post-mortem, using amy hunt's methodology as a lens to examine modern observability practice.

We've all been there: a service that hums along in staging suddenly spikes to 2. 3-second p99 latency in production, and nobody can explain why. What separates a two-day blame storm from a 45-minute resolution is the quality of the hunting process. In this case, the hunt led by amy hunt exposed a silent index corruption in a CockroachDB cluster that only manifested under a specific cross-region read pattern. While the name is a construct, the scenario is pulled directly from real-world workloads our team at Denver Mobile App Developer has seen while migrating monoliths to cloud-native architectures. Let's walk through it step by step.

Key takeaway: amy hunt's troubleshooting sequence proves that with the right telemetry, a single engineer can isolate a needle-in-a-haystack database bug in under an hour - but only if the platform is built for it.

The On-Call Trigger: How Amy Hunt's Incident Started

At 03:17 UTC, PagerDuty fired an alert from the checkout service: "HTTP 5xx rate exceeded 2% for 5 minutes. " The on-call rotation happened to land on amy hunt, whose first action wasn't to examine the service itself. She pulled up the Grafana dashboard pre-filtered for the `payment-gateway` namespace and immediately checked the RED metrics - Rate, Errors, Duration. The error rate was climbing linearly, but latency was the real signal. P99 latency had ballooned from 180ms to 2, and 4s, while request rate remained flatThat ruled out a simple traffic spike; something deeper was at play.

Armed with a hunch, amy hunt queried the service's TraceQL endpoint (part of the Grafana Tempo stack) for traces with status code 500 and duration >1s. She didn't search logs first because - as any SRE knows - logs are context-rich but relation-poor. Traces tell you which particular span caused the failure. Within seconds, she spotted a pattern: every slow trace bottomed out in a `SELECT` against the `user_balances` table, inside the `balance-ledger` microservice. The span attribute `db statement` showed no obvious anti-pattern. So she drilled into the correlated logs via Loki, using the trace ID as the bridge.

This immediate pivot from metrics to traces to logs - the classic observability triad - isn't special by itself. What made amy hunt's approach efficient was the pre-existing instrumentation. The codebase used OpenTelemetry auto-instrumentation for Node js services, which meant she didn't have to ask developers to ship a custom build to get database query telemetry. The takeaway: if you're not shipping traces by default, your on-call engineers are debugging blindfolded. For a deeper configuration walkthrough, see our OpenTelemetry auto-instrumentation setup guide.

Distributed Context Propagation: Why Headers Like W3C Traceparent Matter

When amy hunt isolated the culprit span, she noticed something odd: the trace context headers were present. But the parent span in the `api-gateway` service didn't correctly record the downstream call's outcome. The W3C `traceparent` header was being stripped by an Envoy sidecar that had `tracing: {}` misconfigured in its `EnvoyFilter`. This meant that while Tempo still assembled the trace from independently reported spans, the parent span's status code wasn't updated when the downstream service returned a 500. She only caught the failure because she filtered by the leaf span's status, not the root. This subtle gap highlights why context propagation isn't just a "nice to have" - it's the structural integrity of your tracing data.

The fix required updating the Istio `EnvoyFilter` to pass `traceparent` and `tracestate` headers unmodified. amy hunt rolled out the change via a canary deployment, verifying with a custom smoke test that injected a synthetic trace. Within 15 minutes, root spans accurately reflected downstream failures, closing a diagnostic blind spot that had existed for months. This is exactly the kind of platform hygiene that separates teams who can hunt efficiently from those who blame "the network. " RFC 7230 governs header passing, but the real lesson is operational: always validate that your service mesh isn't silently dropping tracing metadata. The OpenTelemetry specification on context propagation invariants provides the authoritative baseline.

Analyzing the CockroachDB Index Anomaly

Once tracing confirmed that the `balance-ledger` database call was the bottleneck, amy hunt pivoted to database-level observability. She had already deployed the CockroachDB Prometheus exporter. So she queried `sql_conn_latency_bucket` aggregated by node. The histogram showed that latency spikes were cluster-wide, but the `crstore-3` node accounted for 70% of slow queries. Logging into that node, she ran `SHOW TRACE FOR SESSION` on a sample slow query - and found something unexpected. The query planner was performing a full-table scan even though an index existed on `(user_id, timestamp)`. The index statistics were stale.

What amy hunt discovered next felt like a bug. But it was a known edge case documented in CockroachDB's cost-based optimizer docsThe automatic statistics refresh had been silently failing due to a disk-space threshold on one of the nodes, causing the optimizer to choose a sequential scan. A manual `CREATE STATISTICS` command immediately corrected the plan, dropping query time from 2, and 3s to 4msThe underlying cause was a misconfigured `--max-sql-memory` flag that starved the stats collection job. This wasn't a code bug - it was infrastructure entropy, exactly the kind of slow-rot problem that traditional APM tools miss.

From an engineering standpoint, amy hunt's database hunt validates a critical practice: treat database statistics as first-class telemetry. Without the Prometheus exporter, she would have been flying blind. But even with metrics, the correlation required understanding how the optimizer uses histograms. This underscores the need for developers to cultivate a "full-stack" mental model that spans from application code down to the storage engine's internals. The DB Console's Statistic refresh timeline should be part of every on-call runbook.

Log Correlation and the Long Tail of Observability

While the index anomaly was the root cause, amy hunt needed to verify that the fix fully resolved user impact. She turned to Loki, not just to spot errors, but to confirm that a particular class of business-logic logs - a "balance consistency warning" - stopped firing. This log line wasn't an error; it was an `info` event emitted when a read on the balance ledger encountered a discrepancy that triggered a compensating transaction. She had previously instrumented this log with a `trace_id` label, allowing her to cross-reference exactly which user sessions had experienced data inconsistency.

The log correlation revealed that the stale index had caused 14,000 warning events over a 23-minute window. But no actual monetary discrepancies - the compensating transactions fired correctly. This transformed a "maybe we lost data" panic into a "we have proof of consistency" relief. The key lesson from amy hunt's log strategy isn't technical complexity - it's discipline. Every log line she emitted carried structured metadata and a trace ID, making it queryable as a dimension in Loki. If your microservices are still spitting out unstructured strings without context propagation, you're leaving your future on-call self with a painful hunting expedition.

Post-Incident Automation: Writing Playbooks That Codify the Hunt

After the incident, amy hunt didn't just file a post-mortem and move on. She created a runbook automation that would have caught the index statistics failure earlier. Using a low-code rule engine (Grafana Alerting with PromQL), she built an alert that triggers when the 99th percentile query latency for the `balance-ledger` service exceeds 500ms and the `sql statistics, and staleness` metric exceeds a thresholdThe alert routes to the on-call engineer with a direct link to a pre-built Tempo trace query filtered for the faulting service.

She also committed a Terraform module that expressed these alerting rules as code, ensuring that any future service could inherit the same safety net. This "alerting as code" approach is something we've advocated for internally - see our Terraform alerting patterns for SRE for concrete examples. amy hunt's automation didn't just shorten the next outage; it codified the very hunting pattern she had executed manually, turning experiential knowledge into platform capability.

Security Implications of the Trace Visibility Gap

There's a security dimension to this hunt that often goes overlooked. The silent Envoy stripping of trace headers wasn't just an observability bug - it could have masked an attacker exfiltrating data through a downstream service. Because the root span didn't record the HTTP 500 from the leaf, any security information and event management (SIEM) relying on trace data would have missed the failure chain. amy hunt realized that trace context integrity is a data integrity concern, not just a performance diagnostic.

In response, she extended the service mesh configuration to log a warning whenever a `traceparent` header mismatch was detected between proxied services. These warnings fed into a Falco rule that would trigger a low-severity security alert for head-of-line observability failures. This closed the gap between SRE and AppSec, leveraging the same telemetry pipeline for dual visibility. For teams running service meshes, I'd recommend reviewing the Envoy tracing architecture overview to understand exactly which headers your mesh is manipulating.

How Amy Hunt's Approach Reshaped Our CI/CD Observability Gates

As a follow-up, our team adopted a pre-production "trace integrity gate" directly in the CI/CD pipeline. Before any PR merges, a synthetic trace is injected into the staging environment and validated end-to-end: the root span must show the exact status code of the leaf span, with no header stripping and no missing spans. amy hunt wrote the initial PyTest plugin that calls the tracing API and asserts on the assembled trace. This kind of shift-left observability is now part of our platform's test harness.

What started as one engineer's 3 a m debugging session has cascaded into a structural improvement that catches tracing gaps long before they hit production. The concept of a "hunt" - a structured forensic investigation - has become a verb on our team. We now say "let's hunt this outage" as shorthand for the systematic triage pattern amy hunt demonstrated. It's a reminder that great incident response is less about heroics and more about reproducible methodology.

Building a Telemetry Stack Worthy of a Hunt

Can every team replicate amy hunt's 45-minute resolution? Not without the right telemetry stack. The backbone of her investigation was the Grafana LGTM stack (Loki, Grafana, Tempo, Mimir). But the specific tool matters less than the integration. Traces, logs. And metrics must share a common identifier space - in her case, the W3C Trace Context. Without it, the pivot from one signal to another is a manual, error-prone process that