aykroyd

At Denver Mobile App Developer, we manage a fleet of Kubernetes clusters across three cloud providers. Over the past two years, our infrastructure costs grew 22% faster than user traffic. The culprit wasn't a spike in legitimate workload-it was ghost resources: zombie processes, orphaned containers, leaked load balancers, stale DNS records, and unattached persistent volumes that nobody remembered provisioning. These artifacts don't show up on standard dashboards because they rarely fail. They simply exist, quietly consuming CPU, memory, IP space, and budget. After months of manual audits and scripted cleanups, we built a tool that automates the entire lifecycle. We named it aykroyd, after the actor who made ghostbusting famous. This article explains what aykroyd does, how it works under the hood. And what we learned running it in production.

Most engineers are familiar with zombie processes on a single Linux host-processes that have exited but remain in the process table because their parent hasn't called wait(). In distributed systems, the same logical problem appears at every layer. A Kubernetes Job finishes. But the controller leaves behind completed pods because an owner reference was stripped by a faulty mutating webhook. An AWS Elastic Load Balancer remains provisioned after the service that fronted it was deleted. A Cloudflare DNS record points to an IP that no longer exists. None of these issues generate alerts; they just accumulate. Aykroyd is the open-source ghostbuster your infrastructure has been waiting for-it finds and eliminates zombie processes - orphaned containers. And leaked cloud resources before they drain your budget. In the sections that follow, I'll walk through the design, deployment, and operational lessons from our production rollout.

The Ghost Workload Problem in Modern Infrastructure

Ghost workloads are resources that persist beyond their intended lifecycle. In Linux, a zombie process is a well-defined state where the process has terminated but its entry remains because the parent hasn't reaped it. In Kubernetes, the analog is a pod in Completed or Failed state that should have been garbage collected but wasn't-often because a custom controller removed the ownerReferences field or a Job's ttlSecondsAfterFinished was never set. In cloud environments, ghost resources include unattached elastic IPs, orphaned NAT gateways. And security groups referencing deleted instances. In one AWS account audit, we found 14 orphaned NAT gateways costing roughly $32 each per month-$448 monthly for zero traffic.

The financial impact is only part of the story. Ghost resources also create security and compliance risk. An unused IAM role with broad permissions and no attached workload is a credential waiting to be abused. A stale DNS record can be hijacked for phishing or subdomain takeover. And from an SRE perspective, every ghost resource adds noise to capacity planning and inventory systems. Traditional monitoring tools fail here because they focus on behavior-error rates, latency, saturation-not lifecycle. A zombie process consumes almost no CPU. So it doesn't trip a threshold. An orphaned load balancer has no traffic. So it exits the dashboard's view. Detecting these requires a different signal: the delta between what should exist and what actually exists.

Why Traditional Monitoring Tools Miss Ghost Workloads

Prometheus, Grafana, Datadog,? And New Relic are excellent at answering "Is it running correctly? " they're poor at answering "Should it be running at all? " APM tools sample requests and traces. But a ghost resource doesn't receive traffic. Infrastructure monitoring checks CPU, memory - and disk, but an orphaned load balancer has no metrics to scrape. Kubernetes tools like kubectl top show resource usage. But a completed pod with no running containers shows zero for everything. The result is a blind spot that grows silently.

We experimented with several approaches before building aykroyd. First, we wrote cron jobs that queried cloud APIs for unattached resources and empty Kubernetes namespaces. This caught obvious leaks but required constant maintenance and produced false positives. Second, we tried extending Prometheus with custom exporters that reconciled desired state from Terraform. That worked but created a dependency on Terraform state files. Which were often stale or split across teams. We needed a purpose-built system that could correlate control-plane data across Kubernetes, cloud APIs, DNS providers. And process-level signals-without requiring every team to adopt a new workflow. That system became aykroyd.

Enter Aykroyd: A Lifecycle-Aware Detection Engine

Aykroyd is a distributed detection and remediation engine built around a simple principle: every resource must have a declarative owner and a defined terminal state. At its core, aykroyd runs as a lightweight daemon on each node and a Central controller in the cluster. The node agent uses eBPF to monitor process trees and network namespace lifecycle, while the controller reconciles Kubernetes objects, cloud provider inventories. And DNS records. When a resource's owner is missing or its lifecycle state is inconsistent, aykroyd flags it as a candidate ghost.

The detection pipeline has three stages. First, inventory collection pulls data from Kubernetes API (pods, services, persistent volumes, ingresses), cloud APIs (EC2, ELB, EBS, IAM, Route53, Cloudflare). And node-level process listings via eBPF. Second, correlation joins these datasets using identifiers like pod UID - instance ID, security group ID. And ARN. Third, policy evaluation applies rules written in Rego to determine if a resource is orphaned. For example, a rule might say: "An AWS security group is orphaned if no EC2 instance, ENI, or Lambda function references it. And it has been unused for more than 72 hours. " We chose Rego because it allows teams to write readable, testable policies without modifying the core engine-a pattern we borrowed from Open Policy Agent.

One unique design choice: aykroyd doesn't rely on Kubernetes garbage collection as a primary mechanism. The native kubelet garbage collector handles containers and images. But it doesn't understand cloud resources or DNS. Aykroyd operates above that layer, treating Kubernetes as just one control plane among many. This lets it catch cross-plane leaks-for example, an AWS load balancer that remains after the corresponding Kubernetes Service was deleted before the cloud controller could reconcile.

How Aykroyd Correlates Signals Across Control Planes

The real power of aykroyd comes from its correlation engine. In production, we found that a single ghost often leaves traces in multiple systems. A deleted Deployment might leave behind a pod that's still running on a node; that pod might hold an AWS security group; that security group might be referenced by an orphaned network interface. Traditional tools would see three separate anomalies. But aykroyd joins them into one causal chain. We implemented a graph-based correlation model using an embedded Neo4j-like structure in memory (we use a simple adjacency list in Rust for performance, not a full graph database). Each resource is a node; each reference is an edge. When a node's inbound edges drop to zero and its age exceeds a policy threshold, it's flagged.

Here's a concrete example from our staging environment. A developer created a Kubernetes Service of type LoadBalancer. The AWS cloud controller provisioned an ELB and a security group. The developer then deleted the Service via kubectl delete. But a race condition in the cloud controller caused the ELB deletion to fail silently. The ELB remained, but its associated security group was orphaned immediately. Aykroyd's correlation engine detected the ELB with no matching Service, traced its security group, and found that the security group had no remaining instances attached. Within one hour, it opened a remediation ticket (in dry-run mode) showing the full chain and estimated monthly cost. Without aykroyd, that ELB might have persisted for months.

For process-level ghosts, aykroyd uses eBPF to attach to the sched_process_exit tracepoint and track parent-child relationships. When a process exits but its parent fails to reap it, the node agent records the zombie and the parent's PID. It then correlates that parent with a Kubernetes pod using cgroup IDs. This lets aykroyd answer not just "there is a zombie process" but "this zombie belongs to pod checkout-service-7f9c8b owned by Deployment checkout-service, and the parent is a Node js process that leaked children after a SIGTERM handler bug. " That level of attribution is what enables safe automated remediation,