psg

What if you could reimagine workload orchestration not as a series of binary placement decisions,? But as a continuous negotiation between policies, resource telemetry,? And business intent? That's the problem we set out to solve when we built PSG-the Policy-driven Service Grid-a decentralized scheduling and orchestration layer designed for modern, heterogeneous infrastructure. Over the past three years, our team has moved thousands of production services away from static scheduler configurations and toward a model where placement, scaling, and fault recovery are all governed by real-time constraint solving.

The standard Kubernetes scheduler works well for stateless workloads with uniform node profiles. But once you introduce GPU‑accelerated inference pods sharing the same cluster as latency‑critical REST APIs and batch ETL jobs, the default scoring algorithms start making expensive mistakes. We'd see GPU nodes sit idle while CPU‑bound jobs were packed onto memory‑starved instances. And cluster autoscaling would lag behind demand spikes by minutes. PSG was born from that frustration. And it's now running in three of our largest production regions, handling over 400,000 scheduling decisions per minute with a sub‑second policy evaluation latency.

In this article, I'll walk you through the architecture of PSG, explain how its policy engine differs from existing schedulers, share real‑world benchmark data and discuss the operational patterns that make it a viable replacement for-or companion to-Kubernetes' native scheduler. If you've ever traced a production outage back to a resource contention problem that your orchestrator should have prevented, you'll find the concrete patterns here immediately useful.

Why Traditional Schedulers Fall Short in 2025

Modern infrastructure looks nothing like the homogeneous pool of VMs that the original Kubernetes scheduler was designed for. Today's clusters blend x86, ARM64, and specialized accelerator nodes; persistent storage backends with wildly different performance profiles; and workloads that can change their resource requirements mid‑execution. Traditional schedulers like kube‑scheduler operate on a filter‑score‑bind loop that evaluates a static set of predicates and priorities once per pod. That model lacks visibility into future resource claims and can't express complex business constraints such as "keep this deployment's pods in at least three failure domains but never colocate with pods from tenant X. "

Apache Airflow and temporal schedulers solve workflow orchestration, but they're not designed to manage real‑time placement on shared infrastructure. HashiCorp Nomad offers bin‑packing strategies and multiple scheduler types. Yet its policy layer is still imperative: you define job specs and constraints. And Nomad performs a single‑pass optimization. In production, we found that these point‑in‑time decisions lead to cascading resource fragmentation, particularly when workloads scale rapidly and the scheduler's view of node capacity is delayed by polling intervals.

PSG replaces the single‑pass filter‑score model with a continuous feedback loop. Instead of waiting for a pod to be created, PSG agents on each node stream real‑time utilization telemetry to a central policy evaluator. That evaluator solves a constraint satisfaction problem every 500 milliseconds, producing not just placement decisions but also proactive rebalancing orders and pre‑warm signals for the cluster autoscaler. This shift from event‑driven scheduling to state‑driven orchestration is the conceptual heart of PSG.

Defining the PSG Architecture: Core Components

At a high level, PSG consists of four components: the PSG Controller, a fleet of Node Agents, a policy Engine backed by Open Policy Agent (OPA). And a Telemetry Bus built on NATS JetStream. The controller itself is stateless; it subscribes to the Telemetry Bus for node reports and workload submissions, then invokes the Policy Engine to compute a placement or rebalancing plan. All persistent state-scheduling queues, execution history. And policy definitions-lives in a NATS key‑value store with an in‑memory cache for fast access.

The PSG Controller communicates with existing orchestrators like Kubernetes through a standard scheduler extender interface, described in the Kubernetes Scheduler Configuration documentation. When kube‑scheduler fails to find a suitable node after the default filter phase, it calls PSG as a reserve extender. PSG then runs a multi‑dimensional bin‑packing algorithm that considers current and projected resource usage, affinity rules. And compliance policies. If a viable node still isn't found, PSG can trigger a targeted scale‑up request to the cluster autoscaler before the pod's scheduling deadline expires.

Through this design, PSG never replaces the Kubernetes scheduler; it augments it. This allows teams to adopt PSG incrementally, starting with a single namespace and gradually expanding to cover critical workloads. We've run side‑by‑side comparisons where the native scheduler handled 80% of placements and PSG stepped in for the remaining 20%-the ones involving GPU co‑scheduling, anti‑affinity across availability zones. Or memory‑overcommit scenarios,

Policy Engine: From Static Rules to Dynamic Constraints

The Policy Engine is where PSG truly differentiates itself. Instead of hard‑coded predicate functions, every scheduling decision is governed by a set of Rego policies evaluated by OPA. A policy can inspect the full cluster state, historical usage patterns from a Prometheus datasource. And even external APIs-such as a CMDB to determine maintenance windows-before returning an allow/deny decision along with priority scores. We deliberately chose OPA because of its widespread adoption in Kubernetes admission control, which means platform teams already have Rego skills and tooling.

Here's a concrete example from our production environment. A Rego policy prevents GPU‑based model training jobs from being scheduled on nodes where three or more inference pods are already running, unless the training job carries a preemptible label. If the policy detects that the inference pods are part of a deployment marked tier: gold, it further restricts colocation to one per node. This rule simultaneously protects latency‑sensitive workloads and improves GPU utilization-something we could never achieve with node selectors or taints alone. For a deeper look at the policy language, the OPA project's official documentation details how to write such context‑aware rules.

The true power emerges when policies reference the Telemetry Bus. A policy can query a sliding window of CPU steal time, network packet loss, or NVMe throughput degradation, and automatically demote a node's scheduling weight without human intervention. In our testbed, this self‑healing behavior reduced the mean time to recovery from noisy‑neighbor incidents by 11x compared to relying on manual cordon operations.

Node Agents and Resource Telemetry Pipelines

Each node in a PSG‑managed cluster runs a lightweight agent written in Rust-a deliberate choice for minimal memory footprint and zero‑GC pauses. The agent collects standard cgroup metrics via cAdvisor, GPU utilization through NVIDIA's DCGM, and custom hardware counters from IPMI and NVMe‑MI interfaces. This data is batched into Protobuf messages and published to the Telemetry Bus at a configurable interval (typically 200 milliseconds).

The Telemetry Bus itself is a clustered NATS JetStream setup, giving us at‑least‑once delivery semantics and replay capabilities. We discard raw metrics after five minutes, but aggregate histograms are persisted to a ClickHouse database for long‑term analysis. This dual‑store architecture means PSG's policy engine always queries fresh data from NATS. While our SRE team can run historical utilization queries without impacting the critical scheduling path. The agent also exposes an HTTP/2 endpoint for the PSG Controller to push rebalancing commands-migrating a pod to another node without involving the cluster orchestrator directly.

One non‑obvious lesson we learned: the agent must be able to throttle its own telemetry during node‑level contention. We implemented a back‑pressure mechanism where if the node's CPU load exceeds 95%, the agent reduces its collection frequency and emits a "degraded" status to the bus. This prevents the monitoring system from exacerbating resource exhaustion, a failure mode we observed when testing early versions under extreme stress.

How PSG Handles Multi-Tenant Scheduling Without Starvation

Multi‑tenancy is often implemented with ResourceQuotas and LimitRanges in Kubernetes, but those are static allocation policies. They don't adapt to actual consumption; a tenant that requests 80% of the cluster's resources but uses only 20% can still block other tenants if no oversubscription is allowed. PSG treats fairness as a dynamic property by modeling each tenant as a weighted fair queue (WFQ) participant. The WFQ implementation follows the principles laid out in the Linux kernel's Completely Fair Scheduler (CFS), documented in the Linux kernel scheduling design docs.

When a new workload enters the scheduling queue, PSG calculates the tenant's normalized resource usage-CPU, memory, GPU. And IOPS-over the last 30 seconds and compares it against the tenant's fair‑share entitlement. If a tenant has been below its allocation, its queue weight increases, allowing its pods to preempt lower‑priority, over‑consuming tenants even if the absolute priority is lower. This has eliminated the "noisy tenant" problem where a single batch job starves interactive services.

We also introduced a concept called "resource tokens," which are minted based on historical consumption and can be traded between tenants through a simple REST API. A tenant expecting a spike can request tokens from another tenant that's under‑committed, with audit trails stored immutably in NATS. While this mechanism might sound like over‑engineering, in practice it replaced weekly capacity negotiation meetings with a self‑service system that keeps overall cluster CPU utilization above 65% while maintaining p99 latency SLOs.

Integrating PSG with Existing Orchestration Frameworks

PSG isn't a standalone orchestrator; it's designed to sit alongside Kubernetes, Nomad. Or even HashiCorp Nomad's scheduler. We provide a lightweight Kubernetes operator that registers PSG as a scheduler extender using the schedulerName field in Pod specs. For Nomad, we use a task driver plugin that intercepts placement decisions and forwards them to the PSG Controller. The integration surface is deliberately narrow, consisting of a gRPC service with four methods: PlaceWorkload, RebalanceNode, Preempt, ReserveResources.

This modularity allowed us to migrate our legacy Nomad clusters to a hybrid setup where Nomad still manages job lifecycles. But PSG decides which node runs each allocation. The migration took less than two weeks of engineering effort, mostly spent on writing a Go shim that translated Nomad's JSON job specs into the PSG workload model. Once that hook was in place, we could apply Re