trui

The Problem with Black-Box AI Inference in Production

We ship code that touches money, health. And identity every day. Yet the moment a large language model (LLM) enters the dependency chain, determinism vanishes. An innocent update to a system prompt or a minor tweak of a temperature parameter can produce divergent outputs that break downstream parsers, violate compliance rules, or inject hallucinated data into databases. In a recent reliability review for our mobile banking platform, we traced three consecutive false fraud escalations back to an LLM-based classifier that silently started emitting medium confidence values instead of the expected JSON structure-all because the inference provider rolled out a server-side model optimization without notice. None of our existing retry logic, feature flags. Or observability-pipeline alerts caught it for seven hours.

This isn't a rare edge case; it's the default behavior of opaque probabilistic inference. The industry has stacked retries, guardrails. And output validators on top of a fundamentally non-deterministic substrate. Trui takes a different approach. Trui is a transactional reasoner that guarantees deterministic, verifiable outputs from LLM inference by treating a prompt-plus-context as a compiled contract whose execution can be independently replicated and cryptographically attested. Over the next eight sections, I'll walk you through how trui's architecture works, how we integrated it into a production CI/CD pipeline for a mobile feature. And what it means for regulatory attestation of AI-driven decisions.

What Exactly Is Trui? A Transactional Reasoner for Deterministic Outputs

Trui is an open-source runtime and toolchain that wraps arbitrary LLM calls inside a deterministic execution environment. Unlike frameworks such as LangChain or Semantic Kernel-which orchestrate model calls but ultimately leave you at the mercy of stochastic token sampling-trui introduces a compile-and-verify loop. You define your prompt logic in a domain-specific language (DSL) that gets translated into a directed acyclic graph (DAG). Every node in that DAG represents an atomic reasoning step: an LLM invocation with a fixed prompt template, a constrained schema, temperature set to exactly zero. And a pinned model seed. The result is that for a given input context, trui will always produce identical token sequences.

This is more than just freezing parameters. Trui hash-pins the entire inference environment-model binary, runtime library versions, even the hardware architecture via attestation extensions-into a content-addressable manifest. When you call a trui inference, you get back not only the output text but a prover bundle containing the DAG structure, input hashes, intermediate state hashes. And a final Merkle root that can be verified independently by any other trui node. That property makes trui especially attractive for regulated industries where every decision must be explainable and reproducible during an audit.

We first encountered trui through an early-access program while building a HIPAA-compliant symptom triage assistant for a telehealth mobile app. The app needed consistent, medically safe output that could be replayed and challenged. Normal LLM chaining produced dangerous variations: the model would sometimes list medications in alphabetical order and sometimes by severity, breaking the UI layout and confusing clinicians. After adopting trui, we saw zero output drift across 40,000 test cases over two months, even after underlying model weights were patched.

Inside Trui's Architecture: Deterministic DAGs and Merkle Proofs

Trui's execution model borrows heavily from database transaction theory and verifiable computation research. At its core, a trui "program" is a deterministic finite-state machine where transitions are LLM calls whose outputs are validated against JSON Schema definitions before being committed to a Merkle tree. Each call node receives its input from parent nodes via a content-hash chain-a string of bytes that includes the previous node's output hash, the current node's prompt hash and a nonce derived from the global seed. This makes the entire inference graph tamper-evident; modifying one node's output would change its hash. Which invalidates all downstream hashes.

The DAG isn't merely a linear chain. Trui supports branching, conditional logic, and loops with fixed upper bounds (static loop unrolling). For example, a medical triage assistant might need to query a symptom checker up to three times to narrow down a differential diagnosis. Trui ensures the loop unrolling is encoded in the DAG before execution. So the structure is fixed and verifiable ahead of time. We found this static ahead-of-time compilation drastically simplifies debugging-you can inspect the entire plan before any token is generated, making it much easier to spot prompt injection vulnerabilities.

Merkle proofs allow partial verification. If you only care about the final output, you can verify the output node's hash against the root. But if you want to prove that a specific intermediate reasoning step followed a particular logical path, you can request a Merkle proof for that subtree. This is instrumental for compliance teams: we've used it to show that a loan-decision bot never accessed a protected attribute (like race or gender) by revealing only the nodes corresponding to feature transformations. While the raw input stays hashed.

Trui's Approach to Prompt Engineering as a Compiled Contract

In traditional prompt engineering, you tweak a string template and hope for the best. Trui treats the entire prompt pipeline-instructions, few-shot examples, output schema. And any embedded business rules-as a compiled contract. You define it in the Trui Interface Definition Language (TIDL), a declarative syntax that looks like a cross between Protocol Buffers and Jinja2 templates. TIDL compiles down to a binary artifact that is signed and stored in your VCS. From that point on, the contract version is part of the trui invocation; no runtime interpreter can alter the prompt's semantics after compilation.

We learned the hard way that untracked prompt evolution was a major source of our production issues. In one sprint, a data scientist adjusted a classifier prompt to add an emoji for clarity-this alone caused the output to switch from JSON to markdown, breaking our mobile app's parser. With trui's compiled contracts, any such change requires a new contract version. Which must be explicitly rolled out through CI/CD and linked to the mobile client's configuration. Our integration tests now can diff the contract hash against a golden baseline, catching breaking changes before they hit canary.

Additionally, the TIDL compiler performs static analysis on the contract: it checks that all referenced variables are bound, that output schemas are valid. And that loop bounds are within defined limits. This prevents an entire class of runtime errors that plague dynamic template engines. We've open-sourced our TIDL compiler plugin for Visual Studio Code. Which provides real-time validation and autocomplete for trui contracts.

Benchmarking Trui Against LangChain and Semantic Kernel

We ran a side-by-side comparison of trui with LangChain (0. 1, and 0) and Semantic Kernel (dotnet 11) on a common task: extracting structured event information from 10,000 short user-reported incident texts and mapping them to a predefined taxonomy. All three pipelines used the same GPT-4 model snapshot. The key metrics were output schema conformance, latency consistency, and cost overhead. LangChain and Semantic Kernel both produced schema-valid JSON on about 96% of calls. But the remaining 4% required retries with repair prompts, adding an average of 1. 7 seconds per outlier. Trui, with its deterministic contract, delivered 100% schema conformance without retries-every output passed validation on the first call.

Latency was another differentiator. Because trui precompiles the execution graph, it can prefetch model weights and warm up the inference engine's KV cache before the first token is requested. In our benchmarks, trui reduced p99 latency by 22% compared to the dynamically chained approaches, largely because no runtime graph construction or dynamic validation was needed. The cost overhead of generating Merkle proofs added about 3% to the total token generation cost. Which we found acceptable given the audit trail benefits.

Where trui fell short was in flexibility for open-ended creative tasks. If you need a model to free-form brainstorm, deterministic temperature=0 calls can stifle diversity. Trui is not designed for creative exploration; it's built for production inference that must be repeatable. That trade-off is worth it when you're serving features that affect a user's credit score or medical advice.

Integrating Trui Into Existing DevOps and MLOps Pipelines

We integrated trui as a sidecar container in our Kubernetes cluster, sitting between our mobile backend APIs and the LLM inference service (vLLM on A100 GPUs). Trui's architecture uses gRPC streaming for real-time verification. Which we exposed through an Envoy proxy with mTLS. The CI/CD pipeline now includes a trui verify step that checks the signature of each deployed contract against a policy engine (OPA). Only contracts signed by the ML team's keypair are accepted by the trui runtime, preventing unauthorized prompt modifications.

One of the most valuable integrations was with our observability stack. We instrumented trui's Rust-based runtime using OpenTelemetry and exported traces to Grafana Tempo. Each trace carries the contract version hash and the Merkle root, allowing us to correlate application-level errors with a specific inference run. When an unusual spike in hallucination-like behavior occurred, we could trace it back to a specific contract version and node within the DAG, drastically reducing mean time to resolution from days to minutes.

We also built a mobile-side verifier library in Rust, compiled to Kotlin Multiplatform for Android and Swift for iOS, that can independently check the Merkle proof sent from the server. This enables the mobile app to verify that the response it received was generated by the exact contract and model it expects, without trusting the server. This trust-minimized design is especially relevant for mobile wallet apps handling cryptographic keys.

Security Implications: Attestation of Model Integrity with Trui

Supply-chain attacks on machine learning models are no longer theoretical. Malicious actors can poison model weights during training or swap models during deployment. Trui addresses this by integrating with hardware-rooted attestation services through the Intel Trust Domain Extensions (TDX) and AMD SEV-SNP. Before a trui node executes a contract, it verifies a remote attestation report from the inference worker, confirming the model hash, runtime environment. And trui version match the expected manifests. The evidence is included in the prover bundle.

In our production deployment, we set up a policy that rejects any inference whose attestation report doesn't chain back to a specific CI build. This ensures that even if a Kubernetes pod is compromised, an attacker can't inject a backdoored model without also breaking the cryptographic attestation chain. We've open-sourced a Helm chart that automates the setup with the Intel Trust Domain Extensions toolkit.

Trui's design also makes it resilient to prompt injection. Because the contract is compiled and immutable, user-provided data can only be injected into predefined template slots with strict type and length constraints. The TIDL compiler enforces a sandbox model: user input is always treated as data, never as code. Combined with deterministic execution, a trui contract can't be tricked into executing arbitrary instructions, even if a clever jailbreak prompt is embedded in user input-the output path is fixed.