observatory

Most engineers hear "observatory" and picture domed roofs, telescopes, and star charts. In software, the idea is nearly identical: an observatory is a system that continuously watches your digital estate, collects signals most teams miss. And turns raw data into actionable intelligence. Instead of galaxies, you're tracking security headers, TLS configurations, dependency drift, and posture regressions.

The best engineering teams don't treat security and compliance as annual audits; they run an observatory that scans production every hour and fails pull requests that weaken the baseline.

In this post, I will explain what an observatory means in modern engineering, why it's different from standard observability. And how to build one without turning your CI/CD pipeline into a noise machine. The examples come from production systems I have worked on, not marketing slides, so expect specific tools, real numbers. And a few hard-won opinions.

What an Observatory Means for Engineers

An observatory in software engineering is a deliberate, automated practice of measuring the state of your systems against a known good baseline. It isn't a single dashboard. It is a pipeline: scheduled or event-driven scanners, a durable store for results, trend analysis. And remediation workflows tied to the teams that own the code.

Think of the difference between glancing at the sky and running a scheduled astronomical survey. A manual pentest or a quarterly compliance review is the glance. An observatory is the survey. It answers questions like: "Did any service lose its HSTS header this week? " "Which repositories still use an end-of-life Node js version? " "How many public S3 buckets appeared since yesterday? " These are not questions logs or metrics answer well on their own.

The term gained traction thanks to Mozilla Observatory, a free tool that scans websites for security headers and TLS configuration. But the concept has expanded. Today, an observatory can cover cloud posture, supply-chain risk, secret leakage, infrastructure-as-code drift, and even AI model-card compliance. The unifying idea is continuous, comparative assessment.

Mozilla Observatory and the Security Baseline

Mozilla Observatory remains the easiest entry point for web security observatories. Point it at a URL. And it returns a letter grade from A+ to F based on response headers, TLS settings. And a handful of cookie flags. The scoring rubric is public. Which is important: a good observatory doesn't hide its criteria behind a black box. You can inspect the rules, dispute them. And tune them for your risk model.

In production environments, we found that a passing Observatory score is a useful floor, not a ceiling. It catches missing Content-Security-Policy directives, X-Frame-Options omissions, and weak TLS versions. But it won't catch business-logic flaws, authentication bypasses, or供应链 attacks that's why we treat Observatory as one instrument in a larger observatory, not the whole telescope.

If you want to automate this, the Mozilla HTTP Observatory CLI runs inside CI/CD. You can pin a minimum grade and fail builds that slip below it. The CLI also emits JSON. Which makes it trivial to push results into Elasticsearch, S3. Or a Postgres table for trending. We started with a nightly cron scan of all public domains and moved to per-deployment scans once the noise level dropped.

Why Observability isn't Enough Alone

Observability, the three-pillar kind, is about understanding system behavior from outputs. Logs, metrics, and traces tell you what happened, how fast. And where the request traveled. An observatory asks a different question: is the system still configured the way we agreed it should be that's a posture question, not a behavior question.

Here is a concrete example. A metrics dashboard will show that your API is returning 200s with low latency. An observatory will show that the same API is missing a Content-Security-Policy header and is running a Docker image with a critical CVE. Both views are correct, and neither replaces the otherIn fact, the best platforms combine them: use observability to detect runtime anomalies and an observatory to detect configuration drift and exposure.

The tooling split matters. Observability leans on OpenTelemetry, Prometheus, Grafana, and Jaeger. An observatory leans on scanners like Trivy for container vulnerability scanning, OpenSSF Scorecard for repository security posture, OWASP ZAP for dynamic application testing, and Prowler or ScoutSuite for cloud posture. You don't need all of them on day one. But you do need to know which lens each tool provides.

Building Your Own Security Observatory Pipeline

A practical observatory pipeline has four stages: discovery, scanning, persistence, and response. Discovery is the most underestimated. You can't scan what you don't know exists. We maintained a service catalog in Backstage and cross-referenced it against DNS, cloud asset inventories, and repository metadata. The first scan always surfaced "shadow" services that no one had registered.

For scanning, we scheduled jobs with GitHub Actions and AWS Step Functions. Container images were scanned with Trivy at build time and again nightly because new CVEs appear constantly. Web endpoints were scanned with Mozilla Observatory and OWASP ZAP. Repositories were evaluated with OpenSSF Scorecard. Results landed in a central Postgres database with a simple schema: asset_id, scanner, rule_id, severity, finding_json, detected_at, resolved_at.

The response stage is where most observatories die. We learned to route findings to the owning team via Slack or Jira, not a shared security inbox. Each finding carried a due date based on severity and an automatic escalation path. We also added a "false positive" flag with an owner and reason field. Without that escape valve, engineers game the system or ignore it.

Key Metrics Every Observatory Should Track

Raw findings are overwhelming. Trend metrics are what make an observatory useful. We tracked mean time to remediation (MTTR) by severity, percentage of assets with a passing security grade, count of critical or high findings older than 14 days. And the number of new exposed services per week. These numbers told us whether our posture was improving or degrading.

Some metrics map directly to standardsFor web security, we referenced RFC 6797 for HTTP Strict Transport Security (HSTS), RFC 9116 for security txt files, and RFC 9114 for HTTP/3 where relevant. For containers, we tracked unpatched CVEs against the National Vulnerability Database (NVD), and for cloud posture, we used CIS BenchmarksThe key is to anchor your dashboard to documents that other teams and auditors already trust.

• Security header grade: Mozilla Observatory letter score per public domain.
• Dependency freshness: percentage of direct dependencies updated within the last 90 days.
• Secret exposure: unrevoked secrets detected by GitLeaks or TruffleHog.
• Infrastructure drift: Terraform or CloudFormation stacks that differ from production.

We also tracked "observatory coverage," the percentage of known assets actually scanned by at least one tool. A 95 percent coverage metric with zero findings is more honest than a 20 percent coverage metric that looks clean. Coverage is the denominator every security report hides.

Integrating Observatory Scans into CI/CD

The phrase "shift left" is overused. But for an observatory it's the right mental model. If you only scan production, you're documenting failures after deployment. If you scan at build time, you prevent them. We integrated Trivy into our container build pipelines and Mozilla Observatory into our deployment previews. A pull request that introduced a new base image with a critical CVE failed before it could be merged.

GitHub Actions made this straightforward. For example, the HTTP Observatory CLI can run as a step in a workflow that triggers on pull requests to the main branch. You set a grade threshold and let the action fail the job. The same pattern works with Trivy, OpenSSF Scorecard, and tfsec for Terraform. The important detail is version pinning. Scanners change their rules, and a green build yesterday can become a red build today if a new CVE is published. We pinned scanner versions and updated them on a schedule with a dedicated migration PR.

gates need nuance. A hard fail on every low-severity finding will train engineers to bypass the gate. We used severity-based gating: block on critical and high, warn on medium. And log the rest. We also allowed repository-level exemptions with an expiry date. That kept the pipeline fast while preserving accountability. Read our guide to CI/CD security gates

Lessons from Running Observatories at Scale

Scale breaks observatories in predictable ways. The first is noise. When we turned on every scanner at maximum sensitivity, one mid-sized team received 400 findings in a week. Most were informational. They stopped reading. We fixed this by aligning severity with exploitability, not theoretical risk. A missing header on an internal admin tool got a lower score than the same header missing on a public login page.

The second lesson is ownership, and security findings without owners become orphan ticketsWe used CODEOWNERS files, service catalogs. And cloud tags to map every asset to a team. When a finding appeared, it auto-assigned to the right Slack channel. If ownership was unclear, we treated that as a finding of its own and fixed the catalog before fixing the vulnerability.

The third lesson is drift over time. An observatory isn't a project with a finish line. New services appear, old scanners retire, and threat models change. We scheduled quarterly reviews of the observatory itself: which scanners still matter, which metrics moved the needle, and which alerts produced false positives. Treating the observatory as a product, not a script, is what keeps it alive.

The Future of Automated Security Observatories

Security observatories are becoming policy-driven and AI-assisted. Open Policy Agent (OPA) lets teams express security rules as code and evaluate them against live infrastructure. Instead of a dozen custom scanners, you can have a single engine that checks JSON documents against Rego policies. This is especially powerful for Kubernetes admission control and cloud posture management.

Large language models are starting to help with triage they're not yet reliable enough to auto-remediate production systems, but they're useful for summarizing scanner output, suggesting fix commits. And routing findings. We experimented with an LLM layer that read Trivy reports and proposed updated base-image tags or Dockerfile patches. Human review was still required. But the first-draft fix saved hours per finding.

Software Bills of Materials (SBOMs) will also reshape observatories. With a machine-readable inventory of every dependency, an observatory can correlate new CVEs against your actual artifacts in minutes instead of days. The NTIA and CISA guidance on SBOMs, along with SPDX and CycloneDX formats, are making this practical. Learn how SBOMs change vulnerability response

Frequently Asked Questions

What is an observatory in software engineering?

An observatory is an automated system that continuously scans and measures the security, compliance. Or configuration posture of your software assets. It differs from observability because it focuses on static and comparative assessment rather than runtime behavior.

How does Mozilla Observatory score websites?

Mozilla Observatory grades websites from A+ to F based on HTTP response headers, TLS settings, cookie flags, and other security controls. The scoring rules are public. And you can run scans through the website or the open-source CLI.

Can I run observatory scans in CI/CD,

YesTools like the Mozilla HTTP Observatory CLI, Trivy - OWASP ZAP. And OpenSSF Scorecard can run as steps in GitHub Actions - GitLab CI. Or any CI/CD platform. You can gate deployments based on severity thresholds.

What tools complement a security observatory?

Common complements include Trivy for container scanning, OWASP ZAP for dynamic application testing, OpenSSF Scorecard for repository security, Prowler for cloud posture, OPA for policy-as-code, and TruffleHog for secret detection.

How is an observatory different from observability?

Observability explains system behavior using logs, metrics, and traces. An observatory checks whether the system still matches a desired security and configuration baseline, and they work best together

Conclusion: Start Small and Stay Curious

You don't need a custom platform to start running an observatory. Pick one asset class, one scanner, and one metric. Public web endpoints scanned with Mozilla Observatory and tracked for grade regression is a perfectly valid first step. Prove value, reduce noise, then expand to containers, cloud resources, and dependencies.

The teams that get the most value treat the observatory as shared infrastructure, not a security team side project. Engineers own the findings, SREs own the pipeline. And security owns the risk model. When those groups collaborate, the observatory becomes a source of truth rather than a source of dread.

If you haven't run an observatory scan on your production domains lately, do it this week. The results will almost certainly surprise you. And the fixes are usually cheap. The alternative is finding out from a bug bounty report or a compliance audit, and neither is fun

What do you think?

Should observatory findings ever block a deployment automatically,? Or should humans always make the final call on security gates?

What is the single most underrated metric a software observatory should track that most teams currently ignore?

How do you balance full scanning with alert fatigue when the same asset can produce findings from five different tools?