The request for "spiderman" landed on our backlog without context, forcing us to think about web-scale crawlers and the perennial engineering challenge of wrangling the unstructured web. At Denver Mobile App Developer, we have built crawlers for sentiment analysis, competitor monitoring. And training data pipelines. And we kept hitting the same wall: off-the-shelf libraries default to synchronous execution and choke on JavaScript‑heavy pages. We set out to build a respectful, high‑fidelity crawler that behaves like a real browser while respecting the robots txt exclusion protocol to the letter. Most web crawlers fail when faced with modern JavaScript; our spiderman architecture doesn't - here's how we built a polite, scalable crawler that processes 50 million pages daily.
The project, code‑named Spiderman, became a reference implementation for how engineering teams can balance extraction speed with legal and ethical guardrails. Over eighteen months we iterated through three major rewrites, learning hard lessons about headless browser resource leaks, Kafka partition rebalancing. And the paradox of being "too fast" for site administrators. This article unpacks the decisions that matter: the tooling, the architecture. And the operational telemetry required to keep a distributed crawler healthy and compliant.
We will walk you through the genesis of the effort, the concrete technology stack - Playwright - Apache Kafka, MinIO, Prometheus - and the operational patterns we baked into our platform. Every paragraph draws from production incidents, RFCs, and actual dashboards. If you've ever been tasked with extracting data from millions of domains without wrecking your reputation, there's something here you can put into practice tonight.
The Genesis of the Spiderman Project: Why Off-the-Shelf Crawlers Fell Short
Before we wrote a single line of our own code, we stress‑tested Scrapy, Apache Nutch, and several commercial platforms. Scrapy's asynchronous model is elegant for static HTML. But its default middleware can't execute JavaScript, leaving behind any content rendered by React or Vue. Nutch relies on Hadoop and parses robots txt via a plugin that hadn't been updated to reflect RFC 9309. Which introduced the `Crawl‑delay` directive formally. In production, we observed that nearly 40% of the target domains in our media monitoring pipeline were single‑page applications that delivered empty `
` containers to traditional GET requests. 
.The realization forced a fundamental design choice: we needed headless browser instances as first‑class citizens of the crawl loop, not bolted‑on afterthoughts. We also required a queue that could absorb bursts without backpressure. Because a single infinite‑scroll page could spawn thousands of dynamically generated URLs. Existing tools forced us to re‑architect their plumbing, so we decided to build Spiderman from the ground up, focusing on three non‑negotiable pillars: politeness, observability, and modularity.
By giving the project its own identity, the team could reference "Spiderman's crawl budget" or "Spiderman's stale DNS cache" in incident retrospectives without ambiguity. That linguistic anchor turned out to be surprisingly valuable when onboarding new engineers - a clear signal that the crawler is an independent product, not a script someone duct‑taped together.
Architecting for Politeness: Robots txt Parsing and RFC 9309 Compliance
RFC 9309, published in 2022, superseded the informal 1994 convention and gave webmasters a standard way to express crawl preferences. We implemented a parser that follows the specification's BNF grammar exactly, handling `User‑agent`, `Allow`, `Disallow`, `Crawl‑delay`. And the `Sitemap` directive. The parser runs as a sidecar process in each crawl pod. So no request leaves the pod without a fresh robots txt check. We cache parsed profiles in Redis for 30 minutes. But we re‑fetch the file whenever a 429 or 503 response hints that the site's policy may have changed.
The tricky part is respecting `Crawl‑delay` across a distributed fleet of workers. A naive implementation applies the delay per pod, which means a farm of 200 pods would hammer the origin server 200 times faster than intended. Instead, we use a centralized token bucket per host, updated atomically in Redis with Lua scripts. Each pod claims a token before dispatching a request; if no token is available, the request is re‑queued with a back‑off timestamp. This "distributed rate limiter" pattern, documented in our internal playbook Read our guide on distributed rate limiting with Redis, kept our total request rate within 110% of the published crawl‑delay even during deployment scaling events.
We also honor `X‑Robots‑Tag` header directives on dynamically generated resources, a nuance that many frameworks miss. Spiderman parses these headers as they arrive and immediately drops the URL from the frontier if it contains `noindex` or `nofollow`, preventing wasted headless browser sessions. In our last audit, compliance with robots, and txt directives stood at 9997% across 2. 4 million hostnames - a metric we track in our public transparency dashboard,
Headless Browsing at Scale: Playwright and Puppeteer Integration Strategies
After evaluating Puppeteer, Playwright. And Selenium Grid, we settled on Playwright for its superior browser context isolation and network interception API. A single Playwright server process can manage multiple browser contexts, each with its own cache, cookies. And service worker registration. This isolation prevents one site's JavaScript from contaminating another and reduces the attack surface for cross‑domain data leakage. We run Chromium in headless mode inside each pod, pinned to specific CPU sets with cgroups v2 to guarantee fair scheduling.
Memory consumption became the first production headache. A single Chromium tab idle on a memory‑heavy single‑page app can hold 400-600 MB of RAM. And we were running 15 concurrent contexts per pod. We introduced page disposal policies based on idle time and page weight: if a tab's JavaScript heap exceeds 200 MB, we force a garbage collection call via the DevTools protocol and, if that fails, we close the tab and log a structured warning. This alone cut our pod out‑of‑memory restarts by 73%.
We also wrote a custom interception layer using Playwright's route() API to block unnecessary asset categories: images, fonts. And analytics scripts. Block lists are configurable per domain via an internal admin panel, allowing us to fine‑tune for news sites that break when CSS is absent. The cumulative effect is a 40% faster page load and a direct reduction in bandwidth costs from our egress‑heavy cloud provider. For details on writing robust route handlers, consult the Playwright network interception documentation
Distributed Crawl Queuing with Apache Kafka and Priority Scheduling
The crawl frontier - the set of URLs yet to be visited - must handle millions of entries, deduplication. And varying priorities. We chose Apache Kafka for its persistent, partitioned log and its ability to replay messages during debugging. Each URL is published to a topic partitioned by hostname hash, ensuring that all requests to the same host land on the same partition and are processed sequentially, further reinforcing politeness.
We built a custom scheduler that assigns priority tags based on freshness signals: a sitemap URL with a `lastmod` from the past hour gets priority 1. While a deep‑linked blog post from 2018 is priority 3. The scheduler reads a compacted topic of already‑seen fingerprints (a 128‑bit MurmurHash of the canonicalized URL) from a RocksDB state store, achieving sub‑millisecond membership queries without network round‑trips. This pattern mirrors the design of a Kafka Streams stateful topology and allows us to skip re‑insertion of known URLs at the broker level.
As we scaled to 10,000 partitions, partition rebalancing during pod restarts caused bursty gaps in crawl throughput. We moved to a static group membership configuration with `group, and instanceid` and tuned `max, and pollinterval, since ms` to 30 seconds. Which stabilized the consumer group and eliminated the infamous "rebalance storm" that previously consumed 18% of our wall‑clock time. Apache Kafka's consumer configuration reference proved invaluable during this tuning.
Storing Massive HTML Corpora: Object Storage Pipelines with MinIO and Iceberg
Raw HTML payloads from Spiderman's fetches are frequently 300 KB to 2 MB each, generating terabytes weekly. We stream the rendered DOM snapshot and the raw network HAR directly to MinIO object storage, using S3‑compatible multipart uploads. Each object key follows the pattern `
To make this data queryable, we push metadata - URL - fetch timestamp, HTTP status. And a summary of the parsed microdata - into an Apache Iceberg table stored on the same MinIO cluster. Iceberg's hidden partitioning on `fetch_date` means our analysts can run time‑travel queries without understanding the physical layout. One of our data science partners recently joined the HTML corpus with a weather dataset to study correlation between page freshness and local events, running a Spark SQL query that scanned 4. 7 billion rows in 22 seconds. Explore our data lake architecture with Iceberg for mobile app analytics
The dual approach - object storage for raw blobs and a table format for metadata - keeps storage costs linear while enabling ad‑hoc forensic analysis. During a recent crawl of a government transparency portal that reshuffled its URL scheme mid‑crawl, we were able to replay the HAR logs to reconstruct the navigation flow and identify the exact commit that broke our selector chain.
Observability and SRE: Monitoring Crawl Health with Prometheus and Grafana
"How many pages did Spiderman fetch in the last minute? " is a question we answer with a Prometheus counter `spiderman_pages_fetched_total`. But the real insights live in the latency distributions and error budgets. We instrumented a histogram `spiderman_fetch_latency_seconds` with buckets tuned to typical p50 (1. 2s) and p99 (18s) values observed over a month of baselining. Each pod pushes metrics via a Pushgateway. And we alert when the p99 crosses 25 seconds for any single host for more than five minutes.
Alongside metrics, we emit structured logs in JSON with a mandatory `crawl_id` trace ID that propagates through Kafka headers, enabling correlation across the queue, headless browser. And storage layers. When a URL fails repeatedly, we publish an event to a dead‑letter topic and create a Jira ticket via a webhook; the ticket contains the HAR file path, the screenshot of the page at failure (captured by Playwright's `page screenshot()`). And a link to the Grafana dashboard for that host's error budget. This closed loop turned a previously chaotic fire‑drill posture into a methodical SRE practice.
Our on‑call playbook includes a command to pause Spiderman's crawl globally via a feature flag stored in LaunchDarkly. We tested this during a rolling brown‑out drill, successfully halting 12,000 concurrent fetches within 8 seconds. The ability to stop the crawler without destroying state is a direct lesson from operating distributed systems with circuit breakers. How to integrate LaunchDarkly for SRE emergency stop patterns
Ethical Data Extraction: Rate Limiting, User-Agent Rotation,
Need a Custom App Built?
Let's discuss your project and bring your ideas to life.
Contact Me Today →