WHAT THEY ARE
A crawler (also spider, robot, or bot) is a program that walks the public web by requesting URLs, reading the response, extracting more URLs, and repeating. Search engines, archives, price sites, security scanners, and — after about 2023 — large language model companies all run them. A fetcher is the narrower cousin: one shot on behalf of a user (open this link, verify this site), not a continuous walk.
They are not browsers with a person attached. They announce themselves with a User-Agent string, usually including a URL where the operator explains the bot. Honest operators also publish IP ranges so a webmaster can check that “Googlebot” is actually Google.
THE LOOP
Every serious crawler is the same machine with different priorities:
1. Seed list — home pages, sitemaps, last week’s unfinished queue. 2. Frontier — the priority queue of URLs still to fetch. Not a simple FIFO. It ranks by importance, freshness, and politeness. 3. DNS — turn hostnames into addresses. At web scale DNS is its own bottleneck. 4. Fetch — HTTP/1.1 or HTTP/2 GET (sometimes HEAD). Follow redirects. Record status codes. 5. Parse — HTML links, canonical tags, sitemaps, RSS, sometimes rendered JavaScript. 6. Store / index / train — the point of the trip. 7. Enqueue new URLs that pass filters (same scheme, not a trap, not disallowed).
Stop when the budget is spent, the queue is empty, or the operator kills the job. Google’s rendering path adds a second pass: download HTML, hand it to a Web Rendering Service, fetch JS/CSS, build the page as a browser would, then extract what a human would have seen. Rendered assets are cached (Google has described ~30 day WRS cache independent of the site’s Cache-Control) so one stylesheet does not burn the whole host’s crawl budget.
Googlebot’s documented fetch cap as of 2026 is 2 MB per URL excluding PDFs; PDFs go to 64 MB. Other products set their own byte limits on the same crawl fabric. The name “Googlebot” is a 2000s leftover: Search is one client of a shared crawler platform that also feeds Shopping, Ads, and others under different agent strings.
POLITENESS
A crawler that hammers one origin is an accidental denial-of-service. Classic rules:
- One (or few) concurrent connections per host. - Delay between hits on the same server — often a multiple of last download time, or a robots.txt Crawl-delay. - Hash hostnames to worker threads so two threads never pile onto one shop. - Cap total bytes and requests per day per site (crawl budget).
MOMspider (1994) already treated politeness as a design problem. The web never made it optional.
ROBOTS.TXT AND SITEMAPS
robots.txt lives at https://example.com/robots.txt. It is a request, not a lock. Well-behaved automatic crawlers honor it. Malware, scrapers, and some “research” kits do not. Syntax is grouped by User-agent token:
User-agent: * Disallow: /admin/ Allow: / Sitemap: https://example.com/sitemap.xml
User-agent: * is the default. Named agents override. Crawl-delay is widely written and only sometimes obeyed (Google ignores it and uses its own schedule). A sitemap (XML list of URLs plus lastmod) is the polite way to say “here is the inventory.” Crawlers still discover links by walking pages; sitemaps just reduce the scavenger hunt.
robots.txt does not hide a URL from anyone who already has the link. It only asks automatic crawlers not to fetch. For true exclusion you need authentication, noindex, or not putting the document on the public web.
HISTORY IN BRIEF
1993 — World Wide Web Wanderer (Matthew Gray) measures the web’s size. 1994 — WebCrawler, Lycos, MOMspider. First-generation engines and the politeness debate. 1995–98 — AltaVista, Inktomi, early Google. PageRank makes crawl priority a ranking problem, not just a coverage problem. 2000s — Googlebot becomes the default noun. Internet Archive’s Heritrix; Nutch / Hadoop research crawlers. 2008 onward — Common Crawl publishes periodic open snapshots that later become training fuel for many models. 2010s — JavaScript rendering becomes mandatory for modern sites. Mobile-first crawl. Faceted-navigation traps eat budgets. 2023–26 — LLM training crawlers and AI-search crawlers split the robot population. robots.txt files start naming GPTBot and ClaudeBot as often as they name Googlebot.
TYPES OF CRAWLER
Search indexers. Googlebot, Bingbot, Applebot, Yandex, Baiduspider. Goal: discover and refresh pages for a query engine. They re-crawl important URLs often and dusty URLs rarely.
Archivists. Internet Archive, national libraries, Common Crawl (CCBot). Goal: keep a copy. Common Crawl’s dumps are the public well a lot of later AI training quietly drank from.
Training crawlers. Fetch to build model weights. Periodic, often no JS render, invisible in browser analytics. GPTBot, ClaudeBot, Bytespider, Applebot-Extended, Google-Extended, Meta-ExternalAgent, Amazonbot. Blocking them is a policy choice about future models, not about disappearing from search today.
AI search / citation crawlers. Live or near-live index for answers with links. OAI-SearchBot, ChatGPT-User (user-triggered), Claude-SearchBot, PerplexityBot, and Googlebot itself for AI Overviews. Blocking these is an invisibility switch for assistant citations.
User-triggered fetchers. One URL because a person asked. Different robots.txt token. Google documents a whole class of these. They are not the overnight spider.
Commercial scrapers. SEO tools, price watchers, brand monitors. Mix of polite identified bots and unidentified farms.
Hostile or sloppy bots. Credential stuffing, vulnerability scans, copy farms, comment spam. They lie about User-Agent, ignore robots.txt, and rotate IPs. That is not “crawling” in the search-engine sense; it is abuse that happens to use HTTP GET.
MAJOR NAMED AGENTS (2026 SNAPSHOT)
Traffic shares move month to month. Mid-2026 public panels still put Googlebot first but no longer majority; ClaudeBot, Meta-ExternalAgent, and GPTBot take large slices. robots.txt mention-counts often rank GPTBot, ClaudeBot, Google-Extended, CCBot as the most-named AI tokens.
OpenAI — GPTBot (train), OAI-SearchBot (index), ChatGPT-User (live fetch). IP list published as JSON. Anthropic — ClaudeBot, Claude-SearchBot, Claude-User; older logs still show anthropic-ai. Google — Googlebot (Search + Overviews), Google-Extended (Gemini training opt-out token; blocking it does not drop Search), GoogleOther. Microsoft — Bingbot / Copilot path. Apple — Applebot and Applebot-Extended. Meta — Meta-ExternalAgent. ByteDance — Bytespider (Doubao / TikTok AI). Heavily blocked in robots.txt studies. Amazon — Amazonbot. Perplexity — PerplexityBot and Perplexity-User. Common Crawl — CCBot.
The 2026 operational mistake is a blanket “Disallow AI” that also kills OAI-SearchBot or Googlebot. Training and citation are different valves.
CRAWL BUDGET AND SITE DESIGN
Big hosts get a finite number of fetches per day. Waste it on session-IDs, infinite calendars, sort-order URLs, and soft-404s, and the pages you care about go stale. Helps: clean URL scheme, canonical tags, XML sitemaps, stable 301s, real 404s, compressed HTML, not blocking CSS/JS that the renderer needs. Hurts: crawling through a login wall you meant to keep public, or serving a 2 MB homepage of ads before the article.
HOW OPERATORS VERIFY A BOT
User-Agent is costume jewelry. Anyone can send “Googlebot.” Real checks: reverse DNS of the client IP to a name the operator documents, then forward DNS back to the same IP; or match against published IP prefixes (OpenAI GPTBot JSON, Google’s crawler ranges). If it fails that test it is a faker.
LEGAL AND SOCIAL LAYER
robots.txt is convention plus terms of use, not a statute by itself. Courts and statutes on scraping, copyright, and computer-access laws differ by country and year. Site owners who want training opt-out now treat Google-Extended, GPTBot, and CCBot as separate policy knobs. Archives and open-web researchers argue that a public URL is meant to be read. Publishers argue that “read” never meant “copy into a weight matrix.” Both sentences are in play in 2026. This page records the machinery, not the verdict.
STATUS
Crawlers still make search possible. They also now make models possible. Server logs that were 90 percent Google a decade ago are a coalition of search, archive, shop, and AI agents. The old two-file protocol — robots.txt and sitemap.xml — is still the control panel. It was never a wall. It is a labeled door, and in 2026 there are more names on the doorbell than there used to be.
IN ONE SENTENCE
An internet crawler is an automated HTTP walker with a queue, a politeness timer, and a User-Agent — built to index, archive, or train — that asks permission in robots.txt and then copies whatever the public web left unlocked.