What is web crawling and indexing? How crawlers and search indexes work

Crawling and indexing: what each step does

The two words travel together, but they are separate jobs, often run by separate systems, and a page can pass one and fail the other.

The order matters. A page has to be fetched before it can be indexed, so the crawl is the first place to look when a page is missing from search results. The controls are separate too. Blocking a URL in robots.txt stops the engine reading it, not listing it; to keep a page out of results you use noindex, and the engine has to be allowed to fetch the page to see that rule. Bing's webmaster guidelines put it plainly: robots.txt controls crawl access, not indexing.

How a web crawler works, step by step

Every crawler, from a search engine's fleet to a small script, runs the same loop. The textbook description in Introduction to Information Retrieval (Manning, Raghavan and Schütze, Cambridge University Press) follows Mercator, a crawler design that became the basis of several research and commercial crawlers:

1. Start from seeds. The crawl begins with a seed set of known URLs. For a search engine, that is every URL it already knows about, plus sitemaps and URLs sites submit.
2. Take the next URL from the frontier. The URL frontier holds every URL not yet fetched, in priority order.
3. Check robots.txt, then fetch. The crawler resolves the host name with DNS, checks the host's robots.txt rules and downloads the page over HTTP.
4. Parse. It extracts the text for the indexer, and the links with their anchor text.
5. Filter and deduplicate. Each link is made absolute, run through URL filters and dropped if it is already queued or was fetched recently. A page whose content was already seen at another URL is flagged as a copy.
6. Queue and repeat. New URLs join the frontier with a priority. In continuous crawling, the fetched URL goes back in as well, to be fetched again later.

The loop is simple; the scale is not. The same textbook works out that fetching a billion pages in a month means several hundred fetches every second, sustained, while staying polite to every site, surviving malformed pages and avoiding traps. The parts that matter most to a site owner are the frontier, duplicate detection and the recrawl schedule.

The URL frontier decides what is fetched next

The frontier is where a crawler's judgement lives. It orders URLs on two considerations. The first is priority: pages that are useful and change often should be fetched often, so priority combines a quality estimate with how often the page has changed between past visits. Change alone is not enough, because many spam pages change completely on every fetch. The second is politeness.

Many links on a page point to the same host, so a plain priority queue would fire a burst of requests at one server. Mercator-style frontiers prevent that by design: a set of back queues, each holding URLs from a single host, a record of when each host may next be contacted, and one open connection per host at a time. A common heuristic sets the wait before the next request to about ten times as long as the last fetch from that host took, so a server that slows down automatically receives fewer requests.

Open-source crawlers ship with the same instincts. A new Scrapy project waits one second between requests to the same domain (DOWNLOAD_DELAY) and obeys robots.txt (ROBOTSTXT_OBEY) out of the box. Sites can push back in two ways:

• HTTP 429 Too Many Requests, defined in RFC 6585, optionally with a Retry-After header saying how long to wait. A 503 can carry Retry-After too. Common Crawl's crawler uses an adaptive back-off when a site answers 429 or 5xx, and Apple says Applebot's crawl rate adjusts automatically when a site slows down or returns errors.
• Crawl-delay in robots.txt, a non-standard line asking for a pause, in seconds, between requests. Support varies by crawler (see the table below). Bing reads it as one URL per window of 1 to 20 seconds, so Crawl-delay: 5 caps Bingbot at 17,280 URLs a day.

That cap is why a delay is a blunt tool: at 17,280 URLs a day, a site with 100,000 URLs needs almost six days for Bingbot to fetch each page once. Bing's own help recommends against it and points to Crawl Control in Bing Webmaster Tools, which sets a crawl rate for each hour of the day instead.

robots.txt: the rules a crawler reads before fetching

robots.txt is where a site states its rules for bots. Martijn Koster defined the Robots Exclusion Protocol in 1994, and the IETF published it as RFC 9309, a standards-track document, in September 2022. What the standard says, and what a compliant crawler does:

• Location. The file lives at /robots.txt at the top of each host, UTF-8 encoded, as plain text. The file on www.example.com says nothing about shop.example.com.
• Groups. Rules sit under one or more User-agent lines. A crawler looks for a group naming its product token, matching case-insensitively; if none exists, it obeys the * group; if there is no * group either, nothing is disallowed.
• Matching. The most specific rule, the one with the most characters, wins. If an Allow and a Disallow rule are equally specific, Allow wins. * matches any run of characters and $ marks the end of a path.
• Errors. If robots.txt answers with a 4xx status, the crawler may fetch anything. If it answers with a 5xx or the server is unreachable, the crawler must assume everything is disallowed.
• Caching and size. Crawlers should not use a cached copy for more than 24 hours, and must parse at least the first 500 KiB.
• Not security. The RFC states that the rules are not a form of access authorization, and that listing a path in robots.txt makes it public. Protect private areas with authentication.

# Rules for every crawler without its own group
User-agent: *
Disallow: /cart/
Disallow: /search/
Allow: /search/help/

Sitemap: https://www.example.com/sitemap.xml

In this file, /search/help/ stays crawlable because Allow: /search/help/ is the longer, more specific match. Anything beyond User-agent, Allow and Disallow is an extension a crawler may or may not read. Sitemap is the widely supported one. Crawl-delay is not in the standard, which is why Google ignores it while Bing honours it. Google's own reading of the file, including how it treats errors, is covered in how Google crawls a page.

Duplicates and spider traps

The web is full of copies: the same article with and without tracking parameters, print versions, mirrors, syndicated posts. The textbook cites estimates that as many as 40% of web pages duplicate other pages. Crawlers and indexers deal with them at two levels:

• Duplicate URLs. Before a link enters the frontier, it is normalized (a relative link becomes an absolute URL) and checked against the URLs already queued or fetched.
• Duplicate content. After a fetch, the page gets a fingerprint, a short digest of its content, and pages with equal fingerprints are copies. Exact fingerprints miss near-duplicates, such as two pages identical except for a timestamp, so engines use shingling: they compare the sets of overlapping four-word runs in two pages, and pages above a similarity threshold are indexed once.

Your side of the deal is one URL per piece of content, with redirects and rel="canonical" saying which URL you prefer. Google's canonical rules are in the Google guide.

Crawlers also have to survive spider traps: URL spaces that never end, such as a calendar with a "next month" link forever, or filters and sort orders that multiply into millions of URLs. Some traps are built to mislead crawlers; many are accidents of site development. A crawler defends itself with URL filters and limits per host. For your site, the cost is that its attention goes to worthless URLs instead of your real pages.

Recrawling keeps the copy fresh

An index is a snapshot, and the web keeps changing. A continuous crawler puts fetched URLs back in the frontier and tries to revisit each page at a rate that approximates how often it changes, so a news home page is fetched far more often than a policy page last edited years ago. Three mechanisms make revisits cheaper for both sides:

• Conditional requests. The crawler sends If-None-Match with the page's last ETag, or If-Modified-Since with its Last-Modified date. If nothing changed, the server answers 304 Not Modified instead of sending the page again (RFC 9110). Google's crawlers support both headers and Common Crawl supports conditional GET.
• Sitemap lastmod. The Sitemaps protocol lets you state when each URL last changed. It describes changefreq as a hint that crawlers may weigh, not a command they follow.
• Push notifications. IndexNow tells participating engines the moment a URL changes (covered below).

What a search index is and what it stores

A search engine does not search the web when you type a query. It searches its index, a data structure built in advance from the crawled pages. The core of it is the inverted index: instead of listing the words on each page, it lists, for each word, the pages that contain it, the way the index at the back of a book lists pages for each topic.

Building one takes four steps:

1. Collect the documents. The crawler has done this.
2. Tokenize each one. Tokenization chops the text into tokens, usually words, and throws away characters such as punctuation.
3. Normalize the tokens into terms. Lowercasing makes Quick match quick. Stemming reduces foxes to fox, and organizes and organizing to one form. Synonym rules can map leap to jump. The same text analysis runs on every query, so a search for "Foxes leap" looks up fox and jump.
4. Invert. Sort the list of term and document pairs, merge repeats, and split the result into a dictionary of terms and, for each term, a postings list of the documents that contain it.

Three one-line pages show the result. Page 1 reads "Crawlers fetch pages", page 2 "Indexes store pages" and page 3 "Crawlers respect robots.txt":

A query for "crawler pages" looks up two postings lists and intersects them: only page 1 is on both, and no page had to be read to find it. Storing positions lets the engine answer phrase queries by checking that two terms sit side by side, and postings usually record how often each term occurs, which ranking formulas use.

Older systems dropped very common words such as "the" and "of" from the index. The textbook notes that web search engines generally do not use stop lists: titles such as "To be or not to be" consist entirely of them.

What else an index keeps about a page

The word index is only part of the record. Depending on the engine, it also holds:

• the page's title, description, language and other metadata;
• its outgoing links with their anchor text, which the crawler passes to the indexer for use in ranking;
• fetch details: the status code, when the page was fetched, and the ETag or date needed for the next conditional request;
• the content fingerprint used to spot copies, and which URL was chosen to represent a set of duplicates.

At web scale the index is split across many machines. The common approach is to partition by document: each machine indexes a subset of pages, every query goes to all of them, and their top results are merged. When someone searches, the engine runs the query through the same tokenization, fetches and combines the postings, then ranks the matches. Ranking is where engines differ most; for Google's approach, see how Google ranks results. AI assistants with search work from an index too: Bing's guidelines state that its Copilot experiences rely on the same crawling, indexing and ranking foundation as classic Bing search. For Google's AI answers, see the latest SEO trends.

How search engines and AI bots crawl differently

Every major operator documents its crawlers, and the documentation shows four kinds of bot that are easy to confuse:

• Search crawlers fetch pages for a search index: Googlebot, Bingbot, Applebot, OAI-SearchBot, Claude-SearchBot and PerplexityBot.
• Training crawlers collect pages that may be used to train AI models: GPTBot and ClaudeBot. Google and Apple use control tokens instead of separate crawlers: Google-Extended and Applebot-Extended.
• User-triggered fetchers load a page because a person asked an assistant about it: ChatGPT-User, Claude-User and Perplexity-User.
• Archive crawlers build public datasets: Common Crawl's CCBot.

Sources for each row: Bing, Apple, Common Crawl, OpenAI, Anthropic, Perplexity and Google, as documented in September 2026.

Bing also accepts IndexNow pushes (below), and its guidelines define two meta directives for AI use: noarchive keeps a page's content out of Copilot answers and grounding results, and nocache limits Copilot to the URL, title and snippet.

Common Crawl is a non-profit, founded in 2007, that crawls the web and publishes the result as a free, open archive. As of September 2026, its site describes more than 300 billion pages collected over 15 years, with 3 to 5 billion new pages added each month, and says the archive has been cited in over 10,000 research papers. CCBot is built on Apache Nutch, checks robots.txt before fetching and samples each site rather than archiving it whole.

AI crawlers are split by purpose, which is what makes a precise policy possible (next section). Whether a crawler runs JavaScript is often not documented at all, so assume some bots see only the HTML your server sends.

How to control each crawler with robots.txt

Because each bot answers to its own user-agent token, one robots.txt can set a policy per purpose. This example keeps every search crawler, opts out of AI training and the Common Crawl archive, and keeps all bots out of the cart:

# Search engines and AI search
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /cart/

# AI model training and open archives
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

# Everyone else
User-agent: *
Disallow: /cart/

Sitemap: https://www.example.com/sitemap.xml

What decides whether a file like this works:

• Tokens are independent. OpenAI, Anthropic and Perplexity each describe their bots as separate settings. Google-Extended and Applebot-Extended change how Google and Apple may use what Googlebot and Applebot fetch; Google states that Google-Extended does not affect inclusion or ranking in Google Search.
• Changes are not instant. RFC 9309 lets crawlers cache robots.txt for up to 24 hours, and OpenAI and Perplexity both say their systems can take about a day to reflect an edit.
• Every host needs its own file. Anthropic's documentation reminds site owners to repeat an opt-out on every subdomain.
• User-triggered fetchers play by different rules. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page. Anthropic lets you disallow Claude-User like its other bots.
• Page-level rules still apply. Bing and Apple support noindex. Apple also honours nosnippet to keep content out of the AI-generated answers in Siri and Search.

WAF rules and rate limiting at the edge are part of our hosting and maintenance service, for when a request is not enough.

A shared vocabulary is on its way. The IETF's AI Preferences working group is standardizing a way to state preferences such as allowing search but not AI training, attached through a new Content-Usage rule in robots.txt or an HTTP header of the same name. As of September 2026 both documents are still Internet-Drafts, so each company's own token is what its crawlers read today.

Push instead of waiting: how IndexNow works

Crawling is pull: the engine decides when to come back. IndexNow adds push. Microsoft Bing and Yandex created the protocol in October 2021 so that a site can tell search engines the moment a URL is added, updated or deleted. The protocol takes three steps:

1. Publish a key. Generate a key of 8 to 128 characters (letters, digits and hyphens) and host it as a UTF-8 text file named after the key, preferably at the root: https://www.example.com/<key>.txt.
2. Send changed URLs to one endpoint, either one URL as a GET request with url and key parameters or a JSON POST of up to 10,000 URLs.
3. Let the engines share it. The receiving engine fetches your key file to confirm you control the host, and participating engines agree to share submitted URLs with each other.

POST /indexnow HTTP/1.1
Host: api.indexnow.org
Content-Type: application/json; charset=utf-8

{
  "host": "www.example.com",
  "key": "4f1c9e7a2b8d4e6f",
  "urlList": [
    "https://www.example.com/new-page/",
    "https://www.example.com/updated-page/"
  ]
}

Where it fits, according to the IndexNow FAQ:

• Use it with a sitemap, not instead of one. IndexNow is for new and changed pages; the sitemap remains the complete inventory.
• A 200 response means received, not indexed. A 202 means the key is still being checked. Every submitted URL counts toward your site's crawl quota, and the same URL should not be resubmitted many times a day without a real change.
• Check who listens. As of September 2026 the FAQ lists endpoints for Bing, Yandex, Naver, Seznam.cz, Yep and Amazon. Google is not among them, so Google still learns about changes from links and sitemaps.
• Check your platform first. Many CMSs support IndexNow through plugins, and Cloudflare offers it natively, so you may not need to write code.

Read your server logs to see which bots crawl what

Webmaster consoles show you one engine's view. Your access logs show every bot that touched the site, including the ones with no console at all. Apache's Combined Log Format and nginx's predefined combined format, its default, both record the client IP, the time, the request line, the status code and the User-Agent header. That is enough to answer the three questions that matter.

# Requests per crawler (split on quotes, the User-Agent is field 6)
awk -F'"' 'match(tolower($6), /googlebot|bingbot|applebot|oai-searchbot|gptbot|chatgpt-user|claude-searchbot|claudebot|claude-user|perplexitybot|perplexity-user|ccbot/) { print substr(tolower($6), RSTART, RLENGTH) }' access.log | sort | uniq -c | sort -rn

# Status codes one crawler received
awk -F'"' '$6 ~ /bingbot/ { split($3, s, " "); print s[1] }' access.log | sort | uniq -c

# The paths GPTBot requests most
awk -F'"' '$6 ~ /GPTBot/ { split($2, r, " "); print r[2] }' access.log | sort | uniq -c | sort -rn | head -20

What the numbers usually mean:

A user agent proves nothing on its own. The reliable check is a reverse DNS lookup on the IP from your log, then a forward lookup on the name it returns: both must point at each other. Genuine Bingbot addresses resolve to names ending in search.msn.com, Applebot to applebot.apple.com and CCBot to crawl.commoncrawl.org. Apple's documentation uses this example:

host 17.58.101.179
host 17-58-101-179.applebot.apple.com

OpenAI, Perplexity and Anthropic publish their crawlers' IP addresses, so you can match against those lists. Googlebot's version of the check is in the Google guide. Reading logs alongside a crawler and Search Console is how our SEO service audits the crawl, render and index layers of a site.

A bot-friendly checklist for any search engine

These steps help every crawler, not one engine:

1. Serve the main content in the HTML. Bingbot and Applebot can render JavaScript, but CCBot does not run it and many bots do not say either way. Server-rendered or static HTML reaches all of them.
2. Link every page you want found from pages that are already crawled, with ordinary <a href> links, so it enters the frontier.
3. Publish an XML sitemap of canonical URLs with accurate lastmod dates, and name it in robots.txt with a Sitemap: line so every crawler can find it.
4. Keep robots.txt short and deliberate. Block only what has no search value (carts, internal search results, endless filters), never the CSS and JavaScript your pages need to render, and make sure the file answers 200 or 404: a 5xx tells compliant crawlers to stop.
5. Return honest status codes. 200 for real pages, 301 for moved ones, 404 or 410 for removed ones, and 429 or 503 with Retry-After when crawlers must slow down.
6. Keep one URL per piece of content. Redirect duplicates, declare rel="canonical" and keep tracking parameters out of internal links.
7. Support conditional requests. Send ETag or Last-Modified so unchanged pages can be answered with a 304.
8. Notify engines on change with IndexNow or your CMS's integration, and keep the sitemap current for everyone else.
9. Set your AI policy per token, not with one blanket rule, and repeat it on every subdomain.
10. Read the logs every month: requests per bot, status codes per bot, and important paths no bot has requested.

Web crawling and indexing software you can run yourself

The same parts are available as open-source software, for site search, an internal document search or a crawl of your own:

For the search box on your own website you rarely need a crawler at all. Your CMS or build step already knows every page and can feed the index directly, without waiting for a recrawl. A crawler earns its place when the content lives on sites you do not control.