# What is web crawling and indexing? How crawlers and search indexes work

> Crawling is software fetching pages by following links; indexing stores their words in an inverted index so search engines and AI tools can find them.

- URL: https://computese.com/web-crawling-and-indexing-explained/
- Author: Duong Quan Nguyen, CEO, Computese
- Published: 2024-11-14
- Updated: 2026-09-25
- Topics: SEO

## In short
- Crawling is a program fetching pages and following their links; indexing is storing what it found, mainly as an inverted index that maps each word to the pages containing it, so a search never has to reread the web.
- A well-behaved crawler reads robots.txt (RFC 9309) before fetching, keeps one connection per host with a pause between requests, skips duplicates and revisits pages about as often as they change.
- Each bot answers to its own robots.txt token, and training, search and user-requested fetches are separate: blocking GPTBot, ClaudeBot or Google-Extended does not take you out of ChatGPT search, Claude's search or Google Search.
- robots.txt is a request, not access control. Bing, Common Crawl and Anthropic honour Crawl-delay, Google and Apple ignore it, and some user-triggered fetchers may not follow robots.txt at all.
- Your server logs are the ground truth: count requests per bot, check the status codes each one receives, and verify the big crawlers by reverse DNS or published IP lists before trusting a user agent.

Web crawling is software, called a crawler, spider or bot, downloading web pages and following their links to find more. Indexing is what happens next: the text is split into words, and each word is recorded with the pages that contain it, in an inverted index, so a search finds matching pages without rereading the web.

Search engines, web archives and AI companies all run crawlers, and they behave differently. This guide covers how a crawler works as software, what an index stores, how Bingbot, Common Crawl and AI crawlers such as GPTBot, ClaudeBot and PerplexityBot differ and how to control each one, then gives an engine-neutral checklist and a way to read your server logs. For Google's own pipeline (Googlebot, rendering, canonicals and Search Console), read [how the Google search engine works](https://computese.com/how-google-search-engine-works/).

## Crawling and indexing: what each step does

The two words travel together, but they are separate jobs, often run by separate systems, and a page can pass one and fail the other.

| Question           | Crawling                                                | Indexing                                                    |
| ------------------ | ------------------------------------------------------- | ----------------------------------------------------------- |
| What it answers    | Which pages exist, and what do they contain right now?  | Which pages match this word or phrase?                      |
| What does the work | A crawler requesting URLs over HTTP                     | An indexer turning fetched pages into terms                 |
| What it produces   | A store of downloaded pages and a list of new links     | An inverted index, plus stored data about each page         |
| What you control   | robots.txt, links, sitemaps, status codes, server speed | `noindex`, duplicate URLs, the content itself               |
| How it goes wrong  | Blocked, unreachable, too slow, or never linked         | Fetched, but treated as a duplicate, excluded or not useful |

The order matters. A page has to be fetched before it can be indexed, so the crawl is the first place to look when a page is missing from search results. The controls are separate too. Blocking a URL in robots.txt stops the engine reading it, not listing it; to keep a page out of results you use `noindex`, and the engine has to be allowed to fetch the page to see that rule. Bing's [webmaster guidelines](https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a) put it plainly: robots.txt controls crawl access, not indexing.

## How a web crawler works, step by step

Every crawler, from a search engine's fleet to a small script, runs the same loop. The textbook description in [Introduction to Information Retrieval](https://nlp.stanford.edu/IR-book/html/htmledition/crawling-1.html) (Manning, Raghavan and Schütze, Cambridge University Press) follows Mercator, a crawler design that became the basis of several research and commercial crawlers:

1. **Start from seeds.** The crawl begins with a seed set of known URLs. For a search engine, that is every URL it already knows about, plus sitemaps and URLs sites submit.
2. **Take the next URL from the frontier.** The URL frontier holds every URL not yet fetched, in priority order.
3. **Check robots.txt, then fetch.** The crawler [resolves the host name with DNS](https://nlp.stanford.edu/IR-book/html/htmledition/crawler-architecture-1.html), checks the host's robots.txt rules and downloads the page over HTTP.
4. **Parse.** It extracts the text for the indexer, and the links with their anchor text.
5. **Filter and deduplicate.** Each link is made absolute, run through URL filters and dropped if it is already queued or was fetched recently. A page whose content was already seen at another URL is flagged as a copy.
6. **Queue and repeat.** New URLs join the frontier with a priority. In continuous crawling, the fetched URL goes back in as well, to be fetched again later.

The loop is simple; the scale is not. The same textbook works out that fetching a billion pages in a month means several hundred fetches every second, sustained, while staying polite to every site, surviving malformed pages and avoiding traps. The parts that matter most to a site owner are the frontier, duplicate detection and the recrawl schedule.

### The URL frontier decides what is fetched next

The frontier is where a crawler's judgement lives. It orders URLs on [two considerations](https://nlp.stanford.edu/IR-book/html/htmledition/the-url-frontier-1.html). The first is priority: pages that are useful and change often should be fetched often, so priority combines a quality estimate with how often the page has changed between past visits. Change alone is not enough, because many spam pages change completely on every fetch. The second is politeness.

Many links on a page point to the same host, so a plain priority queue would fire a burst of requests at one server. Mercator-style frontiers prevent that by design: a set of back queues, each holding URLs from a single host, a record of when each host may next be contacted, and one open connection per host at a time. A common heuristic sets the wait before the next request to about ten times as long as the last fetch from that host took, so a server that slows down automatically receives fewer requests.

![Link cards pour from a sorting box into three lanes, one per server. A stopwatch on each lane spaces out the requests; the orange one has just let a single card travel to its server.](https://computese.com/images/blog/web-crawling-and-indexing-explained/frontier.29a2a51d08-1536.webp)

*Politeness is built into the queue: every server gets its own lane, and the next request waits for the timer.*

Open-source crawlers ship with the same instincts. A new [Scrapy](https://docs.scrapy.org/en/latest/topics/settings.html) project waits one second between requests to the same domain (`DOWNLOAD_DELAY`) and obeys robots.txt (`ROBOTSTXT_OBEY`) out of the box. Sites can push back in two ways:

- **HTTP 429 Too Many Requests**, defined in [RFC 6585](https://www.rfc-editor.org/rfc/rfc6585.html), optionally with a `Retry-After` header saying how long to wait. A 503 can carry `Retry-After` too. Common Crawl's crawler uses an adaptive back-off when a site answers 429 or 5xx, and Apple says Applebot's crawl rate adjusts automatically when a site slows down or returns errors.
- **`Crawl-delay` in robots.txt**, a non-standard line asking for a pause, in seconds, between requests. Support varies by crawler (see the table below). [Bing](https://www.bing.com/webmasters/help/how-to-report-an-issue-with-bingbot-25c19802) reads it as one URL per window of 1 to 20 seconds, so `Crawl-delay: 5` caps Bingbot at 17,280 URLs a day.

That cap is why a delay is a blunt tool: at 17,280 URLs a day, a site with 100,000 URLs needs almost six days for Bingbot to fetch each page once. Bing's own help [recommends against it](https://www.bing.com/webmasters/help/crawl-error-alerts-e29a3f3e) and points to Crawl Control in Bing Webmaster Tools, which sets a crawl rate for each hour of the day instead.

### robots.txt: the rules a crawler reads before fetching

robots.txt is where a site states its rules for bots. Martijn Koster defined the Robots Exclusion Protocol in 1994, and the IETF published it as [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), a standards-track document, in September 2022. What the standard says, and what a compliant crawler does:

- **Location.** The file lives at `/robots.txt` at the top of each host, UTF-8 encoded, as plain text. The file on `www.example.com` says nothing about `shop.example.com`.
- **Groups.** Rules sit under one or more `User-agent` lines. A crawler looks for a group naming its product token, matching case-insensitively; if none exists, it obeys the `*` group; if there is no `*` group either, nothing is disallowed.
- **Matching.** The most specific rule, the one with the most characters, wins. If an `Allow` and a `Disallow` rule are equally specific, `Allow` wins. `*` matches any run of characters and `$` marks the end of a path.
- **Errors.** If robots.txt answers with a 4xx status, the crawler may fetch anything. If it answers with a 5xx or the server is unreachable, the crawler must assume everything is disallowed.
- **Caching and size.** Crawlers should not use a cached copy for more than 24 hours, and must parse at least the first 500 KiB.
- **Not security.** The RFC states that the rules are not a form of access authorization, and that listing a path in robots.txt makes it public. Protect private areas with authentication.

```text
# Rules for every crawler without its own group
User-agent: *
Disallow: /cart/
Disallow: /search/
Allow: /search/help/

Sitemap: https://www.example.com/sitemap.xml
```

In this file, `/search/help/` stays crawlable because `Allow: /search/help/` is the longer, more specific match. Anything beyond `User-agent`, `Allow` and `Disallow` is an extension a crawler may or may not read. `Sitemap` is the widely supported one. `Crawl-delay` is not in the standard, which is why [Google ignores it](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec) while Bing honours it. Google's own reading of the file, including how it treats errors, is covered in [how Google crawls a page](https://computese.com/how-google-search-engine-works/).

### Duplicates and spider traps

The web is full of copies: the same article with and without tracking parameters, print versions, mirrors, syndicated posts. The textbook cites estimates that as many as 40% of web pages duplicate other pages. Crawlers and indexers deal with them at two levels:

- **Duplicate URLs.** Before a link enters the frontier, it is normalized (a relative link becomes an absolute URL) and checked against the URLs already queued or fetched.
- **Duplicate content.** After a fetch, the page gets a fingerprint, a short digest of its content, and pages with equal fingerprints are copies. Exact fingerprints miss near-duplicates, such as two pages identical except for a timestamp, so engines use [shingling](https://nlp.stanford.edu/IR-book/html/htmledition/near-duplicates-and-shingling-1.html): they compare the sets of overlapping four-word runs in two pages, and pages above a similarity threshold are indexed once.

Your side of the deal is one URL per piece of content, with redirects and `rel="canonical"` saying which URL you prefer. Google's canonical rules are in [the Google guide](https://computese.com/how-google-search-engine-works/).

Crawlers also have to survive [spider traps](https://nlp.stanford.edu/IR-book/html/htmledition/features-a-crawler-must-provide-1.html): URL spaces that never end, such as a calendar with a "next month" link forever, or filters and sort orders that multiply into millions of URLs. Some traps are built to mislead crawlers; many are accidents of site development. A crawler defends itself with URL filters and limits per host. For your site, the cost is that its attention goes to worthless URLs instead of your real pages.

### Recrawling keeps the copy fresh

An index is a snapshot, and the web keeps changing. A continuous crawler puts fetched URLs back in the frontier and tries to [revisit each page](https://nlp.stanford.edu/IR-book/html/htmledition/features-a-crawler-should-provide-1.html) at a rate that approximates how often it changes, so a news home page is fetched far more often than a policy page last edited years ago. Three mechanisms make revisits cheaper for both sides:

- **Conditional requests.** The crawler sends `If-None-Match` with the page's last `ETag`, or `If-Modified-Since` with its `Last-Modified` date. If nothing changed, the server answers `304 Not Modified` instead of sending the page again ([RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html)). Google's crawlers [support both headers](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers) and Common Crawl supports conditional GET.
- **Sitemap `lastmod`.** The [Sitemaps protocol](https://www.sitemaps.org/protocol.html) lets you state when each URL last changed. It describes `changefreq` as a hint that crawlers may weigh, not a command they follow.
- **Push notifications.** IndexNow tells participating engines the moment a URL changes (covered below).

## What a search index is and what it stores

A search engine does not search the web when you type a query. It searches its index, a data structure built in advance from the crawled pages. The core of it is the inverted index: instead of listing the words on each page, it lists, for each word, the pages that contain it, the way the index at the back of a book lists pages for each topic.

Building one takes [four steps](https://nlp.stanford.edu/IR-book/html/htmledition/a-first-take-at-building-an-inverted-index-1.html):

1. **Collect the documents.** The crawler has done this.
2. **Tokenize each one.** Tokenization chops the text into tokens, usually words, and throws away characters such as punctuation.
3. **Normalize the tokens into terms.** Lowercasing makes `Quick` match `quick`. [Stemming reduces](https://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html) `foxes` to `fox`, and `organizes` and `organizing` to one form. Synonym rules can map `leap` to `jump`. The same [text analysis](https://www.elastic.co/docs/manage-data/data-store/text-analysis) runs on every query, so a search for "Foxes leap" looks up `fox` and `jump`.
4. **Invert.** Sort the list of term and document pairs, merge repeats, and split the result into a dictionary of terms and, for each term, a postings list of the documents that contain it.

Three one-line pages show the result. Page 1 reads "Crawlers fetch pages", page 2 "Indexes store pages" and page 3 "Crawlers respect robots.txt":

| Term       | Postings (page: word position) |
| ---------- | ------------------------------ |
| crawler    | 1: 1 · 3: 1                    |
| fetch      | 1: 2                           |
| index      | 2: 1                           |
| page       | 1: 3 · 2: 3                    |
| respect    | 3: 2                           |
| robots.txt | 3: 3                           |
| store      | 2: 2                           |

A query for "crawler pages" looks up two postings lists and [intersects them](https://nlp.stanford.edu/IR-book/html/htmledition/processing-boolean-queries-1.html): only page 1 is on both, and no page had to be read to find it. Storing [positions](https://nlp.stanford.edu/IR-book/html/htmledition/positional-indexes-1.html) lets the engine answer phrase queries by checking that two terms sit side by side, and postings usually record how often each term occurs, which ranking formulas use.

![Three pages of grey text bars feed a tall index box. In two pages the same word is circled; its row in the index is orange and points back to those two pages, while a magnifying glass looks it up.](https://computese.com/images/blog/web-crawling-and-indexing-explained/inverted.402aae30b2-1536.webp)

*An inverted index answers a query by looking up the word, not by reading every page.*

Older systems dropped very common words such as "the" and "of" from the index. The textbook notes that web search engines generally [do not use stop lists](https://nlp.stanford.edu/IR-book/html/htmledition/dropping-common-terms-stop-words-1.html): titles such as "To be or not to be" consist entirely of them.

### What else an index keeps about a page

The word index is only part of the record. Depending on the engine, it also holds:

- the page's title, description, language and other metadata;
- its outgoing links with their anchor text, which the crawler passes to the indexer for use in ranking;
- fetch details: the status code, when the page was fetched, and the `ETag` or date needed for the next conditional request;
- the content fingerprint used to spot copies, and which URL was chosen to represent a set of duplicates.

At web scale the index is split across many machines. The common approach is to [partition by document](https://nlp.stanford.edu/IR-book/html/htmledition/distributing-indexes-1.html): each machine indexes a subset of pages, every query goes to all of them, and their top results are merged. When someone searches, the engine runs the query through the same tokenization, fetches and combines the postings, then ranks the matches. Ranking is where engines differ most; for Google's approach, see [how Google ranks results](https://computese.com/how-google-search-engine-works/). [AI assistants with search](https://computese.com/ais-impact-on-seo-strategies/) work from an index too: Bing's guidelines state that its Copilot experiences rely on the same crawling, indexing and ranking foundation as classic Bing search. For Google's AI answers, see [the latest SEO trends](https://computese.com/latest-seo-trends-2024-boost-your-website-visibility/).

## How search engines and AI bots crawl differently

Every major operator documents its crawlers, and the documentation shows four kinds of bot that are easy to confuse:

- **Search crawlers** fetch pages for a search index: Googlebot, Bingbot, Applebot, OAI-SearchBot, Claude-SearchBot and PerplexityBot.
- **Training crawlers** collect pages that may be used to train AI models: GPTBot and ClaudeBot. Google and Apple use control tokens instead of separate crawlers: Google-Extended and Applebot-Extended.
- **User-triggered fetchers** load a page because a person asked an assistant about it: ChatGPT-User, Claude-User and Perplexity-User.
- **Archive crawlers** build public datasets: Common Crawl's CCBot.

| Bot               | Operator     | What it is for                                       | What to know                                                                |
| ----------------- | ------------ | ---------------------------------------------------- | --------------------------------------------------------------------------- |
| Googlebot         | Google       | Google Search                                        | Ignores `Crawl-delay`; details in the Google guide                          |
| Bingbot           | Microsoft    | Bing search, Copilot and Bing's grounding API        | Honours `Crawl-delay`; renders pages with the current stable Microsoft Edge |
| Applebot          | Apple        | Spotlight, Siri and Safari; may train Apple models   | Ignores `Crawl-delay`; follows Googlebot rules if no Applebot group exists  |
| CCBot             | Common Crawl | Open web archive                                     | Honours `Crawl-delay`; does not run JavaScript                              |
| GPTBot            | OpenAI       | Crawling that may be used to train OpenAI models     | Independent of OAI-SearchBot                                                |
| OAI-SearchBot     | OpenAI       | Pages shown in ChatGPT search                        | Opted-out sites can still appear as navigational links                      |
| ChatGPT-User      | OpenAI       | Pages a ChatGPT user asks for                        | OpenAI says robots.txt rules may not apply                                  |
| ClaudeBot         | Anthropic    | Content that may be used to train Anthropic's models | Honours `Crawl-delay`                                                       |
| Claude-SearchBot  | Anthropic    | Search results in Claude                             | Independent of ClaudeBot                                                    |
| Claude-User       | Anthropic    | Pages a Claude user asks for                         | Can be disallowed in robots.txt                                             |
| PerplexityBot     | Perplexity   | Perplexity search results, not model training        | Perplexity publishes its IP ranges                                          |
| Perplexity-User   | Perplexity   | Pages a Perplexity user asks for                     | Perplexity says it generally ignores robots.txt                             |
| Google-Extended   | Google       | Token only: Gemini training and grounding            | No crawler of its own; no effect on Google Search                           |
| Applebot-Extended | Apple        | Token only: training of Apple's foundation models    | Disallowing it does not stop Applebot crawling for search                   |

Sources for each row: [Bing](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0), [Apple](https://support.apple.com/en-us/119829), [Common Crawl](https://commoncrawl.org/faq), [OpenAI](https://developers.openai.com/api/docs/bots), [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), [Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) and [Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers), as documented in September 2026.

**Bing** also accepts IndexNow pushes (below), and its guidelines define two meta directives for AI use: `noarchive` keeps a page's content out of Copilot answers and grounding results, and `nocache` limits Copilot to the URL, title and snippet.

**Common Crawl** is a non-profit, founded in 2007, that crawls the web and publishes the result as a free, open archive. As of September 2026, [its site](https://commoncrawl.org/) describes more than 300 billion pages collected over 15 years, with 3 to 5 billion new pages added each month, and says the archive has been cited in over 10,000 research papers. CCBot is built on Apache Nutch, checks robots.txt before fetching and samples each site rather than archiving it whole.

**AI crawlers** are split by purpose, which is what makes a precise policy possible (next section). Whether a crawler runs JavaScript is often not documented at all, so assume some bots see only the HTML your server sends.

## How to control each crawler with robots.txt

Because each bot answers to its own user-agent token, one robots.txt can set a policy per purpose. This example keeps every search crawler, opts out of AI training and the Common Crawl archive, and keeps all bots out of the cart:

```text
# Search engines and AI search
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /cart/

# AI model training and open archives
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

# Everyone else
User-agent: *
Disallow: /cart/

Sitemap: https://www.example.com/sitemap.xml
```

What decides whether a file like this works:

- **Tokens are independent.** OpenAI, Anthropic and Perplexity each describe their bots as separate settings. Google-Extended and Applebot-Extended change how Google and Apple may use what Googlebot and Applebot fetch; Google states that Google-Extended does not affect inclusion or ranking in Google Search.
- **Changes are not instant.** RFC 9309 lets crawlers cache robots.txt for up to 24 hours, and OpenAI and Perplexity both say their systems can take about a day to reflect an edit.
- **Every host needs its own file.** Anthropic's documentation reminds site owners to repeat an opt-out on every subdomain.
- **User-triggered fetchers play by different rules.** OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page. Anthropic lets you disallow Claude-User like its other bots.
- **Page-level rules still apply.** Bing and Apple support `noindex`. Apple also honours `nosnippet` to keep content out of the AI-generated answers in Siri and Search.

> [!IMPORTANT]
> robots.txt is a published request, not enforcement. User agents are easy to fake, as [Bing warns](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) and [Common Crawl](https://commoncrawl.org/ccbot) has seen with crawlers posing as CCBot. To actually stop a bot, block it at your firewall or CDN, and use the operator's published IP ranges to tell the real crawler from impostors. Anthropic notes that blocking its IP addresses can stop its bots reading your robots.txt, so it asks site owners to opt out in robots.txt instead.

WAF rules and rate limiting at the edge are part of our [hosting and maintenance](https://computese.com/services/hosting-maintenance/) service, for when a request is not enough.

A shared vocabulary is on its way. The IETF's [AI Preferences working group](https://datatracker.ietf.org/wg/aipref/about/) is standardizing a way to state preferences such as allowing search but not AI training, attached through a new `Content-Usage` rule in robots.txt or an HTTP header of the same name. As of September 2026 both documents are still [Internet-Drafts](https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/), so each company's own token is what its crawlers read today.

## Push instead of waiting: how IndexNow works

Crawling is pull: the engine decides when to come back. IndexNow adds push. Microsoft Bing and Yandex [created the protocol](https://blogs.bing.com/webmaster/2021/10/IndexNow-Instantly-Index-your-web-content-in-Search-Engines/) in October 2021 so that a site can tell search engines the moment a URL is added, updated or deleted. The [protocol](https://www.indexnow.org/documentation) takes three steps:

1. **Publish a key.** Generate a key of 8 to 128 characters (letters, digits and hyphens) and host it as a UTF-8 text file named after the key, preferably at the root: `https://www.example.com/<key>.txt`.
2. **Send changed URLs to one endpoint**, either one URL as a GET request with `url` and `key` parameters or a JSON POST of up to 10,000 URLs.
3. **Let the engines share it.** The receiving engine fetches your key file to confirm you control the host, and participating engines agree to share submitted URLs with each other.

```http
POST /indexnow HTTP/1.1
Host: api.indexnow.org
Content-Type: application/json; charset=utf-8

{
  "host": "www.example.com",
  "key": "4f1c9e7a2b8d4e6f",
  "urlList": [
    "https://www.example.com/new-page/",
    "https://www.example.com/updated-page/"
  ]
}
```

![A web server sends one orange envelope to a search engine box, which passes the notice along dotted arrows to three other engines. A dashed arrow then returns from the first engine to fetch the page.](https://computese.com/images/blog/web-crawling-and-indexing-explained/indexnow.8c178f1990-1536.webp)

*IndexNow turns discovery around: the site announces a change once, and every participating engine hears about it.*

Where it fits, according to the [IndexNow FAQ](https://www.indexnow.org/faq):

- **Use it with a sitemap, not instead of one.** IndexNow is for new and changed pages; the sitemap remains the complete inventory.
- **A 200 response means received, not indexed.** A 202 means the key is still being checked. Every submitted URL counts toward your site's crawl quota, and the same URL should not be resubmitted many times a day without a real change.
- **Check who listens.** As of September 2026 the FAQ lists endpoints for Bing, Yandex, Naver, Seznam.cz, Yep and Amazon. Google is not among them, so Google still learns about changes from links and sitemaps.
- **Check your platform first.** Many CMSs support IndexNow through plugins, and Cloudflare offers it natively, so you may not need to write code.

## Read your server logs to see which bots crawl what

Webmaster consoles show you one engine's view. Your access logs show every bot that touched the site, including the ones with no console at all. Apache's [Combined Log Format](https://httpd.apache.org/docs/2.4/logs.html) and nginx's predefined [`combined` format](https://nginx.org/en/docs/http/ngx_http_log_module.html), its default, both record the client IP, the time, the request line, the status code and the User-Agent header. That is enough to answer the three questions that matter.

```bash
# Requests per crawler (split on quotes, the User-Agent is field 6)
awk -F'"' 'match(tolower($6), /googlebot|bingbot|applebot|oai-searchbot|gptbot|chatgpt-user|claude-searchbot|claudebot|claude-user|perplexitybot|perplexity-user|ccbot/) { print substr(tolower($6), RSTART, RLENGTH) }' access.log | sort | uniq -c | sort -rn

# Status codes one crawler received
awk -F'"' '$6 ~ /bingbot/ { split($3, s, " "); print s[1] }' access.log | sort | uniq -c

# The paths GPTBot requests most
awk -F'"' '$6 ~ /GPTBot/ { split($2, r, " "); print r[2] }' access.log | sort | uniq -c | sort -rn | head -20
```

What the numbers usually mean:

| What you see in the logs                                | What it usually means                          | What to do                                                    |
| ------------------------------------------------------- | ---------------------------------------------- | ------------------------------------------------------------- |
| Most bot requests go to parameter, sort or date URLs    | A spider trap or faceted navigation            | Link to clean URLs; disallow worthless patterns in robots.txt |
| Many 404 or 410 responses for one bot                   | Old links or an outdated sitemap               | Fix internal links, redirect moved pages, update the sitemap  |
| Frequent 429 or 5xx responses                           | The server is shedding load; crawlers back off | Add capacity or caching before blocking bots                  |
| 304 responses on revisits                               | Conditional requests are working               | Nothing: this saves bandwidth on both sides                   |
| Important pages never requested                         | Not linked, or not in the sitemap              | Link them from crawled pages, list them, notify IndexNow      |
| A search engine's user agent from an unexpected network | A fake bot                                     | Verify the IP before allowing or blocking it                  |

A user agent proves nothing on its own. The reliable check is a reverse DNS lookup on the IP from your log, then a forward lookup on the name it returns: both must point at each other. Genuine [Bingbot](https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26) addresses resolve to names ending in `search.msn.com`, Applebot to `applebot.apple.com` and CCBot to `crawl.commoncrawl.org`. Apple's documentation uses this example:

```bash
host 17.58.101.179
host 17-58-101-179.applebot.apple.com
```

OpenAI, Perplexity and Anthropic publish their crawlers' IP addresses, so you can match against those lists. Googlebot's version of the check is in [the Google guide](https://computese.com/how-google-search-engine-works/). Reading logs alongside a crawler and Search Console is how our [SEO service](https://computese.com/services/seo/) audits the crawl, render and index layers of a site.

## A bot-friendly checklist for any search engine

These steps help every crawler, not one engine:

1. **Serve the main content in the HTML.** Bingbot and Applebot can render JavaScript, but CCBot does not run it and many bots do not say either way. Server-rendered or static HTML reaches all of them.
2. **Link every page you want found** from pages that are already crawled, with ordinary `<a href>` links, so it enters the frontier.
3. **Publish an XML sitemap** of canonical URLs with accurate `lastmod` dates, and name it in robots.txt with a `Sitemap:` line so every crawler can find it.
4. **Keep robots.txt short and deliberate.** Block only what has no search value (carts, internal search results, endless filters), never the CSS and JavaScript your pages need to render, and make sure the file answers 200 or 404: a 5xx tells compliant crawlers to stop.
5. **Return honest status codes.** 200 for real pages, 301 for moved ones, 404 or 410 for removed ones, and 429 or 503 with `Retry-After` when crawlers must slow down.
6. **Keep one URL per piece of content.** Redirect duplicates, declare `rel="canonical"` and keep tracking parameters out of internal links.
7. **Support conditional requests.** Send `ETag` or `Last-Modified` so unchanged pages can be answered with a 304.
8. **Notify engines on change** with IndexNow or your CMS's integration, and keep the sitemap current for everyone else.
9. **Set your AI policy per token,** not with one blanket rule, and repeat it on every subdomain.
10. **Read the logs every month:** requests per bot, status codes per bot, and important paths no bot has requested.

## Web crawling and indexing software you can run yourself

The same parts are available as open-source software, for site search, an internal document search or a crawl of your own:

| Software                                                                                                                       | What it is                                                                      | Typical use                              |
| ------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------- | ---------------------------------------- |
| [Apache Nutch](https://nutch.apache.org/)                                                                                      | An extensible, scalable, production web crawler built on Hadoop data structures | Large batch crawls; CCBot is based on it |
| [Heritrix](https://github.com/internetarchive/heritrix3)                                                                       | The Internet Archive's web-scale, archival-quality crawler                      | Web archiving                            |
| [Apache StormCrawler](https://stormcrawler.apache.org/)                                                                        | An SDK for low-latency, distributed crawlers on Apache Storm                    | Continuous crawling                      |
| [Scrapy](https://docs.scrapy.org/en/latest/intro/overview.html)                                                                | A Python framework for crawling sites and extracting structured data            | Focused crawls and scraping              |
| [Apache Lucene](https://lucene.apache.org/core/)                                                                               | A full-text search library in Java, with faceting and vector search             | Search built into an application         |
| [Solr](https://solr.apache.org/), [OpenSearch](https://opensearch.org/), [Elasticsearch](https://www.elastic.co/elasticsearch) | Search servers built on Lucene                                                  | Site, document and log search            |

For the search box on your own website you rarely need a crawler at all. Your CMS or build step already knows every page and can feed the index directly, without waiting for a recrawl. A crawler earns its place when the content lives on sites you do not control.

## Key terms
- **Web crawler**: A program that downloads web pages and follows their links to find more. Also called a spider or bot. Search engines, web archives and AI companies all run them.
- **URL frontier**: The crawler's queue of URLs waiting to be fetched, ordered by priority and split per host so that no single server receives a burst of requests.
- **Politeness**: The rules a crawler follows so it does not overload a site: one connection per host, a pause between requests, and backing off when the server answers 429 or 5xx.
- **robots.txt**: A plain text file at the root of each host that tells crawlers which paths they may fetch. Defined in 1994 and standardized as RFC 9309 in 2022. Compliance is voluntary.
- **Crawl-delay**: A non-standard robots.txt line asking a crawler to wait a number of seconds between requests. Bing, Common Crawl and Anthropic honour it; Google and Apple ignore it.
- **User-agent token**: The name a crawler answers to in robots.txt, such as Bingbot or GPTBot. Google-Extended and Applebot-Extended are tokens with no crawler of their own: they control how fetched content may be used.
- **Inverted index**: The core data structure of a search engine: a dictionary of terms, each pointing to a postings list of the documents, and often the positions, where the term occurs.
- **Tokenization**: Splitting text into tokens, usually words, before indexing. Normalization then lowercases and stems them, so a query matches different forms of the same word.
- **Near-duplicate detection**: Comparing pages by their shingles (overlapping runs of words) rather than exact checksums, so that copies differing only by a date stamp are indexed once.
- **IndexNow**: A push protocol created by Microsoft Bing and Yandex in 2021. A site notifies one participating engine that URLs changed, and that engine shares the notice with the others.

## Common questions

### What is web crawling and indexing in simple terms?

Crawling is a bot visiting pages and following their links, the way a reader follows every reference in a book. Indexing is building the index at the back of the book: recording which pages mention each word, so a search can jump straight to them. A page has to be crawled before it can be indexed, but not every crawled page is indexed.

### Is web crawling the same as web scraping?

They share the machinery but not the goal. A crawler discovers and fetches pages across a site or the web; a scraper extracts specific data, such as prices, from the pages it fetches. Frameworks like Scrapy do both, and the same politeness rules and robots.txt apply to either.

### How long does it take for a new page to be crawled and indexed?

There is no fixed time, and no engine guarantees indexing. The IndexNow project notes that discovery by crawling alone can take days or even weeks. Linking the page from pages that are already crawled, listing it in your sitemap with an accurate lastmod date and notifying IndexNow engines all shorten the wait.

### Can I block AI crawlers without disappearing from search?

Largely, yes, because the operators split their bots by purpose. Disallow the training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) and CCBot for the Common Crawl archive, and keep allowing Googlebot, Bingbot, Applebot, OAI-SearchBot, Claude-SearchBot and PerplexityBot. Fetches that a person triggers in ChatGPT or Perplexity may not follow robots.txt, so those need firewall rules if you must stop them.

### What is web indexing software?

Software that builds an index of documents and answers queries against it. Apache Lucene is the library most open-source search engines are built on, and Apache Solr, OpenSearch and Elasticsearch are search servers built on Lucene. On the crawling side, Apache Nutch, Heritrix, Apache StormCrawler and Scrapy fetch the pages that feed an index.

### Do all crawlers obey robots.txt?

No. RFC 9309 says its rules are not a form of access authorization, and following them is voluntary. The major search and AI crawlers document that they follow robots.txt, some user-triggered fetchers say they may not, and badly behaved bots ignore it or fake a known user agent. Anything private needs authentication, not a Disallow line.

## Sources
1. [Introduction to Information Retrieval, section 20.2: Crawling](https://nlp.stanford.edu/IR-book/html/htmledition/crawling-1.html), Cambridge University Press (Manning, Raghavan, Schütze)
2. [Introduction to Information Retrieval, section 20.2.1: Crawler architecture](https://nlp.stanford.edu/IR-book/html/htmledition/crawler-architecture-1.html), Cambridge University Press (Manning, Raghavan, Schütze)
3. [Introduction to Information Retrieval, section 20.2.3: The URL frontier](https://nlp.stanford.edu/IR-book/html/htmledition/the-url-frontier-1.html), Cambridge University Press (Manning, Raghavan, Schütze)
4. [Scrapy settings: DOWNLOAD_DELAY and ROBOTSTXT_OBEY](https://docs.scrapy.org/en/latest/topics/settings.html), Scrapy documentation
5. [RFC 6585: Additional HTTP Status Codes (429 Too Many Requests)](https://www.rfc-editor.org/rfc/rfc6585.html), IETF
6. [Common Crawl FAQ](https://commoncrawl.org/faq), Common Crawl
7. [About Applebot](https://support.apple.com/en-us/119829), Apple Support
8. [How do I report an issue with Bingbot?](https://www.bing.com/webmasters/help/how-to-report-an-issue-with-bingbot-25c19802), Bing Webmaster Tools
9. [Crawl error alerts](https://www.bing.com/webmasters/help/crawl-error-alerts-e29a3f3e), Bing Webmaster Tools
10. [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html), IETF
11. [Introduction to Information Retrieval, section 20.1.1: Features a crawler must provide](https://nlp.stanford.edu/IR-book/html/htmledition/features-a-crawler-must-provide-1.html), Cambridge University Press (Manning, Raghavan, Schütze)
12. [Introduction to Information Retrieval, section 19.6: Near-duplicates and shingling](https://nlp.stanford.edu/IR-book/html/htmledition/near-duplicates-and-shingling-1.html), Cambridge University Press (Manning, Raghavan, Schütze)
13. [Introduction to Information Retrieval, section 20.1.2: Features a crawler should provide](https://nlp.stanford.edu/IR-book/html/htmledition/features-a-crawler-should-provide-1.html), Cambridge University Press (Manning, Raghavan, Schütze)
14. [RFC 9110: HTTP Semantics](https://www.rfc-editor.org/rfc/rfc9110.html), IETF
15. [Overview of Google crawlers and fetchers: HTTP caching](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers), Google Crawling Infrastructure
16. [Sitemaps XML format](https://www.sitemaps.org/protocol.html), sitemaps.org
17. [Introduction to Information Retrieval, section 1.2: A first take at building an inverted index](https://nlp.stanford.edu/IR-book/html/htmledition/a-first-take-at-building-an-inverted-index-1.html), Cambridge University Press (Manning, Raghavan, Schütze)
18. [Text analysis](https://www.elastic.co/docs/manage-data/data-store/text-analysis), Elastic documentation
19. [Introduction to Information Retrieval, section 2.2.4: Stemming and lemmatization](https://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html), Cambridge University Press (Manning, Raghavan, Schütze)
20. [Introduction to Information Retrieval, section 1.3: Processing Boolean queries](https://nlp.stanford.edu/IR-book/html/htmledition/processing-boolean-queries-1.html), Cambridge University Press (Manning, Raghavan, Schütze)
21. [Introduction to Information Retrieval, section 2.4.2: Positional indexes](https://nlp.stanford.edu/IR-book/html/htmledition/positional-indexes-1.html), Cambridge University Press (Manning, Raghavan, Schütze)
22. [Introduction to Information Retrieval, section 2.2.2: Dropping common terms: stop words](https://nlp.stanford.edu/IR-book/html/htmledition/dropping-common-terms-stop-words-1.html), Cambridge University Press (Manning, Raghavan, Schütze)
23. [Introduction to Information Retrieval, section 20.3: Distributing indexes](https://nlp.stanford.edu/IR-book/html/htmledition/distributing-indexes-1.html), Cambridge University Press (Manning, Raghavan, Schütze)
24. [Bing Webmaster Guidelines](https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a), Bing Webmaster Tools
25. [Which crawlers does Bing use?](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0), Bing Webmaster Tools
26. [Google's common crawlers: Google-Extended](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers), Google Crawling Infrastructure
27. [How Google interprets the robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec), Google Crawling Infrastructure
28. [Common Crawl](https://commoncrawl.org/), Common Crawl
29. [CCBot](https://commoncrawl.org/ccbot), Common Crawl
30. [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots), OpenAI
31. [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), Anthropic (Claude Help Center)
32. [Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers), Perplexity
33. [AI Preferences (aipref) working group](https://datatracker.ietf.org/wg/aipref/about/), IETF Datatracker
34. [draft-ietf-aipref-attach: Associating AI usage preferences with content in HTTP](https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/), IETF Datatracker
35. [IndexNow: Instantly index your web content in search engines](https://blogs.bing.com/webmaster/2021/10/IndexNow-Instantly-Index-your-web-content-in-Search-Engines/), Microsoft Bing Blog
36. [IndexNow documentation](https://www.indexnow.org/documentation), IndexNow.org
37. [IndexNow FAQ](https://www.indexnow.org/faq), IndexNow.org
38. [Log files: Combined Log Format](https://httpd.apache.org/docs/2.4/logs.html), Apache HTTP Server Project
39. [Module ngx_http_log_module](https://nginx.org/en/docs/http/ngx_http_log_module.html), nginx
40. [How to verify Bingbot](https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26), Bing Webmaster Tools
41. [Apache Nutch](https://nutch.apache.org/), Apache Software Foundation
42. [Heritrix](https://github.com/internetarchive/heritrix3), Internet Archive
43. [Apache StormCrawler](https://stormcrawler.apache.org/), Apache Software Foundation
44. [Scrapy at a glance](https://docs.scrapy.org/en/latest/intro/overview.html), Scrapy documentation
45. [Apache Lucene Core](https://lucene.apache.org/core/), Apache Software Foundation
46. [Apache Solr](https://solr.apache.org/), Apache Software Foundation
47. [OpenSearch](https://opensearch.org/), OpenSearch Software Foundation
48. [Elasticsearch](https://www.elastic.co/elasticsearch), Elastic