How the Google search engine works: crawling, indexing and ranking

What the Google search engine is

Google Search is a fully automated search engine. Software called web crawlers explores the web regularly, and the pages it finds are added to Google's index. Google says the vast majority of pages in its results were not submitted by hand: the crawlers found them.

The index is large. According to Google's How Search Works site, it covers hundreds of billions of webpages and is well over 100,000,000 gigabytes in size, organized like the index at the back of a book, with an entry for every word seen on every indexed page. It is also not the only index. Google keeps others built from partnerships, data feeds and the Knowledge Graph, its database of facts about people, places and things, which is why a search can return a book, a transit time or a public dataset next to web pages.

Two rules are worth knowing before anything else:

• Nobody can pay for organic placement. Google does not accept payment to crawl a site more often or to rank it higher. Ads exist, but they are labelled "Sponsored" or "Ad" and shown separately from the results.
• Nothing is guaranteed. Google does not promise to crawl, index or serve any page, even one that follows all of its guidelines. The technical requirements (Googlebot is not blocked, the page returns HTTP 200, the content is indexable) make a page eligible, not included.

How Google Search works, step by step

Google's in-depth guide names three stages: crawling, indexing and serving search results. Two steps inside them matter enough to a site owner to name on their own, so this guide uses five:

1. Discover. Google keeps a list of URLs it knows about, fed by links on pages it has already seen and by sitemaps.
2. Crawl. Googlebot checks the site's robots.txt, requests the URL and downloads the response.
3. Render. Google runs the page's JavaScript in a recent version of Chrome, so it sees what a browser would show.
4. Index. Google analyzes the content, groups duplicates, picks one canonical URL and stores what it learned.
5. Serve. When someone searches, Google finds matching pages in the index, ranks them and builds a results page for that query, language, location and device.

A page can drop out at any step, and technical SEO starts with finding which one:

How Google discovers new pages

There is no central registry of web pages, so Google has to keep looking for them. It calls this URL discovery, and it works three ways. Some URLs are known because Googlebot has visited them before. Others are extracted from links on known pages, as when a category page links to a new blog post. The rest come from sitemaps that site owners submit. Links do most of the work: Googlebot finds new URLs primarily in pages it has already crawled, which is also why a site cannot be kept secret by not linking to it. One visitor clicking out from a "hidden" page can leak its address in another site's referrer log.

For a site owner, discovery comes down to two things.

Crawlable internal links. Every page you want in Google should be reachable from your navigation or from other pages. Google can generally crawl a link only if it is an <a> element with an href attribute. Links inserted with JavaScript work as long as they use that markup; an element that navigates only through a script event, such as <a onclick="..."> or a clickable <span>, may never be followed.

An XML sitemap, where it helps. A sitemap lists the URLs you want in search results. Google says a site of about 500 pages or fewer, linked well internally and without much video, image or news content, might not need one; large sites, new sites with few links from elsewhere and media-heavy sites benefit most. Google adds that most sites still benefit from one, and CMSs such as WordPress usually generate it for you. The rules that matter, from Google's sitemap guide:

• One sitemap holds at most 50,000 URLs or 50MB uncompressed. Larger sites split it and submit a sitemap index file.
• List absolute, canonical URLs, the ones you want shown in results.
• Google ignores <priority> and <changefreq>. It uses <lastmod> when the dates are consistently accurate, so change it only for significant updates to the content, structured data or links, not for a new copyright year.
• Submit it in Search Console's Sitemaps report, or add a Sitemap: line to robots.txt. The old sitemap "ping" URL no longer helps: Google deprecated it in June 2023, and requests to it now return a 404.

A sitemap is a hint. Google may not download it, and listing a URL does not guarantee it will be crawled or indexed.

How Googlebot crawls a page

Once a URL is on the list, Googlebot may fetch it. Googlebot is the generic name for two crawlers, Googlebot Smartphone and Googlebot Desktop. Because Google primarily indexes the mobile version of most sites, most requests come from the smartphone crawler. Both obey the same Googlebot token in robots.txt, so you cannot allow one and block the other.

Googlebot decides algorithmically which sites to crawl, how often and how many pages to fetch from each. It also paces itself by how your server responds: server errors tell it to slow down. For most sites, Google says Googlebot should not visit more than once every few seconds on average.

robots.txt decides what may be crawled

Before fetching a URL, Googlebot reads the host's robots.txt, a plain text file at the root of the site that lists which paths which crawlers may request. The format is the Robots Exclusion Protocol, first defined in 1994 and standardized by the IETF as RFC 9309 in September 2022. A typical file:

User-agent: *
Disallow: /cart/
Disallow: /internal-search/

Sitemap: https://example.com/sitemap.xml

Three details of Google's implementation, as documented in August 2026, catch people out:

• Google reads at most 500 KiB of the file and ignores the rest.
• It caches the file for up to 24 hours, so an edit can take a day to apply.
• A 4xx response (except 429) means "no restrictions". A 5xx response is worse: Google stops crawling the site for the first 12 hours while it retries, then uses the last good copy for up to 30 days.

Status codes tell Googlebot what happened

Google documents how its crawlers treat each HTTP status code:

Googlebot follows up to 10 redirect hops for web content. Among status codes, only 429 and 5xx slow it down: a 401, 403 or 404 has no effect on the crawl rate and gets the URL dropped instead.

Size limits, crawl budget and fake Googlebots

As of its Googlebot documentation (last updated February 2026), Google says Googlebot fetches the first 2MB of an HTML page or other supported file and the first 64MB of a PDF, measured uncompressed. CSS and JavaScript files are fetched separately under the same limit. Anything past the cutoff is never considered for indexing, so keep the main content early in the HTML and large inline data out of it.

Crawl budget is the set of URLs Google can and wants to crawl on one host. It combines crawl capacity, which rises while your server stays fast and falls when it slows down or returns 5xx or 429 errors, with crawl demand, which depends on how many URLs Google knows, how popular they are and how often they change. Google calls it a concern mainly for sites with roughly a million or more unique pages that change weekly, or 10,000 or more that change daily. For everyone else, a current sitemap and a regular look at the Page indexing report are enough. On large sites, the main waste is duplicate URLs from filters and sort orders: consolidate them, and block the ones with no search value in robots.txt.

Finally, other bots often send Googlebot's user agent. Before you block or allow-list it in a firewall, verify the request: a reverse DNS lookup on the IP from your logs should return a host under googlebot.com, google.com or googleusercontent.com, and a forward lookup on that host should return the same IP.

host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

How Google renders JavaScript pages

During the crawl, Google renders the page: it runs the JavaScript it finds in a recent, evergreen version of Chromium, the way your browser does. This matters because many sites build their content in the browser. An app-shell page can arrive as almost empty HTML, and without rendering Google might not see its content at all.

Rendering is queued. Googlebot puts every page that returns HTTP 200 in a render queue, whether or not it uses JavaScript. A page may wait there for a few seconds or longer; then a headless Chromium renders it, Google parses the rendered HTML for more links and uses it to index the page. Pages that return an error status may skip rendering.

What follows for a site owner:

• Put the main content in the server HTML when you can. Google still recommends server-side rendering or pre-rendering: it makes pages faster for users and crawlers, and not all bots run JavaScript.
• Do not block the files a page needs. Google will not render JavaScript from files or pages blocked in robots.txt, and blocking the CSS or scripts a page depends on makes it harder for Google to understand the page.
• Do not rely on JavaScript to remove noindex. When Google finds noindex in the original HTML it may skip rendering, so a script that removes the tag may never run for Google.
• Return real status codes. A single-page app that shows "not found" with HTTP 200 creates soft 404s. Redirect to a URL that returns a 404 from the server, or add noindex to error views.
• Fingerprint file names. Googlebot caches aggressively and its renderer may ignore your caching headers, so a name like main.2bb85551.js makes sure it renders with your current code.

If your templates depend on client-side rendering for their main content, fixing that is usually a front-end change as much as an SEO one. Our web design and development team builds front ends rendered on the server and at the edge.

How Google indexes a page and chooses a canonical

After a page is crawled, Google tries to understand what it is about. It processes the text and key tags and attributes, such as the <title> element and image alt attributes, along with images and video. It also collects signals it may use when serving results, such as the page's language, the country the content is local to and its usability.

The step most site owners never see is canonicalization. The same content often lives at several URLs: the HTTP and HTTPS versions, a desktop and a mobile version, a category page sorted and filtered five ways, a demo copy left open to crawlers. Google clusters pages whose primary content is the same or very similar, then marks the one it judges most complete and useful as the canonical. The canonical is crawled most often and is usually the URL shown in results; the duplicates are crawled less and may be shown in specific contexts, such as a mobile URL to someone on a phone.

What Google keeps ends up in the Google index, a large database hosted on thousands of computers. Indexing is not guaranteed either. Google names low-quality content, robots meta rules that disallow indexing and site designs that make indexing difficult as common reasons a crawled page stays out.

Google also indexes most sites from their mobile version, crawled with Googlebot Smartphone. This is mobile-first indexing: if your mobile page carries less content than the desktop page, Google gets less information and you can expect to lose traffic. Google recommends responsive design, the same HTML at the same URL on every device, as the easiest pattern to implement and maintain.

Tell Google which URL is canonical

Google lists three methods, in order of how strongly they influence its choice:

1. Redirects, a strong signal that the target should be canonical. Use a 301 or 308 for permanent moves, including HTTP to HTTPS.
2. rel="canonical", also a strong signal: a <link> element in the page's <head>, or an HTTP Link header for PDFs and other non-HTML files.
3. Sitemap inclusion, a weak signal.

<link rel="canonical" href="https://example.com/shoes/green-trainer/" />

The methods stack, so combining them helps, but each one is a hint, not a rule: Google may still choose a different canonical. From Google's canonical guide, put a self-referencing canonical on the canonical page, link internally to the canonical URL, never name different canonicals in different places, and do not use robots.txt or noindex to choose between duplicates. Google prefers the HTTPS version as canonical unless its certificate is invalid, it loads insecure resources other than images or it redirects through HTTP, so an expired certificate can cost a page more than a browser warning (renewing an SSL certificate covers that side).

Keep a page out of the index with noindex

To keep a page out of Google, use noindex, either as a robots meta tag or as an X-Robots-Tag response header, which also works for PDFs, images and video files:

<meta name="robots" content="noindex" />

When Googlebot crawls the page and sees the rule, Google drops it from results entirely, whoever links to it. The page must not be blocked in robots.txt, or Google never sees the rule, and a noindex line inside robots.txt is not supported.

How Google ranks and serves results

When someone searches, Google looks up matching pages in the index and returns the results it considers highest quality and most relevant. Ranking is automated. Google says hundreds of factors can play a part, including the searcher's location, language and device, so the same search for a repair shop would return different results in Paris and in Hong Kong. The weight of each factor depends on the query: freshness counts more for a news topic than for a dictionary definition.

Google's How Search Works site groups the key signals into five kinds:

Google also documents the ranking systems behind these signals. BERT, RankBrain and neural matching help it understand how words relate to concepts and intent. Passage ranking lets it judge individual sections of a long page. Freshness systems favour recent content where the query calls for it, and a site diversity system generally shows no more than two listings from the same site in the top results. Names that still circulate in older SEO articles are history: Panda (2011) and Penguin (2012) were folded into Google's core ranking systems in 2015 and 2016, and the helpful content system followed in March 2024.

Features of the Google results page

The results page itself is built per query. Google calls its building blocks visual elements, and the common ones are:

• Text results, the classic listing: a title link, a snippet and an attribution line with the site name, favicon and a readable URL, sometimes with sitelinks or a date.
• Rich results, which add graphical or interactive elements such as review stars or recipe details, usually from structured data in the page.
• Image and video results, more likely for queries that seek images or video.
• Exploration features, such as "People also ask", that help people refine the search.
• Knowledge panels, information boxes for people, places, organizations and things in the Knowledge Graph, generated automatically from sources across the web.
• Local results, for queries with local intent.
• AI Overviews and AI Mode, covered in the next section.

Which elements appear depends on the query: a search for bicycle repair shops will likely show local results and no images, while a search for a bicycle style will likely show images and no local results. You influence the text result through the page itself. Google builds title links from the <title> element, the main visible title, headings and other sources, and it creates snippets mainly from the page content, using the meta description when that describes the page better. Rich results usually depend on structured data, which should match what the page visibly shows.

Where AI Overviews and AI Mode fit

AI Overviews and AI Mode are part of Google Search, and their supporting links come from the same index. Google's documentation for site owners (last updated December 2025) is short:

• To be shown as a supporting link, a page must be indexed and eligible to appear in Search with a snippet. There are no additional technical requirements, no special schema.org markup and no new machine-readable or AI text files to create.
• Both features may use a "query fan-out" technique: issuing several related searches across subtopics and sources to build a response, which lets them show a wider set of links than a classic results page.
• AI Overviews appear only when Google's systems judge them useful beyond the classic results, and often do not trigger.
• The controls are the ones you already have. robots.txt rules for Googlebot govern crawling for Search, AI features included, and nosnippet, data-nosnippet, max-snippet and noindex limit what is shown. Google-Extended is a separate control for AI training and grounding in some of Google's other systems.
• Clicks and impressions from AI features are counted in Search Console's Performance report, under the "Web" search type.

In other words, the path into an AI answer on Google is the path this guide describes: be discoverable, crawlable, renderable and indexed, and be the most useful page for the question.

Check what Google sees with Search Console

Search Console is Google's free toolset for site owners. Verify your site, then use two tools before anything else.

The URL Inspection tool shows one URL two ways. The indexed view reports whether the URL is on Google, when it was last crawled, and the Google-selected canonical next to the canonical you declared. The live test fetches the page now and shows the rendered HTML, a screenshot, the response headers and the resources that loaded. It cannot tell whether the page will be treated as a duplicate. After fixing a page you can click Request indexing, within a daily quota; asking again does not make it faster, and Google says crawling can take from a few days to a few weeks. For many pages, submit a sitemap with accurate <lastmod> dates instead.

The Page indexing report lists every URL Google knows in your property and why each one is or is not indexed. For a site under 500 pages, Google suggests starting with a search such as site:example.com and opening the report only if key pages are missing. The statuses you will meet most often:

Check the Crawl Stats report for server errors and slow responses, and the Performance report for the queries each page appears for. If whole sections of a site sit in the excluded rows and the reason is not obvious, that is where our SEO service starts: the crawl, render and index layers, audited with a crawler, your server logs and Search Console together.