Technical SEO: what it is and how to audit crawling, rendering and indexing

What technical SEO covers, and where it stops

Google Search works in three stages, crawling, indexing and serving, and not every page makes it through each one. Pages built with JavaScript add a step between the first two: rendering. Technical SEO owns everything up to the index, plus the page-level signals that are engineering problems rather than writing problems: HTTPS, loading speed and valid structured data. Titles, headings and the copy itself are on-page work, covered in on-page SEO.

Google publishes the floor. A page is eligible for indexing when it meets three technical requirements: Googlebot is not blocked from it, it returns HTTP 200, and it has indexable content. Meeting them does not guarantee indexing. The Bing Webmaster Guidelines rest on the same crawl and index foundation, and say the same practices also support a URL's eligibility for Copilot answers and citations. The sections below follow a URL through discovery and crawling, rendering and indexing, then cover the page-level checks and site migrations.

Control crawling with robots.txt and indexing with noindex

These are two different switches. robots.txt decides what a crawler may fetch. Google's introduction to robots.txt says it exists mainly to avoid overloading your site and is not a mechanism for keeping a page out of Google: a disallowed URL can still be indexed from links pointing to it, and then shows in results without a description. The same page warns against blocking CSS and JavaScript files that Google needs to understand the page.

To keep a page out of the index, let it be crawled and serve a noindex rule, either as a robots meta tag or as an X-Robots-Tag HTTP header. The header form works for PDFs and other files that have no HTML head. Google does not support noindex inside robots.txt.

# https://example.com/robots.txt
User-agent: *
Disallow: /cart/
Disallow: /*?sort=

Sitemap: https://example.com/sitemap.xml

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex

robots.txt has been an IETF standard since RFC 9309 was published in September 2022. Google's interpretation of it adds details that matter in an audit:

• Google reads the first 500 KiB of the file and ignores the rest, and generally caches it for up to 24 hours.
• The most specific rule, measured by path length, wins; when rules conflict, Google applies the least restrictive one.
• A 4xx response other than 429 counts as "no robots.txt", so everything may be crawled.
• A 5xx response stops Google crawling the whole site for the first 12 hours; after that it uses the last good copy for up to 30 days.

Search Console's robots.txt report shows the files Google found for your top 20 hosts, when each was last fetched and any parse errors, and lets you request a recrawl after an urgent fix. Bing draws the same line: its guidelines say robots.txt controls crawl access, not indexing, and that NOINDEX is what keeps a URL out of Bing and Copilot.

Keep XML sitemaps accurate

A sitemap tells search engines which URLs you want indexed and when they changed. Google's sitemap rules are short:

• One sitemap holds at most 50,000 URLs or 50 MB uncompressed. Split larger sets and list the files in a sitemap index.
• Use absolute URLs and UTF-8, and host the file at the site root.
• Google ignores <priority> and <changefreq>. It uses <lastmod> only when the value is consistently and verifiably accurate: the date of the last significant change to the main content, structured data or links, not a new copyright year.
• Submitting a sitemap is a hint, not an instruction. Submit it in Search Console's Sitemaps report, reference it with a Sitemap: line in robots.txt, or both.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/services/seo/</loc>
    <lastmod>2026-09-18</lastmod>
  </url>
</urlset>

List only URLs you want in results: each should return 200, be indexable and name itself as canonical. Sitemap inclusion is also a canonical signal (see below), so listing a duplicate argues against your own canonical tag. Bing asks for more on freshness than Google does: its guidelines want canonical URLs only, deleted or redirected URLs removed promptly, accurate lastmod values and ETag headers, and an IndexNow notification whenever a URL is added, updated or removed.

Return the right status codes, redirects and 404s

The status code is the first thing a crawler reads, and Google's crawlers act on it before they look at the content:

Why two codes for each? Under HTTP semantics (RFC 9110), a client may turn a POST into a GET when it follows a 301 or 302; 308 and 307 exist for when that change is unwanted. Google treats each pair alike, yet other clients read the difference, so send the code that means what you intend. Google ranks redirect methods by how reliably it interprets them: server-side redirects first, then an instant meta refresh, and JavaScript redirects only when neither is possible.

Chains are the usual problem. Googlebot follows up to 10 hops, but Google's site move guide asks you to redirect straight to the final destination or, failing that, to keep chains ideally to three hops and fewer than five, because each hop adds latency and not every client follows long chains. Google's crawl budget guide adds that long chains hurt crawling.

A soft 404 is a page that tells the visitor the content does not exist, or has no main content at all, while returning 200. Google's troubleshooting guide lists typical causes: a missing server-side include, a broken database connection, an empty internal search result or a JavaScript file that failed to load. Google excludes these pages and lists them in the Page indexing report. The fix follows the truth: return 404 or 410 if the content is gone, a 301 if it moved, and repair the page if it is real but reached Googlebot broken. Redirecting many old URLs to the home page is not a fix either; Google may treat that as a soft 404 as well.

Bing is stricter about temporary redirects: its guidelines reserve 302 for changes shorter than two days, and ask for redirects rather than canonical tags when content moves. To see exactly what a URL returns, follow it from the command line:

curl -sIL https://example.com/old-page/ | grep -iE "^(HTTP|location)"

Consolidate duplicate URLs with canonical tags

Most sites serve the same content at several URLs: with and without www, over HTTP and HTTPS, with and without a trailing slash, with tracking or sort parameters. Canonicalization is Google clustering those duplicates and choosing one URL to represent them. The chosen URL is crawled most often; the others are crawled less. Your preference is a hint, not a rule.

Google ranks the signals you control by strength: redirects and rel="canonical" annotations are strong, sitemap inclusion is weak, and they add up when they agree. The same page lists what goes wrong:

• Using robots.txt or the URL removal tool to pick a canonical. A disallowed duplicate can still be indexed without its content, and the removal tool hides every version.
• Sending conflicting signals, such as one URL in the sitemap and another in rel="canonical".
• Using noindex to choose between duplicates, which removes the page from Search entirely.
• Relative URLs or URL fragments as canonicals.
• Internal links that point at a duplicate instead of the canonical.

Every indexable page should carry a self-referencing canonical with an absolute URL. For a PDF, send it as an HTTP header:

<link rel="canonical" href="https://example.com/services/seo/" />

Link: <https://example.com/downloads/white-paper.pdf>; rel="canonical"

When Google disagrees with you, the URL Inspection tool shows both the user-declared and the Google-selected canonical, and the Page indexing report groups such pages under "Duplicate, Google chose different canonical than user". Check that the declared canonical really is a duplicate: Google never picks one that is not similar to the page. Bing makes the same point from the other side: canonical tags signal a preference, but they do not replace fixing duplication at the source, through URL management and consistent URL structures.

Make JavaScript content visible in the rendered HTML

Google processes JavaScript pages in three phases: crawling, rendering and indexing. Every page that returns 200 goes into a render queue, where it may wait a few seconds or longer; then a headless, evergreen Chromium runs the JavaScript, and Google reads links and content from the rendered HTML. Google still recommends server-side rendering or pre-rendering, because it is faster for users and crawlers and not every bot runs JavaScript. Bing's guidelines go further and warn against hiding critical content behind client-side rendering.

The vocabulary, as web.dev defines it:

• Server-side rendering (SSR): the server sends HTML, rather than JavaScript, in response to each request.
• Static rendering: the HTML for each URL is generated ahead of time, at build, and can be served from a CDN.
• Prerendering: a client-side app is run at build time to capture its first state as HTML; it still needs JavaScript to become interactive.
• Client-side rendering (CSR): the browser builds the page by running JavaScript that changes the DOM.
• Hydration: client-side scripts add state and interactivity to server-rendered HTML.

Dynamic rendering, which serves crawlers a pre-rendered copy while users get the client-side app, is now described by Google as a workaround and not a recommended solution; use server-side rendering, static rendering or hydration instead. Frameworks such as Next.js and Nuxt offer both static and server-side rendering, so the choice is made per route: static for pages that change rarely, server-rendered where content is personal or changes constantly, client-side only for views that should not rank, such as dashboards behind a login. Our web design and development builds choose static, incremental or server rendering this way, per route.

Whatever the architecture, the rendered HTML must contain:

1. A unique <title> and meta description.
2. Exactly one rel="canonical", identical to the one in the server HTML. JavaScript must not change it.
3. The robots meta tag you intend. If the server HTML says noindex, Google may skip rendering, so removing it with JavaScript may not work.
4. Links as <a href> elements with real URLs, routed with the History API rather than # fragments.
5. The main content, including anything lazy-loaded. Google does not load content that needs a swipe, click or typing to appear.
6. The hreflang links, inside a well-formed <head>.
7. A meaningful status code. A single-page app that cannot send a 404 should redirect to a URL that does, or add noindex to its error view.

Two limits sit underneath this. Google indexes the mobile version of your content, crawled with a smartphone user agent, so the mobile HTML needs the same content and metadata as the desktop version; the layout side is covered in mobile optimization. And as of September 2026, Googlebot fetches only the first 2 MB of an HTML file for Google Search, measured uncompressed, with each CSS and JavaScript file fetched separately under the same limit; whatever comes after the cut-off is never considered for indexing. Google also caches resources aggressively and may ignore caching headers, so fingerprinted file names such as main.2bb85551.js keep it from rendering with stale code.

To check a page, run a live test in the URL Inspection tool: it shows a screenshot of the rendered page, the page code and the resources it loaded. Compare that with what curl returns from the server. Anything that exists only in the rendered version depends on the render queue, and on every other crawler being able to run your JavaScript.

Mark language and regional versions with hreflang

hreflang tells search engines that several URLs are the same page in different languages or for different regions, so each searcher gets the right one. Google accepts it in three equivalent forms: link elements in the HTML head, HTTP headers (useful for PDFs) or the XML sitemap. Pick one; using all three adds maintenance and no benefit.

The rules that break most implementations:

• Return links. Each version lists itself and every other version. If two pages do not point to each other, Google ignores the pair.
• Valid codes. A language code from ISO 639-1, optionally followed by a region from ISO 3166-1 Alpha 2: en, en-CA, fr-CA. A region on its own is invalid (be means Belarusian, not Belgium), and codes outside those standards, such as es-419, are not supported.
• Absolute URLs, including https://.
• x-default for the page that serves everyone else, such as a language selector.
• Canonicals in the same language. Each version's canonical should be a page in the same language, as Google's canonical guide asks; normally that is the page itself. A page that only translates the header and footer counts as a duplicate of the original.

<link
  rel="alternate"
  hreflang="en-CA"
  href="https://example.com/en-ca/pricing/"
/>
<link
  rel="alternate"
  hreflang="fr-CA"
  href="https://example.com/fr-ca/tarifs/"
/>
<link rel="alternate" hreflang="en" href="https://example.com/en/pricing/" />
<link
  rel="alternate"
  hreflang="x-default"
  href="https://example.com/pricing/"
/>

Google does not use hreflang or the HTML lang attribute to detect a page's language; it determines the language with its own algorithms. hreflang only connects versions, so an audit crawls every alternate and checks that each returns 200, is its own canonical and carries the full set.

Handle pagination and faceted navigation

Long lists create URLs faster than anything else on a site. For pagination, Google's guidance is:

• Give every page its own URL, such as ?page=2, and its own canonical. Do not point pages 2 and beyond at page 1.
• Link each page to the next with an <a href>, and consider linking every page back to the first.
• Keep page numbers out of # fragments, which Google ignores.
• Do not rely on rel="next" and rel="prev": Google no longer uses them, although other search engines may.
• Put real paginated URLs behind "Load more" buttons and infinite scroll, because Google's crawlers do not click buttons.

Faceted navigation, the filters for colour, size, price and brand, is worse, because every combination of parameters is a new URL. Google's faceted navigation guide names the two costs: crawlers spend their time on useless URLs, and so discover new, useful ones more slowly. If filtered pages do not need to rank, keep crawlers out of them with robots.txt, or move the filters into URL fragments, which Google generally does not crawl. A canonical pointing at the unfiltered page, or nofollow on filter links, also helps, but Google calls both less effective in the long term.

User-agent: *
Disallow: /*?*colour=
Disallow: /*?*size=
Disallow: /*?*price=

If some filter pages should rank, keep the space finite: separate parameters with &, keep path-based filters in a fixed order with no duplicates, and return a real 404 for combinations with no results instead of redirecting to a generic error page. Which filter pages deserve indexing on a store is covered in e-commerce SEO.

Know when crawl budget matters

Crawl budget is the set of URLs Google can and wants to crawl on a host. Google's crawl budget guide is written for three kinds of site, with the numbers given as rough estimates rather than thresholds:

• large sites, with 1 million or more unique pages whose content changes about once a week;
• medium or larger sites, with 10,000 or more unique pages whose content changes daily;
• sites where a large share of URLs sit in Search Console as "Discovered - currently not indexed".

If your new pages are crawled the day you publish them, you do not need to manage crawl budget. Otherwise you are working with two quantities. The crawl capacity limit rises when your server answers quickly and consistently and falls when it slows down or returns 5xx or 429 responses; it is shared by all of Google's crawlers. Crawl demand depends on how many URLs Google believes you have, how popular they are and how stale its copies are. Each hostname has its own budget.

What frees budget, from the same guide:

1. Consolidate duplicate content, so crawling goes to unique pages rather than unique URLs.
2. Block URLs you never want crawled in robots.txt. Do not use noindex for this: Google still requests the page to see the rule.
3. Return 404 or 410 for pages that are gone, and fix soft 404s, which keep being crawled.
4. Keep sitemaps current with lastmod, and avoid long redirect chains.
5. Make pages fast to load, and answer unchanged pages with 304 Not Modified.

Blocking pages in robots.txt to shift budget to others temporarily does not work: Google only moves freed capacity elsewhere if it was already hitting your capacity limit. The Crawl Stats report shows crawl requests by response, file type and purpose, and host status for robots.txt fetches, DNS and connectivity; Google says a site with fewer than a thousand pages should not need it. Bing describes the same economy: it allocates crawl capacity by site health, efficiency, signal quality and crawl value, and warns that low-value URLs delay the indexing of important ones.

Read server logs and verify Googlebot

Search Console summarizes crawling; your access logs record every request a crawler made. nginx's predefined combined log format already captures the client address, time, request line, status, bytes sent, referrer and user agent, and the $request_time variable adds how long each response took. That is enough to answer the questions that matter:

• What share of Googlebot's requests goes to parameter URLs, redirects and errors?
• Which important sections are rarely crawled, and how soon are new URLs fetched after publishing?
• Which URLs return 404 or 5xx to crawlers but not to you?
• Do response times rise when crawl volume rises?

User-agent strings are easy to fake, so verify a request before you trust a line that says Googlebot. Google's method is a DNS round trip: run a reverse DNS lookup on the IP address, check that the name is in googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that name and confirm it returns the same address. At scale, match addresses against Google's published IP lists instead, which are JSON files in CIDR notation: common-crawlers.json covers Googlebot, with separate files for special-case crawlers and user-triggered fetchers.

# Status codes served to requests claiming to be Googlebot (combined log format)
grep "Googlebot" access.log | awk '{print $9}' | sort | uniq -c | sort -rn

# Verify one address: reverse lookup, then forward lookup
host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

For Bing, the reverse lookup must return a name ending in search.msn.com and the forward lookup must return the same IP. Bing also publishes a list of Bingbot addresses and asks you to refresh it daily rather than hard-code ranges.

Check HTTPS, Core Web Vitals and structured data

HTTPS

Google prefers the HTTPS version of a page as canonical unless the certificate is invalid, the page loads insecure resources other than images, it redirects through HTTP, or its canonical points to HTTP. Invalid certificates and HTTPS-to-HTTP redirects make Google prefer HTTP very strongly, and HSTS cannot override that. The checks follow: every HTTP URL redirects to HTTPS in one hop, the certificate covers every host name you serve, and sitemaps, canonicals and hreflang list only HTTPS URLs. Keeping the certificate current is covered in how to renew an SSL certificate.

Serving pages securely is one of the questions in Google's page experience self-assessment, and browsers are raising the stakes. In October 2025 Google announced that Chrome 154, due in October 2026, turns on "Always Use Secure Connections" for public sites by default, so Chrome asks the user before the first visit to a public site without HTTPS.

Core Web Vitals

Core Web Vitals measure loading, responsiveness and visual stability on real visits. The thresholds are a Largest Contentful Paint (LCP) of 2.5 seconds or less, an Interaction to Next Paint (INP) of 200 milliseconds or less and a Cumulative Layout Shift (CLS) of 0.1 or less, met at the 75th percentile of page loads, with mobile and desktop measured separately. INP replaced First Input Delay as a Core Web Vital in 2024. Google says Core Web Vitals are used by its ranking systems, and in the same place that good scores do not guarantee top rankings and that chasing a perfect score for SEO alone may not be the best use of your time.

Treat them as a template problem. Search Console's Core Web Vitals report uses field data from the CrUX report, anonymized timings from real visits, groups similar URLs together and rates each group by its worst metric. One oversized hero image or heavy third-party script in a shared template therefore shows up as a whole group of failing URLs.

Structured data

Structured data only helps when it is valid and true. Google recommends JSON-LD as the easiest format to maintain (Microdata and RDFa are equally fine when valid), reads it even when JavaScript injects it, and asks you to test with the Rich Results Test during development and watch the rich result reports after deployment, because templating or serving changes can break it. Its structured data guidelines set the rules: no rich result without every required property, markup must describe content visible on the page, valid markup still guarantees nothing, and misleading markup can bring a manual action. Bing also requires markup to match visible content. Which schema types fit which page is covered in semantic SEO.

Migrate a site without losing its search traffic

A migration is where technical SEO either pays for itself or fails in public. Google's site move guide comes down to a redirect map built before launch and checked after it:

1. Inventory every old URL from sitemaps, server logs, analytics, the links report in Search Console and the CMS. Include images, video, CSS and JavaScript files.
2. Decide each URL's fate: keep it, 301 it to one new page, merge it into a page that replaces it, or let it return 404 or 410 when nothing does. Do not send many old URLs to the home page; Google may treat that as a soft 404.
3. Prepare the new site: self-referencing canonicals, hreflang and internal links on the new URLs, and a new sitemap.
4. Test the map on staging: every old URL answers with one 301 or 308 hop to a page that returns 200.
5. Launch, changing one thing at a time. Google recommends moving the domain, the CMS and the design as separate steps. For a domain or subdomain change, file a Change of Address in Search Console; it is not needed for HTTP to HTTPS, www or path changes.
6. Monitor after cutover: submit the old and new sitemaps and watch indexed URLs move from one to the other, and read the logs for crawlers hitting 404s.
7. Keep the redirects for as long as possible, generally at least a year.

Expect movement anyway. For a medium-sized site, Google says it can take a few weeks or more before the new URLs replace the old ones in results, and longer for large sites. The map itself is easy to test with a script:

# redirects.csv: one "old_url,new_url" pair per line
while IFS=, read -r old new; do
  hop=$(curl -s -o /dev/null -w "%{http_code} %{redirect_url}" "$old")
  final=$(curl -s -o /dev/null -w "%{http_code}" "$new")
  case "$hop" in
    "301 $new" | "308 $new") [ "$final" = "200" ] || echo "TARGET $new returns $final" ;;
    *) echo "REDIRECT $old returns $hop" ;;
  esac
done < redirects.csv

A technical SEO audit checklist

Work from the top down: a crawling problem makes every row below it meaningless for the URLs it affects. A site crawler such as Screaming Frog SEO Spider or Sitebulb, set to a smartphone user agent, covers most rows; Search Console and your server logs cover the rest.

If you would rather have this done than do it, our SEO service audits the crawl, render and index layers with a crawler, your server logs and Search Console together, and runs migrations from a one-to-one redirect map that is tested before cutover and monitored after it.