How Websites Get Crawled and Indexed

#01What are the three stages?

Google explains Search in three stages (Google):

1. Crawling. Google downloads text, images and videos from pages it has found, using automated programs called crawlers.
2. Indexing. Google analyses the page's content and key tags, such as the title element and image alt text, and stores the information in its index.
3. Serving. When someone searches, Google returns the information it judges most relevant.

Not every page makes it through each stage. Google also states plainly that it doesn't accept payment to crawl a site more often or rank it higher (Google).

#02How do search engines discover pages?

There's no central list of all web pages. Google calls the process of finding them "URL discovery". It finds new pages mainly in three ways (Google):

• Pages it already knows, which it revisits.
• Links from known pages to new ones, such as a category page linking to a new post.
• Sitemaps you submit: a file listing the URLs you want crawled (Google).

Bing also encourages sitemaps and, for faster updates, supports IndexNow. IndexNow is a protocol for notifying participating search engines when URLs change (IndexNow). See XML sitemaps for AI discovery and IndexNow.

Our view: good internal linking is the most reliable discovery method you control. A page with no links pointing to it is hard for any crawler to find.

#03What can stop a page being crawled?

Google lists common access problems: server problems, network issues, and robots.txt rules blocking Googlebot (Google). Pages behind a login also can't be crawled.

robots.txt is a file at the root of your site that tells crawlers which paths they may fetch (Google). It controls crawling, not indexing. To keep a page out of Google's index, Google says to use a noindex rule, and the page must not be blocked by robots.txt or Google can't see the rule (Google).

Google also adjusts its crawl speed to avoid overloading a site. Server errors such as HTTP 500 signal it to slow down (Google). Very large sites may need to think about crawl budget (Google).

#04What is rendering, and why does JavaScript matter?

During crawling, Google renders the page and runs its JavaScript using a recent version of Chrome, much as your browser does (Google). This matters because many sites build their content with JavaScript. Google has a guide to JavaScript SEO for this reason (Google).

AI crawlers may behave differently. A study by Vercel and MERJ found that the major AI crawlers it examined fetched JavaScript files but didn't execute them (Vercel). Our view: make sure your important content, prices and structured data are in the HTML the server sends, not added later by scripts. See JavaScript rendering for AI crawlers.

#05What happens during indexing?

After crawling, Google works out what the page is about. It processes the text, the title element, alt attributes, images and videos (Google).

Google also checks for duplicates. It groups pages with similar content and picks the most representative one as the canonical, which is the version that may be shown in results (Google). You can indicate your preferred URL with a rel="canonical" link and other signals (Google). See canonicals and duplicates.

Indexing isn't guaranteed. Google notes that not every page it processes will be indexed (Google). Thin, duplicated or low-quality pages are common reasons.

#06How do AI crawlers fit in?

AI companies run their own bots and document them. OpenAI, for example, lists separate user agents for search (OAI-SearchBot), training (GPTBot) and user-initiated fetches (ChatGPT-User). It explains how each uses robots.txt (OpenAI). Perplexity documents PerplexityBot and Perplexity-User (Perplexity). Google uses its own common crawlers and also offers a separate control, Google-Extended, for some AI uses (Google).

Some AI products also rely on a search engine's index. Our view: being properly crawled and indexed by Google and Bing remains a foundation for AI visibility. Blocking an AI company's search bot in robots.txt can keep you out of its answers. See configuring robots.txt for AI crawlers and the list of AI crawler user agents.

#07How do you check that your pages are crawled and indexed?

• Google Search Console URL Inspection shows whether Google has indexed a URL. You can also test the live URL (Google).
• Bing Webmaster Tools gives the same view for Bing. See Bing Webmaster Tools for AI visibility.
• Server logs show which bots actually fetched which pages. See reading server logs for AI bot visits.
• View the source of a page (not the rendered page) to confirm key content is in the HTML.

Fictional check: a company launches a new pricing page but doesn't link to it from the menu or add it to the sitemap. Two weeks later, URL Inspection shows the URL is "unknown to Google". Adding a menu link and updating the sitemap gives crawlers a path to it.

#08Quick checklist

• Important pages are linked from other pages
• An up-to-date XML sitemap is submitted
• robots.txt doesn't block pages or bots you want
• Pages you want hidden use noindex, not just robots.txt
• Key content is in server-rendered HTML
• Each page has one clear canonical URL
• The server responds quickly without errors

#09Where next?

Read SEO basics in the AI era, then the technical GEO audit checklist.

#10From MentionX data: which pages engines citeOriginal data

Method. Read-only aggregate queries over MentionX scans from 22 August to 29 October 2026 (UTC): 2,217 engine runs (2,154 completed) across 18 anonymised projects and six engines.

Crawling and indexing are the first step for both search and AI answers: a page that isn't fetched can't be cited. In the MentionX sample, cited URLs come from a range of page types:

Some engines cite far more often than others, and those that cite depend on pages they can retrieve:

Our view: every cited page type in the table had to be discovered, fetched and read first. Basic crawlability is the cheapest way to make sure your best pages are in the pool engines draw from.

Limits. A convenience sample of mostly B2B software projects, run through engine APIs, which can behave differently from the consumer apps. Perplexity, Copilot and DeepSeek figures come from a single day (26 October 2026), so treat them as a snapshot, not a trend.

#11Worked example: Ledgerline finds a pricing page that isn't indexed

Ledgerline is a fictional accounts-payable tool. Figures are illustrative.

Ledgerline notices that answers about its pricing quote an old blog post rather than its pricing page. In Search Console, the URL Inspection tool reports the pricing page as "Excluded by 'noindex' tag", left over from a redesign.

The team removes the noindex tag, checks that robots.txt doesn't block the page and confirms that the prices appear in the raw HTML rather than loading through JavaScript. It adds the page to the XML sitemap, links to it from the main navigation and requests indexing.

It then checks server logs and sees search and AI crawlers fetching the page within days. A month later, prompt checks show the pricing page cited more often than the old post, which the team updates with a link to current pricing.

#12How do sitemaps help?

An XML sitemap lists the pages you want search engines to know about. It helps discovery, especially for new pages or pages with few internal links, but it doesn't guarantee indexing. Include only canonical pages that return a 200 status, keep it updated automatically and submit it in Search Console. Reference it in robots.txt so other crawlers can find it.

#13How do internal links affect crawling?

Crawlers follow links to discover pages, so pages linked from your navigation and related articles are found faster. A page with no internal links, sometimes called an orphan page, may be found late or not at all. Use normal HTML links with descriptive text, and link important pages from more than one place.

#14Why might a crawled page still not be indexed?

Search engines don't index every page they fetch. A page may be left out because it duplicates another page, has little content, points to a different canonical URL or carries a noindex tag. Search Console shows a reason for each excluded page. Our view: treat "crawled but not indexed" as a quality signal, and improve or merge the page before resubmitting it.

#15What should you check after a site launch?

Crawl the site with a tool that reports status codes, canonical tags and noindex tags. Check that robots.txt allows the pages you want found. Confirm that important content is in the HTML before JavaScript runs. Submit the sitemap, then watch Search Console's indexing report for several weeks for unexpected drops.

#16How do redirects affect crawling?

When a page moves, a permanent redirect tells search engines where it went. Update internal links to point to the new URL directly, and avoid chains of several redirects, which waste crawl effort and can be dropped by some crawlers. Keep redirects in place for as long as old links might still be used.

#17Common questions

Does a sitemap guarantee indexing? No. It helps discovery, but search engines still decide which pages to index.

Can AI crawlers read JavaScript? Some can't. Put key content in the server-rendered HTML so every crawler can read it.

How long does indexing take? It varies from days to weeks. Requesting indexing in Search Console can help for important pages.