Crawling and indexing are consistently conflated in SEO conversations, even by practitioners who have been doing this for years. The two terms describe two distinct stages of how Google processes your site, and the distinction matters practically: a page that isn't being crawled has a different problem than a page that's being crawled but not indexed. Diagnosing the wrong stage leads to fixing the wrong thing. This guide covers what each stage actually involves, how Google moves through them, and how to identify and fix problems at each step.

What Crawling Is

Crawling is Google's process of discovering and downloading the content of web pages. Googlebot, Google's crawler, starts from a set of known URLs and follows links from those pages to discover new ones. When Googlebot visits a URL, it downloads the page's HTML, follows the links it finds there, and adds new discovered URLs to the crawl queue for future visits. This is how Google builds its map of the web.

Crawling is not the same as reading or understanding content. It's fetching. Googlebot downloads the raw HTML of a page and passes it to Google's processing pipeline. What happens to that content afterward is indexing, a separate step.

What Controls Whether Googlebot Crawls a Page

Several factors determine whether Googlebot crawls a specific URL.

robots.txt. A Disallow rule in robots.txt instructs Googlebot not to crawl matching URLs. Critically, robots.txt controls crawling, not indexing. A page blocked by robots.txt can still appear in Google's index if other pages link to it, because Google can infer the page exists from those links even without crawling it. This is a common source of confusion: robots.txt blocking doesn't guarantee a page won't be indexed.

Internal links. Pages that aren't linked from any other crawled page are harder for Googlebot to discover. Orphan pages (pages with no internal links pointing to them) may be crawled infrequently or not at all, depending on whether they appear in a sitemap or have external backlinks. This is why internal linking architecture is a crawlability concern as much as a PageRank concern.

XML sitemap. A sitemap explicitly tells Google which URLs you want crawled. It doesn't guarantee those URLs will be crawled immediately or at all, but it provides a direct list that Googlebot can use rather than relying entirely on link discovery. Submitting a sitemap through Google Search Console and keeping it current is the most direct signal to Googlebot about which URLs matter on your site.

Crawl budget. Google allocates a finite crawl budget to each site based on the site's authority, server response speed, and the freshness of its content. As covered in the crawl budget guide, sites with many low-value pages (thin content, duplicate pages, parameter URLs, redirect chains) waste crawl budget on URLs that don't need indexing, which can leave important new content undiscovered for longer than necessary.

What Indexing Is

Indexing is the process of analyzing crawled content and adding it to Google's search index. After Googlebot downloads a page's HTML, Google's systems render the page (executing JavaScript where necessary), extract and analyze the text content, identify entities, understand the page's topic and intent, evaluate quality signals, and decide whether the page is worth adding to the index.

This is the stage where content quality becomes directly relevant. A page can be perfectly crawlable and still not get indexed if Google's quality systems determine it's not useful enough to include. The most common reasons Google crawls a page but doesn't index it:

The page has a noindex directive. A <meta name="robots" content="noindex"> tag in the page's HTML, or a noindex value in the X-Robots-Tag HTTP response header, explicitly instructs Google not to index the page. Unlike robots.txt (which blocks crawling), noindex is respected at the indexing stage: Google crawls the page to discover the noindex directive, then doesn't add it to the index. The Indexability Checker surfaces both meta robots directives and X-Robots-Tag header values for any URL, making it straightforward to confirm whether a page is inadvertently noindexed.

The page is a near-duplicate of another page. Google's canonicalization systems identify when multiple URLs contain substantially similar content and choose one to index as the canonical. The others may be crawled but won't be indexed independently. This is why keyword cannibalization and thin content at scale suppress indexing: Google indexes one version and ignores the rest.

The page has a canonical tag pointing elsewhere. A <link rel="canonical"> tag telling Google the canonical version of a page is a different URL instructs Google to index the canonical rather than the page being crawled. If the canonical URL is wrong (pointing to a different page, a redirect, or a noindexed page), the canonical implementation is broken and neither page may get indexed correctly.

The content is low quality. Google's quality systems may determine that a page is too thin, too generic, or too similar to other content already in the index to be worth adding. This is the mechanism behind Google's Helpful Content system: pages that don't demonstrate genuine value may be crawled repeatedly but never stably indexed, or indexed briefly and then dropped.

The Third Stage: Ranking

Crawling gets a page discovered. Indexing gets it into Google's database. Ranking is the third stage: determining where the page appears in search results for specific queries. A page can be indexed without ranking meaningfully, which is the situation for most new pages before they've accumulated authority signals, backlinks, and engagement data. Ranking problems are a different class of issue from crawling or indexing problems, and they require different diagnostics.

The practical diagnostic sequence for a page that isn't appearing in search results: first confirm it's indexed (use the site:yourdomain.com/specific-page search operator or Search Console's URL Inspection tool). If not indexed, determine whether it's a crawling problem or an indexing problem. If indexed but not ranking well, the issue is in the ranking signals: content quality, authority, backlinks, and on-page optimization.

How to Check Whether a Page Is Crawled and Indexed

Google Search Console URL Inspection. The URL Inspection tool in Search Console is the most authoritative source for crawl and index status for any specific URL. It shows the last time Googlebot crawled the page, whether it's indexed, whether a canonical has been selected (and whether that canonical matches what you specified), and any coverage issues. For any page where you suspect a crawling or indexing problem, URL Inspection is the first tool to reach for.

The site: operator. Searching site:yourdomain.com/page-url in Google shows whether that URL is currently indexed. It's a quick spot check, though Search Console's URL Inspection is more detailed and reliable for diagnosis.

The Indexability Checker. The Indexability Checker fetches a URL and checks for all the signals that would prevent or interfere with indexing: noindex meta tags, X-Robots-Tag noindex headers, canonical tags (and whether they're self-referencing or pointing elsewhere), robots.txt blocking, and redirect status. Running a page through it surfaces every indexability signal in one place rather than checking each in a separate tool.

Common Crawl and Index Problems by Symptom

New page not appearing in Google after weeks. First check whether it's indexed via URL Inspection. If not indexed, check the URL Inspection "Coverage" section for the specific reason Google gives. Crawled but not indexed often indicates thin content or a near-duplicate issue. Discovered but not crawled means it's in the queue but hasn't been fetched yet, often resolved by requesting indexing via URL Inspection. Not indexed with no stated reason usually means a robots.txt block or a noindex directive worth checking with the Indexability Checker.

Indexed yesterday, not today. Sudden deindexing of previously indexed pages is usually caused by a recently introduced noindex tag (from a CMS update or template change), a robots.txt change, or a canonical tag change. Check the page's HTML source for noindex meta tags, use the Indexability Checker to verify no header-level noindex is present, and validate the robots.txt file with the Robots.txt Tester.

Sitemap pages not getting indexed despite being crawled. When Search Console shows pages as "Submitted in sitemap but not indexed," the most common causes are thin or duplicate content, canonical mismatches, and noindex directives the sitemap submission can't override. A full technical audit with the SEO Audit Tool surfaces all of these across multiple pages simultaneously, making it efficient to identify patterns in which pages are failing to index rather than diagnosing each one individually.