Quick answer
Use rel=canonical when a URL is a duplicate or variant of another page and you want signals merged into the main version. Use noindex when a page is useful to visitors but should not appear in search; it must stay crawlable so Google can see the tag. Use robots.txt only to stop crawling of URLs with no search value; it does not reliably keep pages out of Google. For moved pages, use a 301 redirect.
Key takeaways
- Canonical merges duplicates; noindex removes a page; robots.txt stops crawling.
- Never block a noindexed page in robots.txt: Google cannot see the tag.
- A robots.txt-blocked URL can still be indexed if other pages link to it.
- Give each paginated page its own canonical, not page 1.
- URL Inspection shows the canonical Google actually chose.
Canonical tags, noindex and robots.txt all seem to “hide” pages from Google. They do three different jobs, and mixing them up is one of the most common technical mistakes we find in audits. This guide explains each one and gives a simple decision table.
It is part of the groundwork in our technical SEO services.
What does each one actually do?
rel=canonical: “this other URL is the main version”
A canonical tag tells Google which of several similar URLs you prefer to show in search. The duplicates can still be crawled; their signals are consolidated into the main version. Google calls it a strong hint, not a rule, in its canonical guide.
noindex: “do not show this page in search”
A noindex robots meta tag or HTTP header tells Google to drop the page from results. Google has to crawl the page to see the tag, so the page must not be blocked in robots.txt. That is spelled out in Google’s noindex documentation.
robots.txt: “do not crawl this”
robots.txt controls crawling, not indexing. Google’s robots.txt guide says it is not a way to keep a page out of Google: a blocked URL can still be indexed, without its content, if other pages link to it.
Which one should you use?
Ask what you want to happen to the page in search, then pick the tool from the table below.
- The page is a copy or variant of another page (tracking parameters, print versions, a product in two categories): canonical to the main version.
- The page is useful to visitors but should not appear in search (thank-you pages, internal search results, thin tag archives): noindex, and keep it crawlable.
- The URLs waste crawling and have no search value (endless filter combinations, admin or cart paths): robots.txt disallow, once they are not indexed.
- The page has moved for good: a 301 redirect, not any of the three.
Which signal for which situation
| Situation | Use | Crawlable? | Effect in search |
|---|---|---|---|
| Duplicate or variant URL | rel=canonical to main URL | Yes | Signals merge to main URL |
| Useful page, not for search | noindex (meta or header) | Yes, must be | Dropped from results |
| Crawl waste, no search value | robots.txt Disallow | No | Not crawled; can still be indexed if linked |
| Page moved permanently | 301 redirect | Yes | Old URL replaced by new |
| Private content | Password or login | No | Not accessible |
What are the most common mistakes?
- noindex plus robots.txt block. Google never sees the noindex, so the URL can stay indexed. Remove the block until the page drops out.
- Canonical pointing to a noindexed or redirected page. Mixed signals; Google may ignore both.
- Every page canonicalised to the homepage. A theme or plugin default that tells Google your whole site is a duplicate.
- Paginated pages canonicalised to page 1. Google’s pagination guidance says to give each page its own canonical, so products on page 2 can still be found.
- Staging sites left open, or live sites left noindexed after launch. Check both on every release.
- Sitemaps listing non-canonical or noindexed URLs. List only the pages you want indexed.
Old blog content is a common place these problems pile up; see our guide to consolidating old blog posts.
How do you check what Google decided?
Use the URL Inspection tool in Search Console. It shows whether the page is indexed, the canonical you declared and the canonical Google selected. If they differ, your signals disagree: check redirects, internal links and the sitemap.
The Page indexing report groups URLs by reason, such as “Excluded by noindex tag”, “Blocked by robots.txt” or “Duplicate, Google chose different canonical than user”. Work through it page type by page type.
On online stores, filters and variants make this decision constantly. Our ecommerce SEO services and our category and product page guide cover the store-specific rules. For a full checklist, see our technical SEO audit checklist, and for a real rescue, our Jeep Nepal case study.
Want a second opinion on your site? Ask us for a free SEO review and we will tell you what to fix first, in plain language.
Explore More
Reviewed by
K.K. Bhatta
K.K. Bhatta is a general digital marketer at Queens Digital Agency: hands-on across SEO, paid media, and whatever platform changes next. A lifelong learner rather than a one-topic specialist, he tests ideas on live campaigns before writing about them, and stays skeptical of anything that has not actually been tried.
Every claim in this article was checked against a live source before publishing.
A canonical says another URL is the main version and merges signals into it. Noindex removes the page from search results entirely.
Avoid it. They send conflicting signals. Choose one based on whether the page is a duplicate (canonical) or should not be in search at all (noindex).
No. It stops crawling. A blocked URL can still appear in results if other pages link to it. Use noindex to remove a page.
No. Google treats it as a strong signal and may choose a different canonical if your other signals disagree.
Sources and references
Facts in this guide come from the sources below, checked live on 8 November 2026. Everything else is our own working practice, stated as such.