Blocked by robots.txt

What does 'Blocked by robots.txt' mean?

'Blocked by robots.txt' means a Disallow rule in your site's robots.txt file is preventing Googlebot from crawling the URL, so it cannot be indexed through normal crawling. To fix it, open yourdomain.com/robots.txt, find the Disallow rule that matches the path, and remove or narrow it so the URL is allowed. Test the URL in the Search Console robots.txt report or URL Inspection, then Request Indexing and submit via IndexNow.

Blocked from crawling, not the same as noindex

robots.txt controls crawling, not indexing. A Disallow rule stops Googlebot from fetching the page at all. Importantly, a blocked URL can still appear in results (without a description) if other pages link to it — but Google cannot read its content, so it usually will not rank well.

If you want a page kept out of the index, do not block it in robots.txt — Google must be able to crawl it to see a noindex tag. Blocking and noindexing the same URL is self-defeating.

Find the rule that's blocking you

Open yourdomain.com/robots.txt in a browser.
Look for User-agent: * or User-agent: Googlebot blocks, then any Disallow: lines whose path matches your URL.
Remember that Disallow: /blog would block /blog and everything under it; a trailing pattern like Disallow: /*? can block all URLs with query strings.
Use the URL Inspection tool — it names the exact robots.txt line that blocks the URL.

Fix it

Remove or narrow the Disallow rule so the URL (and its CSS/JS assets) can be crawled.
If you only need to block part of a path, use a more specific Disallow instead of a broad one.
Add an Allow: rule to carve out an exception under a broader Disallow if needed.
Re-test in URL Inspection until it shows crawling is allowed, then Request Indexing.
Submit the URL via IndexNow so Bing and Yandex re-crawl promptly.

Put it into practice

Generate your IndexNow key, build the submission payload and check index status — free, in your browser.

Open the free tool

Frequently asked questions

Will removing a robots.txt block index my page automatically?+

It lets Google crawl the page again, which is the prerequisite for indexing — but Google still decides based on quality and signals. Request Indexing and submit via IndexNow to speed up the re-crawl.

Why does a robots.txt-blocked URL still show in Google?+

If other pages link to a blocked URL, Google may index the URL itself without content (a bare link with no description). To truly remove it, allow crawling and add a noindex tag, or use the Removals tool.

Should I block pages with robots.txt or noindex?+

Use noindex (with crawling allowed) when you want a page out of the index. Use robots.txt mainly to save crawl budget on sections that have no SEO value, like faceted filters or internal scripts.