Explains crawl control with robots rules and sitemaps
robots.txt controls what is crawled, a noindex rule what is indexed, and a sitemap tells which URLs the site wants found; blocking crawling does not remove a URL from the index.
Can AI do this?
Not tested yet
No AI benchmark covers this knobit yet. New results are announced in the Can AI Do the Job? newsletter.
What it looks like in practice
Expected result: robots.txt controls what is crawled, a noindex rule what is indexed, and a sitemap tells which URLs the site wants found; blocking crawling does not remove a URL from the index.
How it is assessed: Answers a written question on the robots rules and XML sitemap correctly
Related knobits
Needed for
All 7 knobits of Enhance website visibility.
Source: knowledge the ESCO skill 'enhance website visibility' applies in DO process activities P2.6, P2.13, P3.2; Google Search Central documentation; checked on 18 November 2026 against Google Search Central, robots.txt introduction