Skilirium

Explains crawl control with robots rules and sitemaps

E-252 · Knowledge knobit · Activity: Enhance website visibility · Competence: Search engine optimisation

robots.txt controls what is crawled, a noindex rule what is indexed, and a sitemap tells which URLs the site wants found; blocking crawling does not remove a URL from the index.

Can AI do this?

Not tested yet

No AI benchmark covers this knobit yet. New results are announced in the Can AI Do the Job? newsletter.

What it looks like in practice

Expected result: robots.txt controls what is crawled, a noindex rule what is indexed, and a sitemap tells which URLs the site wants found; blocking crawling does not remove a URL from the index.

How it is assessed: Answers a written question on the robots rules and XML sitemap correctly

Related knobits

Needed for

All 7 knobits of Enhance website visibility.

Source: knowledge the ESCO skill 'enhance website visibility' applies in DO process activities P2.6, P2.13, P3.2; Google Search Central documentation; checked on 18 November 2026 against Google Search Central, robots.txt introduction