Explains how a search engine crawls, indexes and serves pages
A page is discovered and fetched (crawling), analysed and stored (indexing), and returned for a query (serving); a page can fail at each stage for a different reason.
Can AI do this?
Not tested yet
No AI benchmark covers this knobit yet. New results are announced in the Can AI Do the Job? newsletter.
What it looks like in practice
Expected result: A page is discovered and fetched (crawling), analysed and stored (indexing), and returned for a query (serving); a page can fail at each stage for a different reason.
How it is assessed: Answers a written question on the crawling, indexing and serving stages of web search correctly
Related knobits
Needed for
- E-104 Designs a click-through experiment on search result appearance
- E-202 Designs internal linking between page types
- E-204 Diagnoses non-indexed URLs
All 8 knobits of Conduct search engine optimisation.
Source: knowledge the ESCO skill 'conduct search engine optimisation' applies in DO process activities P0.2, P2.13, P3.5; Google Search Central documentation; checked on 18 November 2026 against Google Search Central, How Search works