Crawling
Fetching pages and following discovered links. A crawler can only assess content it can reach and process.
Indexing
Processing content for inclusion in a search engine’s searchable collection. Fetching a page or listing it in a sitemap does not guarantee inclusion.
Canonical URL
The preferred address for a piece of content when several URLs are similar. A canonical declaration is a signal, not a redirect.
noindex
A robots directive requesting that a supporting engine exclude a page from its index. The engine must be able to retrieve the directive.
robots.txt
A file at the site root containing crawl instructions for cooperating bots. It is not password protection and is not equivalent to noindex.
HTTP status
A response code describing the outcome of a request: for example 200 for success, 404 for not found and 503 for temporary service unavailability.
Redirect
A response that sends the client to another URL. 301 and 308 indicate a permanent move; 302 and 307 indicate a temporary one.
Origin
The combination of scheme, hostname and port. HTTPS and HTTP addresses, or two different subdomains, have different origins.
XML
A structured text format using elements and attributes. Sitemap XML must be well-formed; arbitrary HTML is not a substitute.
Sitemap index
An XML file pointing to child sitemap files. It organises multiple files instead of listing page URLs directly.
lastmod
Optional sitemap metadata for the actual last modification of a page. It should not be reset just because the sitemap was regenerated.
hreflang
An annotation connecting equivalent language or regional versions of a page. It does not translate a page or force indexing.
Query parameter
A value following the question mark in a URL. It may identify distinct content or merely tracking information; removing it blindly can change the destination.
URL fragment
The part after #, often pointing to a section of a page. This generator removes fragments when deduplicating discovered URLs.
Orphan page
A page with no internal links leading to it from the part of the site being explored. A link-following crawler may never discover it.
Sources and further reading
Reviewed 2 November 2026. Platform behaviour can change; consult the linked documentation for your setup.