sitemap.download

CRAWLING AND LIMITS

Why are subdomains or HTTP links skipped?

Origin means protocol, hostname and port together. HTTPS and HTTP, or a main host and a subdomain, are separate origins. The first page’s redirects establish the crawl origin. Later redirects outside it are not followed. Enter the preferred final website address when starting.

For example, https://www.example.com and https://example.com are different origins. Check whether your own navigation consistently points to the preferred host before interpreting skipped links as a crawler error.

← Search all 59 questions