sitemap.download

CRAWLING AND LIMITS

Which URL should I use to start a crawl?

Enter the full public address, including https:// or http://. Starting from a homepage usually exposes more navigation links. Starting from an article or section explores whatever same-origin links are reachable from there.

Use the preferred final address after redirects if you know it. The tool does not enforce a section or path boundary: a crawl beginning under /blog can still follow links to other areas on the same origin. Do not submit private URLs or access tokens.

← Search all 59 questions