DOCUMENTATION / KNOWLEDGE BASE
Questions about sitemaps?
Find your answer.
59 practical answers about creating, reviewing and publishing sitemaps — and understanding what the crawler can and cannot do.
59 questions across 8 topics.
Try fewer words, a different phrase or select All topics. You can also clear the filters to browse every answer.
Sitemap basics
Is a sitemap an indexing or ranking guarantee?
No. It helps supporting crawlers discover content.
What is an XML sitemap?
An XML sitemap is a machine-readable inventory of public page URLs.
Does a small website need a sitemap?
A small site with clear internal links may already be easy for crawlers to explore.
What is the difference between an XML and an HTML sitemap?
An XML sitemap is structured for automated processing.
Is this sitemap generator free?
Yes. The generator is a free, independent project with no account requirement.
Does a sitemap contain the text of my pages?
No. A page sitemap lists locations, not the text of your articles or products.
Crawling and limits
Why are some pages missing?
The crawler can only follow HTML links reachable from the start URL on the same origin.
Why did crawling stop at 1,000 pages?
The safety limit is 1,000 page attempts per run, including failed requests.
Why are subdomains or HTTP links skipped?
Origin means protocol, hostname and port together.
Does this generator follow robots.txt?
This version does not interpret robots.txt rules.
Which URL should I use to start a crawl?
Enter the full public address, including https:// or http://.
Can I crawl password-protected pages?
No authenticated crawl is provided. The proxy does not forward your browser login cookies or authorization headers to the target.
Does the crawler run JavaScript on the target website?
No. It reads the returned HTML without running the target’s scripts.
Does the crawler import an existing sitemap?
No. This version starts from an HTML page and follows discovered links.
Can I restrict a crawl to a single folder?
There is no folder-only setting in this version.
Can this tool include PDFs or other non-HTML files?
This crawler accepts HTML pages and skips common non-page extensions such as PDF, ZIP and media files.
Errors and troubleshooting
What do HTTP errors and timeouts mean?
A 403 may indicate blocked automated requests; 404 means the page was not found; 429 usually means too many requests; 5xx responses indicate a server problem.
What should I do after “Too many requests”?
Wait at least a minute before starting another run.
Why does crawling take a long time?
Pages are fetched one at a time with a short pause between requests.
What does “too many redirects” mean?
The proxy follows at most five redirects for a page.
Why is my website URL rejected?
The service accepts only public HTTP or HTTPS website addresses on their standard ports.
What does “does not return an HTML page” mean?
The proxy checks the response Content-Type. It expects text/html or application/xhtml+xml.
Why does the crawl stop when the first page fails?
The first successful page establishes the crawl origin and supplies the first links.
What if my website blocks the crawler’s requests?
Security systems may reject automated requests from the proxy even when a normal browser can access the same page.
URLs and indexing
Are URL parameters and canonical tags handled automatically?
Fragments are removed and exact duplicate URLs are deduplicated.
Which URLs should I exclude from my sitemap?
Review login screens, baskets, internal search results, staging addresses and duplicate tracking variants.
Should a noindex page appear in a sitemap?
A sitemap normally lists pages intended for discovery and indexing.
What is the difference between robots.txt and noindex?
robots.txt provides crawl rules for cooperating crawlers.
What is a canonical URL, and does this tool detect it?
A canonical URL identifies the preferred version of content when multiple addresses are similar or equivalent.
Are URL fragments such as #contact included?
The generator removes the fragment part of discovered URLs before deduplication.
Should I remove all query parameters?
No. A parameter can identify useful content, such as a page of results, or merely duplicate the same page with tracking information.
Can I change the order of URLs in the download?
This interface does not provide manual sorting.
Images
Why are some images missing?
Only supported HTML image attributes are inspected.
Can I remove individual images?
The review list selects pages. In image mode, unchecking a page removes all image references attached to it.
Can an image sitemap contain CDN image URLs?
The extractor accepts HTTP(S) image locations even when they use a different origin from the containing page.
Why are placeholder images included?
A page may contain a placeholder in src and a final image in data-src or srcset.
Does the generator download or store my images?
No. Image mode collects image URLs from fetched HTML.
Can I create a page sitemap and an image sitemap?
Yes. Run the generator once in Page URLs mode and again in Images mode.
Review and download
Why are there fewer entries than processed pages?
A processed page may be excluded because it has a recognised noindex directive, resolves to an already included URL, has no extracted images in image mode, or was unchecked in the review list.
Can I stop and still download?
Yes. Use Stop crawling.
What if my XML file is too large?
The download is limited to 50 MB of uncompressed XML.
How do I choose which pages to download?
When a crawl finishes or is stopped with eligible results, the review list shows selected checkboxes.
Can I add or edit URLs in the results list?
The results list currently supports selecting or excluding collected pages, not typing in new URLs or editing their addresses.
Why is the download button missing?
No download is offered until eligible results exist and at least one page is selected.
Can I resume a stopped crawl?
This version does not save a crawl checkpoint or offer a resume control.
Does deselecting a URL delete the page from my website?
No. Selection changes only the XML generated in this tab.
Publishing and maintenance
How often should I regenerate it?
Update it after publishing or deleting pages, changing URLs or moving a website.
Where should I upload sitemap.xml?
Upload it to the website whose URLs it describes, preferably at the root so it is available as https://your-domain.example/sitemap.xml.
How do I add my sitemap to robots.txt?
Add a Sitemap line containing the full public file URL to the target website’s existing robots.txt.
Does this tool submit my sitemap to search engines?
No. It generates a downloadable file and leaves publishing and submission to you.
Should I replace my website platform’s existing sitemap?
Not automatically. Many content platforms maintain sitemaps as their content changes.
Why does my browser say the XML has no style information?
That message usually means the browser is displaying raw XML without a stylesheet.
How can I check whether my published sitemap works?
Open the exact public address while signed out.
Will my downloaded sitemap update automatically?
No. A download is a snapshot of the selected results from that crawl.
Privacy and sessions
Why did my results disappear?
Crawl data and selections live only in this browser tab’s memory.
Does all processing happen only in my browser?
The browser controls the queue, parses the returned HTML and creates the XML.
Are FAQ searches and sitemap selections saved?
The FAQ search filters the already loaded question list in your browser.
Does the proxy send my cookies to the target website?
The proxy constructs its own outbound request.
Can I use the generator without an internet connection?
A crawl requires a connection to this website’s proxy and to the target website.