Crawl a Website

Enter one URL, choose a crawl limit, and discover links on the same site or root domain. Some sites block browser crawlers or proxy requests.

Starting URL

Max Pages

Scan entire site
Follow same-site links until the queue is empty or the max page limit is reached. Turn off to scan only the starting page and its direct links. Include subdomains
Treat pages on the same root domain as in-scope, such as www.example.com and blog.example.com. Fast crawl mode
Finish early when a sitemap, homepage, or repeated batches have already found a strong URL set. Turn off for a deeper crawl.

URLs (one per line)

Change Frequency

Priority

Last Modified

🗺️ Sitemap.xml Generator

Crawl a Website

Generated Sitemap