Website URL Extractor

It starts at the URL you enter, reads the links in the page's HTML and follows the ones that point to the same site, breadth first, so pages closer to the start are crawled first. It stops when it reaches your page limit, your depth limit, or runs out of new links.

The crawler reads the HTML the server sends and doesn't run JavaScript, so links that are only added by scripts aren't seen. It also can't reach pages that need a login, pages no other page links to, pages beyond the depth or page limit, and paths blocked by robots.txt when that option is on. Compare the results with your sitemap to find orphan pages.

200 means the page loaded. 404 and 410 mean it doesn't exist, which usually points to a broken internal link. 5xx codes are server errors. Redirects are followed automatically, so a page that redirects shows the status of its final destination, and the final URL appears in the export.

Up to 500 pages per run, 100 by default. Pages are fetched four at a time through our server, because browsers can't read other websites directly. For larger sites, crawl one section at a time by starting from a section URL and setting a lower depth.

Yes, by default. The tool reads the site's robots.txt and skips paths that are disallowed for crawlers in general. Skipped URLs are listed with the status Blocked so you can see what was left out. You can turn this off to check your own site.

External links are collected and counted, along with the first page they were found on, but they aren't fetched, so their status isn't checked. Export them to review where your site links out to.