About the Website Crawler
Starting from a URL you choose, this website crawler follows internal links the way a search engine would, respects your robots.txt, and stops at 100 pages. For every URL it records the status code, title, meta description, H1, canonical, noindex status, click depth and internal link count, then highlights duplicates and broken internal links.
The crawl is also compared with your XML sitemap, so you can see sitemap URLs the crawler never reached and crawled pages missing from the sitemap. Because it only checks public URLs, it suits small business sites, blogs and sections of larger sites. Export everything as CSV for a spreadsheet audit, or generate a fresh XML sitemap from the indexable pages it found. Run the crawler again after fixes to confirm the problems are gone.
How to use it
- Enter your homepage or the section URL you want the crawler to start from.
- Start the crawl and let it follow internal links up to 100 pages.
- Filter the results for errors, noindex pages, duplicates and sitemap differences.
- Export the CSV or download an XML sitemap built from indexable crawled pages.
Frequently asked questions
How is a website crawler different from a single-page audit?
A single-page audit looks deeply at one URL. A crawler moves across your site through internal links and collects a smaller set of data from many pages, which reveals patterns: duplicate titles, broken internal links, pages buried too deep, or noindex tags left on by mistake. Use the crawler to find problem pages and the audit to inspect them in detail.
Why did the crawler find fewer pages than my site has?
The crawler only reaches pages linked internally from the start URL, stops at 100 pages, and skips anything your robots.txt disallows. Pages that are only reachable through search forms, JavaScript-only navigation or external links will not be found. If important pages are missing, check whether they are linked from other pages and listed in your sitemap.
Does crawling my site affect its performance?
The crawl is limited to 100 pages and fetches them at a modest pace, which is similar to light visitor traffic for most hosting. It requests only the HTML of each page, not every image or script. If your server is very fragile or rate-limits bots, crawl a smaller section by starting from a deeper URL.