How Groager Crawls and Audits Your Website

Every Website Intelligence audit in Groager starts the same way: a real crawl of your actual site, not a guess.

How discovery works

The crawler starts at your homepage and follows internal links breadth-first, the same way a search engine would. It also checks your sitemap.xml and reconciles what it finds there against what it actually reaches by crawling -- pages that are in your sitemap but never linked from anywhere are flagged as sitemap orphans, a real, common technical SEO issue.

Respecting robots.txt

Before a single page is fetched, Groager parses your robots.txt and applies the same longest-match precedence rules search engines use. If a path is disallowed, it is never crawled -- full stop. This is not configurable per-audit; it is a hard rule.

Crawl-delay

If your site declares a Crawl-delay directive, Groager throttles itself to match it, even if that makes the audit slower. Politeness is not optional.

From raw pages to real findings

Once pages are crawled, Groager does not just count them -- it builds a real link graph (crawled_pages + link_graph_edges), then derives evidence-based findings:

• Broken internal links, with real inbound-link counts and (when Google Search Console is connected) real organic traffic exposure
• Orphan pages, categorized by how they were actually discovered -- crawl-only, sitemap-only, or external-referral-only
• Redirect chains and duplicate content, detected from the real crawled HTML

If a metric cannot be calculated from real evidence with enough confidence, Groager returns insufficient_data instead of a fabricated number. That is a deliberate design choice, not a limitation we are hiding.

Optional JavaScript rendering

For JS-heavy sites, Groager can optionally render pages with a real headless browser before analyzing them -- off by default, and even when enabled it never bypasses robots.txt or crawl limits. It also compares raw HTML links against rendered links, so you can see exactly how much of your navigation depends on JavaScript execution.

The result is an audit you can actually trust: every score traces back to evidence you can inspect, not a black box.