Full Site Crawl and Redirect Chain Detection for Multi-Site Networks

A site that looks fine in the browser can be silently broken for search engines. Redirect chains add latency and dilute link equity. Canonical mismatches tell Google to ignore your pages. Orphan pages that exist in your sitemap but have no internal links never get crawled. Duplicate content across pages confuses ranking signals.

These problems multiply across a multi-site network. One misconfigured redirect rule on a single site is a minor issue. The same misconfiguration replicated across 16 monoclone sites is a network-wide indexing failure.

The Site Analyzer includes a full site crawl feature and a redirect chain detector built specifically for diagnosing these problems at scale.

Full Site Crawl: What It Does

The site crawl feature reads your sitemap.xml and crawls up to 50 pages from it. For each page, it checks HTTP status, canonical tags, internal links, meta tags, and content structure. The result is a complete health report for your site — not a sample of three pages, but a comprehensive scan of your actual published content.

What the crawl detects:

Broken pages (4xx/5xx) — Pages listed in your sitemap that return error status codes. These are pages Google is trying to index but cannot reach. Every broken page in your sitemap wastes crawl budget and signals poor site maintenance.

Redirect chains — Pages that redirect to another URL, which redirects to another URL, which finally resolves. Each hop adds latency and loses a percentage of link equity. A three-hop chain can lose 15-20% of the original page's ranking power.

Canonical mismatches — Pages where the canonical tag points to a different URL than the one being served. In monoclone networks, this is the most common and most damaging misconfiguration.

Orphan pages — Pages that appear in your sitemap but are not linked from any other page on the site. Search engines treat orphan pages as low-priority because no internal links point to them.

Redirect Chain Detector

The redirect chain detector tests four specific redirect paths that cause problems on content networks:

HTTP to HTTPS redirect — Does http://yourdomain.com correctly redirect to https://yourdomain.com in a single hop?

http://the100dollarnetwork.com
  → 301 → https://the100dollarnetwork.com  ✓ (single hop)

WWW to non-WWW (or vice versa) — Does https://www.yourdomain.com redirect to https://yourdomain.com cleanly?

https://www.the100dollarnetwork.com
  → 301 → https://the100dollarnetwork.com  ✓ (single hop)

Index.html paths — Does https://yourdomain.com/index.html redirect to the root, or does it serve duplicate content?

https://the100dollarnetwork.com/index.html
  → 301 → https://the100dollarnetwork.com  ✓ (single hop)

https://the100dollarnetwork.com/chapter-1/index.html
  → 301 → https://the100dollarnetwork.com/chapter-1/  ✓ (single hop)

Combined chains — The worst case: http://www.yourdomain.com/index.html triggers three redirects in sequence.

http://www.the100dollarnetwork.com/index.html
  → 301 → https://www.the100dollarnetwork.com/index.html
  → 301 → https://the100dollarnetwork.com/index.html
  → 301 → https://the100dollarnetwork.com/
  ✗ Three hops — fix your redirect rules

Each hop is logged with its status code and destination. You see the exact chain and know exactly which redirect rule needs to be consolidated.

Duplicate Content Detector

The crawl includes a duplicate content analysis using Jaccard similarity. For every pair of crawled pages, the tool compares their text content and flags pairs that exceed a similarity threshold.

Common causes of duplicate content in site networks:

• Boilerplate-heavy pages where the template content outweighs the unique content
• Paginated sections that repeat the same introductory paragraphs
• Tag and category pages that display the same excerpts in different combinations
• Monoclone sites where placeholder content was never replaced

The duplicate content report shows each flagged pair with its similarity score, so you can prioritize which pages need the most differentiation.

Hosting and Indexing Health Checks

Beyond individual page analysis, the crawl evaluates site-level hosting configuration:

Canonical domain consistency — Are all canonical tags using the same base domain? A single page pointing its canonical to www.yourdomain.com while the rest use yourdomain.com creates a split-indexing signal.

Sitemap domain validation — Does every URL in sitemap.xml match the domain serving the sitemap? This catches the monoclone copy-paste error where a site's sitemap still references the template's original domain.

Robots.txt analysis — Is robots.txt accessible, properly formatted, and not blocking critical paths? The tool checks for common mistakes like blocking /assets/ (which blocks CSS and JS that Google needs to render your pages) or referencing the wrong sitemap URL.

# Common robots.txt mistake in monoclone networks
User-agent: *
Sitemap: https://wrong-domain.com/sitemap.xml  # Should be this domain

Multi-Provider Fix Guides

When the tool detects a redirect chain or hosting misconfiguration, it provides fix instructions specific to your hosting provider. Each issue includes remediation steps for the five most common static hosting platforms.

Netlify — Redirect rules go in netlify.toml or a _redirects file:

[[redirects]]
  from = "https://www.the100dollarnetwork.com/*"
  to = "https://the100dollarnetwork.com/:splat"
  status = 301
  force = true

Vercel — Redirect rules in vercel.json:

{
  "redirects": [
    {
      "source": "/:path(.*)",
      "has": [{ "type": "host", "value": "www.the100dollarnetwork.com" }],
      "destination": "https://the100dollarnetwork.com/:path",
      "permanent": true
    }
  ]
}

Cloudflare — Page Rules or bulk redirects in the dashboard, or _redirects for Cloudflare Pages.

Apache — .htaccess rewrite rules:

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

RewriteCond %{HTTP_HOST} ^www\.(.+)$ [NC]
RewriteRule ^(.*)$ https://%1/$1 [L,R=301]

Nginx — Server block redirect configuration:

server {
    listen 80;
    server_name the100dollarnetwork.com www.the100dollarnetwork.com;
    return 301 https://the100dollarnetwork.com$request_uri;
}

server {
    listen 443 ssl;
    server_name www.the100dollarnetwork.com;
    return 301 https://the100dollarnetwork.com$request_uri;
}

LLM Fix Prompt for Automated Remediation

Each detected issue includes a pre-built LLM Fix Prompt — a structured prompt you can paste into ChatGPT, Claude, or any coding assistant to generate the exact fix for your specific hosting setup.

The prompt includes the issue type, the current (broken) behavior, the desired behavior, and your hosting provider. The LLM generates the configuration code you need to paste into your project. This turns a diagnostic report into actionable fixes in under a minute per issue.

For network operators running monoclone architectures, a single LLM-generated fix in your shared template resolves the issue across every site on the next deploy.

Where These Tools Fit in the Network Stack

The Site Crawl and Redirect Chain Detector handle the technical infrastructure layer. They answer the question: "Is my hosting configured correctly so that search engines can actually reach and index my content?"

The $97 Launch Chapter 40 covers netlify.toml configuration — the redirect rules, headers, and build settings that prevent these issues from appearing in the first place. If you are setting up a new network site, start there.

The $20 Dollar Agency Chapter 4 covers Technical SEO — the broader category of crawlability, indexability, and site architecture that these tools audit. If your crawl report shows structural issues beyond simple redirect fixes, that chapter walks through the full remediation process.

Run a crawl now at jwatte.com/tools/. Enter your domain, let the tool scan your sitemap, and see exactly what search engines encounter when they try to index your site. Fix the redirect chains first — they are the fastest wins with the highest impact on crawl efficiency and ranking performance.