Matcha Growth 2026 Technical SEO Audit Dataset: 312 Sites, Robots.txt and Indexing Patterns

Across the last 12 months we audited 312 websites for technical SEO issues, with a focus on robots.txt configuration, Google Search Console coverage warnings, and crawl-budget patterns. The headline finding: 79% of “Indexed, though blocked by robots.txt” URLs on ecommerce sites in our dataset were faceted-nav or parameter variants of an unfiltered category page. This piece walks through the methodology, six findings about how robots.txt and indexing break in the wild, and the root causes we see most often.

Methodology

This piece draws on 312 technical SEO audits we have conducted across the last 12 months, performed under Matcha Growth (matchagrowth.com client work) and prior agency engagements at Pariveda Solutions (Toyota, Lexus, TurboTax, and HealthMarkets), Gr0, and Freestyle Creative. Audits range from single-page brochure sites to multi-million-URL enterprise storefronts, with concentration in ecommerce, SaaS, and publishing verticals.

Each audit included a Google Search Console coverage review, a robots.txt parse, and a sampled crawl of flagged URLs. Stats reflect the rate of finding each pattern across our audit sample, not population prevalence. Sites that hire a technical SEO consultant skew toward having recognized problems, and our sample over-represents English-language sites in retail, media, and SaaS verticals.

Findings at a glance

The six findings below come from Matcha Growth’s 2026 Technical SEO Audit Dataset (312 sites audited in the last 12 months).

• Finding 1: 23% of audited sites had at least one URL flagged as “Indexed, though blocked by robots.txt”
• Finding 2: 71% of flagged sites trace to one of three root causes
• Finding 3: 12% of audited sites had ever clicked GSC’s “Validate Fix” button
• Finding 4: 87% of flagged URLs that owners wanted deindexed had Disallow-only as the prior fix attempt
• Finding 5: 79% of flagged URLs on ecommerce sites were parameter or faceted-nav variants of an unfiltered category page
• Finding 6: Only 8% of audited sites had any prior server-log analysis on file

Each finding is broken out in detail in the sections below.

Finding 1: 23% of audited sites had at least one URL flagged as “Indexed, though blocked by robots.txt”

Across our 312 audited sites, 72 carried at least one URL with this Search Console status at the time of engagement. The pattern concentrated in ecommerce and large publishers, where dynamic URL generation outpaces editorial review of robots.txt rules. For sites under 5,000 indexable URLs, the rate fell under 8%.

The status itself is a contradiction: Google has indexed the URL while telling site owners it cannot crawl it. That contradiction is the source of the confusion that brings most owners to an audit.

Finding 2: 71% of flagged sites trace to one of three root causes

Of the 72 flagged sites, 51 had their robots.txt block originate from one of three sources: an inherited staging Disallow that carried over to production, a CMS-generated robots.txt default (typically from a Yoast or Rank Math preset), or a Disallow rule pasted from a third-party SEO audit tool’s recommendation. The remaining 29% spanned a long tail of bespoke causes. None of the three concentrated causes require log analysis or advanced tooling to detect.

Finding 3: 12% of audited sites had ever clicked GSC’s “Validate Fix” button

Only 38 of the 312 sites had any record of using Search Console’s validation flow on a coverage status. The button is the canonical signal to Google that a fix has shipped, and it is the fastest way to trigger a recrawl on flagged URLs. The other 88% had either ignored the flow or assumed it would run automatically. The cost of skipping validation is silent: the flag persists in Search Console for weeks longer than it needs to.

Finding 4: 87% of flagged URLs that owners wanted deindexed had Disallow-only as the prior fix attempt

When we asked the 72 flagged-site owners how they had tried to fix the issue before engaging us, 63 reported using a robots.txt Disallow as their only intervention. Only 6% had ever correctly combined an allow-crawl directive with a noindex tag on the URL itself. The Disallow-only approach is the most common fix the open web recommends, and it is the wrong one for this specific Search Console status.

Disallow tells Google not to crawl, which prevents Google from seeing the noindex tag that would actually remove the URL from the index. The correct sequence is to allow crawl, serve a noindex tag, wait for Google to recrawl, then add the Disallow back if needed.

Finding 5: 79% of flagged URLs on ecommerce sites were parameter or faceted-nav variants of an unfiltered category page

On the 38 ecommerce sites in our flagged-site subset, the median flagged URL count was 1,400, and 79% of those URLs resolved to color, size, price, or sort variants of a parent category. One Shopify storefront we audited carried over 14,000 flagged parameter URLs at intake. The fix here is structural, not rule-based: internal facet links need rel-canonical pointing back to the unfiltered parent, and the parameter Disallow needs to come off so Google can see the canonical and consolidate.

Finding 6: Only 8% of audited sites had any prior server-log analysis on file

Across the full 312-site dataset, 25 sites had ever run server-log analysis to verify which URLs Googlebot was actually requesting. On sites over 100,000 URLs the rate climbed to 21%, but it was still a minority practice even at enterprise scale. Server logs are the only source of truth for what Google is crawling, and their absence is why most robots.txt rules ship without anyone verifying whether they took effect.

What we see in the field

A Shopify storefront we audited carried more than 14,000 flagged “Indexed, though blocked by robots.txt” URLs at the time of engagement. Every flagged URL was a parameter variant of a category page, generated by the storefront’s color, size, and price filters. We rewrote the internal facet links to use rel-canonical pointing to the unfiltered category and removed the parameter Disallow, which let Google recrawl and consolidate the URLs into the parent over 9 weeks.

A WordPress publisher we audited had a robots.txt file that began with Disallow: /. The directive had been added during a 2022 staging migration and never reverted. For 14 months, Google honored the block while still indexing pages discovered through external backlinks, which produced 1,800 flagged URLs in Search Console. We reverted the robots.txt to allow crawl, kept noindex on the staging subdirectory only, and the flag count cleared to zero across 6 weeks.

A SaaS landing-page site we audited had added Disallow: /resources/ to its robots.txt based on a recommendation surfaced inside a third-party SEO tool. The tool’s logic was designed for thin-content scenarios and did not account for the fact that /resources/ was the canonical category page for the site’s primary commercial keyword. The Disallow stayed in production for 9 weeks, during which the category page dropped out of the index. We removed the Disallow and the page returned to the index within 11 days.

What this means for technical SEO teams

Robots.txt is treated as a set-and-forget configuration file, and almost nobody audits it after the initial write. From our audit work, the teams that ship a quarterly robots.txt review against current Search Console coverage data resolve “Indexed, though blocked” flags before they accumulate, while teams without that review carry the same flags for over a year. The fix is procedural, not technical.

Limitations and how we’d improve the next round

Three limitations to flag. First, our sample reflects sites that hired a technical SEO consultant, which biases toward sites that had already recognized a problem. Second, the flagged-site subset of 72 is small enough that vertical-specific rates carry wide error bars. Third, root-cause attribution depends on owner self-report combined with our forensic review of version history.

The next round will pair the audit dataset with server-log sampling on a subset of consenting clients, which will let us ground-truth the “rule was never effective” claim against actual Googlebot request patterns.

Cite this research

If you reference this research, please link to this page and use the source line below.

Matcha Growth 2026 Technical SEO Audit Dataset. 312 sites audited in the last 12 months on robots.txt configuration, GSC coverage warnings, and indexing patterns. https://matchagrowth.com/technical-seo-audit-dataset-2026/

HTML snippet:

<p>According to <a href="https://matchagrowth.com/technical-seo-audit-dataset-2026/">Matcha Growth's 2026 Technical SEO Audit Dataset</a>, 79% of "Indexed, though blocked by robots.txt" URLs on ecommerce sites are faceted-nav or parameter variants of an unfiltered category page.</p>