Faceted Navigation SEO: Crawl & Indexation

Stage 1 · Define the system

Start with the URL space, not with a robots.txt rule

A filter interface can be small on screen and enormous in crawl space. Before changing directives, model what URLs the UI can generate and how crawlers can discover them.

Google's current faceted-navigation documentation describes the core failure mode plainly: parameterized facets can create a very large or effectively infinite URL space. Crawlers usually cannot know whether a new combination is useful until they request it, so uncontrolled facets can cause overcrawling and slow discovery of genuinely useful URLs. The first diagnostic object is therefore not a single URL; it is the URL grammar that generates the cohort.

SourceGoogle Crawling Infrastructure — Managing crawling of faceted navigation URLs · Google Crawling Infrastructure — Crawl Budget Management

Build the facet inventory before you touch crawl controls

1
Map every URL-producing state
From representative category pages, record each facet key, allowed values, multi-select behavior, sorting, pagination, tracking parameters and whether parameter order can change. Include states created only after JavaScript interaction.
PathCategory template → filter UI → generated href/history URL → server response
Pass criterionYou can enumerate the parameters and path segments that create distinct URLs and explain which UI actions generate each one.
SourceGoogle Crawling Infrastructure — Managing crawling of faceted navigation URLs · Google Search Central — URL structure best practices for Google Search
2
Separate facets from other URL mutations
Do not put filtering, sort order, pagination and campaign parameters into one undifferentiated bucket. They can produce similar-looking URLs but may need different crawl, indexing and canonical behavior.
Pass criterionEvery observed URL mutation has a named class and an owner: filter, sort, pagination, tracking or application state.
SourceGoogle Search Central — Pagination, incremental page loading, and their impact on Google Search · Google Search Central — How to specify a canonical URL with rel=canonical and other methods
3
Estimate combinatorial growth
Multiply the available values only as a rough upper-bound exercise; then compare that theoretical space with URLs actually exposed in links, sitemaps, logs and Search Console. The calculation is diagnostic, not a traffic forecast.
Pass criterionYou have both a theoretical URL-space risk and an observed discovery/crawl sample, kept explicitly separate.
SourceGoogle Crawling Infrastructure — Managing crawling of faceted navigation URLs · Google Search Console Help — Crawl Stats report

Custom diagram

A few facet controls can multiply into a crawl graph

The diagram separates a base category from filter dimensions and the resulting URL states, showing why eligibility decisions must happen before every combination becomes a crawlable link.

The SEO risk comes from the number of discoverable URL states, not from the number of visible filter controls.

Stage 2 · Decide before directives

Give every filter state an explicit crawl and index role

The durable decision is not “robots or canonical?” It is whether a URL state should exist for search at all, and what each layer must do once that role is chosen.

My recommended framework is a URL eligibility contract to keep implementation decisions consistent. It is an engineering policy, not a Google-defined taxonomy. Each state gets one intended role, and then discovery, response status, index directive, canonical signal, internal links and sitemap inclusion are checked against that role. This prevents contradictory combinations such as a URL blocked in robots.txt while the team expects Google to recrawl it and observe a noindex tag.

SourceGoogle Search Central — Block Search indexing with noindex · Google Search Central — How to specify a canonical URL with rel=canonical and other methods · Google Search Central — Build and submit a sitemap

URL eligibility contract for common faceted-navigation states
State	Crawl	Index intent	Canonical / sitemap	Typical implementation	Validation
Indexable search landing page	Allowed	Eligible	Self-canonical; include canonical URL in sitemap	Stable URL, meaningful inventory, crawlable internal links	200, indexable, self-canonical, linked, present in intended sitemap cohort
Crawlable duplicate/support variant	Allowed when needed	Not independently preferred	Canonical to preferred URL; omit duplicate from sitemap	Keep only when UX/technical need justifies fetchability	Declared canonical is consistent; internal links favor preferred URL
User-only filter state	Normally prevent broad crawl discovery	No standalone search landing-page role	Omit from sitemap	Button/form state, fragment when appropriate, or robots pattern for parameter URLs	Users can filter; unwanted URL cohort stops expanding in crawl evidence
Noindex exception	Allowed	Excluded from Search after Google sees directive	Do not use as a crawl-budget substitute	meta robots noindex or X-Robots-Tag noindex	Google can fetch the URL and observe noindex
Empty / impossible state	Fetch may occur until learned	Not a valid page	No sitemap; no canonical rescue	Return 404 at the same URL and remove links that generate it	Consistent 404; no recurring internal discovery path

SourceGoogle Crawling Infrastructure — Managing crawling of faceted navigation URLs · Google Crawling Infrastructure — Crawl Budget Management · Google Search Central — How to specify a canonical URL with rel=canonical and other methods · Google Search Central — Block Search indexing with noindex · Google Search Central — Build and submit a sitemap

Custom diagram

URL eligibility is a decision tree, not a directive lookup

A decision tree routes a filter state toward an indexable landing page, a crawlable duplicate, a user-only state, a noindex exception or a 404 based on user/search purpose and response validity.

Choose the role first; directives are implementation details of that role.

Stage 3 · Measure observed demand

Measure actual Googlebot demand before estimating crawl waste

Theoretical combinations tell you where risk exists. Verified server requests tell you whether Googlebot is spending requests on those states today.

Use raw access logs when available because they let you group requests by exact URL patterns, status codes and time. Do not trust the user-agent string by itself: Google documents reverse/forward DNS checks and published IP ranges for verifying crawler requests. Search Console's Settings → Crawl stats report is useful for request trends, response breakdowns and host issues, but its example URLs are not a complete substitute for server logs.

SourceGoogle Crawling Infrastructure — Verify requests from Google crawlers and fetchers · Google Search Console Help — Crawl Stats report

VERIFIED · Group verified Googlebot requests by facet signature

#!/usr/bin/env python3
"""Summarize verified Googlebot requests by faceted-navigation parameter signature.

Input CSV columns:
  timestamp,url,status,user_agent,verified_googlebot

`verified_googlebot` must be produced upstream by reverse/forward DNS validation
or by matching Google's published crawler IP ranges. User-agent text alone is not
sufficient evidence that a request came from Google.
"""

from __future__ import annotations

import csv
import sys
from collections import Counter
from pathlib import Path
from urllib.parse import parse_qsl, urlsplit

FACET_KEYS = {
    "brand",
    "color",
    "size",
    "price",
    "material",
    "rating",
    "sort",
}
TRUE_VALUES = {"1", "true", "yes", "verified"}


def facet_signature(url: str) -> str:
    """Return a stable, order-independent signature for known facet keys."""
    query = parse_qsl(urlsplit(url).query, keep_blank_values=True)
    keys = sorted({key.lower() for key, _ in query if key.lower() in FACET_KEYS})
    return "+".join(keys) if keys else "no-known-facets"


def summarize(csv_path: Path) -> Counter[tuple[str, str]]:
    counts: Counter[tuple[str, str]] = Counter()
    with csv_path.open(newline="", encoding="utf-8") as handle:
        reader = csv.DictReader(handle)
        required = {"url", "status", "verified_googlebot"}
        missing = required.difference(reader.fieldnames or [])
        if missing:
            raise ValueError(f"Missing CSV columns: {', '.join(sorted(missing))}")

        for row in reader:
            if row["verified_googlebot"].strip().lower() not in TRUE_VALUES:
                continue
            signature = facet_signature(row["url"])
            status = row["status"].strip() or "unknown"
            counts[(signature, status)] += 1
    return counts


def main() -> int:
    if len(sys.argv) != 2:
        print("Usage: python facet-log-classifier.py verified-requests.csv", file=sys.stderr)
        return 2

    counts = summarize(Path(sys.argv[1]))
    print("requests\tstatus\tfacet_signature")
    for (signature, status), requests in sorted(
        counts.items(), key=lambda item: (-item[1], item[0][0], item[0][1])
    ):
        print(f"{requests}\t{status}\t{signature}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

VERIFIED on Python 3.13.5 on 2026-09-01 against the synthetic CSV shipped with the research package. The script intentionally requires an upstream verified_googlebot flag; do not classify production traffic as Googlebot from the user-agent string alone.

SourceGoogle Crawling Infrastructure — Verify requests from Google crawlers and fetchers

Create a before-change crawl baseline

1
Verify crawler identity before aggregation
For production analysis, validate source IPs using Google's documented reverse/forward DNS process or published crawler IP ranges before marking rows as verified Googlebot.
PathServer access logs → source IP → Google crawler verification
Pass criterionFacet counts are based on verified crawler requests, not on an untrusted user-agent match.
SourceGoogle Crawling Infrastructure — Verify requests from Google crawlers and fetchers
2
Group by behavior, not by raw URL
Normalize representative parameters into signatures such as brand+color, size+sort or price. Keep HTTP status alongside the signature so error and redirect patterns remain visible.
Pass criterionYou can name the highest-request facet cohorts and their dominant status codes without manually reviewing thousands of full URLs.
3
Cross-check the site-wide trend in Crawl Stats
Compare request volume, response types and host availability around the same dates. Treat the report as high-level context; use logs for the exact facet cohort when you need URL-level completeness.
PathSearch Console → Settings → Crawl stats
Pass criterionThe log finding and Search Console trend do not materially contradict each other, or the discrepancy is documented for investigation.
SourceGoogle Search Console Help — Crawl Stats report

Stage 4 · Promote useful states

Only promote filter combinations that solve a stable search task

Some filter pages deserve their own search landing-page role. The useful question is not whether a facet exists, but whether the resulting page has a durable purpose distinct from the parent category and from other filter states.

The Google guidance used for this article does not provide a numeric threshold for when a filter combination deserves indexing. That decision remains site-specific. My screening rule is to require a stable user/search task, a durable result set or inventory concept, a URL that can remain stable, and enough distinct page purpose to justify being treated as its own landing page. Search demand can support the decision, but it does not override an empty, volatile or near-duplicate experience.

SourceGoogle Crawling Infrastructure — Managing crawling of faceted navigation URLs · Google Search Central — How to specify a canonical URL with rel=canonical and other methods

Evidence I want before an indexable facet becomes a landing page

Purpose: the combination answers a recognizable browsing or search need rather than exposing an arbitrary permutation.
Inventory: the result set is useful enough to remain a real destination and does not frequently collapse to zero or one irrelevant item.
Identity: the page is meaningfully distinguishable from the parent category and nearby facet combinations; a canonical tag cannot manufacture that distinction.
Durability: the URL grammar and parameter/path order are stable enough to link, monitor and keep consistent across releases.
Discovery: important landing pages can receive normal crawlable internal links rather than existing only inside a search box or transient client-side state.
Operations: the page can be kept self-canonical, indexable, included in the right sitemap cohort and monitored when inventory changes.

SourceGoogle Search Central — Help Google understand your ecommerce website structure · Google Search Central — Link best practices for Google · Google Search Central — How to specify a canonical URL with rel=canonical and other methods · Google Search Central — Build and submit a sitemap

Once a facet is promoted, treat it as part of your information architecture. Link to it consistently from relevant crawlable pages and avoid splitting signals across equivalent URL permutations. The same principle appears in Google's canonical guidance and is explored more deeply in Metricum Lab's internal linking architecture guide.

SourceGoogle Search Central — How to specify a canonical URL with rel=canonical and other methods · Google Search Central — Link best practices for Google

Stage 5 · Reduce URL discovery

Keep user-only filters useful without exposing every permutation

If a filter state has no independent search role, preserve the UX while removing the crawlable URL expansion path. This is where robots.txt, fragments and interaction design belong.

Google's faceted-navigation guidance gives two direct approaches when filtered states do not need to be potentially indexed: disallow their crawl patterns in robots.txt, or use URL fragments for filter state because Google Search generally does not crawl/index fragment states as separate URLs. Google also notes that canonical and nofollow are generally less effective long-term crawl controls for this problem. Use fragments only for states that truly do not need independent indexing; Google's broader URL guidance says fragments should not represent separately indexable content.

SourceGoogle Crawling Infrastructure — Managing crawling of faceted navigation URLs · Google Search Central — URL structure best practices for Google Search

STATICALLY VERIFIED · Example robots.txt policy for user-only query states

# Example policy: query-string states used only for UX stay out of crawl.
# Keep indexable facet landing pages on dedicated, crawlable URLs instead.
User-agent: Googlebot
Disallow: /*?*sort=
Disallow: /*?*price=
Disallow: /*?*size=

Sitemap: https://www.example.com/sitemap.xml

STATICALLY VERIFIED against Google's current robots.txt syntax and wildcard support on 2026-09-01. This is an illustrative policy, not a rule to paste unchanged: parameter names, order and existing Allow/Disallow rules must be tested against the production URL grammar. The indexable examples in this guide use dedicated crawlable landing-page URLs rather than exceptions inside these blocked patterns.

SourceGoogle Crawling Infrastructure — Managing crawling of faceted navigation URLs · Google Crawling Infrastructure — How Google interprets the robots.txt specification

Custom diagram

Control faceted navigation as a stack of independent layers

The diagram separates URL generation and discovery, crawl permission, index eligibility, canonical preference and post-release validation so one mechanism is not expected to do another layer's job.

robots.txt controls crawling; noindex controls index eligibility after crawl; canonical expresses a preferred representative; sitemaps and internal links reinforce the URLs you actually want discovered.

Stage 6 · Build canonical landing pages

Make indexable facet pages look intentional to both users and crawlers

Once a filter state earns a search role, stop treating it as a disposable parameter variant. Give it stable discovery, response and canonical behavior.

For a facet landing page that you want available to Search, keep the URL crawlable, return a normal successful response, use a stable parameter/path convention, and make canonical signals agree. Google's canonical documentation treats redirects and rel=canonical as strong signals and sitemap inclusion as a weaker signal; it also recommends linking internally to the canonical URL rather than duplicate variants. A self-canonical is not a guarantee of Google's selection, but it removes avoidable ambiguity.

SourceGoogle Search Central — How to specify a canonical URL with rel=canonical and other methods · Google Search Central — Build and submit a sitemap · Google Crawling Infrastructure — Managing crawling of faceted navigation URLs

Landing-page acceptance criteria

Returns 200 for a valid, non-empty state and does not redirect merely because the combination is unpopular.
Has one stable preferred URL; equivalent parameter-order permutations are not all linked or placed in the sitemap.
Uses a self-referencing canonical on the preferred page and points duplicate/support variants to the intended canonical when consolidation is appropriate.
Appears in the sitemap only if the URL is one you actually want Google to consider for Search.
Receives contextual <a href> links from relevant category/navigation surfaces when it matters for discovery.
Keeps the main product/category graph reachable even if user-only filters are implemented with buttons, forms or JavaScript state.

SourceGoogle Search Central — How to specify a canonical URL with rel=canonical and other methods · Google Search Central — Build and submit a sitemap · Google Search Central — Link best practices for Google · Google Search Central — Help Google understand your ecommerce website structure

Do not use canonical as a shortcut for an uncontrolled URL generator. Google notes that canonicalization can reduce crawling of non-canonical versions over time, but its faceted-navigation guidance still considers robots/fragments more effective crawl controls when those filtered URLs do not need to be crawled. The implementation should therefore reduce unwanted discovery at the source instead of relying on canonical cleanup after millions of variants exist.

SourceGoogle Crawling Infrastructure — Managing crawling of faceted navigation URLs · Google Search Central — How to specify a canonical URL with rel=canonical and other methods

Stage 7 · Close dead branches

Return 404 for empty or impossible states; normalize equivalent URL permutations

A faceted system needs explicit failure and normalization behavior. Empty, duplicate-filter or impossible states should not stay as soft destinations, while equivalent parameter-order URLs should converge on one identity.

Google's faceted-navigation guidance specifically recommends a 404 response when a filter combination returns no results or is otherwise nonsensical, and says not to redirect such URLs to a generic empty listing. This lets crawlers learn that the state is not a valid resource. The engineering work is to make this response deterministic across server rendering, API failures and client-side transitions.

SourceGoogle Crawling Infrastructure — Managing crawling of faceted navigation URLs

Failure-state rules worth encoding in automated tests
Observed state	Preferred behavior	Why	Verification
Valid combination with useful inventory	200 + intended eligibility policy	It is a real user destination	Status, canonical, robots meta, product set and links match contract
Zero-result combination	404	No valid resource exists at this state	HTTP response remains 404 in SSR/direct request and is not converted to 200 by the app shell
Impossible or unknown facet value	404	Prevents arbitrary parameter values from becoming valid crawl branches	Random invalid values consistently return 404
Same state in different parameter order	Normalize to one URL or consolidate duplicates	Avoids multiple identities for one result set	Crawler/log sample shows one preferred order/URL
Sort-only variant	Normally user-only / non-indexable policy	Ordering changes often do not create a distinct landing-page purpose	Not in sitemap; crawl/index behavior matches contract

SourceGoogle Crawling Infrastructure — Managing crawling of faceted navigation URLs · Google Search Central — Pagination, incremental page loading, and their impact on Google Search · Google Search Central — How to specify a canonical URL with rel=canonical and other methods

Stage 8 · Preserve product discovery

Do not solve crawl control by hiding the catalog from crawlable links

It is possible to stop facet URL explosion and accidentally make products harder to discover. Crawl control and core catalog discovery must be validated separately.

Google's ecommerce guidance recommends making products reachable through site navigation and notes that Googlebot generally does not submit searches into a site's search box. Its link guidance says normal crawlable links are usually <a href> elements; event-only pseudo-links are not a reliable replacement. Therefore a non-indexable filter UI can use buttons/forms/fragments for state, while the underlying category → subcategory → product graph still needs crawlable paths or another deliberate discovery mechanism such as a sitemap.

SourceGoogle Search Central — Help Google understand your ecommerce website structure · Google Search Central — Link best practices for Google · Google Search Central — Build and submit a sitemap

Regression-test discovery after changing the filter UI

1
Crawl from the category hierarchy without executing filter interactions
Start from the same seed URLs a search crawler can discover and verify that important categories, subcategories and products remain reachable through crawlable links or are deliberately represented in a sitemap.
Pass criterionCritical product/detail URLs are still discoverable without requiring a crawler to click filter buttons or submit an internal search form.
SourceGoogle Search Central — Help Google understand your ecommerce website structure · Google Search Central — Link best practices for Google · Google Search Central — Build and submit a sitemap
2
Open indexable facet URLs directly
For promoted landing pages, test a fresh direct request and rendered output. If client-side routing changes the URL, make sure the server/SSR path resolves the same intended resource instead of depending on a prior UI state.
Pass criterionA promoted facet URL is independently loadable and returns the same intended content/metadata when requested directly.
SourceGoogle Search Central — URL structure best practices for Google Search
3
Keep user-only state out of the crawl graph
Inspect rendered anchors and URL changes after using non-indexable filters. The UX can remain interactive, but it should not emit thousands of new crawlable links for states the eligibility contract excludes.
Pass criterionUser-only filters work for people while the rendered link graph exposes only the URLs intended for crawl discovery.
SourceGoogle Crawling Infrastructure — Managing crawling of faceted navigation URLs · Google Search Central — Link best practices for Google

If filters are JavaScript-driven, validate the rendered DOM and direct-route behavior rather than assuming the framework will make the right crawl/index decision. Metricum Lab's JavaScript SEO debugging guide covers the separate crawl → server response → render → index evidence chain for client-rendered applications.

SourceGoogle Search Central — Link best practices for Google · Google Search Central — URL structure best practices for Google Search

Stage 9 · Verify the same cohorts

Validate crawl reduction and landing-page health as separate outcomes

A successful rollout should reduce unwanted facet crawling without sacrificing discovery and indexability of the filter pages you intentionally kept.

Do not declare success because a crawler now reports fewer URLs. Compare the same parameter signatures and landing-page cohorts before and after release. Google also cautions that freeing crawl capacity does not guarantee that the saved requests will be reallocated to other URLs unless the site was already constrained by its crawl capacity limit. The defensible outcome is therefore less unwanted crawling plus healthy intended URLs, not a promised percentage increase in crawling elsewhere.

SourceGoogle Crawling Infrastructure — Crawl Budget Management · Google Search Console Help — Crawl Stats report

Post-release validation loop

1
Re-run the verified log cohorts
Use the same signatures, verified crawler criteria and comparable date windows as the baseline. Check request counts, status-code mix and newly appearing parameter patterns.
Pass criterionUnwanted facet cohorts stop expanding or materially decline, while valid landing-page and product crawling remains observable.
SourceGoogle Crawling Infrastructure — Verify requests from Google crawlers and fetchers
2
Check Crawl Stats for site-wide side effects
Review crawl requests, response breakdowns and host availability around deployment. A crawl-control release should not coincide with a new serving problem that explains the change instead.
PathSearch Console → Settings → Crawl stats
Pass criterionThe crawl change is not explained by host errors, availability degradation or another unrelated site-wide event.
SourceGoogle Search Console Help — Crawl Stats report
3
Sample each eligibility state
Test indexable landing pages, crawlable duplicate variants, user-only states, noindex exceptions and invalid/empty combinations separately. Verify status, robots access, meta robots, canonical, links and sitemap membership against the contract.
Pass criterionEvery sampled URL behaves according to its assigned state; contradictory signals are treated as defects, not exceptions.
SourceGoogle Crawling Infrastructure — Managing crawling of faceted navigation URLs · Google Search Central — Block Search indexing with noindex · Google Search Central — How to specify a canonical URL with rel=canonical and other methods · Google Search Central — Build and submit a sitemap
4
Inspect indexation as a cohort, not a one-URL anecdote
Use Search Console to monitor representative landing-page cohorts and investigate unexpected canonical/index states. A single URL can lag or change for unrelated reasons, so preserve cohort-level evidence and the release date.
Pass criterionImportant landing-page cohorts remain eligible/discoverable and there is no new systematic indexation regression after crawl controls ship.
SourceGoogle Search Central — How to specify a canonical URL with rel=canonical and other methods

Stage 10 · Make the policy durable

Make faceted navigation a release gate, not a one-time cleanup

Facet systems change with catalog attributes, merchandising, frontend rewrites and localization. The control policy has to ship with those changes.

Release checklist for every new facet or URL rule

The new facet has an explicit eligibility state before launch.
Valid, invalid, zero-result and multi-select states have deterministic HTTP behavior.
Parameter/path order and normalization rules are covered by tests.
User-only states do not add uncontrolled crawlable links or sitemap URLs.
Indexable landing pages remain directly loadable, self-canonical and intentionally linked.
robots.txt changes are tested against representative production URL patterns and do not block resources/pages that must be fetched.
Noindex exceptions remain crawlable so the directive can be observed.
Sitemap generation includes only the canonical URLs intended for Search.
A verified Googlebot/log cohort and Search Console checkpoint are scheduled after release.
New facet behavior is checked on mobile, SSR/direct requests and JavaScript-enhanced journeys where applicable.

SourceGoogle Crawling Infrastructure — Managing crawling of faceted navigation URLs · Google Search Central — Block Search indexing with noindex · Google Search Central — How to specify a canonical URL with rel=canonical and other methods · Google Search Central — Build and submit a sitemap · Google Search Central — Help Google understand your ecommerce website structure

The practical principle is simple: control the URL inventory before you try to clean up crawler behavior after the fact. Start by mapping one representative category and writing the eligibility contract for its filter states. The critical limitation is that no generic rule can decide which facets deserve organic landing pages for every business; that requires real inventory, query and conversion context. Once the role is chosen, however, the implementation is testable: URL generation, crawl permissions, status codes, canonical signals, internal links and sitemap membership can all be validated against the same contract.

SourceGoogle Crawling Infrastructure — Managing crawling of faceted navigation URLs · Google Crawling Infrastructure — Crawl Budget Management · Google Search Central — How to specify a canonical URL with rel=canonical and other methods

If Search Console later reports valuable facet pages as Crawled — currently not indexed, diagnose that as a separate index-selection problem after first proving that the faceted-navigation policy is behaving as designed. Crawl control narrows the URL space; it does not guarantee index selection.

SourceGoogle Crawling Infrastructure — Crawl Budget Management · Google Search Central — How to specify a canonical URL with rel=canonical and other methods

strategy branches in Google guidance

for empty or nonsensical combinations

standard separator for query parameters

Build the facet inventory before you touch crawl controls

Map every URL-producing state

Separate facets from other URL mutations

Estimate combinatorial growth

A few facet controls can multiply into a crawl graph

URL eligibility is a decision tree, not a directive lookup

VERIFIED · Group verified Googlebot requests by facet signature

Create a before-change crawl baseline

Verify crawler identity before aggregation

Group by behavior, not by raw URL

Cross-check the site-wide trend in Crawl Stats

Evidence I want before an indexable facet becomes a landing page

STATICALLY VERIFIED · Example robots.txt policy for user-only query states

Control faceted navigation as a stack of independent layers

Landing-page acceptance criteria

Regression-test discovery after changing the filter UI

Crawl from the category hierarchy without executing filter interactions

Open indexable facet URLs directly

Keep user-only state out of the crawl graph

Post-release validation loop

Re-run the verified log cohorts

Check Crawl Stats for site-wide side effects

Sample each eligibility state

Inspect indexation as a cohort, not a one-URL anecdote

Release checklist for every new facet or URL rule

Sources

Turn an exploding filter graph into a testable crawl policy