SEO Log File Analysis: Verify Googlebot and Find Crawl Gaps

A request log answers a narrow question: what reached this logging layer, and what response did that layer record? It does not show whether Google rendered the page, chose it as canonical, or kept it in the index. Even a 200 response is only evidence of serving, not successful indexing.

Sources:[5] Google Crawling Infrastructure[6] Google Search Console Help

Start with a decision you could change. For a catalog release, that might be whether newly published category pages are being requested at all. For a migration, it might be whether a retired route still receives crawler requests. “Find crawl waste” is too loose: it encourages a large count to become a verdict before you have decided which pages should be crawled.

Sources:[3] Google Search Console Help[6] Google Search Console Help

Sources:[1] Google Crawling Infrastructure[6] Google Search Console Help

Use Search Console → Settings → Crawl stats as a cross-check, not as the source CSV for this audit. Its URL examples are not exhaustive. A mismatch with your logs is a reason to check scope, dates and capture completeness before calling either dataset wrong.

Sources:[3] Google Search Console Help

When a CDN sits in front of the origin, decide which layer represents the question. For crawler-facing responses, use an edge export when it is available and sufficiently complete. In Cloudflare’s HTTP requests dataset, EdgeResponseStatus is the status returned to the client; OriginResponseStatus describes a different leg. Do not concatenate both as though they were separate visits.

Sources:[7] Cloudflare Docs

An origin-only report can still be useful, but label it that way. A URL absent from that report may have been served at the edge. Equally, an edge export with sampling or gaps cannot support a claim that a particular URL received no requests. Keep a small acquisition note alongside the file: host scope, logging layer, UTC interval, sampling policy, missing periods and export filters.

Sources:[7] Cloudflare Docs

Keep duration fields out of the first coverage pass unless you have verified their units and boundaries. NGINX $request_time, for example, is measured in seconds from the first client bytes to the log write after the last response bytes are sent. Calling it “TTFB in milliseconds” would change both the quantity and its unit.

Sources:[8] NGINX

A Googlebot-looking user-agent is a claim made by the client. Google warns that it can be spoofed. For batch work, match the recorded client IP against the published common-crawler ranges and then identify the crawler product from the user-agent. Keep Search Googlebot separate from image/video crawlers, AdsBot and user-triggered inspection tools.

Sources:[1] Google Crawling Infrastructure[4] Google Search Central

The verification documentation checked on 29 September 2026 points to common-crawlers.json. Save a copy with its retrieval date and hash. The audit records the snapshot hash and its creationTime, so a second analyst can identify the same input instead of silently using a newer list.

Sources:[1] Google Crawling Infrastructure

def matches_ranges(address: str, networks: list[Any]) -> bool:
    ip = ipaddress.ip_address(address)
    return any(ip.version == net.version and ip in net for net in networks)

Sources:[9] Python documentation

For one suspicious address, follow the official reverse-DNS procedure: resolve the IP to a hostname, check the permitted Google domain boundary, resolve that hostname forward, and confirm that the original address is returned. A hostname merely containing the word “google” is not sufficient. Keep the commands and responses with the case rather than replacing the evidence with a boolean.

Sources:[1] Google Crawling Infrastructure

Request volume and URL coverage answer different questions. Repeated requests to one page increase volume without expanding coverage. Use a left join from the owned inventory to the filtered requests, keeping URLs with no match. Starting with the log and joining outward makes the absent pages disappear from your report.

For this workflow, observed coverage is distinct eligible URLs with at least one verified HTML GET / all eligible URLs in the fixed cohort. A separate serving measure requires a 200 or 304 response. These are audit definitions, not Google thresholds. A requested page that returned 503 is observed but does not pass the serving check.

Keep 304 separate from redirects. It indicates that the cached representation can be reused; lumping every 3xx into a “redirect waste” bucket misclassifies it. At the same time, a 200 can still contain a soft error page, so inspect response content before calling an affected template healthy.

Sources:[5] Google Crawling Infrastructure

Treat exact URL identity as a join contract. Do not strip all query strings or merge trailing-slash variants merely to increase the match rate. If the export and inventory use different encodings or host representations, reconcile them with a documented mapping. Preserve the original URL so a suspicious group can be traced back to individual requests.

My priority order is serving failures, missed important pages, then unwanted URL families. It puts a failing destination ahead of a large but possibly harmless request bucket. The finding should name the page cohort and the competing explanation, not just the largest bar in a chart.

Sources:[2] Google Crawling Infrastructure[5] Google Crawling Infrastructure

A reduction in requests to unimportant URLs is not proof that Google will spend the difference on your preferred pages. Google describes both crawl capacity and crawl demand, and explicitly warns against treating temporary robots.txt blocking as a way to reallocate budget. Diagnose the constraint before setting a crawl-volume target.

Sources:[2] Google Crawling Infrastructure

Likewise, noindex is not a crawl-saving instruction: Google must request a crawlable page to encounter it. When the issue is filter URL proliferation, use the separate faceted navigation policy guide. When requests succeed but index inclusion remains the concern, move to the indexation investigation rather than stretching log evidence beyond its scope.

Sources:[2] Google Crawling Infrastructure

The SEO log file analysis kit contains a standard-library Python script, a normalized request CSV, a four-URL inventory, a deliberately synthetic range list, tests and a handoff template. No database account or external service is required. It does not fetch your site or modify crawl controls.

Expected results: 4 eligible URLs, 3 observed URLs, 2 URLs served with 200 or 304. The six verified HTML GETs include one search URL outside the inventory. The asset request is kept out of HTML coverage; the spoofed user-agent, inspection-tool request, duplicate export and out-of-window row do not inflate the result. The HEAD request is reported separately.

Read summary.json before opening coverage.csv. The summary records filters and discarded-row categories; the CSV preserves every inventory URL, its observed and successful request counts, first/last observations and statuses. If the fixture differs, stop and resolve the input or code change. A pleasing percentage is not a substitute for reproducing the known result.

For production, replace all three inputs with your normalized log export, owned inventory and the saved official range snapshot. Remove --allow-synthetic-ranges; choose your real interval and a new output directory. The tool intentionally rejects duplicate inventory URLs, malformed rows and conflicting request IDs instead of silently repairing them. Resolve those issues at the source and rerun.

Before changing anything, write a one-sentence hypothesis. For example: “The new category URLs are linked only after an unavailable interaction, which may explain the absence of qualifying requests in this complete edge window.” The absence is an observation; the explanation is a hypothesis. The next step is to inspect the links and access conditions, not to state that Google has rejected the pages.

Do not require equal crawl volume in the before and after periods. New content, demand and site changes can alter the population. Compare response failure rates and coverage with their denominators, and describe any change in eligibility. If a new window is incomplete, mark it incomplete instead of converting its missing requests into an apparent improvement.

The first useful deliverable is a small list of important URLs whose observed behavior conflicts with their intended role. Start there. Logs can show that a serving or discovery problem deserves attention; closing the SEO question may still require rendering checks and index evidence. Keep those conclusions separate in the handoff.