Capability: Serpdex (Indexation Audit)

Alex Chen, Senior SEO Strategist 8+ years in technical SEO and search engine optimization

What is it

An indexation audit is a systematic process that compares the URLs your site wants indexed (submitted via sitemap) against what Google actually has in its live search index. The critical gap—often called the "indexation delta"—can reveal massive hidden leakage of crawl budget and content authority.

Consider this common scenario:

• Sitemap submitted URLs: 12,000
• GSC Coverage (discovered URLs): 9,800
• Live indexed (via site: search or URL Inspection): 5,400

That gap of 4,400 URLs between "discovered" and "indexed" represents content that Google knows about but has chosen not to show in search results. Without an indexation audit, you are flying blind.

The "Serpdex" capability automates this entire workflow. When you Run Serpdex for my site, the system pulls the full URL list from your sitemaps, supplements it with Google Search Console (GSC) coverage data, and then checks each unique URL against Google’s index using a programmable URL Inspection API. The output is a granular, categorized report: which pages are indexed, which are excluded (with the specific exclusion reason—e.g., "Crawled - currently not indexed," "Duplicate without user-selected canonical," "Blocked by robots.txt"), and which the system could not verify due to quota or error limits.

This is far more precise than a loose site: search, which often returns incomplete results, shows stale URLs, or fails on large sites.

When to use it

An indexation audit is not a "set and forget" task. You should trigger a Run Serpdex for my site scan under these concrete conditions:

• After a site migration (domain change, protocol change, or platform move): Redirects can break, sitemaps can go stale, and 302 chains can prevent indexation. Run the audit within 48 hours of the migration completion to catch orphaned or non-indexed pages before traffic drops.
• After a large content push (e.g., adding 500+ pages): Not all new content gets indexed quickly, especially if Google considers it thin or duplicative. The audit reveals which new URLs were picked up and which were silently excluded.
• When organic traffic drops unexpectedly by 20%+: A sudden indexation dump (e.g., a "not selected" wave after a core update) is often the root cause. The audit diagnoses whether the traffic drop is algorithmic or indexation-related.
• When you suspect duplicate content or canonical issues: If multiple versions of the same page exist (www vs. non-www, trailing slashes, session IDs), the audit will show which canonical URL Google chose to index and which were consolidated.
• As a periodic health check (monthly for sites > 10,000 pages): Indexation lags, site errors, and Google policy changes can degrade indexation over time without drastic traffic signals. Monthly scans create a baseline.

Where does it run

The Run Serpdex for my site capability operates inside a specialized SEO intelligence platform—a unified orchestration environment that integrates:

• A sitemap fetcher that parses XML sitemaps (including nested sitemaps) and extracts all <loc> URLs.
• A GSC API connector that pulls the coverage, discovery, and performance reports for the property, deduplicating URLs already found in sitemaps.
• An indexation checker that makes authenticated URL Inspection API calls (one per URL, respecting Google’s daily quota of 2,000 queries per property per day for verified owners).
• A report engine that cross-references the collected data and categorizes results using the official Google indexation status codes.

The platform runs on-demand (triggered via a button click or API call) or on a scheduled cron job. All processing happens server-side; you do not need to keep your browser open or manually copy-paste URLs. The system respects rate limits and logs each API response for debugging.

How it works

The following numbered steps describe the exact workflow when you Run Serpdex for my site:

1. URL Collection (Union)

• The system fetches all sitemap URLs for the verified property. For a site with multiple sitemaps (e.g., sitemap-posts.xml, sitemap-products.xml, sitemap-categories.xml), it recursively follows the sitemap index file.
• Simultaneously, it retrieves the GSC "Coverage" report (the "URLs discovered" and "URLs submitted but not indexed" tables). These two sources are merged into a single deduplicated list of unique URLs.

Example: A site with 15,000 sitemap URLs and 2,000 GSC-only URLs yields a combined pool of 17,000.

1. Batched Quota Budgeting

• If the total pool exceeds the daily API quota (2,000 queries/property/day), the system intelligently prioritizes: it checks the GSC "submitted but not indexed" URLs first, then the remaining sitemap URLs, and last the "discovered but not submitted" ones. This ensures the most critical gaps are checked within the budget.
• The system batches requests (50 URLs per batch) to stay within API rate limits (1 query per second for most Google services).

1. URL Inspection (Simulated "Live Test")

• For each URL in the prioritized list, the system calls the URL Inspection API endpoint. This returns the live indexation status, including:
• INDEXED (the URL is in the index)
• INDEXED_NOT_SELECTED (indexed but not served in search results—e.g., soft 404, canonical redirect)
• CRAWLED_NOT_INDEXED
• NOT_CRAWLED
• EXCLUDED with a specific reason (robots.txt block, noindex tag, 404, 503, etc.)
• Each response is stored raw for later debugging.

1. Categorization and Delta Detection

• The results are grouped into three tiers:
• Indexed: URLs the API confirms are in the index.
• Not Indexed (with reason): URLs that are discovered but excluded (e.g., "Crawled – currently not indexed," "Duplicate – submitted URL not selected as canonical").
• Unverified: URLs that could not be checked (quota exhausted, API error, timeout).
• A comparison is made against the original sitemap submission: "sitemap submitted but not indexed" is the most actionable group.

1. Final Report

• The system generates a summary dashboard showing:
• Total submitted URLs
• Total indexed
• Indexation rate (%)
• Breakdown of exclusion reasons
• A detailed CSV/export with per-URL status
• This is not a simple site: search—it uses the full API and covers URLs not visible in any site: query (e.g., parameter-laden URLs, paginated series).

Trade-offs and counter-arguments

No indexation audit is perfect. The following table outlines the primary limitations of the Serpdex workflow and practical mitigations.

Limitation	Mitigation
Daily API quota (2,000 URLs/property)	Divide large sites into multiple batches across days, or switch to a sampling strategy (e.g., the system automatically prioritizes high-value pages). For sites > 50,000 URLs, consider using the GSC "discovered" list as a proxy instead of full sitemap check.
False negatives (URL appears indexed in API but not showing in results)	The API state "INDEXED" means Google acknowledges the URL exists, but it may not serve it in results if the page is soft-404 or devalued. Cross-reference with GSC Performance data (impressions > 0) to confirm actual visibility.
Indexing lag (URL submitted yesterday, API still shows "not indexed")	The API reflects near-real-time state, but indexation can take 24–72 hours for new URLs. Run the audit at least 7 days after a content push to avoid false alerts.
Cannot check every URL on extremely large sites (> 100,000)	Use the "sitemap-only" mode (limited to sitemap URLs) rather than "sitemap + GSC" union. For the rest, rely on GSC coverage reports, which aggregate counts without per-URL checks.
API errors (500s, quota resets at midnight UTC)	The system automatically retries failed requests with exponential backoff (max 3 retries) and logs errors for manual review. Plan runs early in the UTC day to maximize quota window.

FAQ

How often should I run this? For sites under 10,000 pages, a monthly scan is sufficient unless you make a major change (migration, large content push). For larger sites (50,000+ pages), schedule a weekly sampling scan and a full audit monthly. Avoid running more than once every 48 hours to respect quota and avoid noise from indexing lag.

What causes pages to be "Crawled - currently not indexed"? This is one of the most common and frustrating statuses. It means Google has crawled the URL but decided, at least temporarily, not to add it to the index. Common causes: thin content, low authority, technical issues (broken structured data), or a page that duplicates another existing URL. Fixes include improving content quality, adding internal links, and ensuring canonical tags are correct.

Does this fix indexing issues or just detect them? Detection only. Serpdex is a diagnostic tool—it tells you what is broken and why (e.g., "excluded due to noindex tag" or "crawled but not selected"). You must then take remediation steps: remove noindex tags, fix 404s, improve content, update sitemaps, or submit reconsideration requests. The report provides the list of URLs requiring action.

How does this differ from a site: search? A site: search is a loose query that often returns a subset of indexed pages (Google limits results to a few hundred per query). It does not show excluded URLs, does not reveal why a page is excluded, and misses URLs with complex parameters. Serpdex uses the authoritative URL Inspection API, which returns the full status and reason for each specific URL.

Can I run this on a site I don't own (e.g., a competitor)? No. The URL Inspection API requires verification via Google Search Console (ownership via DNS record, HTML file, or Google Analytics/Google Tag Manager account). Without verification, the API will refuse all requests. For competitor analysis, you must rely on indexation proxies (like comparing site: results or using third-party crawl tools), which are less accurate.

Takeaway

An indexation audit is the first step in fixing unseen content gaps. The Serpdex capability bridges the blind spot between what you submit and what Google actually serves, giving you a prioritized, per-URL action list. Run it after any major site change, treat it as a recurring health check, and never assume your sitemap equals your index.