Monitor Indexing With URL Inspection API

The URL Inspection API gives you a proxy for Google’s internal index view, but it is not a real-time rank tracker, and its quota is tight—engineering teams must build a sampling strategy around coverage states, canonical validation, and property-level scope shifts to avoid false negatives and denial-of-service bans.

The Problem

Founders and engineering leads often treat the URL Inspection API as a “Google say yes/no” check. They hit it hourly for every new URL, trigger alarms when the API returns NOT_IN_INDEX, and don’t account for the fact that the API reflects the state of the index as of the last crawl, not the moment you hit the endpoint. Worse, they treat the result as a rank signal—e.g., “URL is indexed, so we’re ranking”—which the API never claims to provide.

The real pain is threefold: quota exhaustion (the free tier allows only 2,000 queries per day per property, per Google Search Console documentation), sampling bias (inspecting only the top 100 URLs by traffic misses the long tail of discoverability issues), and scope mismatch (a URL might be indexed at the domain-level property but not in a country-specific property, or vice versa). Without a structured, quota-aware sampling strategy, you either burn your daily budget on a few high-value URLs or miss indexation regressions that affect 80% of your content.

Core Framework

Key Principle 1: The API is a snapshot of the last known index state, not a live endpoint

The URL Inspection API returns the coverage state (e.g., INDEXED, NOT_IN_INDEX, DUPLICATE, CRAWL_ERROR) based on data that Google’s indexing pipeline has already processed. According to Google’s official documentation, the API “provides information about the status of a URL as seen in Google’s index, including the most recent crawl and indexing information.” That means if you publish a page and immediately call the API, you will get a NOT_IN_INDEX response until Google’s crawler visits and processes the URL—which can take hours or days. This is not a failure of your deployment; it’s a timing constraint.

Example: A SaaS startup publishes a product page and immediately runs a curl against the API. They get NOT_IN_INDEX and trigger a PagerDuty alert. The real issue: they should have waited for a crawl event (or used the urlInspection.index endpoint after a submission, which still isn’t real-time). The API is a diagnostic tool, not a deployment verification.

Key Principle 2: Quota is per property, per day, and must be stratified by URL priority

The official quota is 2,000 queries per day per Search Console property (web property or domain property). This is not per user; it’s shared across all API calls for that property. You must design a sampling strategy that allocates limited queries across the URL corpus in a way that maximizes coverage of potential indexation problems.

Mental model: Treat the API quota as a budget. Evaluate your corpus size. If you have 100,000 URLs, you cannot inspect them all daily. You need to sample:

• High-risk URLs (newly published, recently updated, or flagged by logs as noindex)
• Stratified random sample from each coverage state (e.g., 10% of INDEXED, 50% of NOT_IN_INDEX if you have few)
• Seasonal / event-driven URLs (campaign pages, landing pages with short TTL)

A naive flat rate (e.g., inspect the first 2,000 URLs alphabetically) will miss the dynamic changes that matter.

Step-by-Step Execution

Step 1: Define property scope and validate API access

Before writing any code, confirm you have the correct Search Console property. The URL Inspection API works on properties (domain properties like sc-domain:example.com or URL-prefix properties like https://www.example.com/). If you have multiple properties (e.g., one for each country subfolder), you must run the API against each property separately. The API does not aggregate across properties.

Action: Create a list of all Search Console properties you need to monitor. For each, verify that you have the owner or user permission (API requires authentication via OAuth 2.0). Use the Google Search Console API’s sites.list method to confirm property existence.

Code snippet (Python): from google.oauth2 import service_account from googleapiclient.discovery import build

SCOPES = ['https://www.googleapis.com/auth/webmasters.readonly'] creds = service_account.Credentials.from_service_account_file('key.json', scopes=SCOPES) service = build('searchconsole', 'v1', credentials=creds)

List all properties

sites = service.sites().list().execute() for site in sites.get('siteEntry', []): print(site['siteUrl'], site['permissionLevel'])

Step 2: Implement quota-aware sampling with a priority tier system

Your sampling algorithm must respect the daily quota. Build a priority queue:

• Tier 1 (high priority): URLs that are newly published (last 24 hours) or URLs that recently changed status in your CMS (e.g., noindex removed). Allocate 20% of daily quota to these.
• Tier 2 (medium priority): URLs that have been in NOT_IN_INDEX or DUPLICATE for more than 7 days. Allocate 30%.
• Tier 3 (low priority): Random sample from the full corpus, stratified by coverage state. Allocate 50%.

Use a scheduled job (e.g., cron hourly) that pulls from the queue, but never exceed the cumulative daily quota. Log all API calls to a database with a timestamp and quotaUsed column.

Example strategy for 2,000 daily quota:

Step 3: Parse and store coverage state, canonical fields, and indexation signals

For each inspected URL, the API returns a response object with fields including:

• inspectionResult.coverageState (e.g., INDEXED, NOT_IN_INDEX, DUPLICATE, CRAWL_ERROR)
• inspectionResult.indexStatusResult.verdict (e.g., PASS, FAIL)
• inspectionResult.inspectionResultLink (a link to the Search Console UI for that URL)
• inspectionResult.canonical (the canonical URL as Google sees it)
• inspectionResult.robotsTxtState (whether the URL is blocked by robots.txt)
• inspectionResult.sitemap (sitemap that submitted the URL)

Critical: Store both the coverageState and the canonical field. If the canonical differs from the URL you inspected, that indicates a consolidation issue. Log the canonical to detect soft 404s or redirect loops.

Example database schema: sql CREATE TABLE url_inspections ( url TEXT PRIMARY KEY, property TEXT, timestamp TIMESTAMP, coverage_state TEXT, canonical TEXT, robots_txt_state TEXT, verdict TEXT, created_at TIMESTAMP DEFAULT NOW() );

Step 4: Set up alerting on state transitions, not static states

The most valuable signal is a change in indexation status. If a URL was INDEXED yesterday and is now NOT_IN_INDEX, that’s a problem. But if it has always been NOT_IN_INDEX, it’s noise.

Action: For each URL, compare the latest coverage state with the previous value from your database. If the state transitions from INDEXED to anything else, send an alert (e.g., Slack, PagerDuty). Set a cooldown per URL (e.g., max 1 alert per hour) to avoid storm.

Alert criteria: - INDEXED → NOT_IN_INDEX (critical – possible deindexation) - INDEXED → DUPLICATE (high – canonical mismatch) - INDEXED → CRAWL_ERROR (high – server or robots issue) - Any state → PAGE_NOT_FOUND (404) (medium – content removed)

Do not alert on NOT_IN_INDEX → NOT_IN_INDEX (no change). Also, ignore transitions from NULL (first inspection) because you have no baseline.

Step 5: Validate canonical fields and detect accidental noindex/robots block

The API returns robotsTxtState and indexStatusResult.verdict separately. A common cause of NOT_IN_INDEX is a noindex meta tag or robots.txt rule. The API can tell you which one.

Action: For each URL that is NOT_IN_INDEX, check the robotsTxtState and verdict. If robotsTxtState is ALLOWED but verdict is FAIL (with reason NOINDEX), then the page has a noindex directive. Cross-reference with your CMS – if the page should be indexable, this is a bug.

Example alert: { "url": "https://example.com/pricing", "coverageState": "NOT_IN_INDEX", "robotsTxtState": "ALLOWED", "verdict": "FAIL", "indexStatusResult": { "verdict": "FAIL", "coveringStates": ["NOINDEX"] } } → Trigger script to remove noindex tag.

Step 6: Build a performance dashboard with time-series of coverage state proportions

Use the stored data to create a daily rollup per property. Track the percentage of inspected URLs that are:

• INDEXED
• NOT_IN_INDEX
• DUPLICATE
• CRAWL_ERROR

Plot these over time. A sudden drop in INDEXED percentage (e.g., from 95% to 85%) indicates a systematic issue (e.g., a new noindex rule, a robots.txt disallow, or a server failure).

Dashboard metrics (example table):

A 4.7% drop in INDEXED is a red flag. Investigate the URLs that changed.

Step 7: Separate indexation monitoring from rank tracking

The URL Inspection API does not return ranking position, search appearance, or any metric comparable to a SERP position. According to Google’s documentation, the API provides “information about the status of a URL as seen in Google’s index” – it is a coverage tool, not a ranking tool. Engineers must resist the temptation to interpret INDEXED as “ranked high” or NOT_IN_INDEX as “ranked zero.”

Action: Create a separate system for rank tracking (e.g., using Google Search Console’s Search Analytics API or a third-party tool). Do not mix the two. In your monitoring playbook, make it explicit: “The URL Inspection API tells you if Google has indexed the page, not where it appears.” Use this to diagnose indexation issues, not to measure SEO performance.

Common Mistakes

• ❌ Treating the API as real-time: Calling the API milliseconds after publishing a page and expecting INDEXED. The API reflects the last crawl, not the current state. Wait at least 24 hours post-submission.
• ❌ Ignoring quota limits: Running 2,001 queries on a single property will return a 429 Quota Exceeded error. The quota resets at midnight Pacific Time. Use a distributed token bucket (e.g., 2,000 tokens per day, replenished hourly) to avoid bursts.
• ❌ Sampling only the top URLs: If you only inspect the 500 most-visited URLs, you miss indexation problems on the long tail (e.g., blog archives, category pages). A stratified random sample is essential.
• ❌ Not storing canonical field: The canonical URL can differ from the inspected URL. If you ignore it, you might think a page is indexed when it’s actually being consolidated to a different URL. This leads to false positives.
• ❌ Alerting on every state change without cooldown: A single URL flipping between INDEXED and NOT_IN_INDEX due to transient server errors can cause alert fatigue. Add a cooldown (e.g., 1 alert per URL per 6 hours) and require two consecutive non-INDEXED states before firing.

Metrics to Track

• Indexation rate (%): Number of inspected URLs that are INDEXED divided by total inspected. Target: >95% for stable content, >80% for new content.
• Canonical mismatch rate (%): Number of inspected URLs where the canonical URL is different from the inspected URL. Target: <5% (high mismatch indicates duplicate content or server misconfiguration).
• Stuck URL count: Number of URLs that remain NOT_IN_INDEX for more than 7 days. Target: 0 (address as soon as possible).
• Quota usage (%): Percentage of daily quota used. Target: <90% to leave room for manual inspections and emergencies.
• Alert frequency: Number of alerts per day. Target: <10 (if higher, your alerting logic is too sensitive or you have a real problem).

Checklist

• [ ] List all Search Console properties and verify OAuth permissions.
• [ ] Implement a token bucket rate limiter that respects 2,000 qpd per property.
• [ ] Build a priority queue with tiers for new URLs, stuck URLs, and random sample.
• [ ] Store full API response (coverage state, canonical, robotsTxtState, verdict) in a time-series database.
• [ ] Set up alerts for state transitions: INDEXED → NOT_IN_INDEX, INDEXED → DUPLICATE, INDEXED → CRAWL_ERROR.
• [ ] Create a dashboard showing daily % indexation per property with a 7-day moving average.
• [ ] Document the separation between indexation monitoring and rank tracking in your team’s runbook.
• [ ] Implement a weekly review of stuck URLs and canonical mismatches.

How to Implement a Quota-Aware Sampling Script in Python

1. Fetch the list of all URLs from your sitemap or CMS. Use a paginated query (e.g., 1000 URLs per page). Store in a PostgreSQL table with a last_inspected timestamp.
2. Calculate the remaining quota for the day. Use a Redis counter with a TTL reset at midnight UTC. Key: quota_used:{property}. Increment by 1 each API call.
3. Select URLs from the priority queue. Write a SQL query:

   SELECT url FROM urls
   WHERE property = 'sc-domain:example.com'
   ORDER BY
     CASE
       WHEN created_at > NOW() - INTERVAL '24 hours' THEN 0
       WHEN last_coverage_state != 'INDEXED' THEN 1
       ELSE 2
     END,
     RANDOM()
   LIMIT :remaining_quota;

1. Call the API for each selected URL. Use the urlInspection.index method with siteUrl and inspectionUrl. Handle errors (429 – sleep and retry with exponential backoff).
2. Update your database with the response. Also update last_inspected and last_coverage_state.
3. Log the quota usage. After each successful call, increment the Redis counter.
4. Run the script hourly via cron. Set the LIMIT to (remaining_quota / 24) to spread usage evenly.

Frequently Asked Questions

Why does the API sometimes return “NOT_IN_INDEX” for a URL that appears in Google search results?

The API shows the state of the URL as seen in the index at the time of the last crawl, not the current search results. A URL may be indexed but not yet refreshed in the API’s cache. Also, the API may return NOT_IN_INDEX if the URL is canonicalized to a different version (the API will show the canonical URL, not the original). In that case, the original URL is not in the index, but the canonical is.

Can I use the URL Inspection API to check if a page is ranking for a specific keyword?

No. The API has no field for ranking position, keyword, or search appearance. It is strictly a coverage diagnostic. For keyword-level data, use the Search Analytics API (query fields) or a third-party rank tracker.

How do I handle the 2,000 queries per day limit for a site with 500,000 URLs?

You cannot inspect all URLs daily. Use a stratified sampling approach: allocate 200 queries to new/changed URLs, 600 to randomly selected URLs that are currently NOT_IN_INDEX, and the remaining 1,200 to a random sample of the entire corpus (weighted by coverage state). Cycle through the corpus over a week (e.g., 500,000 URLs / 1,200 per day = ~417 days to cover everything once). Accept that you will not have daily coverage of every URL.

What does the canonical field tell me that the coverage state doesn’t?

The canonical field reveals which URL Google considers the primary version. If you inspect https://example.com/page?utm_source=foo and the API returns canonical https://example.com/page, then Google has consolidated the parameterized URL into the clean URL. This is normal for canonicalization, but if the canonical points to a different domain or a 404 page, you have a problem.

How do I set up alerts for quota exhaustion?

Monitor the quota usage counter. If it reaches 1,900 (95% of 2,000), send a warning. If it hits 2,000, log an error but do not alert—the quota will reset the next day. Instead, configure your sampling script to stop early and queue the remaining URLs for the next day.

Is the API free? Are there any paid tiers?

The URL Inspection API is free with the 2,000 queries per day per property limit. Google does not offer a paid tier for higher limits. To increase capacity, you would need to use multiple properties (e.g., break a large site into sub-properties) or rely on other data sources like crawl logs.

Sources

1. Google Search Central, URL Inspection API Overview
2. Google Search Central, Usage Limits
3. Google Search Central, Search Console API – Sites
4. Google Search Central, Coverage State Documentation
5. Google Search Central, Index Status Result
6. Google Search Central, Canonical URLs
7. Google Search Central, robots.txt and Search Console

Using NQZAI for This Playbook

NQZAI’s SEO monitoring platform can automate the quota-aware sampling strategy described in this playbook. Its built-in connection to the URL Inspection API handles OAuth token refresh and rate limiting, so your engineering team does not need to build a token bucket from scratch. The dashboard automatically tracks coverage state proportions over time and triggers alerts on state transitions using configurable thresholds. NQZAI also provides a separate module for Search Analytics data, ensuring that indexation monitoring and rank tracking remain distinct—preventing the common mistake of conflating the two. By integrating NQZAI, you can enforce the principles of property scope, stratified sampling, and canonical field validation without writing custom database pipelines.