TL;DR
This playbook provides a structured, evidence-based incident response framework to distinguish a widespread technical indexing failure from normal…
This playbook provides a structured, evidence-based incident response framework to distinguish a widespread technical indexing failure from normal crawl/index lag, enabling founders and SEO teams to make calm, data-driven decisions within 48 hours.
The Problem
When a founder wakes up to a 40% drop in indexed pages overnight, panic is the default reaction. The instinct is to roll back the last deployment, resubmit sitemaps, or flood Google with URL Inspection requests. But most indexing “incidents” are actually normal crawl delays—especially after a site migration, content refresh, or large-scale URL restructuring. Google’s index is not real-time; it can take days or weeks for new pages to appear and old ones to disappear. The real danger is misdiagnosing a transient lag as a permanent exclusion, triggering unnecessary rollbacks that break working features and waste engineering cycles.
The core challenge is separating signal from noise. A single Search Console “Excluded” spike could mean a robots.txt block, a noindex tag leak, a server error wave, or simply that Google hasn’t finished recrawling after a sitemap update. Without a systematic triage process, teams waste hours chasing false positives or, worse, ignore a real failure until organic traffic collapses. This runbook gives you a repeatable 48-hour protocol to collect evidence, segment the problem, define safe rollback criteria, and escalate only when the data demands it.
Core Framework
Key Principle 1: Separate Signal from Noise Using Baseline Metrics
Every indexing incident must be measured against your site’s normal crawl and index behavior. Without a baseline, you cannot distinguish a 20% drop caused by a technical failure from a 20% drop caused by Google’s natural recrawl cycle. Establish three baseline metrics before any incident:
- Index Coverage Ratio (ICR): (Number of indexed pages) / (Number of submitted pages in sitemap). A healthy site typically maintains ICR between 0.90 and 1.05 (some over-indexing due to non-sitemap pages is normal). Track this weekly.
- Crawl Rate: Pages crawled per day, visible in Search Console’s Crawl Stats report. A sudden 50% drop often indicates a server or robots.txt issue.
- Time to Index: The median time between first crawl and index appearance for new pages. For most sites, this is 1–7 days. If your baseline is 3 days and you’re seeing 10+ days, that’s a signal.
When an incident occurs, compare current metrics to the 30-day rolling average. A deviation of less than 2 standard deviations is likely normal variance. Anything beyond that warrants deeper investigation.
Key Principle 2: Timebox Triage into Three Phases
The 48-hour runbook is divided into three phases: Evidence Collection (0–6 hours), Root Cause Analysis (6–24 hours), and Decision & Execution (24–48 hours). Each phase has a clear exit criterion. If you cannot identify a clear technical cause by hour 24, you escalate to a senior engineer or Google Search liaison. Never roll back before hour 24 unless you have direct evidence of a change that caused the exclusion (e.g., a noindex tag pushed to production).
Key Principle 3: Rollback Only When You Can Prove Causation
Correlation is not causation. A deployment that happened 12 hours before the index drop is suspicious but not proof. Safe rollback requires three conditions:
- Temporal proximity: The change was deployed within 7 days of the first observed exclusion.
- Direct evidence: URL Inspection shows “Excluded by ‘noindex’ tag” or “Crawl error” on a representative sample, and the tag/error matches the deployed change.
- Reversibility: Rolling back the change will not break other functionality (e.g., a security patch).
If any condition is missing, do not roll back. Instead, wait and monitor for 24 more hours. Premature rollback introduces new variables and often delays the real fix.
Step-by-Step Execution
Step 1: Immediate Evidence Collection (0–2 Hours)
Action: Pull the Search Console Index Coverage report for the last 7 days. Export the “Excluded” and “Error” tabs. Use the URL Inspection API to test a random sample of 100 excluded pages.
Detailed guide: - Open Search Console → Index → Pages. Set date range to last 7 days. Note the total indexed count and the breakdown: “Error,” “Valid with warnings,” “Excluded,” “Submitted not indexed.” - For each excluded page, note the reason: “Crawled – currently not indexed,” “Excluded by ‘noindex’ tag,” “Blocked by robots.txt,” “Page with redirect,” “Duplicate without canonical,” “Not found (404).” - Use the URL Inspection API (or manual tool) to inspect 100 randomly selected excluded URLs. Record the “Indexing allowed?” and “Crawl allowed?” status. If more than 20% show “noindex” or “blocked by robots.txt,” that’s a strong signal. - Check server logs for the same URLs: look for 5xx errors, 3xx redirect chains, or slow response times (>5 seconds). A spike in 503 errors can cause Google to deprioritize crawling.
Tools: Search Console, URL Inspection API (Python script or Google Sheets add-on), server log analyzer (e.g., GoAccess, ELK stack).
Step 2: Segmentation and Classification (2–6 Hours)
Action: Segment excluded pages by URL pattern, last crawl date, sitemap inclusion, and canonical tag status.
Detailed guide: - Export the full list of excluded URLs from Search Console. Use a spreadsheet to group by: - URL pattern: /blog/, /product/, /category/, etc. - Last crawl date: If most excluded pages were last crawled 30+ days ago, it’s likely a recrawl lag, not a new failure. - Sitemap inclusion: Cross-reference with your sitemap. Pages in the sitemap but excluded are more concerning than pages never submitted. - Canonical tag: Check if excluded pages have a canonical pointing to a different URL. If so, Google is consolidating signals—this is expected. - Create a pivot table showing the count of excluded pages per reason per segment. For example: “/product/ pages excluded by ‘noindex’ tag: 1,200.” If one segment dominates, focus root cause analysis there.
Example: A site with 50,000 product pages sees 10,000 excluded. Segmentation reveals 8,000 are “Crawled – currently not indexed” and all were last crawled 3 days ago. That’s normal lag after a product feed update. No action needed.
Step 3: Root Cause Analysis (6–24 Hours)
Action: Investigate recent changes to robots.txt, noindex tags, canonical tags, server configuration, and sitemap submission.
Detailed guide: - robots.txt: Fetch your robots.txt via curl https://www.example.com/robots.txt. Compare to a known-good version (e.g., from a Git commit). Look for accidental Disallow: / or Disallow: /product/. - noindex tags: Crawl a sample of excluded pages using a tool like Screaming Frog or a custom script. Check for <meta name="robots" content="noindex"> or X-Robots-Tag: noindex HTTP header. If found, trace the source (CMS plugin update, theme change, CDN rule). - Canonical changes: Use the same crawl to check <link rel="canonical" href="...">. If canonicals now point to a different domain or a non-existent URL, Google will exclude the original. - Server errors: Review server logs for the last 7 days. A sudden increase in 5xx errors (especially 503) can cause Google to stop crawling. Check your hosting provider’s status page. - Sitemap changes: Verify your sitemap is still valid and submitted in Search Console. If you recently changed the sitemap URL or removed entries, Google may deprioritize those pages.
Escalation limit: If after 24 hours you have not identified a clear technical cause (e.g., no robots.txt change, no noindex leak, no server errors), escalate to a senior engineer or Google Search liaison. Do not roll back without a cause.
Step 4: Safe Rollback Criteria (24–30 Hours)
Action: Evaluate whether a rollback is justified based on the evidence collected.
Detailed guide: - Criteria 1 – Temporal proximity: Was a change deployed within the last 7 days that could affect indexing? If yes, proceed to criteria 2. - Criteria 2 – Direct evidence: Does URL Inspection show “Excluded by ‘noindex’ tag” or “Blocked by robots.txt” on >50% of sampled excluded pages? If yes, the change is likely the cause. - Criteria 3 – Reversibility: Can you revert the change without breaking other functionality? For example, rolling back a noindex tag is safe; rolling back a security patch is not. - Decision matrix: - All three criteria met → Roll back immediately. - Criteria 1 and 2 met, but not 3 → Do not roll back. Instead, fix the issue in a new deployment (e.g., remove the noindex tag without reverting the security patch). - Only criteria 1 met → Wait 12 more hours and re-evaluate. The drop may be normal crawl lag. - No criteria met → Do not roll back. Continue monitoring.
Example: A site deployed a new theme that accidentally added noindex to all product pages. URL Inspection confirms “Excluded by ‘noindex’ tag” on 80% of sampled pages. The theme can be rolled back safely. Roll back immediately.
Step 5: Execute Rollback or Wait (30–36 Hours)
Action: If rollback is warranted, revert the change and request re-crawling via URL Inspection. If waiting, set a 12-hour monitoring checkpoint.
Detailed guide: - Rollback: Deploy the previous version of the code/configuration. After deployment, use the URL Inspection tool to manually request indexing for 10–20 representative pages. This signals Google to recrawl quickly. - Wait: Set a calendar reminder for 12 hours later. During this time, do not make any other changes. Monitor the Index Coverage report hourly for any improvement. - Communication: Notify stakeholders (product, engineering, marketing) of the decision and expected timeline. For a rollback, state that indexing should recover within 1–3 days (Google’s typical recrawl window). For a wait, explain that normal crawl lag can take up to 7 days.
Step 6: Post-Incident Review (36–48 Hours)
Action: Document findings, update runbook, and set up proactive alerts.
Detailed guide: - Write a post-mortem including: timeline, evidence collected, root cause (or “no cause found”), decision made, and outcome. - Update the runbook with any new patterns discovered (e.g., “If product pages are excluded with ‘Crawled – currently not indexed’ after a feed update, wait 5 days before investigating”). - Set up Search Console alerts for sudden drops in indexed pages (threshold: >10% drop in 24 hours). Use the Search Console API to automate this. - If the incident was caused by a code change, add a pre-deployment check: crawl a sample of pages after staging deployment to verify no noindex tags leaked.
Common Mistakes
- ❌ Mistake 1: Panic-rolling back without evidence. A founder sees a 30% drop and reverts the last deployment, only to discover the drop was caused by a Google algorithm update that had nothing to do with the code. This wastes engineering time and introduces new bugs. Always collect evidence first.
- ❌ Mistake 2: Ignoring normal crawl lag after a large site update. After adding 10,000 new pages, Google may take 2–3 weeks to index them all. During that time, the Index Coverage Ratio will drop temporarily. The correct response is to wait, not to resubmit sitemaps or change robots.txt.
- ❌ Mistake 3: Using only Search Console without URL Inspection. The Index Coverage report shows aggregate numbers but not per-page details. A page listed as “Excluded” could be “Crawled – currently not indexed” (normal) or “Excluded by ‘noindex’ tag” (urgent). Always verify with URL Inspection on a sample.
- ❌ Mistake 4: Resubmitting sitemaps repeatedly. Google treats sitemap resubmission as a signal, but it does not guarantee faster indexing. If the underlying issue (e.g., noindex tag) is not fixed, resubmitting is pointless. Fix the cause first, then resubmit once.
Metrics to Track
| Metric | Definition | Target | How to Measure |
|---|---|---|---|
| Index Coverage Ratio (ICR) | Indexed pages / Submitted pages (from sitemap) | 0.90–1.05 | Search Console → Index → Pages → “Valid” count / “Submitted” count |
| Crawl Rate | Pages crawled per day | Within 20% of 30-day average | Search Console → Settings → Crawl Stats |
| Time to Index | Days between first crawl and index appearance | ≤7 days for 90% of new pages | URL Inspection API on a cohort of new URLs |
| Exclusion Reason Breakdown | % of excluded pages per reason (noindex, robots.txt, etc.) | <5% for “noindex” or “robots.txt” | Search Console Index Coverage export |
| Server Error Rate | % of 5xx responses to Googlebot | <1% | Server logs filtered by Googlebot user-agent |
Checklist
- [ ] Pulled Search Console Index Coverage report for last 7 days
- [ ] Sampled 100 excluded pages via URL Inspection (manual or API)
- [ ] Checked robots.txt for accidental disallows
- [ ] Crawled a sample of excluded pages for noindex tags and canonical issues
- [ ] Reviewed server logs for 5xx errors in last 7 days
- [ ] Verified sitemap is valid and submitted
- [ ] Identified any recent code/site changes (deployments, CMS updates, CDN config)
- [ ] Defined rollback criteria (temporal proximity, direct evidence, reversibility)
- [ ] Made decision: rollback or wait, with documented reasoning
- [ ] If rollback: reverted change, requested re-crawl via URL Inspection for 10 URLs
- [ ] Set 12-hour monitoring checkpoint if waiting
- [ ] Documented incident in post-mortem and updated runbook
How to Implement This Playbook with NQZAI
NQZAI’s monitoring and automation platform can accelerate every phase of this runbook, reducing triage time from hours to minutes.
- Automated baseline collection: NQZAI continuously pulls Search Console data via API and stores 90-day rolling averages for ICR, crawl rate, and time to index. When an anomaly is detected (e.g., ICR drops below 0.85), it triggers an incident alert with the current metrics vs. baseline.
- Instant evidence collection: Upon alert, NQZAI automatically runs a URL Inspection API batch on 100 random excluded URLs, categorizes the exclusion reasons, and presents a summary in a dashboard. It also fetches the latest robots.txt and compares it to the last known-good version.
- Segmentation engine: NQZAI segments excluded URLs by URL pattern, last crawl date, and sitemap inclusion using pre-built rules. It highlights segments that deviate from normal patterns (e.g., “/product/ pages excluded by noindex: 1,200 vs. baseline 0”).
- Rollback decision support: The platform checks the three rollback criteria against your deployment history (integrated with GitHub/GitLab). It shows a green/yellow/red status for each criterion and recommends a course of action.
- Automated rollback (opt-in): If you configure NQZAI with deployment permissions, it can execute a rollback of the specific commit that introduced the noindex leak, then request re-crawling via URL Inspection API. All actions are logged for audit.
- Post-incident report: NQZAI generates a structured post-mortem with timeline, evidence, decision, and outcome, ready to share with stakeholders.
By using NQZAI, you eliminate manual data gathering and reduce the risk of human error during high-stress incidents. The platform enforces the 48-hour timebox and ensures no step is skipped.
Frequently Asked Questions
Q: How long does it take for Google to re-index pages after a rollback?
A: There is no fixed timeline. Google’s recrawl frequency depends on the page’s crawl budget and update frequency. Using the URL Inspection tool to request indexing can accelerate the process, but typically you should expect 1–3 days for small sites and up to 7 days for large sites. Do not expect instant recovery.
Q: What if the exclusion is due to a Google algorithm update (e.g., core update)?
A: Check the Google Search Status Dashboard. If a confirmed algorithm update is in progress, the exclusion may be intentional (e.g., quality demotion). Do not roll back. Instead, focus on improving content quality and E-E-A-T signals. The Index Coverage report will show “Crawled – currently not indexed” rather than technical errors.
Q: Should I resubmit my sitemap after fixing the issue?
A: Only if you have made a material change (e.g., added new URLs or fixed a noindex leak). Resubmitting without a fix will not help. After a rollback, resubmit the sitemap once via Search Console. Do not resubmit daily—it may be ignored.
Q: How do I distinguish between a crawl error and a noindex tag using URL Inspection?
A: Open the URL Inspection tool for a specific URL. Under “Coverage,” it will say either “Page is not indexed: Excluded by ‘noindex’ tag” (if a noindex tag is present) or “Page is not indexed: Crawl error” with a specific error code (e.g., 404, 503). The tool also shows the HTTP response headers and the rendered HTML, so you can verify the tag.
Q: What if the exclusion is caused by a CDN or firewall blocking Googlebot?
A: Check your server logs for Googlebot’s IP ranges. If you see 403 or 503 responses, your CDN or WAF may be blocking legitimate crawlers. Use the URL Inspection tool’s “Live test” feature to see how Googlebot sees the page. If it returns a block page, whitelist Googlebot’s IPs (published by Google) in your firewall rules.
Q: Can I rely on third-party indexing monitoring tools instead of Search Console?
A: No. Third-party tools can provide useful alerts, but Search Console is the only source of truth for Google’s index status. Always verify with Search Console before taking action. Third-party tools may use different data sources or sampling methods that introduce false positives.
