How healthy is the site for crawling and indexation?

This is a technical SEO question — the kind that usually shows up from sEO lead with developers. It rarely has a one-line answer, because the honest version of “How healthy is the site for crawling and indexation” is a shortlist of rival explanations, not a single cause. The job is to work through that shortlist with evidence and stop as soon as one of them is confirmed — not to write a report that mentions all of them.

The rival explanations

Direct answer: Discovery, rendering, and indexation are three separate systems, and five distinct failure modes — robots/noindex blocks, sitemap-listed URLs Google refuses, rendering gaps, orphan/soft-404 pages, and conflicting hreflang/canonical hints — can each independently make a 'healthy-looking' sitemap misleading.

Treat these as competitors, not a checklist. The point of naming five up front is to stop the first plausible-sounding one from becoming the story before the others have been checked.

• robots.txt or noindex is blocking a directory we think is live.
• Sitemaps list URLs Google refuses (canonicalised, redirected, thin).
• Rendered HTML lacks the primary copy or links that users see.
• Orphan and soft-404 URLs dilute crawl.
• Hreflang / canonical hints conflict with the sitemap.

What the evidence has to show

Direct answer: GSC Indexing and robots-tester data, a text-versus-JS-render crawl diff, Googlebot log status codes by directory, a submitted-vs-indexed count, and spot-checked fetch-as-Google on client-rendered templates are what confirm which of the five failure modes is actually live.

None of the five above survives on a hunch. Here is what actually needs pulling before any of them can be ruled in or out:

• GSC Indexing, sitemaps, robots tester; inspect 15 representative URLs live vs indexed.
• Crawl in text and JS-render modes; diff word count and internal links.
• Log lines for Googlebot 2xx / 3xx / 4xx by directory.
• Count indexable vs submitted vs indexed.
• Spot-check fetch as Google on templates that rely on client-side render.

The decision rule

Direct answer: Healthy means: money templates are indexable, render-equivalent, in the sitemap, and receiving Googlebot 200s. If submitted >> indexed, stop adding URLs and fix the exclusion rules first.

What to tell the people around you

Direct answer: The team needs robots/sitemap access and a freeze on new parameterized URLs until the exclusion rules are written — not a general reassurance that 'indexing looks fine'.

The analysis is not finished until it produces something a non-specialist can act on. That means naming the situation, the cost of getting the first move wrong, and a specific ask — not a summary of the investigation.

• Situation — Discovery, rendering, and indexation are three different systems. A green sitemap ping does not mean the pages are in the index as we intend.
• So what — Content and PR spend on URLs Google cannot settle on is wasted. Indexation health is a gate, not a nice-to-have.
• The ask — Access to robots, sitemap generation, and a log sample. Freeze new parameterised landing pages until rules are written.

Technical SEO diagnoses. Platform engineering implements rules.

How to act on this

1. Pull GSC Indexing, sitemap, and robots-tester reports, then inspect 15 representative URLs live versus indexed.
2. Crawl the site in both text and JS-render modes and diff word count and internal links between the two.
3. Sample Googlebot log lines by directory to see the real 2xx/3xx/4xx split Google is actually receiving.
4. Compare the submitted, indexable, and indexed counts — a large submitted-versus-indexed gap means stop adding URLs and fix exclusion rules first.
5. Spot-check fetch-as-Google on any template that relies on client-side rendering before trusting its indexation status.

Frequently asked questions

Does a successful sitemap ping mean the URLs are indexed?

No — a sitemap ping only confirms Google received the file. Indexation is a separate decision Google makes per URL, which is exactly why the submitted-vs-indexed count is checked directly rather than inferred from the ping response.

What's the difference between crawled and indexed?

Crawled means Googlebot fetched the URL; indexed means Google chose to store and potentially rank it. A URL can be crawled repeatedly and still excluded from the index if it looks thin, duplicate, or low-value.

Should we noindex or robots.txt-block low-value URLs?

Neither substitutes for the other — robots.txt stops crawling, noindex/canonical control indexation. Use the one that matches whether you want Google to see the page at all or just to skip it in results.

How do we know if a rendering gap is actually hurting us?

Diff the text-crawl and JS-render-crawl word counts and internal-link sets. If the rendered version is meaningfully thinner or missing links the client-side version has, Google may be indexing a stripped-down page.

What triggers the 'freeze new URLs' recommendation?

A submitted-count that badly outpaces the indexed count on a template — adding more URLs in that same pattern only grows the invisible set faster than the exclusion rules can be written.

Can a site be 'too healthy' — indexing URLs it shouldn't?

Yes — over-indexation (thin, near-duplicate, or low-value URLs all getting indexed) is its own health problem, which is why the submitted-vs-indexed comparison looks for both too-few and unexpectedly-many indexed URLs.

Can a site pass every check here and still rank poorly?

Yes — this diagnosis only confirms Google can discover, render, and store the pages correctly. Ranking depends on relevance and quality signals this check doesn't measure, which is why it's a prerequisite, not a ranking guarantee.

Sources

1. Google Search Console overview — Search Central
2. Robots.txt introduction and guide — Search Central
3. Web crawler — Wikipedia
4. Search engine indexing — Wikipedia

Where nqzai fits

nqzai runs this same rival-hypothesis framework against your own connected Search Console, Analytics, and audit history, and returns a keep / change / stop decision with the evidence named — including which of the explanations above it could not test, and what to connect to close that gap. No extra cost for the analysis itself; it reads measurements already on file.

Ask nqzai: “How healthy is the site for crawling and indexation?”