What should our robots.txt and sitemap actually contain?

This is a technical SEO question — the kind that usually shows up from developers, technical SEO interview. It rarely has a one-line answer, because the honest version of “What should our robots.txt and sitemap actually contain” is a shortlist of rival explanations, not a single cause. The job is to work through that shortlist with evidence and stop as soon as one of them is confirmed — not to write a report that mentions all of them.

The rival explanations

Direct answer: Robots and sitemap problems trace to five specific misuses — treating robots.txt as a noindex substitute (it isn't one), sitemaps listing redirected/canonicalized/noindexed URLs, meta robots and X-Robots-Tag disagreeing with robots.txt on the same path, a staging rule leaking to production (or the reverse), and parameter URL space being both allowed by robots.txt and advertised by internal search.

Treat these as competitors, not a checklist. The point of naming five up front is to stop the first plausible-sounding one from becoming the story before the others have been checked.

• robots.txt is being used as a noindex substitute (it is not).
• Sitemaps include redirected, canonicalised, or noindexed URLs.
• Meta robots, X-Robots-Tag, and robots.txt disagree on the same path.
• A staging rule leaked to production or vice versa.
• Parameter URL space is allowed by robots and advertised by internal search.

What the evidence has to show

Direct answer: The current robots.txt, all sitemap indexes, and a full crawl of meta-robots/x-robots directives, GSC's robots tester and sitemap-processing-error reports, log confirmation that Googlebot actually respects the rules, a staging-versus-production directive diff, and a money-URL checklist for allowed/indexable/sitemap-listed status are what confirms the three systems agree.

None of the five above survives on a hunch. Here is what actually needs pulling before any of them can be ruled in or out:

• Current robots.txt, all sitemap indexes, and a crawl of meta robots / x-robots.
• GSC robots tester + sitemap processing errors.
• Log confirmation that Googlebot respects the rules.
• Diff of staging vs production directives.
• List of money URLs and whether each is allowed, indexable, and in the sitemap.

The decision rule

Direct answer: robots.txt governs crawl, not indexation. noindex / canonical govern indexation. Sitemaps list only canonical, indexable, 200 URLs. When two directives conflict, rewrite until a crawler and GSC tell the same story.

What to tell the people around you

Direct answer: Platform and SRE need SEO ownership of robots/sitemap generation rules and a change-control note on any edit — not an assumption that these files are 'set and forget' once created.

The analysis is not finished until it produces something a non-specialist can act on. That means naming the situation, the cost of getting the first move wrong, and a specific ask — not a summary of the investigation.

• Situation — Three files/headers are being asked to do one job. They have different jobs. The policy is the answer.
• So what — A single conflicting line can hide a revenue directory. This is cheap to get right and expensive to debug later.
• The ask — SEO ownership of robots and sitemap generation rules. A change-control note on any edit.

Technical SEO writes policy. Platform generates sitemaps. SRE reviews robots.

How to act on this

1. Pull the current robots.txt, every sitemap index, and a full crawl of meta-robots and X-Robots-Tag directives.
2. Check GSC's robots tester and sitemap-processing-error reports for existing conflicts.
3. Confirm via log data that Googlebot is actually respecting the rules as written, not just that the file is technically valid.
4. Diff staging directives against production to catch a leaked staging rule (or vice versa) before it causes damage.
5. For every money URL, confirm it is simultaneously allowed by robots.txt, indexable (no conflicting noindex/canonical), and listed correctly in the sitemap — rewrite whichever file disagrees.

Frequently asked questions

Can robots.txt be used to remove a page from search results?

No — robots.txt only controls crawling, not indexation. A URL blocked by robots.txt can still appear in results (typically with no snippet) if Google learns about it another way; noindex or removal tools are the correct approach for de-indexing.

Should sitemaps include every URL on the site?

No — sitemaps should list only canonical, indexable, 200-status URLs. Including redirected, canonicalized-away, or noindexed URLs actively confuses Google about which version of a page is authoritative.

What happens when meta robots and robots.txt disagree on the same page?

This is one of the five named failure patterns — a page blocked by robots.txt can't have its meta robots tag read at all (since Google can't crawl it to see the tag), which often produces exactly the opposite of the intended result.

How do we prevent a staging rule from leaking to production?

Diff staging and production robots/meta-robots directives as a standard part of any deployment process, not just when a problem is suspected — this is one of the five root causes and it's fully preventable with a routine check.

Who should own changes to robots.txt and sitemap generation logic?

SEO should own the written policy, with a required change-control note on any edit — an unreviewed change to either file can silently hide a revenue-generating directory from Google.

How often should robots.txt and sitemaps be reviewed after the initial policy is set?

On every significant template or IA change — a policy written once and never revisited is exactly how a staging rule or a new parameterized URL type ends up conflicting with the original intent.

Do image and video sitemaps need the same canonical/indexable rule as page sitemaps?

Yes — the same principle applies: only list canonical, accessible, intended-to-be-indexed assets, since including redirected or blocked media creates the same kind of conflicting signal as a page sitemap would.

Sources

1. Robots.txt introduction and guide — Search Central
2. Build and submit a sitemap — Search Central
3. Robots exclusion standard — Wikipedia
4. Site map — Wikipedia

Where nqzai fits

nqzai runs this same rival-hypothesis framework against your own connected Search Console, Analytics, and audit history, and returns a keep / change / stop decision with the evidence named — including which of the explanations above it could not test, and what to connect to close that gap. No extra cost for the analysis itself; it reads measurements already on file.

Ask nqzai: “What should our robots.txt and sitemap actually contain?”