Robots.txt and sitemaps: a practical publishing guide

A robots.txt file and an XML sitemap both help search crawlers, but they solve different problems. This guide grew out of maintaining those files for 404cache and the accompanying Robots.txt & Sitemap Generator. The project-specific parts below link to public source; the general crawler guidance links to search-engine documentation.

The short distinction

• robots.txt tells compliant crawlers which paths they may request. It is crawl control, not access control.
• An XML sitemap lists canonical URLs you want search engines to discover, with optional metadata such as a meaningful last-modified date.

Neither file guarantees indexing or ranking. Do not put secrets behind a Disallow rule: the path is publicly readable and other systems can still request it. Google documents this distinction in its robots.txt guidance, and the sitemap protocol defines the supported XML fields at sitemaps.org.

A minimal safe pair

For a site at https://example.com, a permissive file can be as small as:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

A corresponding sitemap starts with absolute, canonical URLs:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-09-13</lastmod>
  </url>
</urlset>

Only publish a lastmod value when it represents a substantive page change. A build timestamp applied to every URL makes the field less useful.

How 404cache handles the files

This site does not rely on an editor manually copying every route into XML. app/sitemap.ts collects public routes, indexable legacy articles, migrated Guides and Build Notes, and registered Tools. Draft and noindex articles are excluded.

The Next.js metadata route creates a sitemap during the static build. The repository's scripts/post-build.mjs then examines the export so generated static routes are represented and pages declaring noindex are not accidentally added back. app/robots.ts provides the crawler rules and sitemap location.

Those links document the implementation. They do not prove how another framework or host behaves.

Publishing checklist

1. Choose canonical URLs first. Sitemap entries should match each page's canonical URL, including hostname and path casing.
2. Exclude non-public states. Drafts, private previews, error pages, and deliberately noindex pages do not belong in the sitemap.
3. Use robots rules for crawl management only. Protect private content with authentication or network access controls.
4. Validate the text and XML. Check for malformed directives, invalid XML characters, relative URLs, and accidental staging domains.
5. Serve predictable URLs. Conventionally these are /robots.txt and /sitemap.xml at the site root.
6. Submit and monitor. Search engines can discover the sitemap from robots.txt, but their webmaster tools provide parsing and coverage feedback. Google's sitemap documentation describes its current submission methods.
7. Regenerate on structural changes. A new public route, redirect, canonical change, or noindex decision should be reflected in the next build.

Generate a starting point

The 404cache Robots.txt & Sitemap Generator can produce both files in the browser. Treat its output as a starting point: review every disallowed path and canonical URL against the deployment before publishing it.

For an automated site, source the route list from the same content model that produces the pages. That reduces drift more effectively than maintaining a second hand-written list.