Tutorial: Sitemap and robots.txt in Next.js

Sitemaps and robots.txt tell crawlers what exists and what to skip. This tutorial implements both with Next.js App Router conventions.

What you will build

• `app/sitemap.ts` that lists static routes and published blog posts
• `app/robots.ts` that allows public pages and references the sitemap
• A Search Console submission checklist

Prerequisites

• Next.js 13.3+ (Metadata routes)
• Absolute site URL in env
• Blog posts available from your data layer or CMS

Step-by-step

1. Create `app/sitemap.ts` exporting a default function that returns `MetadataRoute.Sitemap`.
2. Include high-priority routes: `/`, `/services`, `/work`, `/about`, `/contact`, `/blog`, `/career`.
3. Map published posts to `/blog/[slug]` with `lastModified` from `publishedAt`.
4. Create `app/robots.ts` with `allow: "/"`, disallow admin/private paths, and `sitemap` URL.
5. Deploy and open `/sitemap.xml` and `/robots.txt` in the browser.
6. Submit the sitemap URL in Google Search Console.

sitemap.ts

import type { MetadataRoute } from "next";
import { getAllPosts } from "@/lib/data/blog";

const site = process.env.NEXT_PUBLIC_SITE_URL!;

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const posts = await getAllPosts();

  const staticRoutes = ["", "/services", "/work", "/about", "/contact", "/blog", "/career"].map(
    (path) => ({
      url: `${site}${path}`,
      lastModified: new Date(),
      changeFrequency: "weekly" as const,
      priority: path === "" ? 1 : 0.8,
    }),
  );

  const blogRoutes = posts.map((post) => ({
    url: `${site}/blog/${post.slug}`,
    lastModified: new Date(post.publishedAt),
    changeFrequency: "monthly" as const,
    priority: 0.7,
  }));

  return [...staticRoutes, ...blogRoutes];
}

robots.ts

import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  const site = process.env.NEXT_PUBLIC_SITE_URL!;
  return {
    rules: {
      userAgent: "*",
      allow: "/",
      disallow: ["/admin", "/api/"],
    },
    sitemap: `${site}/sitemap.xml`,
  };
}

Verify

• `/sitemap.xml` returns XML without auth cookies required
• Every sitemap URL returns 200 and matches its canonical
• Search Console → Sitemaps shows “Success” after processing

Tips

• Exclude draft, archived, and thank-you pages
• Keep sitemap under 50,000 URLs (split if you grow past that)
• Re-deploy or on-demand revalidate when posts publish

Next steps

Combine sitemaps with the metadata tutorial so every listed URL also has a strong title, description, and Open Graph image.