Google Indexing in Astro: Incremental Cache-Driven Submissions

For static blogs with rapidly growing content, Google Search Console (GSC) often presents a familiar bottleneck: “Discovered – currently not indexed.” Crawlers discover URLs via sitemaps or inbound links, but due to limited domain authority and tight crawl budgets, long-tail technical articles languish in a crawl queue for weeks or months.

While manually requesting indexing via GSC URL Inspection bumps priority, it is capped at roughly 10–15 URLs per day and is tedious to maintain. Moving to programmatic submission is the logical step, but without disciplined filtering and incremental diffing, developers quickly exhaust API limits. This article details how to integrate build-time metadata from Astro with a lightweight local state cache to establish a zero-waste submission pipeline.

Understanding Google Indexing API and Quota Constraints

Google provides the Web Search Indexing API to allow site owners to notify search engines of page modifications (URL_UPDATED) or removals (URL_DELETED). While officially recommended for short-lived content such as JobPosting and BroadcastEvent, developer and SEO communities have widely adopted it to wake Googlebot, routinely triggering high-priority crawling and rendering for regular pages within hours to two days.

However, each Google Cloud project receives a default free quota of 200 requests per day. On a multilingual static site with articles, project profiles, pagination, and taxonomy pages, the total URL count easily exceeds 300. Blindly submitting the entire site on every deployment introduces several problems:

1. Immediate Quota Exhaustion: A single pipeline run triggers HTTP 429 errors, stranding remaining pages until the quota window resets.
2. Redundant Crawl Overhead: Repeatedly submitting unchanged articles expends crawl attention that should focus on newly published writing.
3. Budget Dilution by Thin Pages: Tag archives and pagination directories consume quota that belongs on in-depth technical guides.

Level 1 Filter: Pruning Taxonomies to Focus on Core Content

Before dispatching network requests, low-value directories must be filtered out. In an Astro architecture driven by Content Collections, genuine content assets reside in localized article and project paths:

function isEligibleContentPage(url) {
  if (!url) return false;

  // Prune thin taxonomy collections and pagination
  if (url.includes('/tag/') || url.includes('/page/')) {
    return false;
  }

  // Exact match for blog articles and project showcases, excluding /blog/ and /projects/ indexes
  const isBlogArticle =
    /^https:\/\/zengliangyi\.com\/(?:[a-z]{2}\/)?blog\/[a-z0-9-]+(?:\/)?$/.test(url) &&
    !url.endsWith('/blog/') &&
    !url.endsWith('/blog');

  const isProjectPage =
    /^https:\/\/zengliangyi\.com\/(?:[a-z]{2}\/)?projects\/[a-z0-9-]+(?:\/)?$/.test(url) &&
    !url.endsWith('/projects/') &&
    !url.endsWith('/projects');

  return isBlogArticle || isProjectPage;
}

This pruning reduces hundreds of URLs to 130+ core long-tail documents with rich structural markup and deep content, protecting the API quota from being consumed by empty archives.

Level 2 Diffing: Inspecting Build-Time Sitemap Lastmod

Reliable change detection cannot depend on filesystem modification dates (mtime), which reset across clean CI clones and Docker container builds.

In the Astro Rocket content model, articles in the blog collection represent the primary knowledge base. During build time, src/content.config.ts enforces publishedAt and optional updatedAt headers. Integration hooks (scripts/article-sitemap.mjs) inspect compiled HTML heads for BlogPosting structured data, injecting deterministic ISO timestamps into <lastmod> nodes inside dist/client/sitemap-0.xml:

<url>
  <loc>https://zengliangyi.com/en/blog/tailwind-v4-vite-setup/</loc>
  <lastmod>2026-10-04T00:00:00.000Z</lastmod>
</url>

The indexing script parses this generated artifact directly, providing an authoritative timestamp for every candidate URL without inspecting markdown files a second time. For articles with timestamps, this delivers exact change detection; for project showcases that currently lack sitemap dates, the state cache below provides robust initial enrollment and safety defaults.

Level 3 Persistence: Local State Caching and Decision Branching

We maintain a minimal state record in .indexing-cache.json (tracked in .gitignore to prevent committing deployment-local timestamps). Each entry tracks the last submission date alongside the lastmod value present during that submission:

{
  "https://zengliangyi.com/en/blog/tailwind-v4-vite-setup/": {
    "lastmod": "2026-10-04T00:00:00.000Z",
    "submittedAt": "2026-10-13T07:16:30.000Z"
  }
}

The submission logic evaluates each page against the state file:

const pending = [];

for (const item of allItems) {
  const cached = cache[item.url];

  if (!cached) {
    // 1. Unindexed or newly created article
    pending.push({ ...item, reason: 'new/unsubmitted' });
  } else if (item.lastmod && cached.lastmod && new Date(item.lastmod) > new Date(cached.lastmod)) {
    // 2. Substantial modification reflected in updatedAt
    pending.push({ ...item, reason: 'updated' });
  } else if (!cached.lastmod && item.lastmod) {
    // 3. Previously submitted without lastmod, now enriched with timestamp
    pending.push({ ...item, reason: 'lastmod-attached' });
  } else {
    // 4. Unmodified document, skip
    skippedCount++;
  }
}

This pipeline delivers concrete benefits:

• Graceful Backfill: On Day 1, the script processes up to the 200-item ceiling; on Day 2, it resumes automatically with unsubmitted items without repeating yesterday’s batch.
• Instant Silent Runs: When no content changes occur between builds, the script exits cleanly in milliseconds without generating network traffic.
• Targeted Push on Revision: When an article receives code corrections or additional diagrams with an updated updatedAt header, only that specific URL is targeted.

Formalizing Agent and Contributor Workflows

The integrity of an automated pipeline relies entirely on the accuracy of upstream metadata. If an author or AI coding agent refactors an article without bumping its timestamp, the incremental diff engine will overlook the change.

To prevent this drift, the project rulebook in AGENTS.md explicitly codifies the maintenance requirement:

Content updates require updatedAt: When modifying an existing blog post or project, update or set updatedAt: YYYY-MM-DD in its frontmatter. This controls <lastmod> in the sitemap and informs SEO indexers of content changes.

With this convention enforced, both manual commits and agentic pairing workflows maintain correct timestamps, completing the loop from markdown edits to build-time sitemaps, incremental cache evaluation, and prompt Googlebot indexing.