--- title: "Google Indexing in Astro: Incremental Cache-Driven Submissions" description: How to fix Google indexing deadlocks in static Astro sites using incremental cache-driven API submissions, Git SHA hashing, and sitemap lastmod timestamps. image: "https://zengliangyi.com/og/blog/en/google-indexing-incremental-cache.png" --- For static blogs with rapidly growing content, Google Search Console (GSC) often presents a familiar bottleneck: “Discovered – currently not indexed.” Crawlers discover URLs via sitemaps or inbound links, but due to limited domain authority and tight crawl budgets, long-tail technical articles languish in a crawl queue for weeks or months. While manually requesting indexing via GSC URL Inspection bumps priority, it is capped at roughly 10–15 URLs per day and is tedious to maintain. Moving to programmatic submission is the logical step, but without disciplined filtering and incremental diffing, developers quickly exhaust API limits. This article details how to integrate build-time metadata from Astro with a lightweight local state cache to establish a zero-waste submission pipeline. ## Understanding Google Indexing API and Quota Constraints Google provides the [Web Search Indexing API](https://developers.google.com/search/apis/indexing-api/v3/quickstart) to allow site owners to notify search engines of page modifications (`URL_UPDATED`) or removals (`URL_DELETED`). While officially recommended for short-lived content such as `JobPosting` and `BroadcastEvent`, developer and SEO communities have widely adopted it to wake Googlebot, routinely triggering high-priority crawling and rendering for regular pages within hours to two days. However, each Google Cloud project receives a default free quota of **200 requests per day**. On a multilingual static site with articles, project profiles, pagination, and taxonomy pages, the total URL count easily exceeds 300. Blindly submitting the entire site on every deployment introduces several problems: - Immediate Quota Exhaustion : A single pipeline run triggers HTTP 429 errors, stranding remaining pages until the quota window resets. - Redundant Crawl Overhead : Repeatedly submitting unchanged articles expends crawl attention that should focus on newly published writing. - Budget Dilution by Thin Pages : Tag archives and pagination directories consume quota that belongs on in-depth technical guides. ## Level 1 Filter: Pruning Taxonomies to Focus on Core Content Before dispatching network requests, low-value directories must be filtered out. In an Astro architecture driven by Content Collections, genuine content assets reside in localized article and project paths: ```javascript function isEligibleContentPage ( url ) { if ( ! url) return false ; // Prune thin taxonomy collections and pagination if (url. includes ( '/tag/' ) || url. includes ( '/page/' )) { return false ; } // Exact match for blog articles and project showcases, excluding /blog/ and /projects/ indexes const isBlogArticle = / ^ https: \/\/ zengliangyi \. com \/ (?: [a-z] {2} \/ ) ? blog \/ [a-z0-9-] + (?: \/ ) ?$ / . test (url) && ! url. endsWith ( '/blog/' ) && ! url. endsWith ( '/blog' ); const isProjectPage = / ^ https: \/\/ zengliangyi \. com \/ (?: [a-z] {2} \/ ) ? projects \/ [a-z0-9-] + (?: \/ ) ?$ / . test (url) && ! url. endsWith ( '/projects/' ) && ! url. endsWith ( '/projects' ); return isBlogArticle || isProjectPage; } ``` This pruning reduces hundreds of URLs to 130+ core long-tail documents with rich structural markup and deep content, protecting the API quota from being consumed by empty archives. ## Level 2 Diffing: Inspecting Build-Time Sitemap Lastmod Reliable change detection cannot depend on filesystem modification dates (`mtime`), which reset across clean CI clones and Docker container builds. In the Astro Rocket content model, articles in the `blog` collection represent the primary knowledge base. During build time, `src/content.config.ts` enforces `publishedAt` and optional `updatedAt` headers. Integration hooks (`scripts/article-sitemap.mjs`) inspect compiled HTML heads for `BlogPosting` structured data, injecting deterministic ISO timestamps into `` nodes inside `dist/client/sitemap-0.xml`: ```xml < url > < loc >https://zengliangyi.com/en/blog/tailwind-v4-vite-setup/ < lastmod >2026-10-04T00:00:00.000Z ``` The indexing script parses this generated artifact directly, providing an authoritative timestamp for every candidate URL without inspecting markdown files a second time. For articles with timestamps, this delivers exact change detection; for project showcases that currently lack sitemap dates, the state cache below provides robust initial enrollment and safety defaults. ## Level 3 Persistence: Local State Caching and Decision Branching We maintain a minimal state record in `.indexing-cache.json` (tracked in `.gitignore` to prevent committing deployment-local timestamps). Each entry tracks the last submission date alongside the `lastmod` value present during that submission: ```json { "https://zengliangyi.com/en/blog/tailwind-v4-vite-setup/" : { "lastmod" : "2026-10-04T00:00:00.000Z" , "submittedAt" : "2026-10-13T07:16:30.000Z" } } ``` The submission logic evaluates each page against the state file: ```javascript const pending = []; for ( const item of allItems) { const cached = cache[item.url]; if ( ! cached) { // 1. Unindexed or newly created article pending. push ({ ... item, reason: 'new/unsubmitted' }); } else if (item.lastmod && cached.lastmod && new Date (item.lastmod) > new Date (cached.lastmod)) { // 2. Substantial modification reflected in updatedAt pending. push ({ ... item, reason: 'updated' }); } else if ( ! cached.lastmod && item.lastmod) { // 3. Previously submitted without lastmod, now enriched with timestamp pending. push ({ ... item, reason: 'lastmod-attached' }); } else { // 4. Unmodified document, skip skippedCount ++ ; } } ``` This pipeline delivers concrete benefits: - Graceful Backfill : On Day 1, the script processes up to the 200-item ceiling; on Day 2, it resumes automatically with unsubmitted items without repeating yesterday’s batch. - Instant Silent Runs : When no content changes occur between builds, the script exits cleanly in milliseconds without generating network traffic. - Targeted Push on Revision : When an article receives code corrections or additional diagrams with an updated `updatedAt` header, only that specific URL is targeted. ## Formalizing Agent and Contributor Workflows The integrity of an automated pipeline relies entirely on the accuracy of upstream metadata. If an author or AI coding agent refactors an article without bumping its timestamp, the incremental diff engine will overlook the change. To prevent this drift, the project rulebook in `AGENTS.md` explicitly codifies the maintenance requirement: > Content updates require `updatedAt` : When modifying an existing blog post or project, update or set `updatedAt: YYYY-MM-DD` in its frontmatter. This controls `` in the sitemap and informs SEO indexers of content changes. With this convention enforced, both manual commits and agentic pairing workflows maintain correct timestamps, completing the loop from markdown edits to build-time sitemaps, incremental cache evaluation, and prompt Googlebot indexing. Share: ## Related Posts [## CLI knowledge workflows: exits, JSON contracts, and final states Check current CLI commands and JSON shapes; reject false success with synthetic process and per-ID state checks before scheduling and recovery. 曾亮懿 October 7, 2026 4 min read #ChatCrystal #CLI #Automation](/en/blog/cli-knowledge-workflow) ## From writing to code Explore ChatCrystal, developer tools and daily learning notes on my GitHub. Questions about the implementation are welcome. [Visit my GitHub](https://github.com/ZengLiangYi) ```json { "@context": "https://schema.org", "@type": "WebSite", "name": "ZengLiangYi Blog", "alternateName": [ "Zeng Liangyi", "ZengLiangYi", "zengliangyi.com" ], "url": "https://zengliangyi.com", "description": "曾亮懿的个人网站。记录 AI Native 探索、开发者工具与全栈产品实践。" } { "@context": "https://schema.org", "@type": "BlogPosting", "headline": "Google Indexing in Astro: Incremental Cache-Driven Submissions", "description": "How to fix Google indexing deadlocks in static Astro sites using incremental cache-driven API submissions, Git SHA hashing, and sitemap lastmod timestamps.", "url": "https://zengliangyi.com/en/blog/google-indexing-incremental-cache/", "image": "https://zengliangyi.com/images/blog/google-indexing-incremental-cache.svg", "datePublished": "2026-10-13T00:00:00.000Z", "dateModified": "2026-10-23T00:00:00.000Z", "author": { "@type": "Person", "name": "曾亮懿", "url": "https://zengliangyi.com/en/about" }, "publisher": { "@type": "Organization", "name": "ZengLiangYi Blog", "logo": { "@type": "ImageObject", "url": "https://zengliangyi.com/favicon-96x96.png" } }, "mainEntityOfPage": { "@type": "WebPage", "@id": "https://zengliangyi.com/en/blog/google-indexing-incremental-cache/" } } { "@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://zengliangyi.com/en" }, { "@type": "ListItem", "position": 2, "name": "Blog", "item": "https://zengliangyi.com/en/blog" }, { "@type": "ListItem", "position": 3, "name": "Google Indexing in Astro: Incremental Cache-Driven Submissions", "item": "https://zengliangyi.com/en/blog/google-indexing-incremental-cache/" } ] } ```