--- name: al-367-stop-ai-agents-from-scraping-your-website-without-hurting description: "More people now paste a site's URLs into a coding agent and ask it to copy everything, which can cost the owner data, server and API spend. On Oct 28, 2026 a founder posted on X that when he returned 403 errors, the agents spoofed a Googlebot user agent, probed his /api/ routes and launched headless browsers, so he built a 6-layer 'AI Agent Firewall' and open-sourced it as a SKILL.md file. Its core rules: don't send a bare 403 or 429 (agents treat it as a puzzle to retry), serve an HTTP 200 page with no data that tells the agent scraping isn't allowed and links to the official product, verify Googlebot by IP and network rather than user agent, lock sitemaps and public JSON APIs, add per-IP rate limits and test the setup by attacking your own URLs. The post passed 19,000 views and 650 bookmarks within a day. The prompt below runs a safe version of that audit on your own site. Use when the user wants help with ai scraper blocking (security) for a small business." --- # Stop AI agents from scraping your website without hurting your Google rankings: have a coding agent audit how your site answers bots, verify real Googlebot by IP, and serve scrapers a polite no instead of an error they retry More people now paste a site's URLs into a coding agent and ask it to copy everything, which can cost the owner data, server and API spend. On Oct 28, 2026 a founder posted on X that when he returned 403 errors, the agents spoofed a Googlebot user agent, probed his /api/ routes and launched headless browsers, so he built a 6-layer 'AI Agent Firewall' and open-sourced it as a SKILL.md file. Its core rules: don't send a bare 403 or 429 (agents treat it as a puzzle to retry), serve an HTTP 200 page with no data that tells the agent scraping isn't allowed and links to the official product, verify Googlebot by IP and network rather than user agent, lock sitemaps and public JSON APIs, add per-IP rate limits and test the setup by attacking your own URLs. The post passed 19,000 views and 650 bookmarks within a day. The prompt below runs a safe version of that audit on your own site. ## Inputs to collect first Ask the user for anything below that they haven't already given you. Don't guess missing facts. - Domain, what's valuable on it (e.g. listings, prices, a dataset)… - Signup page, API page or product URL ## Instructions Do the task below with the user's inputs in place of each [bracket]. Help me protect my own website from AI agents and scripts that scrape it, without hurting search indexing. My site: [domain, what's valuable on it (e.g. listings, prices, a dataset), stack (e.g. Next.js on Cloudflare, WordPress, Express)]. Where I want scrapers sent instead: [signup page, API page or product URL]. 1) Audit first, change nothing: list which pages, sitemaps and /api/ routes expose the valuable data or trigger expensive work, and how each currently responds to a normal browser, curl, a fake 'Googlebot' user agent and a headless browser. 2) Propose a layered plan: a robots.txt and llms.txt policy (which crawlers to allow and which to disallow), verifying real Googlebot by IP and network rather than user agent, a polite HTTP 200 notice page with no data rows and noindex headers for blocked requests, browser-header checks, per-IP rate limits, and locking public JSON endpoints. 3) For each layer, explain the risk of blocking real visitors, accessibility tools or search engines, and how to test that Google can still crawl my pages. 4) Write the code changes for my stack as a reviewable diff, behind a setting I can switch off. 5) Give me a test checklist I can run only against my own site. Don't deploy anything or touch any site I don't own; show me every change first. ## Safety checks (share these with the user) - Check every figure, price and total against your own records before anything is sent, posted or booked. A person approves refunds, discounts, payments and price changes. - Read and edit the draft before it reaches a customer or goes public. If you later automate this prompt, start with "draft and notify me", not "send automatically". - Check names, dates, numbers and any claim against your own source material before you use the answer. AI can misread or invent details. - Say clearly which parts need a person to review or approve before anything is sent, paid, posted or filed. ## About this skill - Works with: Claude Code, Cursor, Codex, Cloudflare Workers - Based on this real-world example: https://x.com/illyism/status/2115357045264425386 - Free prompt from AgentPlaybook Real Estate: https://agentic-library.netlify.app/prompt/al-367-stop-ai-agents-from-scraping-your-website-without-hurting-yo/ - To install in an agent that supports Agent Skills, save this file as SKILL.md in a folder with the same name as `name` above (for example in your skills directory). Install steps for Claude.ai, Claude Code, Cursor and Codex: https://agentic-library.netlify.app/skills/