{"uid":"cap_nQvoeiMl5CQxLVSZwm0_L","slug":"ai-crawler-access-checker-00314844","name":"AI Crawler Access Checker","description":"Given a website domain, reports which AI crawlers may fetch its home page under its robots.txt (RFC 9309): GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and more, each allowed or blocked, plus Googlebot, listed sitemaps, whether /llms.txt is published and page-level noindex/nosnippet, with fixes. Static check, no browser. Not charged if robots.txt cannot be read (no response, or a bot block).","url":"https://x402.agentharbor.agency/ai-crawlers?utm_source=zero.xyz","method":"GET","headers":{},"bodySchema":{"type":"object","$schema":"https://json-schema.org/draft/2020-12/schema","required":["input"],"properties":{"input":{"type":"object","required":["type","method"],"properties":{"type":{"type":"string","const":"http"},"method":{"enum":["GET","HEAD","DELETE"],"type":"string"},"queryParams":{"type":"object","required":["domain"],"properties":{"domain":{"type":"string","pattern":"^[A-Za-z0-9]([A-Za-z0-9-]{0,61}[A-Za-z0-9])?(\\.[A-Za-z0-9]([A-Za-z0-9-]{0,61}[A-Za-z0-9])?)+$","maxLength":253,"description":"Bare public hostname, e.g. example.com"}},"additionalProperties":false}},"additionalProperties":false},"output":{"type":"object","required":["type"],"properties":{"type":{"type":"string"},"example":{"type":"object"}}}}},"responseSchema":null,"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.01","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.01/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.01","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.01","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_WZf_6VVrG_QnKM2_cXf8m","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.01","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Given a domain, checks which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, etc.) are allowed or blocked by its robots.txt, and reports sitemaps, llms.txt presence, and noindex/nosnippet signals.","exampleAgentPrompt":"Can you check which AI crawlers are allowed or blocked on openai.com — I want to know if GPTBot, ClaudeBot, and PerplexityBot can access it, and whether the site has an llms.txt file?","exampleUseCases":[{"title":"Competitor AI visibility audit","prompt":"Check what AI crawlers can access stripe.com — I want to know if GPTBot, ClaudeBot, Google-Extended, and PerplexityBot are allowed or blocked in their robots.txt, and whether they publish an llms.txt."},{"title":"Own site crawler access check","prompt":"Can you tell me which AI bots are blocked from crawling my site at mybrand.com? I want to know the status for ChatGPT-User, ClaudeBot, and PerplexityBot, and whether my pages have noindex or nosnippet set."},{"title":"AI search indexing troubleshooting","prompt":"My content isn't showing up in AI-powered search engines — can you check the robots.txt on techblog.io and tell me if GPTBot, OAI-SearchBot, or Applebot-Extended are being blocked, and if there are any nosnippet directives?"}],"resultDescription":"Returns per-crawler allow/block status for major AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, and others), Googlebot status, listed sitemaps, whether /llms.txt is published, page-level noindex/nosnippet flags, and suggested fixes. No charge if robots.txt cannot be read.","failureModes":["robots.txt unreachable or blocked — endpoint returns no charge and indicates the file could not be read","invalid domain format — rejected by schema pattern validation","domain resolves but returns no robots.txt — treated as unreadable","network timeout on robots.txt fetch — no charge applied"],"whenToPreferThis":"Use this endpoint when you need a fast, static check of AI crawler permissions for any public domain without spinning up a browser. Ideal for auditing AI search visibility, diagnosing indexing issues, or monitoring competitors' robots.txt policies. Prefer this over general SEO audit tools when specifically targeting AI crawler access rules and llms.txt presence.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-15T06:30:24.747Z","isFirstParty":false,"canonicalSlug":"ai-crawler-access-checker-00314844"}