{"uid":"cap_u7kqBUJgzJXZ8yMkYUfgw","slug":"aeo-ai-crawler-access-matrix-44a86e3d","name":"AEO AI-Crawler Access Matrix","description":"robots.txt AI-crawler access matrix: for the 2026 cohort of AI and answer-engine crawlers (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended and more) reports allowed, blocked or unlisted per bot, whether the default rule blocks everything, declared sitemaps and a score. One network operation.","url":"https://payai.agentstools.dev/aeo/crawlers","method":"GET","headers":{},"bodySchema":{"type":"object","$schema":"https://json-schema.org/draft/2020-12/schema","required":["input"],"properties":{"input":{"type":"object","required":["type","method"],"properties":{"type":{"type":"string","const":"http"},"method":{"enum":["GET","HEAD","DELETE"],"type":"string"},"queryParams":{"type":"object","required":[],"properties":{"url":{"type":"string","description":"Target URL to audit (e.g. https://example.com)"},"host":{"type":"string","description":"Alternative input: a hostname"},"domain":{"type":"string","description":"Alternative input: a bare domain (e.g. example.com)"}}}},"additionalProperties":false},"output":{"type":"object","required":["type"],"properties":{"type":{"type":"string"},"example":{"type":"object"}}}}},"responseSchema":null,"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.005","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.005/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.005","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.005","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_WjsBZFxPhRoPZ_I1KtJrk","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.005","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Audits a website's robots.txt to report allowed, blocked, or unlisted status for 2026-cohort AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.) with a composite score.","exampleAgentPrompt":"Can you check whether AI crawlers like GPTBot, ClaudeBot, and PerplexityBot are allowed or blocked on openai.com — and give me the overall AEO score?","exampleUseCases":[{"title":"Pre-launch AI visibility check","prompt":"Before we launch the new marketing site at acme.com, can you pull the robots.txt AI-crawler matrix and tell me if GPTBot, ClaudeBot, and PerplexityBot are allowed in, blocked, or not listed at all?"},{"title":"Competitor AI indexing audit","prompt":"I want to know if my competitor perplexity.ai is blocking any of the major AI answer-engine bots — can you run the crawler access audit and show me their AEO score?"},{"title":"Diagnosing missing AI traffic","prompt":"Our site techblog.example.com isn't showing up in AI-generated answers — can you check the robots.txt crawler matrix to see if any of the 2026 AI bots are blocked or if the default rule is locking them all out?"}],"resultDescription":"Returns per-bot access status (allowed, blocked, or unlisted) for the full 2026 cohort of AI and answer-engine crawlers (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, and more), a flag indicating whether a catch-all default rule blocks all crawlers, a list of declared sitemaps, and a composite AEO score summarizing overall AI crawler friendliness.","failureModes":["No robots.txt present — endpoint may return empty or default-permissive interpretation","Invalid or unreachable URL/host — returns an error indicating the target could not be fetched","Malformed domain input — rejected if neither url, host, nor domain resolves","robots.txt too large or non-standard format — partial parse results may lower confidence in output"],"whenToPreferThis":"Choose this endpoint when you need a structured, per-bot breakdown of AI crawler access from a single robots.txt fetch — especially when auditing for answer engine optimization (AEO) readiness, diagnosing missing AI-search visibility, or comparing competitor crawler policies. It covers the full 2026 AI-bot cohort in one call, making it faster than manually parsing robots.txt and more comprehensive than generic SEO crawl checkers that don't track LLM-specific bots.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-09-15T07:11:43.149Z","isFirstParty":false}