{"uid":"cap_JGeJ0HTmWoaga5Yyz_iv8","slug":"milliapi-ai-robots-txt-checker-a34380d2","name":"MilliAPI AI Robots.txt Checker","description":"Tiny machine-readable APIs that AI agents can discover, pay for with x402 USDC on Base, and use instantly. Services start at $0.001 per call.","url":"https://milliapi.com/api/ai-robots-check","method":"GET","headers":{},"bodySchema":null,"responseSchema":{"type":"json","example":{"origin":"https://example.com","robotsTxt":{"status":404,"present":false},"aiCrawlerHomepageAccess":{"GPTBot":{"reason":"No matching robots.txt group","allowed":true}}}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.001","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.001/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.001","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.001","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_C33cpZl_uT0a9jXJMM0bV","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.001","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Checks a website's robots.txt to determine whether major AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot, Applebot-Extended) are allowed or blocked on its homepage.","exampleAgentPrompt":"Can you check whether the major AI crawlers like GPTBot and ClaudeBot are allowed to crawl https://example.com based on its robots.txt?","exampleUseCases":[{"title":"Competitor AI accessibility audit","prompt":"Check whether GPTBot, ClaudeBot, and PerplexityBot are allowed to crawl https://competitor.com — I want to know if they're blocking AI indexing."},{"title":"Pre-publication crawl policy verification","prompt":"Before we publish our new content strategy, can you verify whether our site at https://mybrand.com currently allows or blocks major AI crawlers in robots.txt?"},{"title":"AI bot access monitoring for client site","prompt":"My client's site is https://clientsite.io — can you check which AI crawlers like Google-Extended and Applebot-Extended are permitted or blocked in their robots.txt?"}],"resultDescription":"Returns a JSON object with the origin URL, the HTTP status and presence of the robots.txt file, and per-bot access results for each major AI crawler (GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot, Applebot-Extended), including whether each is allowed and the reason derived from the robots.txt rules.","failureModes":["Target URL is not a valid public HTTPS URL — returns validation error","robots.txt returns 404 or is absent — reflected in response with present:false","Target site is unreachable or times out — may return network error","Malformed robots.txt on the target site — parsing may yield incomplete results","Rate limiting or access restrictions on the target site may affect result accuracy"],"whenToPreferThis":"Use this endpoint when you need to quickly determine whether a specific website's robots.txt permits or blocks major AI crawlers without manually fetching and parsing the file. It is ideal for competitive analysis, SEO audits, compliance checks, and agent workflows that need to assess AI accessibility of external sites. Prefer this over writing custom robots.txt parsers or using general web scraping when you need structured, per-bot permission data across the most relevant AI crawlers.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-09-14T12:32:57.877Z","isFirstParty":false}