{"uid":"cap_aUbLMEs3ssTxAx0OPsT4Z","slug":"sitecheck-ai-crawler-policy-checker-23beac3c","name":"SiteCheck AI Crawler Policy Checker","description":"For eleven AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others): allowed or blocked for the page's path under robots.txt, plus the noai and noimageai opt-outs in meta and X-Robots-Tag, and whether llms.txt exists. Reads the page as static HTML; no JavaScript runs.","url":"https://api.sitecheck-api.workers.dev/api/ai-crawlers?utm_source=zero.xyz","method":"GET","headers":{},"bodySchema":{"type":"object","$schema":"https://json-schema.org/draft/2020-12/schema","required":["input"],"properties":{"input":{"type":"object","required":["type","method"],"properties":{"type":{"type":"string","const":"http"},"method":{"enum":["GET"],"type":"string"},"queryParams":{"type":"object","required":["url"],"properties":{"url":{"type":"string","description":"Page address, https://…"}}}},"additionalProperties":false},"output":{"type":"object","required":["type"],"properties":{"type":{"type":"string"},"example":{"type":"object"}}}}},"responseSchema":{"type":"json","example":{"url":"https://developers.cloudflare.com/","path":"/","optOut":{"noai":false,"noimageai":false,"metaRobots":null,"xRobotsTag":null},"source":"https://developers.cloudflare.com/robots.txt and the page (static HTML, no JavaScript run)","llmsTxt":true,"crawlers":[{"name":"GPTBot","allowed":true},{"name":"ChatGPT-User","allowed":true},{"name":"OAI-SearchBot","allowed":true}],"checkedAt":"2026-10-14T07:09:42.733Z","robotsTxt":{"exists":true}}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.002","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.002/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.002","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.002","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_gTSy6_0oPBlpeK-6dLTy1","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.002","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Checks whether a given webpage allows or blocks AI crawlers by inspecting its robots.txt, meta robots tags, X-Robots-Tag headers, and llms.txt file","exampleAgentPrompt":"Can you check whether https://developers.cloudflare.com/ allows AI crawlers like GPTBot and ChatGPT-User to access it, including what its robots.txt and llms.txt say?","exampleUseCases":[{"title":"Pre-crawl AI permission check","prompt":"Before I scrape https://openai.com/ with my AI agent, can you check whether it allows GPTBot and OAI-SearchBot in its robots.txt and whether it has an llms.txt file?"},{"title":"AI training data opt-out audit","prompt":"I want to know if https://nytimes.com/ has opted out of AI image training or AI crawling — check its robots.txt, meta robots tags, and X-Robots-Tag for any AI opt-out signals."},{"title":"Dataset source compliance verification","prompt":"I'm building a dataset from web pages and need to confirm https://huggingface.co/ permits AI crawlers — can you check its crawler policy including llms.txt and any noai or noimageai directives?"}],"resultDescription":"Returns a JSON object containing the page URL, whether llms.txt exists, a list of named AI crawlers (e.g. GPTBot, ChatGPT-User, OAI-SearchBot) with their allowed/blocked status, robots.txt existence, opt-out flags (noai, noimageai, metaRobots, xRobotsTag), the data source (robots.txt + static HTML), and a timestamp of when the check was performed.","failureModes":["URL not reachable or returns non-200 — robots.txt may be assumed absent","Malformed URL input causes validation error","Site uses JavaScript-rendered robots directives that static HTML parsing misses (no JS execution)","robots.txt exists but is malformed or uses non-standard syntax — partial parsing may occur","Rate limits or network timeouts on the target server may cause incomplete results"],"whenToPreferThis":"Choose this endpoint when you need to programmatically verify whether a specific webpage or domain permits AI crawlers before scraping, indexing, or training on its content. It is the right choice when you need granular per-bot permission data (GPTBot, ChatGPT-User, OAI-SearchBot) and want to check both robots.txt and llms.txt in one call. Prefer it over generic robots.txt parsers when you specifically care about AI crawler opt-out signals (noai, noimageai) and meta/header-level directives.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-15T06:38:40.055Z","isFirstParty":false,"canonicalSlug":"sitecheck-ai-crawler-policy-checker-23beac3c"}