{"uid":"cap_diLnsypQ_ch6JVWCEFAh8","slug":"crawlwitness-ai-usage-permission-check-4968f969","name":"CrawlWitness AI Usage Permission Check","description":"What a website allows for AI (robots.txt per AI bot, RSL, IETF AIPREF, TDMRep, noai, llms.txt, C2PA), with signed EAS receipts anchored on Base and an attested history of every signal. Paid calls use x402 (USDC on Base or Solana): no account, no API key.","url":"https://api.crawlwitness.com/v1/can-use?utm_source=zero.xyz","method":"GET","headers":{},"bodySchema":{"type":"object","$schema":"https://json-schema.org/draft/2020-12/schema","required":["input"],"properties":{"input":{"type":"object","required":["type","method"],"properties":{"type":{"type":"string","const":"http"},"method":{"enum":["GET"],"type":"string"},"queryParams":{"type":"object","required":["domain","bot"],"properties":{"bot":{"enum":["GPTBot","ClaudeBot","anthropic-ai","Google-Extended","GoogleOther","CloudVertexBot","Applebot-Extended","meta-externalagent","Amazonbot","amazon-kendra-","Bytespider","CCBot","cohere-training-data-crawler","AI2Bot","Diffbot","Timpibot","ImagesiftBot","omgili","PetalBot","KimiBot","ICC-Crawler","Cotoyogi","Reflectionbot","CloudflareBrowserRenderingCrawler","LinerBot","FishBot","Novellum","bigsur.ai","NavuBot","QualifiedBot","atlassian-bot","make.com","magpie-crawler","Awario","w4mwnpbXf3MFAbxOkJRw","SemrushBot-OCOB","SemrushBot-SWA","BorderxBot","SelectikaScraper","CitibotSiteCrawler","AdpResearchBot","WARDBot","ygs-scraper-bot","OAI-SearchBot","Claude-SearchBot","PerplexityBot","Applebot","Amzn-SearchBot","Bravebot","Cloudflare-AI-Search","Cloudflare-AI-Search-External","DuckAssistBot","YouBot","KernelSearchBot","ShapBot","alphalens-bot","Anomura","Element451Bot","ChatGPT-User","Claude-User","Perplexity-User","meta-externalfetcher","meta-externaltest","muse-","MistralAI-User","cohere-ai","Google-Agent","GrokAgent","Devin","ApifyWebsiteContentCrawler","Instapaper","Retool","QATechBot","TwinAgent","CledaraBot","HarkBot","HIFIBot","ChathiveCrawler","EasyScan","Nava","ZipchatBot"],"type":"string","description":"AI user-agent token, e.g. GPTBot, ClaudeBot, Google-Extended, CCBot"},"use":{"enum":["train","rag","search"],"type":"string","description":"Intended use (default train)"},"domain":{"type":"string","maxLength":253,"minLength":3,"description":"Domain, e.g. nytimes.com"}}}},"additionalProperties":false},"output":{"type":"object","required":["type"],"properties":{"type":{"type":"string"},"example":{"type":"object"}}}}},"responseSchema":{"type":"json","example":{"bot":"GPTBot","use":"train","pago":{"recibo":{"firma":"0x…"},"transaccion":"0x…"},"domain":"nytimes.com","method":"can-use/1","allowed":false,"reasons":["robots.txt disallows GPTBot at the site root"],"bot_role":"training","decision":"not_allowed","operator":"OpenAI"}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.001","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.001/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.001","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.001","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_w1hWsqNueC1SuVgm3O8AF","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.001","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Returns a yes/no verdict on whether a specific AI bot may crawl, train on, or RAG-ingest a given domain, by checking robots.txt, IETF AIPREF, TDMRep, noai, and RSL signals, with a signed receipt.","exampleAgentPrompt":"Before we ingest articles from wsj.com into our RAG pipeline using ClaudeBot, check if the site actually permits that — I need a signed receipt confirming the rag use is allowed.","exampleUseCases":[{"title":"Pre-training data compliance check","prompt":"We're building a training dataset and want to scrape wikipedia.org using GPTBot — can you verify that Wikipedia actually allows AI training use and get us a signed receipt we can keep for compliance?"},{"title":"RAG pipeline gate for enterprise ingestion","prompt":"Before our pipeline indexes any page from ft.com using PerplexityBot for RAG, automatically check whether that domain permits it so we don't ingest content we're not allowed to use."},{"title":"AI search bot permission audit","prompt":"I need to know if reddit.com allows OAI-SearchBot to use their content for search purposes — check their robots.txt and any AIPREF or noai signals and tell me the verdict."}],"resultDescription":"A yes/no permission verdict indicating whether the specified AI bot may perform the requested use (train, rag, or search) on the given domain, along with the policy sources consulted (robots.txt, IETF AIPREF, TDMRep, noai, RSL) and a cryptographically signed receipt that can be stored as a compliance record.","failureModes":["400 error for invalid or malformed domain (not charged)","400 error for unknown or unsupported bot token (not charged)","Domain has no explicit AI policy signals — verdict may default to permissive or restrictive depending on interpretation","Network or DNS resolution failure for the target domain","robots.txt fetch timeout causing delayed or failed verdict"],"whenToPreferThis":"Choose this endpoint when you need a fast, authoritative, multi-signal permission check before crawling, training on, or ingesting a domain into a RAG pipeline, and you need a signed receipt for compliance auditing. It aggregates robots.txt, IETF AIPREF, TDMRep, noai, and RSL signals in a single call, saving you from implementing each standard yourself. Prefer it over rolling your own robots.txt parser when AI-specific policy signals (AIPREF, noai) matter and when legal/compliance auditability requires a signed artifact.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-11-01T06:38:22.924Z","isFirstParty":false,"canonicalSlug":"crawlwitness-ai-usage-permission-check-4968f969"}