{"uid":"cap_ec-4l8FFfJxS9JkGxoFWy","slug":"ai-crawler-access-triage-ce9c1933","name":"AI Crawler Access Triage","description":"Check how websites treat AI crawlers, in bulk. POST {\"urls\":[...]} (up to 200) to get per-URL access verdicts for GPTBot, ClaudeBot, and a normal browser — served, refused, or tollgated — from a standing census, with a summary. Unknown URLs are added to the census and returned free; you are charged only for URLs already known. For agents deciding what they can fetch, and for spotting robots.txt, paywall, or bot-block changes over time.","url":"https://tizona-outpost.venusberg.workers.dev/v1/oracle/triage","method":"POST","headers":{},"bodySchema":{"type":"object","properties":{"urls":{"type":"array","items":{"type":"string"}}}},"responseSchema":null,"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.001","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.001/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.001","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.001","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_yBLzPq953tPMk2cD35WD-","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.001","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Bulk-checks how websites treat AI crawlers (GPTBot, ClaudeBot) and normal browsers, returning per-URL access verdicts (served, refused, or tollgated) from a standing census.","exampleAgentPrompt":"Check these 50 URLs and tell me which ones block GPTBot or ClaudeBot versus which ones serve them normally — I need to know if any are tollgating or outright refusing AI crawlers before I try to fetch them.","exampleUseCases":[{"title":"Pre-fetch AI crawler permission check","prompt":"Before my agent tries to scrape these 30 news sites, can you check which ones actually allow GPTBot and ClaudeBot through, and flag any that are paywalled or bot-blocked?"},{"title":"Competitor content access audit","prompt":"I have a list of 100 competitor websites — can you tell me which ones refuse AI crawlers, which ones tollgate them, and which ones serve content normally to GPTBot and ClaudeBot?"},{"title":"Spotting new bot-block rollouts","prompt":"Run a triage on these URLs we've been monitoring — I want to know if any sites that previously allowed AI crawlers have started blocking or tollgating GPTBot or ClaudeBot since we last checked."}],"resultDescription":"A per-URL breakdown of access verdicts for GPTBot, ClaudeBot, and a normal browser — each labeled as 'served', 'refused', or 'tollgated' — plus an overall summary. Unknown URLs are added to the census and returned free; already-known URLs incur a charge.","failureModes":["Exceeding 200 URLs per request returns an error","Malformed URLs in the array may produce null or error verdicts per entry","URLs very recently added to the census may have stale or incomplete data","Network timeouts to the census backend may delay or drop responses"],"whenToPreferThis":"Use this endpoint when you need to bulk-assess AI crawler accessibility across many URLs before attempting to fetch them, or when you want to detect robots.txt, paywall, or bot-block changes over time. It is more efficient than checking each URL individually and covers GPTBot, ClaudeBot, and normal browser access in a single call.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-09-14T12:42:42.802Z","isFirstParty":false}