{"uid":"cap__9R998mW7T3Z9mhEYQs16","slug":"tariffmonkee-crawler-diligence-check-390cec43","name":"TariffMonkee Crawler Diligence Check","description":"AI crawler monetization check for a domain: per-crawler robots.txt rules for 18 named AI crawlers and whether access is already sold through TollBit, RSL, x402 or a 402 response. Verdict: unmonetized-crawler-magnet, already-monetized, blocked-not-monetized or low-appeal. A revenue figure appears only when the caller supplies a rate and volume.","url":"https://tariffmonkee.com/paid/crawler-diligence?utm_source=zero.xyz","method":"GET","headers":{},"bodySchema":{"type":"object","$schema":"https://json-schema.org/draft/2020-12/schema","required":["input"],"properties":{"input":{"type":"object","required":["type","method"],"properties":{"type":{"type":"string","const":"http"},"method":{"enum":["GET"],"type":"string"},"queryParams":{"type":"object","required":["domain"],"properties":{"domain":{"type":"string","description":"Bare domain to screen, no scheme."},"monthly_crawls":{"type":"number","description":"Your assumed monthly AI crawler hits. Required together with rate_per_crawl."},"rate_per_crawl":{"type":"number","description":"Your assumed USD per crawl. Required together with monthly_crawls. No rate is assumed for you: none is published industry-wide."}}}},"additionalProperties":false},"output":{"type":"object","required":["type"],"properties":{"type":{"type":"string"},"example":{"type":"object","required":["domain","verdict","confidence","reachable","robots_txt_found","crawler_rules","rules_summary","monetization","content_signals","regulatory_note","caveats","checked_at"],"properties":{"domain":{"type":"string"},"caveats":{"type":"array","items":{"type":"string"}},"verdict":{"type":"string"},"reachable":{"type":"boolean"},"checked_at":{"type":"string"},"confidence":{"type":"string"},"monetization":{"type":"object","required":["cloudflare_ai_crawl_control","tollbit","rsl_license","x402","returns_402_to_crawler","evidence"],"properties":{"x402":{"type":"boolean"},"tollbit":{"type":"boolean"},"evidence":{"type":"array"},"rsl_license":{"type":"boolean"},"returns_402_to_crawler":{"type":"boolean"},"cloudflare_ai_crawl_control":{"type":"boolean"}}},"crawler_rules":{"type":"array","items":{"type":"object","required":["crawler","operator","kind","status","detail"],"properties":{"kind":{"type":"string"},"detail":{"type":"string"},"status":{"type":"string"},"crawler":{"type":"string"},"operator":{"type":"string"}}}},"rules_summary":{"type":"object","required":["allowed","blocked","unlisted"],"properties":{"allowed":{"type":"number"},"blocked":{"type":"number"},"unlisted":{"type":"number"}}},"content_signals":{"type":"object","required":["has_llms_txt","article_like","detail"],"properties":{"detail":{"type":"string"},"article_like":{"type":"boolean"},"has_llms_txt":{"type":"boolean"}}},"regulatory_note":{"type":"string"},"revenue_estimate":{},"robots_txt_found":{"type":"boolean"}}}}}}},"responseSchema":{"type":"json","example":{"domain":"example.com","caveats":["Screening depth is an external probe. It reports what the site publicly declares and how it responds, not what its server logs actually contain.","robots.txt is a request, not enforcement. A crawler that ignores it will not appear blocked here and will not appear in these results at all.","No revenue figure is produced unless you supply both a rate and a crawl volume. There is no industry rate table: TollBit and Cloudflare prices are set by each publisher individually and are not published."],"verdict":"low-appeal","reachable":true,"checked_at":"2026-10-18T08:07:32.297Z","confidence":"low","monetization":{"x402":false,"tollbit":false,"evidence":[],"rsl_license":false,"returns_402_to_crawler":false,"cloudflare_ai_crawl_control":false},"crawler_rules":[{"kind":"training","detail":"Not named in robots.txt, so crawling is permitted by default","status":"unlisted","crawler":"GPTBot","operator":"OpenAI"},{"kind":"search","detail":"Not named in robots.txt, so crawling is permitted by default","status":"unlisted","crawler":"OAI-SearchBot","operator":"OpenAI"},{"kind":"agent","detail":"Not named in robots.txt, so crawling is permitted by default","status":"unlisted","crawler":"ChatGPT-User","operator":"OpenAI"}],"rules_summary":{"allowed":18,"blocked":0,"unlisted":18},"content_signals":{"detail":"Homepage does not look article-like; may be an app, storefront, or landing page","article_like":false,"has_llms_txt":false},"regulatory_note":"From 2026-09-15 Cloudflare blocks mixed-use AI crawlers by default on ad-hosting pages for new customers, new sites, and all free-plan customers. Cloudflare has also replaced per-fetch Pay Per Crawl with Pay Per Use, which pays on content used in AI answers rather than per fetch. Both materially aff...","revenue_estimate":null,"robots_txt_found":false}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.04","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.04/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.04","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.04","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_wbRFHOES8PrKrbJuEKK_u","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.04","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Runs due-diligence checks on a URL to assess how well it behaves for web crawlers, returning signals useful for evaluating a site's crawlability and trustworthiness.","exampleAgentPrompt":"Before my agent scrapes that site, can you run a crawler diligence check on https://example.com to see if it allows bots and looks trustworthy?","exampleUseCases":[{"title":"Pre-crawl safety check for agent","prompt":"I'm about to have my agent fetch a bunch of pages from newssite.com — can you run a crawler diligence check on it first to make sure it won't block us and seems legit?"},{"title":"Competitor research site vetting","prompt":"I want to scrape pricing data from competitor.com, but first check whether that domain is crawler-friendly and doesn't have any red flags that would get my bot blocked."},{"title":"Content pipeline URL triage","prompt":"We're ingesting URLs from third-party feeds and need to filter out ones that block crawlers or look shady — can you run a diligence check on https://suspiciousblog.net before we add it to the pipeline?"}],"resultDescription":"A structured report on the target URL's crawler diligence, including signals about robots.txt rules, bot-blocking behavior, site trustworthiness indicators, and overall crawlability assessment — helping an agent decide whether it is safe and permitted to proceed with fetching content from the domain.","failureModes":["URL is unreachable or returns a network error, resulting in incomplete report","Target site actively blocks the diligence probe, returning limited or no data","Malformed or non-HTTP URL input causes a request validation error","Rate limiting or timeout on the target domain prevents full signal collection","The site uses advanced bot detection that obscures true crawlability status"],"whenToPreferThis":"Choose this endpoint when you need a quick, pay-per-call assessment of whether a URL is safe and permitted for automated crawling — especially before committing to a full scrape or adding a domain to a content pipeline. Prefer it over manual inspection when you need programmatic, structured crawler compliance signals without setting up accounts or subscriptions. It complements browser-rendering or content-extraction endpoints by acting as a gatekeeper check first.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-28T12:35:40.569Z","isFirstParty":false,"canonicalSlug":"tariffmonkee-crawler-diligence-check-390cec43"}