{"uid":"cap__9R998mW7T3Z9mhEYQs16","slug":"tariffmonkee-crawler-diligence-check-390cec43","name":"TariffMonkee Crawler Diligence Check","description":"Small, single-purpose web data APIs that AI agents pay for per call in USDC over the x402 protocol (HTTP 402). No accounts, no API keys, no subscriptions.","url":"https://tariffmonkee.com/paid/crawler-diligence","method":"GET","headers":{},"bodySchema":{"properties":{"input":{"type":"object","required":["type","method"],"properties":{"type":{"type":"string","const":"http"},"method":{"enum":["GET"],"type":"string"}},"additionalProperties":false}}},"responseSchema":null,"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.04","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.04/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.04","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.04","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_wbRFHOES8PrKrbJuEKK_u","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.04","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Runs due-diligence checks on a URL to assess how well it behaves for web crawlers, returning signals useful for evaluating a site's crawlability and trustworthiness.","exampleAgentPrompt":"Before my agent scrapes that site, can you run a crawler diligence check on https://example.com to see if it allows bots and looks trustworthy?","exampleUseCases":[{"title":"Pre-crawl safety check for agent","prompt":"I'm about to have my agent fetch a bunch of pages from newssite.com — can you run a crawler diligence check on it first to make sure it won't block us and seems legit?"},{"title":"Competitor research site vetting","prompt":"I want to scrape pricing data from competitor.com, but first check whether that domain is crawler-friendly and doesn't have any red flags that would get my bot blocked."},{"title":"Content pipeline URL triage","prompt":"We're ingesting URLs from third-party feeds and need to filter out ones that block crawlers or look shady — can you run a diligence check on https://suspiciousblog.net before we add it to the pipeline?"}],"resultDescription":"A structured report on the target URL's crawler diligence, including signals about robots.txt rules, bot-blocking behavior, site trustworthiness indicators, and overall crawlability assessment — helping an agent decide whether it is safe and permitted to proceed with fetching content from the domain.","failureModes":["URL is unreachable or returns a network error, resulting in incomplete report","Target site actively blocks the diligence probe, returning limited or no data","Malformed or non-HTTP URL input causes a request validation error","Rate limiting or timeout on the target domain prevents full signal collection","The site uses advanced bot detection that obscures true crawlability status"],"whenToPreferThis":"Choose this endpoint when you need a quick, pay-per-call assessment of whether a URL is safe and permitted for automated crawling — especially before committing to a full scrape or adding a domain to a content pipeline. Prefer it over manual inspection when you need programmatic, structured crawler compliance signals without setting up accounts or subscriptions. It complements browser-rendering or content-extraction endpoints by acting as a gatekeeper check first.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-09-13T18:43:44.060Z","isFirstParty":false}