{"uid":"cap_9jSVbEvBSMI2CMmM66Z0n","slug":"check-robots-txt-crawl-rules-c9fa41e1","name":"Check robots.txt crawl rules","description":"Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule that matched, the user-agent group, the crawl-delay and any declared sitemaps. Implements the Robots Exclusion Protocol (RFC 9309) with longest-match-wins, Allow-over-Disallow tie-breaking and * / $ wildcards.","url":"https://web.openverbs.com/v1/robots?utm_source=zero.xyz","method":"POST","headers":{},"bodySchema":{"type":"object","properties":{"url":{"type":"string","format":"uri","maxLength":2048,"description":"Public http(s) URL to check. Its origin's /robots.txt is fetched and its path is the first path evaluated."},"paths":{"type":"array","items":{"type":"string","maxLength":2048,"minLength":1},"maxItems":50,"description":"Optional extra paths (or same-origin URLs) to check in the same call, e.g. [\"/admin\", \"/search?q=x\"]. Up to 50."},"userAgent":{"type":"string","maxLength":200,"minLength":1,"description":"User-agent token to evaluate rules for, e.g. \"Googlebot\". Defaults to \"*\"."}}},"responseSchema":null,"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.006","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.006/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.006","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.006","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_QW-NjfEE8s52rA2vMaeq3","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.006","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Fetches a site's robots.txt and evaluates whether one or more paths are allowed or disallowed for a given user-agent.","exampleAgentPrompt":"Before I scrape https://example.com/blog and /admin, can you check whether those paths are allowed for a user-agent called 'MyBot' according to their robots.txt?","exampleUseCases":[{"title":"Pre-scrape crawlability check","prompt":"I'm about to scrape https://techcrunch.com/articles and /search — can you check if those paths are allowed for a generic wildcard user-agent in their robots.txt before I start?"},{"title":"SEO bot access audit","prompt":"Can you check whether Googlebot is allowed to access https://myshop.com/products, /checkout, and /account according to the site's robots.txt?"},{"title":"AI crawler policy check","prompt":"I want to know if the site at https://publisher.com blocks GPTBot or similar AI crawlers — check /articles and /feed for me."}],"resultDescription":"The endpoint returns the evaluated allow/disallow status for the primary URL path and any additional paths specified, based on the site's fetched robots.txt rules matched against the given user-agent token.","failureModes":["robots.txt not found (site returns 404 or 500 for /robots.txt) — treated as fully allowed per standard convention","Invalid or unreachable URL returns an error","URL scheme not http/https is rejected","More than 50 paths in a single call exceeds the limit","Malformed user-agent string may cause unexpected matching"],"whenToPreferThis":"Use this endpoint when you need to programmatically verify whether crawling specific paths on a site is permitted before making requests, especially for SEO audits, bot compliance checks, or pre-scrape validation. Prefer it over manually parsing robots.txt when checking multiple paths or multiple user-agents in a single call.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-27T19:03:34.244Z","isFirstParty":false,"canonicalSlug":"check-robots-txt-crawl-rules-c9fa41e1"}