{"uid":"cap_HX2rRDgwDHmV6WB6gEhBG","slug":"402utils-robots-txt-compliance-checker-af66e5d5","name":"402utils Robots.txt Compliance Checker","description":"Robots.txt compliance check for polite crawlers and AI agents (GPTBot, ClaudeBot, ...). Pass a page URL and optionally your crawler's product token; returns whether the URL may be fetched, the winning Allow/Disallow rule, crawl-delay and sitemap URLs. Full RFC 9309 matching: wildcards, $ anchors, longest-match, merged groups. A 4xx robots.txt allows all; 5xx or unreachable hosts report disallowed. Cached per host for 1 hour.","url":"https://402utils.com/v1/robots-check?utm_source=zero.xyz","method":"GET","headers":{},"bodySchema":{"type":"object","$schema":"https://json-schema.org/draft/2020-12/schema","required":["input"],"properties":{"input":{"type":"object","required":["type","method"],"properties":{"type":{"type":"string","const":"http"},"method":{"enum":["GET"],"type":"string"},"queryParams":{"type":"object","required":["url"],"properties":{"url":{"type":"string","format":"uri","description":"Public http(s) page URL to test (required)."},"userAgent":{"type":"string","default":"*","description":"Crawler product token, e.g. \"GPTBot\". Defaults to \"*\"."}},"description":"Query string only."}},"additionalProperties":false},"output":{"type":"object","required":["type"],"properties":{"type":{"type":"string"},"example":{"type":"object","required":["url","userAgent","allowed","robotsUrl","robotsStatus","sitemaps"],"properties":{"url":{"type":"string"},"allowed":{"type":"boolean"},"sitemaps":{"type":"array","items":{"type":"string"}},"robotsUrl":{"type":"string"},"userAgent":{"type":"string"},"crawlDelay":{"type":"number","description":"Crawl-delay of the matched group, in seconds."},"matchedRule":{"type":"string","description":"Winning rule as written, e.g. \"Disallow: /search\". Absent when no rule matched."},"robotsStatus":{"type":"integer","description":"HTTP status of robots.txt; 0 when it was unreachable (network failure ⇒ full disallow per RFC 9309)."}}}}}}},"responseSchema":{"type":"json","example":{"url":"https://example.com/search?q=demo","allowed":false,"sitemaps":["https://example.com/sitemap.xml"],"robotsUrl":"https://example.com/robots.txt","userAgent":"GPTBot","matchedRule":"Disallow: /search","robotsStatus":200}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.001","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":{"p10Cents":"0.1000","medianCents":"0.1000","p90Cents":"0.1000","minCents":"0.1000","maxCents":"0.1000","p95Cents":"0.1000","sampleCount":1,"varies":false,"failureChargeRate":1},"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.001/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.001","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.001","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_sUrky264VF5FxfmEDOzvB","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.001","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Checks whether a given URL is allowed or disallowed for a specific crawler bot by fetching and parsing the host's robots.txt per RFC 9309","exampleAgentPrompt":"Before I scrape https://example.com/blog/article-123, can you check if GPTBot is allowed to fetch that URL according to the site's robots.txt?","exampleUseCases":[{"title":"Verify AI crawler access compliance","prompt":"I need to crawl a bunch of product pages for our research. Can you check if ClaudeBot is allowed on each domain's robots.txt before we start?"},{"title":"Discover sitemaps and crawl rules","prompt":"We're building a search index for academic papers. Can you look up the robots.txt for these research repository sites and tell me what sitemaps they publish plus the crawl-delay rules?"},{"title":"Batch scraping with bot permission checks","prompt":"Before we scrape competitor pricing data from these e-commerce sites, can you verify which ones actually allow our crawler bot according to their robots.txt rules?"}],"resultDescription":"Returns a JSON object with: allowed (boolean), matchedRule (the winning Allow/Disallow directive as written), crawlDelay (seconds from the matched group), sitemaps (array of sitemap URLs declared in robots.txt), robotsUrl (URL of the fetched robots.txt), robotsStatus (HTTP status of the robots.txt response, 0 if unreachable). A 4xx robots.txt status means all crawling is allowed; 5xx or unreachable means disallowed per RFC 9309.","failureModes":["Host unreachable — robotsStatus is 0, allowed is false per RFC 9309","robots.txt returns 5xx — allowed is false per RFC 9309","robots.txt returns 4xx — allowed is true (permissive fallback)","Invalid or malformed URL input — likely 4xx API error","Robots.txt cache miss causes slight latency increase on first fetch per host"],"whenToPreferThis":"Use this endpoint when an AI agent or crawler needs to verify RFC 9309-compliant robots.txt access rules before fetching a URL, especially when you need wildcard/anchor matching, longest-match resolution, crawl-delay extraction, and sitemap discovery all in one call. Prefer over manual robots.txt parsing when you need full RFC 9309 compliance including merged groups, $ anchors, and proper 4xx/5xx fallback behavior. Cached per host for 1 hour, making it efficient for batches of URLs on the same domain.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-12T12:40:47.450Z","isFirstParty":false,"canonicalSlug":"402utils-robots-txt-compliance-checker-af66e5d5"}