{"uid":"cap_Z-KbbJ1ndZiWyaHa-t38A","slug":"sitecheck-robots-txt-checker-740aed61","name":"SiteCheck Robots.txt Checker","description":"Checks whether a URL's path may be crawled under robots.txt (RFC 9309): the verdict for a user agent (default: *, Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot), the rule that decided it and its line, crawl-delay, sitemaps and the matching group's lines. Longest rule wins, allow wins a tie, * and $ work. No robots.txt means everything is allowed; a server error is a 503, not charged.","url":"https://api.sitecheck-api.workers.dev/api/robots?utm_source=zero.xyz","method":"GET","headers":{},"bodySchema":{"type":"object","$schema":"https://json-schema.org/draft/2020-12/schema","required":["input"],"properties":{"input":{"type":"object","required":["type","method"],"properties":{"type":{"type":"string","const":"http"},"method":{"enum":["GET"],"type":"string"},"queryParams":{"type":"object","required":["url"],"properties":{"url":{"type":"string","description":"A page URL on the site whose robots.txt is checked"},"userAgent":{"type":"string","description":"One crawler name to check, e.g. Googlebot (optional; default lists the common crawlers)"}}}},"additionalProperties":false},"output":{"type":"object","required":["type"],"properties":{"type":{"type":"string"},"example":{"type":"object"}}}}},"responseSchema":{"type":"json","example":{"url":"https://www.google.com/search?q=x","note":null,"path":"/search?q=x","origin":"https://www.google.com","source":"https://www.google.com/robots.txt, RFC 9309 matching (longest rule wins, allow wins a tie)","status":200,"results":[{"rule":{"line":3,"type":"disallow","pattern":"/search"},"group":{"lines":["User-agent: *","User-agent: Yandex","Disallow: /search"],"userAgent":"*"},"allowed":false,"userAgent":"*","crawlDelay":null},{"rule":{"line":3,"type":"disallow","pattern":"/search"},"group":{"lines":["User-agent: *","User-agent: Yandex","Disallow: /search"],"userAgent":"*"},"allowed":false,"userAgent":"Googlebot","crawlDelay":null},{"rule":{"line":3,"type":"disallow","pattern":"/search"},"group":{"lines":["User-agent: *","User-agent: Yandex","Disallow: /search"],"userAgent":"*"},"allowed":false,"userAgent":"Bingbot","crawlDelay":null}],"sitemaps":["https://www.google.com/sitemap.xml"],"checkedAt":"2026-10-15T14:07:52.191Z"}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.001","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.001/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.001","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.001","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_Flmw6hsrU7FWJtMwwPlWB","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.001","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Fetches and parses a site's robots.txt file to determine which crawlers are allowed or blocked from accessing a given URL path","exampleAgentPrompt":"Can you check whether Googlebot is allowed to crawl https://www.example.com/products according to that site's robots.txt?","exampleUseCases":[{"title":"SEO audit for blocked pages","prompt":"I'm auditing our site's SEO — can you check if Googlebot is allowed to crawl https://shop.mystore.com/checkout based on our robots.txt?"},{"title":"Competitor crawl policy research","prompt":"I want to know whether Bingbot is blocked from accessing https://www.competitor.com/pricing — can you check their robots.txt?"},{"title":"Scraping compliance check","prompt":"Before I scrape https://www.news-site.com/articles, can you check whether any common crawlers are allowed there and list any sitemaps their robots.txt mentions?"}],"resultDescription":"Returns the parsed robots.txt evaluation for the given URL, including: whether each crawler (default: common crawlers like Googlebot, Bingbot, Yandex, etc.) is allowed or disallowed, the specific matching rule and line number, the user-agent group it falls under, any crawl delay, discovered sitemap URLs, and a timestamp of when the check was performed.","failureModes":["robots.txt not found (404) — site may not have one, returns 200 with empty rules","URL is malformed — validation error returned","Site is unreachable or times out — network error","Specified user agent doesn't match any group — falls back to wildcard rules"],"whenToPreferThis":"Use this endpoint when you need to programmatically verify crawl permissions for a specific URL and crawler combination, especially during SEO audits, before building web scrapers, or when checking competitor crawl policies. It parses robots.txt with RFC 9309-compliant longest-match rules, making it more accurate than naive text parsing.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-20T12:48:09.706Z","isFirstParty":false,"canonicalSlug":"sitecheck-robots-txt-checker-740aed61"}