{"uid":"cap_vsdTAiLgtEbk5Ju3InLOx","slug":"tanod-web-robots-txt-checker-65f00d2e","name":"Tanod Web – robots.txt Checker","description":"Web tools: search, page to Markdown or screenshot, company profile and jobs, sitemaps, RDAP, IP.","url":"https://web.tanod.dev/v1/robots?utm_source=zero.xyz","method":"POST","headers":{},"bodySchema":{"type":"object","properties":{"url":{"type":"string","maxLength":2048,"minLength":1,"description":"Public http(s) URL to test; <origin>/robots.txt is fetched and the URL's path is matched against it."},"user_agent":{"type":"string","default":"*","maxLength":200,"minLength":1,"description":"Product token (e.g. Googlebot) or a full User-Agent string; '*' (default) = any crawler."}}},"responseSchema":{"type":"json","example":{"url":"https://www.google.com/search","path":"/search","fetch":"ok","group":"*","stats":{"lines":264,"rules":245,"groups":5,"ignored_lines":0},"reason":"most specific matching rule: disallow /search","allowed":false,"sitemaps":["https://www.google.com/sitemap.xml"],"truncated":false,"robots_url":"https://www.google.com/robots.txt","user_agent":"Googlebot","crawl_delay":null,"http_status":200,"matched_rule":{"line":3,"type":"disallow","pattern":"/search"},"product_token":"googlebot","sitemaps_total":1,"untrusted_content":true}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.001","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.001/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.001","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.001","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_DlI3QZhfBNDGKNq9EkllZ","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.001","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Fetches a URL's robots.txt and checks whether a given path is allowed or disallowed for a specified user-agent/crawler.","exampleAgentPrompt":"Can you check whether Googlebot is allowed to crawl https://www.example.com/blog/private-post — fetch the robots.txt and tell me which rule applies and whether it's blocked?","exampleUseCases":[{"title":"SEO audit for blocked pages","prompt":"I need to know if our product pages at https://shop.example.com/products are being blocked from Google's crawler — check robots.txt for Googlebot and tell me the matched rule."},{"title":"Scraper pre-flight compliance check","prompt":"Before I run my scraper, can you check if https://news.ycombinator.com/item?id=12345 is allowed for a generic crawler in robots.txt, and also list any sitemaps you find?"},{"title":"Competitor site crawl policy research","prompt":"What does https://www.competitor.com/robots.txt say about crawling their /pricing page — is it allowed, what's the crawl delay, and which user-agent groups does it define?"}],"resultDescription":"Returns whether the URL path is allowed or disallowed for the specified crawler, the exact matched rule (line number, type, pattern), robots.txt parse stats (lines, rules, groups), list of sitemap URLs, crawl delay, the raw robots.txt URL, HTTP status of the robots.txt fetch, and whether the content was truncated.","failureModes":["robots.txt unreachable or returns non-200 HTTP status — fetch field will indicate error","URL malformed or missing scheme — input validation error","robots.txt too large causing truncation (truncated: true in response)","No matching rule found — defaults to allowed","User-agent product token not recognized — falls back to wildcard group"],"whenToPreferThis":"Use this endpoint when you need a definitive, parsed robots.txt compliance check for a specific URL and crawler, especially when you need the exact matched rule, crawl delay, and sitemap discovery in one call. Prefer over manual parsing or generic fetch tools when you need structured output with rule attribution.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-13T12:43:37.254Z","isFirstParty":false,"canonicalSlug":"tanod-web-robots-txt-checker-65f00d2e"}