{"uid":"cap_fY6z7UCP0br1nhiiCndzJ","slug":"robots-txt-linter-and-allow-disallow-analyzer-fb2f57e5","name":"robots.txt Linter and Allow/Disallow Analyzer","description":"Lints and validates a caller-supplied robots.txt against RFC 9309 and reports every check, not only the failures: group structure, product token charset, path patterns, special characters, other records, size limit. Given a user agent and URLs it recomputes the allow/disallow decision per URL, naming the winning rule, its octet length and the tiebreak that applied. Where the RFC is silent or contradicts itself, the reading used is stated. Makes no outbound request.","url":"https://api.ozdreamtools.de/api/robots/lint?utm_source=zero.xyz","method":"POST","headers":{},"bodySchema":{"type":"object","properties":{"urls":{"type":"array","items":{"type":"string"},"description":"URLs or paths whose access decision should be recomputed, at most 50"},"robots_txt":{"type":"string","description":"Contents of the robots.txt to check"},"user_agent":{"type":"string","description":"Product token to evaluate for, e.g. 'GPTBot'. A full User-Agent header is resolved to the token it contains. Required when 'urls' is given."}}},"responseSchema":null,"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.05","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.05/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.05","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.05","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_2bzi_tTdkl49WTYwcpuxv","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.05","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Validates a robots.txt file against RFC 9309, reporting all checks (not just failures), and computes per-URL allow/disallow decisions with winning rule details for a given user agent.","exampleAgentPrompt":"Can you lint this robots.txt content and tell me whether Googlebot is allowed to access /blog/private and /admin — I want to know which rule wins and why: 'User-agent: *\\nDisallow: /admin\\nAllow: /blog'","exampleUseCases":[{"title":"Debug blocked Googlebot URLs","prompt":"Here's my robots.txt — can you check if it's RFC 9309 compliant and tell me whether Googlebot is allowed or disallowed from /shop/checkout and /sitemap.xml, including which rule wins each decision?"},{"title":"Pre-deploy robots.txt audit","prompt":"Before I push this robots.txt to production, can you run a full lint against RFC 9309 and report every check — group structure, product token charset, path patterns, special characters, and whether I'm within the size limit?"},{"title":"Diagnose crawl budget issue","prompt":"My site is getting unexpectedly blocked for certain crawlers. Here's my robots.txt — can you check the allow/disallow decisions for the user agent 'bingbot' across these URLs: /products, /products/sale, and /internal/api, and tell me what rule and tiebreak applies to each?"}],"resultDescription":"A detailed lint report covering every RFC 9309 check (group structure, product token charset, path patterns, special characters, other records, size limit) with pass/fail status for each; plus per-URL allow/disallow decisions for the specified user agent, naming the winning rule, its octet length, and the tiebreak logic applied, along with notes on any RFC ambiguities and the interpretation used.","failureModes":["Malformed or missing robots.txt content returns a parsing error","Invalid or missing user agent causes decision computation to fail","Empty URL list means no per-URL decisions are returned","Extremely large robots.txt content may hit size validation limits","RFC ambiguity edge cases are noted but may produce unexpected interpretations"],"whenToPreferThis":"Choose this endpoint when you need exhaustive RFC 9309 compliance reporting on a robots.txt — not just error detection but confirmation of every passing check too — and when you need to trace exactly why a specific URL is allowed or disallowed for a named crawler, including the winning rule and tiebreak. It makes no outbound requests and processes only caller-supplied content, making it suitable for CI/CD pipelines, pre-deploy audits, or debugging without exposing live infrastructure.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-10T18:40:23.400Z","isFirstParty":false,"canonicalSlug":"robots-txt-linter-and-allow-disallow-analyzer-fb2f57e5"}