{"uid":"cap_DJvginbKN-N9Y2rYX49rd","slug":"kairostamp-robots-txt-checker-0dbb99c3","name":"Kairostamp Robots.txt Checker","description":"robots.txt check: fetches the origin's /robots.txt server-side and reports whether a given URL path is allowed or blocked for a user agent, with the matched rule, the robots URL, the HTTP status, a timestamp and a tamper-evident receipt. Query parameters: url (required), user_agent (optional, defaults to *). Only robots.txt is fetched; no crawling. Pair with GET /api/v1/link-map or /api/v1/sitemap to plan an allowed fetch.","url":"https://kairostamp.com/api/v1/robots-check","method":"GET","headers":{},"bodySchema":null,"responseSchema":null,"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.005","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.001/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.001","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.001","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_qpekouoLeb1Ti1oS7xqs0","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.001","costPer":"request","priority":0,"asset":"0x036CbD53842c5426634e7929541eC2318f3dCF7e","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Server-side fetches a site's robots.txt and evaluates whether a given URL path is allowed or blocked for a specified user agent, returning the decision, matched rule, HTTP status, and a tamper-evident timestamped receipt.","exampleAgentPrompt":"Before you fetch that page at https://example.com/pricing, can you check whether GPTBot is allowed to crawl it according to the site's robots.txt?","exampleUseCases":[{"title":"Pre-crawl permission check for AI bot","prompt":"Check if GPTBot is allowed to scrape https://techcrunch.com/2024/01/ai-news before my agent fetches it — I need to know the matched robots.txt rule and whether it's blocked."},{"title":"Audit site crawl policy for Googlebot","prompt":"Look up the robots.txt for https://shop.example.com/products and tell me whether Googlebot is permitted to crawl that path, and what rule matched."},{"title":"Validate crawlability before sitemap traversal","prompt":"I'm about to map out https://docs.acme.io — first check if the wildcard user agent is allowed to hit /docs/api/reference according to the robots.txt, and give me a timestamped receipt I can keep as proof."}],"resultDescription":"Returns a JSON object with the crawl decision (allowed, blocked, or unknown), the user agent evaluated, the matched robots.txt rule, the URL of the robots.txt file fetched, its HTTP status code, a fetch timestamp, a UUID receipt ID, and a tamper-evident chain hash for auditability.","failureModes":["robots.txt not found (404) — decision returned as 'unknown' with null matched_rule","target domain unreachable — fetch fails, robots_status null","malformed URL input — validation error before fetch","robots.txt fetch times out — unknown decision returned","user_agent token too long (over 200 chars) — rejected by schema validation"],"whenToPreferThis":"Use this endpoint when an agent needs to verify crawl permissions for a specific URL before fetching it, especially when a tamper-evident audit receipt is required. Prefer it over parsing robots.txt manually — it handles server-side fetching, rule matching, and receipt generation in one call. Combine with the link-map or sitemap endpoints on the same platform to plan compliant crawls.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-09-16T00:41:54.854Z","isFirstParty":false}