{"uid":"cap_ow4T9x9OMpCBr0d3QJHOY","slug":"agentread-robots-691dc810","name":"agentread /robots","description":"Read a URL your own fetch failed on and get the page text back, or a verdict explaining why it cannot be read. Built for agents that just got a 403, 429, empty body or JS shell.","url":"https://agentread.midi-thumb.workers.dev/robots?utm_source=zero.xyz","method":"GET","headers":{},"bodySchema":{"type":"object","$schema":"https://json-schema.org/draft/2020-12/schema","required":["input"],"properties":{"input":{"type":"object","required":["type"],"properties":{"type":{"const":"http"},"method":{"const":"GET"}},"additionalProperties":true},"output":{"type":"object","additionalProperties":true}}},"responseSchema":{"type":"json","example":{}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.01","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.01/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.01","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.01","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_Qa9mSAKli6CWuGVwbXz8n","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.01","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Fetches and returns the robots.txt content from a target domain when a direct agent fetch failed, providing the raw text or a verdict on why it cannot be read.","exampleAgentPrompt":"I tried to fetch the robots.txt for example.com but got a 403 — can you use agentread to retrieve it for me and show me what it says?","exampleUseCases":[{"title":"Recover blocked robots.txt fetch","prompt":"My fetch of https://example.com/robots.txt returned a 403 — can you pull it through agentread and tell me which paths are disallowed?"},{"title":"Check crawl rules before scraping","prompt":"Before I start scraping techcrunch.com, can you grab their robots.txt via agentread so I can see which sections I'm allowed to crawl?"},{"title":"Diagnose empty robots.txt response","prompt":"I got an empty body when I tried to read https://shop.acme.com/robots.txt — can you try fetching it through agentread and tell me what it actually contains, or why it can't be read?"}],"resultDescription":"Returns the raw text content of the fetched robots.txt page, or if the URL cannot be retrieved, a structured verdict explaining the reason (e.g. blocked, JS-only, inaccessible). Response is JSON with the page text or failure explanation.","failureModes":["Target URL is genuinely inaccessible even via proxy — returns a verdict string explaining the block","URL resolves to a JavaScript shell with no static content — returns a verdict indicating JS-only rendering","Rate limit or IP ban persists even through the proxy layer — returns error verdict","Malformed or missing input URL — returns validation error","Target server returns empty body — returns verdict noting empty response"],"whenToPreferThis":"Use this endpoint specifically when your own HTTP fetch of a robots.txt URL has already failed with a 403, 429, empty body, or JavaScript shell response, and you need a fallback proxy fetch. It is purpose-built for the agent failure-recovery use case rather than general web scraping or structured data extraction.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-31T18:36:54.191Z","isFirstParty":false,"canonicalSlug":"agentread-robots-691dc810"}