{"uid":"cap_T4qlFn_Oo_3eL0-5_2lRn","slug":"agent-services-x402-robots-txt-checker-61207b77","name":"agent-services x402 robots.txt checker","description":"Paid per-call web tools for AI agents: DNS, TLS, robots.txt, hashing, JavaScript rendering and screenshots. Payment over x402 (USDC on Base).","url":"https://49112-hjhjtioxvirqvcvo.splox.app/robots?utm_source=zero.xyz","method":"GET","headers":{},"bodySchema":null,"responseSchema":{"type":"json","example":{"robots":true,"disallow":["/admin"],"sitemaps":["https://example.com/sitemap.xml"]}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.0005","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"registry","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.0005/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.0005","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.0005","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_ekxwZDwdhW6Hznd76Tvyk","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.0005","costPer":"request","priority":0,"asset":null,"unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Fetches and parses the robots.txt file for a given domain, returning crawl permissions and sitemap locations, paid per-call via x402 USDC.","exampleAgentPrompt":"Can you check the robots.txt for example.com and tell me which paths are disallowed and whether there are any sitemaps listed?","exampleUseCases":[{"title":"Pre-scrape crawl permission check","prompt":"Before I scrape techcrunch.com, can you check their robots.txt to see which paths are off-limits and if there's a sitemap I should use instead?"},{"title":"SEO audit for sitemap discovery","prompt":"I'm doing an SEO audit on shopify.com — can you pull their robots.txt and list all the sitemaps they've declared there?"},{"title":"Verify bot access policy for competitor site","prompt":"Can you check whether robots.txt on competitor.com disallows access to their /pricing or /checkout pages for automated bots?"}],"resultDescription":"A JSON object containing a boolean indicating whether crawling is permitted, an array of disallowed path patterns from the robots.txt, and an array of sitemap URLs declared in the file.","failureModes":["Domain does not have a robots.txt file — may return empty or default-permit response","Network timeout reaching the target domain","Invalid or malformed domain input causes request failure","robots.txt is too large or malformed to parse correctly","Target site blocks the fetcher IP, returning a non-200 status"],"whenToPreferThis":"Use this endpoint when an AI agent needs to programmatically inspect a website's robots.txt before crawling, scraping, or indexing — especially when you need both disallow rules and sitemap locations in a single structured response. Prefer this over manual fetch when you need parsed, structured output rather than raw text, and when you want pay-per-call pricing without a subscription.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-02T18:35:28.669Z","isFirstParty":false,"canonicalSlug":"agent-services-x402-robots-txt-checker-61207b77"}