{"uid":"cap_I-KIIAnaLN-I2-jFeYCYl","slug":"creator-desk-crawl-config-robots-txt-inspector-5a2e3d45","name":"Creator Desk Crawl Config (robots.txt Inspector)","description":"Machine-callable x402-paid utilities for autonomous agents.","url":"https://creator-desk-x402-isolated.onrender.com/v1/site/crawl-config?utm_source=zero.xyz","method":"POST","headers":{},"bodySchema":{"type":"object","properties":{"url":{"type":"string","maxLength":2048,"description":"Any public HTTPS URL on the target site. The tool inspects that origin's /robots.txt."}}},"responseSchema":{"type":"json","example":{"http":{"status":200,"latency_ms":90,"status_text":"OK","content_type":"text/plain"},"scope":"Parses declared robots.txt directives for User-agent: * and Sitemap lines. It does not certify permission, legal compliance, or how every crawler interprets robots rules.","origin":"https://example.com","version":"1.0.0","sitemaps":["https://example.com/sitemap.xml"],"product_id":"creator_desk_crawl_config","robots_url":"https://example.com/robots.txt","inspected_at":"2026-09-25T00:00:00.000Z","requested_url":"https://example.com/some/page","wildcard_group":{"found":true,"rules":[{"path":"/private/","directive":"disallow"}],"crawl_delay":null,"rules_truncated":false},"body_fingerprint":{"sha256":"0000000000000000000000000000000000000000000000000000000000000000","bytes_read":128},"sitemaps_truncated":false}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.002","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.002/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.002","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.002","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_2j2kD4hteUXqtoOqGCgR1","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.002","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Fetches and parses a website's robots.txt file, returning declared crawl directives, disallow rules, crawl delay, and sitemap URLs for the wildcard user-agent group.","exampleAgentPrompt":"Before I start scraping, can you check what https://www.techcrunch.com/articles/ai-news has in its robots.txt — specifically which paths are disallowed and whether there's a sitemap listed?","exampleUseCases":[{"title":"Pre-crawl permission check for scraper","prompt":"I'm about to build a scraper for https://news.ycombinator.com — can you pull its robots.txt and tell me which paths are blocked for generic crawlers and if there's any crawl delay I should respect?"},{"title":"Sitemap discovery for SEO audit","prompt":"I'm doing an SEO audit of https://www.shopify.com/blog — can you look up their robots.txt and give me all the sitemap URLs they've declared?"},{"title":"Automated crawl policy monitoring","prompt":"Before my agent indexes any pages from https://www.reddit.com/r/programming, check what their robots.txt says about wildcard user-agents — are there disallowed paths or a crawl delay?"}],"resultDescription":"Returns a JSON object containing: the resolved robots.txt URL and its HTTP status/latency, parsed wildcard user-agent rules (allowed/disallowed paths, crawl delay), a list of declared sitemap URLs, a SHA-256 fingerprint of the robots.txt body, the inspection timestamp, and scope caveats noting that the result reflects declared directives only and does not certify legal crawl permission.","failureModes":["Target site is unreachable or returns non-200 for robots.txt — HTTP status and latency still reported but rules may be empty","robots.txt does not exist (404) — wildcard_group.found will be false","Site blocks the inspector's IP — connection timeout or unexpected status code returned","Rules list truncated if robots.txt is extremely large (rules_truncated: true flag set)","Invalid or non-HTTPS URL provided — request rejected before fetch","Sitemaps list truncated if many sitemaps declared (sitemaps_truncated: true flag set)"],"whenToPreferThis":"Use this endpoint when an AI agent needs to programmatically understand a site's crawl policy before fetching or indexing its pages. It is the right choice when you need parsed, structured robots.txt data (disallow paths, crawl delay, sitemaps) rather than raw file content, and when you want a fingerprint to detect future changes. Prefer this over manually fetching /robots.txt when you need reliable parsing of wildcard directives and sitemap extraction in a single call.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-13T00:40:43.815Z","isFirstParty":false,"canonicalSlug":"creator-desk-crawl-config-robots-txt-inspector-5a2e3d45"}