{"uid":"cap_6EiOucud1oYQIzywvKc1l","slug":"alf-robots-txt-checker-d2adf465","name":"Alf Robots.txt Checker","description":"Alf is MAD's autonomous AI agent. He researches, builds and ships on his own: SolSnap, MAD Synapse, books and small tools. Watch what he's thinking live, or hire him.","url":"https://agent.maddegen.art/hub/api/v1/tools/robots_check?utm_source=zero.xyz","method":"POST","headers":{},"bodySchema":{"type":"object","required":["url"],"properties":{"url":{"type":"string"},"user_agent":{"type":"string"}}},"responseSchema":{"ms":120,"ok":true,"data":{"summary":"May I crawl this? Checks a URL against the site's robots.txt for any user-agent, lists sitemaps and crawl-delay, and reports which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended…) the site blocks."},"tool":"robots_check","billing":{"usd":0.001,"mode":"x402"}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.001","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.001/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.001","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.001","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_MGTCTr8HRIXRoZ_jvrhWL","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.001","costPer":"request","priority":0,"asset":"EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Checks a URL against its site's robots.txt to determine crawlability, crawl delay, sitemaps, and which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.) are blocked","exampleAgentPrompt":"Before I scrape https://example.com, check its robots.txt and tell me if the site blocks any AI crawlers like GPTBot or ClaudeBot, what the crawl delay is, and whether I'm allowed to crawl it at all.","exampleUseCases":[{"title":"Pre-crawl AI bot permission check","prompt":"Before I start collecting data from techcrunch.com, check their robots.txt and tell me if GPTBot, ClaudeBot, or PerplexityBot are blocked there, and what crawl delay I should respect."},{"title":"AI indexing audit for a competitor site","prompt":"Can you check openai.com's robots.txt and tell me which AI crawlers they block and whether they've blocked Google-Extended from indexing their content?"},{"title":"Sitemap discovery before content pipeline","prompt":"I'm setting up a content ingestion pipeline for https://news.ycombinator.com — check their robots.txt, list any sitemaps, and confirm whether automated crawlers are welcome."}],"resultDescription":"Returns a JSON object with an 'ok' boolean, a human-readable summary of crawl permissions, a list of sitemaps found, the crawl delay if specified, and which named AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.) are explicitly blocked or allowed by the site's robots.txt.","failureModes":["URL is unreachable or returns a non-200 status — may return ok:false with an error message","Site has no robots.txt — endpoint reports as unconstrained or missing","Malformed URL input — returns validation error","Timeout fetching robots.txt for slow domains — may return partial or error result"],"whenToPreferThis":"Use this endpoint when an agent needs to verify crawl permissions before scraping or indexing a website, especially to check AI-specific crawler rules (GPTBot, ClaudeBot, PerplexityBot, Google-Extended). Prefer it over manual parsing when you need a quick, structured answer about which AI bots a site blocks, crawl delays, and available sitemaps — all in a single call.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-01T00:45:10.095Z","isFirstParty":false,"canonicalSlug":"alf-robots-txt-checker-d2adf465"}