{"uid":"cap_MBsavFQj6pY80TJyzTGWT","slug":"vextorium-robots-txt-parser-f281873e","name":"Vextorium Robots.txt Parser","description":"Parse a site's robots.txt: whether a user-agent may crawl a path, which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot...) are blocked, and declared sitemaps. Check before your agent scrapes.","url":"https://api.vextorium.com/robots-txt-pago?utm_source=zero.xyz","method":"POST","headers":{},"bodySchema":{"type":"object","properties":{"url":{"type":"string"},"ruta":{"type":"string","description":"Path to test (default /)"},"agente":{"type":"string","description":"User-agent to test (optional)"}}},"responseSchema":{"type":"json","example":{"sitio":"https://www.nytimes.com","grupos":63,"bots_ia":{"CCBot":"bloqueado","GPTBot":"bloqueado","YouBot":"bloqueado","Diffbot":"bloqueado","Amazonbot":"permitido","ClaudeBot":"bloqueado","cohere-ai":"bloqueado","Bytespider":"bloqueado","Claude-User":"bloqueado","ChatGPT-User":"bloqueado","anthropic-ai":"bloqueado","DuckAssistBot":"bloqueado","OAI-SearchBot":"bloqueado","PerplexityBot":"bloqueado","MistralAI-User":"permitido","Google-Extended":"bloqueado","Perplexity-User":"bloqueado","Claude-SearchBot":"bloqueado","Applebot-Extended":"bloqueado","meta-externalagent":"bloqueado"},"consulta":{"ruta":"/section/world","agente":"GPTBot","permitido":false,"grupo_aplicado":["gptbot"]},"sitemaps":["https://www.nytimes.com/sitemaps/new/news.xml.gz","https://www.nytimes.com/sitemaps/new/sitemap.xml.gz","https://www.nytimes.com/sitemaps/new/collections.xml.gz","https://www.nytimes.com/sitemaps/new/video.xml.gz","https://www.nytimes.com/sitemaps/new/cooking.xml.gz","https://www.nytimes.com/sitemaps/new/recipe-collects.xml.gz","https://www.nytimes.com/sitemaps/new/regions.xml","https://www.nytimes.com/sitemaps/new/best-sellers.xml","https://www.nytimes.com/sitemaps/new/subscription-landing-pages.xml","https://www.nytimes.com/sitemaps/new/weather.xml.gz","https://www.nytimes.com/sitemaps/new/espanol.xml.gz","https://www.nytimes.com/sitemaps/new/espanol-collects.xml.gz","https://www.nytimes.com/wirecutter/sitemapindex.xml","https://www.nytimes.com/athletic/sitemap-videos-index.xml","https://www.nytimes.com/athletic/sitemap-authors.xml","https://www.nytimes.com/athletic/sitemap-verticals.xml","https://www.nytimes.com/athletic/sitemap-teams.xml","https://www.nytimes.com/athletic/sitemap-cities.xml","https://www.nytimes.com/athletic/sitemap-players.xml","https://www.nytimes.com/athletic/sitemap-tags.xml","https://www.nytimes.com/athletic/sitemap-stats.xml","https://www.nytimes.com/athletic/sitemap-schedule.xml","https://www.nytimes.com/athletic/sitemap-roster.xml","https://www.nytimes.com/athletic/sitemap.xml","https://www.nytimes.com/games-assets/v2/assets/sitemap/games.xml"],"robots_txt":true,"bots_ia_bloqueados":18}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.003","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.003/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.003","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.003","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_0nhQbjVuDjELbMQhCoFoX","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.003","costPer":"request","priority":0,"asset":"EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Fetches and parses a website's robots.txt, reporting which AI crawlers are blocked or allowed, whether a specific path is crawlable by a given user-agent, and declared sitemaps.","exampleAgentPrompt":"Before scraping nytimes.com, check its robots.txt and tell me whether GPTBot is allowed, whether the path /section/world is accessible to it, and list any declared sitemaps.","exampleUseCases":[{"title":"Pre-scrape compliance check for agent","prompt":"I'm about to scrape techcrunch.com — can you check its robots.txt and tell me whether ClaudeBot is allowed and whether the path /startups is accessible?"},{"title":"Audit which AI bots a site blocks","prompt":"I want to know which AI crawlers are blocked on openai.com — check GPTBot, ClaudeBot, PerplexityBot, Google-Extended and any others listed in their robots.txt."},{"title":"Find sitemaps for a content pipeline","prompt":"Pull the robots.txt for bbc.com and give me all the sitemap URLs it declares so I can feed them into my content indexer."}],"resultDescription":"A JSON object containing: the site URL, number of rule groups, a per-AI-bot status map (e.g. GPTBot: blocked/allowed) covering ~20 named bots, a query result showing whether a specific path is permitted for a given user-agent and which rule group applied, a list of sitemap URLs, and the raw robots.txt text.","failureModes":["Site has no robots.txt — empty or null result returned","Domain does not resolve or is unreachable — fetch error","Malformed robots.txt — parsing may produce incomplete group data","Invalid URL input — API returns validation error","Rate limiting or network timeout on the target site"],"whenToPreferThis":"Use this endpoint when an AI agent or scraper needs to check crawl permissions before accessing a site, audit how a site treats AI bots specifically, or discover sitemaps for content indexing. It is purpose-built for AI-bot-awareness (covering ~20 named AI crawlers) rather than being a generic SEO robots.txt tool.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-16T18:46:40.550Z","isFirstParty":false,"canonicalSlug":"vextorium-robots-txt-parser-f281873e"}