{"uid":"cap_877VvdA5k2lX2Pk7wz_f6","slug":"robots-txt-sitemap-analyzer-34747aad","name":"robots.txt & Sitemap Analyzer","description":"Pay-per-call data and tools for AI agents, $0.002-0.01 per call in USDC via x402 (Base, Polygon or Solana), no account or API key; MAX tiers ($0.05-0.25) run deeper jobs. Agent toolkit: web search, search + read top pages, any page as clean markdown with a prompt-injection check, which-x402-API-to-call recommender, seller check. Data: Google Maps places with contacts, YouTube video data and transcripts, job listings (Indeed, LinkedIn, 13 boards), Google Ads Transparency, TikTok comments, Pinterest, Airbnb, Redfin, AliExpress, app stores, live sports, website and SEO checks, Shopify store tech stacks, and onchain token / address risk checks.","url":"https://agent-data.centorprat.workers.dev/tools/robots-sitemap-analyzer?utm_source=zero.xyz","method":"POST","headers":{},"bodySchema":{"type":"object","properties":{"domains":{"type":"array","items":{"type":"string"},"maxItems":3,"description":"Domains to analyze, e.g. [\"example.com\"]. One report per domain."}}},"responseSchema":{"type":"json","example":{"items":[{"note":"Read-only fetch of robots.txt and the XML sitemap(s). Recommendations, not a verdict.","domain":"apple.com","signal":"ok-with-recommendations","findings":["Sitemap URLs have no <lastmod> - crawlers can't tell what changed."],"sitemaps":[{"url":"https://www.apple.com/shop/sitemap.xml","type":"sitemapindex","found":true,"valid":true,"sample":["https://www.apple.com.cn/shop/sitemap-index.xml","https://www.apple.com/ae/shop/sitemap-index.xml","https://www.apple.com/at/shop/sitemap-index.xml","https://www.apple.com/au/shop/sitemap-index.xml","https://www.apple.com/be-fr/shop/sitemap-index.xml"],"childSitemaps":47},{"url":"https://www.apple.com/autopush/sitemap/sitemap-index.xml","type":"sitemapindex","found":true,"valid":true,"sample":["https://www.apple.com/autopush/sitemap/ae-ar/sitemap-image.xml","https://www.apple.com/autopush/sitemap/ae-ar/sitemap-video.xml","https://www.apple.com/autopush/sitemap/ae-ar/sitemap.xml","https://www.apple.com/autopush/sitemap/ae/sitemap-image.xml","https://www.apple.com/autopush/sitemap/ae/sitemap-video.xml"],"childSitemaps":293},{"url":"https://www.apple.com/newsroom/sitemap.xml","type":"urlset","found":true,"valid":true,"urlCount":2601,"withLastmod":0,"newestLastmod":null}],"checkedAt":"2026-10-24T13:49:33+00:00","robotsTxt":{"found":true,"groups":6,"crawlDelay":null,"userAgents":["*","Baiduspider","HaoSouSpider","Sogou inst spider","Sogou spider2","Sogou web spider"],"sitemapsDeclared":["https://www.apple.com/shop/sitemap.xml","https://www.apple.com/autopush/sitemap/sitemap-index.xml","https://www.apple.com/newsroom/sitemap.xml","https://www.apple.com/retail/sitemap/sitemap.xml","https://www.apple.com/today/sitemap-index.xml"],"blocksAllCrawlers":false,"totalDisallowRules":9}}]}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.02","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.02/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.02","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.02","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_WLernhnntxchh9BQ2iiBa","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.02","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Parses robots.txt and XML sitemaps for given domains, reporting crawlability rules, sitemap structure, and flagged issues","exampleAgentPrompt":"Can you analyze the robots.txt and sitemaps for example.com and openai.com — tell me if they block crawlers, what sitemaps they declare, and flag any crawlability issues?","exampleUseCases":[{"title":"Pre-scrape crawlability audit","prompt":"Before I start scraping nytimes.com and theguardian.com, can you check their robots.txt files to see if they block automated bots, what crawl-delay they set, and what sitemaps they list?"},{"title":"SEO technical site audit","prompt":"I'm doing a technical SEO audit on shopify.com — can you pull the robots.txt rules and sitemap structure, including URL counts and lastmod coverage, and flag any issues?"},{"title":"Competitor sitemap reconnaissance","prompt":"Check the sitemaps and robots.txt for hubspot.com, salesforce.com, and marketo.com — I want to know how many URLs each has indexed and whether they restrict any important sections from crawlers."}],"resultDescription":"For each domain: parsed robots.txt findings (user-agent rules, allow/disallow paths, crawl-delay, declared sitemaps, blocks-all flag) and sitemap analysis (URL counts, nested sitemaps, lastmod coverage), plus a crawlability signal and a list of flagged issues. Failed domain lookups are not charged.","failureModes":["Domain does not exist or is unreachable — lookup fails, not charged","robots.txt absent or returns non-200 status — noted in findings","Sitemap URL declared but returns error or malformed XML — flagged as issue","More than 3 domains submitted — request rejected at input validation","Network timeout fetching robots.txt or sitemap — lookup marked as failed"],"whenToPreferThis":"Choose this endpoint when you need a structured, parsed breakdown of robots.txt directives and sitemap metadata for up to 3 domains in a single call, especially for SEO audits, pre-crawl checks, or competitive research. It is keyless and read-only, making it low-friction. Prefer it over manual fetching when you need normalized, issue-flagged output rather than raw file content.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-28T06:43:01.946Z","isFirstParty":false,"canonicalSlug":"robots-txt-sitemap-analyzer-34747aad"}