{"uid":"cap_E7uuVDONmxEgJwx9kvHHh","slug":"agentrxi-sitemap-crawler-ea46fb99","name":"AgentRXI Sitemap Crawler","description":"List every page on a website: reads robots.txt and all sitemaps (including sitemap indexes), with last-modified dates; falls back to on-page links. Up to 5000 URLs. Also callable as GET with the same fields as query parameters.","url":"https://agentrxi.com/v1/sitemap?utm_source=zero.xyz","method":"GET","headers":{},"bodySchema":{"type":"object","$schema":"https://json-schema.org/draft/2020-12/schema","required":["input"],"properties":{"input":{"type":"object","required":["type","method"],"properties":{"type":{"type":"string","const":"http"},"method":{"enum":["GET"],"type":"string"},"queryParams":{"type":"object","required":["url"],"properties":{"url":{"type":"string"},"limit":{"type":"integer","description":"1 to 5000, default 500"}}}},"additionalProperties":false},"output":{"type":"object","required":["type"],"properties":{"type":{"type":"string"},"example":{"type":"object"}}}}},"responseSchema":null,"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.02","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.02/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.02","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.02","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_5GcKOJMfxP0-GxN2i5D4T","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.02","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Fetches and returns up to 5000 URLs from a website by reading its robots.txt, sitemaps, and sitemap indexes, with last-modified dates, falling back to on-page link discovery.","exampleAgentPrompt":"Can you list all the pages on https://example.com — use the sitemap crawler and return up to 1000 URLs with their last-modified dates?","exampleUseCases":[{"title":"Full site URL inventory for SEO audit","prompt":"Pull every URL from https://techblog.io — I want the complete list of pages with last-modified dates, up to 3000 URLs, so I can do an SEO content audit."},{"title":"Competitor site structure analysis","prompt":"List all the pages on https://competitor-brand.com for me — I want to see their full site structure and how many pages they have, up to 5000 URLs."},{"title":"Broken link detection preparation","prompt":"Get me all the URLs on https://mystore.com so I can check them for broken links — fetch up to 2000 pages including last-modified dates."}],"resultDescription":"A list of up to 5000 URLs discovered on the target website, sourced from robots.txt directives, XML sitemaps (including sitemap indexes), and on-page links as a fallback. Each entry typically includes the URL and its last-modified date where available.","failureModes":["Target site blocks crawler user-agents — returns empty or partial URL list","No sitemap or robots.txt present and on-page link fallback returns few URLs","URL limit too low for large sites — results truncated at specified limit","Invalid or unreachable URL — returns an error","Sitemap XML is malformed — partial parse results"],"whenToPreferThis":"Use this endpoint when you need a comprehensive list of all pages on a domain without writing a custom crawler. It is ideal for SEO audits, content inventories, competitive analysis, and link-checking workflows. Prefer it over general web scraping when you want structured URL discovery with last-modified metadata rather than page content.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-15T06:39:13.678Z","isFirstParty":false,"canonicalSlug":"agentrxi-sitemap-crawler-ea46fb99"}