{"uid":"cap_duJ33p2PdF4WjMlGI8-wa","slug":"agent-hub-website-sitemap-crawler-fe09228c","name":"Agent Hub Website Sitemap Crawler","description":"List every page on a website: reads robots.txt and all sitemaps (including sitemap indexes), with last-modified dates; falls back to on-page links. Up to 5000 URLs. Also callable as GET with the same fields as query parameters.","url":"https://agents.torbrooknews.com/v1/sitemap?utm_source=zero.xyz","method":"GET","headers":{},"bodySchema":{"type":"object","$schema":"https://json-schema.org/draft/2020-12/schema","required":["input"],"properties":{"input":{"type":"object","required":["type","method"],"properties":{"type":{"type":"string","const":"http"},"method":{"enum":["GET"],"type":"string"},"queryParams":{"type":"object","required":["url"],"properties":{"url":{"type":"string"},"limit":{"type":"integer","description":"1 to 5000, default 500"}}}},"additionalProperties":false},"output":{"type":"object","required":["type"],"properties":{"type":{"type":"string"},"example":{"type":"object"}}}}},"responseSchema":null,"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.02","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.02/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.02","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.02","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_IPpN-H7qhb2a7qXeMDlp_","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.02","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Crawls a website's robots.txt and all sitemaps (including sitemap indexes) to list up to 5000 URLs with last-modified dates, falling back to on-page link discovery.","exampleAgentPrompt":"Can you list all the pages on example.com — pull the sitemap and robots.txt and give me up to 2000 URLs with their last-modified dates?","exampleUseCases":[{"title":"SEO audit of competitor website","prompt":"List all pages on competitor.com — I want to see their full site structure with last-modified dates, up to 5000 URLs, so I can audit their content coverage."},{"title":"Content migration URL inventory","prompt":"I'm migrating myoldsite.com to a new platform and need a complete list of every page on the site. Can you crawl the sitemap and return all URLs with their last-modified dates, up to 5000?"},{"title":"Monitoring site for new content","prompt":"Pull all the current URLs from techblog.io — up to 1000 pages — so I can compare them next week and see which new articles were published."}],"resultDescription":"A list of up to 5000 URLs found on the target website, sourced from robots.txt, sitemap.xml, sitemap index files, and/or on-page links as fallback. Each entry typically includes the page URL and its last-modified date where available.","failureModes":["Target website blocks crawlers or returns 403/401, resulting in no URLs returned","Website has no sitemap and on-page link fallback finds very few pages","URL limit exceeded — response capped at the requested limit (default 500, max 5000)","Invalid or unreachable URL provided returns an error","Very large sitemap indexes may time out before full enumeration"],"whenToPreferThis":"Choose this endpoint when you need a structured, comprehensive list of all pages on a website without writing a custom crawler. It is preferable to manual scraping because it intelligently reads robots.txt and nested sitemap indexes first, falling back to link discovery only when needed. Best for SEO audits, content inventories, site migrations, or seeding a web crawl pipeline.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-13T12:45:00.704Z","isFirstParty":false,"canonicalSlug":"agent-hub-website-sitemap-crawler-fe09228c"}