{"uid":"cap_Zf9DFsmsE4xonjqLE1QPf","slug":"agent-utility-suite-site-map-index-1901d59d","name":"Agent Utility Suite – Site Map Index","description":"List a website's pages from any URL on it: reads robots.txt and XML sitemaps (indexes and .xml.gz included), or the home page's links when there is no sitemap. Returns URLs with last-modified dates; filter by search words or path prefix, up to 1,000 URLs. Find pricing, docs, blog or careers pages before scraping them. Not charged when nothing is found. No account or API key. $0.002 USDC per call.","url":"https://x402-agent-api.onrender.com/api/v1/site-map?utm_source=zero.xyz","method":"POST","headers":{},"bodySchema":{"type":"object","properties":{"url":{"type":"string","format":"uri","maxLength":2048,"description":"The website, or any page on it, e.g. https://example.com. Its origin (scheme and host) is mapped."},"limit":{"type":"integer","default":200,"maximum":1000,"minimum":1,"description":"How many URLs to return (1-1,000, default 200)."},"search":{"type":"string","maxLength":200,"minLength":1,"description":"Only URLs containing every word of this text (case-insensitive) in the URL, or in the title or lastmod when known, e.g. \"pricing\" or \"blog 2026\"."},"path_prefix":{"type":"string","maxLength":500,"minLength":1,"description":"Only URLs whose path starts with this, e.g. \"/blog/\" or \"/docs\"."}}},"responseSchema":{"type":"json","example":{"site":"https://example.com","urls":[{"url":"https://example.com/blog/agent-payments","title":null,"lastmod":"2026-09-30T08:12:00+00:00"},{"url":"https://example.com/blog/x402-launch","title":null,"lastmod":"2026-09-12T14:03:00+00:00"},{"url":"https://example.com/blog/hello-world","title":null,"lastmod":"2026-08-01"}],"count":3,"source":"sitemap","scanned":412,"truncated":true,"next_calls":[{"why":"Read the first 3 of these pages in one call.","tool":"scrape_batch","price":"$0.008","arguments":{"urls":["https://example.com/blog/agent-payments","https://example.com/blog/x402-launch","https://example.com/blog/hello-world"]}},{"why":"Read the first page as Markdown.","tool":"scrape_markdown","price":"$0.002","arguments":{"url":"https://example.com/blog/agent-payments"}}],"total_found":57,"sitemaps_read":["https://example.com/sitemap_index.xml","https://example.com/post-sitemap.xml"]}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.002","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.002/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.002","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.002","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_702F04Ohf7MyWBn6rDaq9","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.002","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Returns a filtered, paginated list of URLs from a website's sitemap, optionally filtered by keyword or path prefix","exampleAgentPrompt":"Can you list all the pages under the /blog/ path on stripe.com — show me up to 50 URLs, and filter for ones that mention 'payments' somewhere in the URL or title?","exampleUseCases":[{"title":"Discover competitor pricing pages","prompt":"List all pages on acme-competitor.com that contain the word 'pricing' in the URL or title — return up to 100 results so I can review their pricing structure."},{"title":"Find docs pages for scraping","prompt":"Give me all URLs under the /docs/ path on docs.example.com, up to 200 pages, so I can decide which ones to read in detail."},{"title":"Index a company blog for research","prompt":"Show me all blog posts on hubspot.com — list up to 500 URLs from paths starting with /blog/ so I can find relevant articles about inbound marketing."}],"resultDescription":"A JSON object containing the canonical site origin, an array of URL entries (each with URL, optional title, and optional lastmod timestamp), the total count returned, the source type (sitemap or crawl), how many URLs were scanned internally, whether results were truncated, the total number found, the list of sitemaps read, and suggested next_calls with recommended tools and arguments for reading the returned pages.","failureModes":["Website has no sitemap — source may fall back to crawl with reduced coverage","URL is malformed or unreachable — returns error","limit exceeds 1000 — capped or rejected","search keyword yields zero matches — empty URLs array returned","Site blocks crawlers — partial or empty results","Payment not completed — 402 response requiring USDC payment on Base before retry"],"whenToPreferThis":"Use this endpoint when you need to discover what pages exist on a website before deciding which to read, or when you want to find pages matching a keyword or path prefix without scraping the whole site. It is the right first step before batch-reading pages with scrape_markdown or scrape_batch. Prefer it over generic search engines when you need a structured, complete index of a specific domain rather than a ranked search result list.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-10T06:41:19.505Z","isFirstParty":false,"canonicalSlug":"agent-utility-suite-site-map-index-1901d59d"}