{"uid":"cap_LAzzrU-0cHtylJHpCbwQR","slug":"agent402-tools-site-map-discovery-5f6ceba0","name":"Agent402.Tools Site Map Discovery","description":"Discover a website's URLs in one call: reads robots.txt, its declared sitemap(s) (sitemap indexes and gzipped sitemaps included, /sitemap.xml as the fallback) and the start page's internal links, then returns a same-host, normalized, deduplicated list (up to 500) with an optional substring filter. Hard budgets: at most 6 fetches, 15 seconds, 5 MB. Use it to pick which pages to crawl or extract next.","url":"https://agent402.tools/api/site-map","method":"POST","headers":{},"bodySchema":{"type":"object","properties":{"url":{"type":"string","description":"Start URL (the site's homepage or any page on it)"},"limit":{"type":"integer","description":"Max URLs to return, 1-500 (default 100)"},"search":{"type":"string","description":"Optional case-insensitive substring filter applied to the discovered URLs"},"includeSubdomains":{"type":"boolean","description":"Also keep URLs on subdomains of the start site (default false; www and bare host always count as one site)"}}},"responseSchema":{"type":"json","example":{"url":"https://www.iana.org/","host":"www.iana.org","urls":["https://www.iana.org/","https://www.iana.org/domains","https://www.iana.org/numbers","https://www.iana.org/protocols"],"total":120,"search":null,"source":"robots.txt, sitemap(s) and start-page links, fetched live","fetches":3,"sources":{"links":24,"sitemap":96},"warnings":[],"fetchedAt":"2026-08-22T00:00:00.000Z","truncated":true,"sitemapsRead":1}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.005","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.005/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.005","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.005","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_Py9Ms1axyR8ofWydjD0GN","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.005","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Crawls a website and returns a structured list of discovered URLs from its sitemap, robots.txt, and page links","exampleAgentPrompt":"Can you crawl https://www.example.com and give me a list of up to 200 of its pages, but only ones that include the word 'blog' in the URL?","exampleUseCases":[{"title":"Audit all pages on a company site","prompt":"Go to https://www.acme.com and discover all the pages on it — I want a full list of up to 500 URLs so I can audit their site structure."},{"title":"Find product pages for scraping","prompt":"Crawl https://shop.example.com and return up to 100 URLs that contain '/products/' in the path — I need to identify all the product pages."},{"title":"Index a competitor's blog","prompt":"Fetch the sitemap of https://competitorblog.com and list up to 300 URLs, including any subdomains, so I can see everything they've published."}],"resultDescription":"A JSON object containing the start URL, host, an array of discovered URLs (from robots.txt, sitemaps, and page links), total URL count, optional search filter applied, data sources used, number of HTTP fetches made, per-source URL counts, any warnings, fetch timestamp, and whether the result was truncated due to the limit.","failureModes":["Site has no sitemap or robots.txt, resulting in fewer discovered URLs or empty list","Start URL is unreachable or returns non-200 status, causing fetch failure","Limit parameter outside 1-500 range may be rejected or clamped","Search filter returns zero matches if substring doesn't appear in any URL","Subdomain URLs missed if includeSubdomains is false (default)","Large sites may be truncated when total URLs exceed the requested limit"],"whenToPreferThis":"Use this endpoint when you need to discover the full set of URLs for a given website quickly without running a full recursive spider — it synthesizes robots.txt, sitemap XML, and page links in a single call. Prefer it over manual sitemap parsing when you want a normalized, deduplicated list with optional substring filtering. It's ideal for site auditing, content indexing, competitor research, and feeding downstream scraping pipelines.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-09-14T00:52:09.587Z","isFirstParty":false}