{"uid":"cap_Nc9M5CqY9u3MZS2IiZAoZ","slug":"agent402-site-crawl-9eb353f5","name":"Agent402 Site Crawl","description":"Crawl a website from a starting URL and return each page as clean markdown: breadth-first over internal links, bounded by page count and depth, honouring robots.txt, with per-page title, status, depth and outbound links. Use it when an agent needs a whole section of a site rather than one known page.","url":"https://agent402.tools/api/site-crawl","method":"POST","headers":{},"bodySchema":{"type":"object","properties":{"url":{"type":"string","description":"Start URL"},"limit":{"type":"integer","description":"Max pages to fetch, 1-20 (default 10); failed fetches count toward it"},"format":{"enum":["markdown","text"],"type":"string","description":"Page content format (default markdown)"},"maxDepth":{"type":"integer","description":"Link depth from the start URL, 0-2 (default 1)"},"sameHost":{"type":"boolean","description":"true (default): stay on the start host (www and bare host count as one); false: also follow subdomains of the start site"},"excludePatterns":{"type":"array","items":{"type":"string"},"description":"Never follow links whose URL contains any of these substrings (max 20)"},"includePatterns":{"type":"array","items":{"type":"string"},"description":"Only follow links whose URL contains at least one of these substrings (max 20)"},"maxCharsPerPage":{"type":"integer","description":"Cap on content characters per page, 200-20000 (default 8000)"}}},"responseSchema":{"type":"json","example":{"url":"https://example.com/","pages":[{"url":"https://example.com/","depth":0,"links":[],"title":"Example Domain","status":200,"content":"# Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\n[Learn more](https://iana.org/domains/example)","contentChars":166}],"format":"markdown","queued":0,"source":"live fetch over internal links (breadth-first), robots.txt honoured for Agent402Bot","crawled":1,"fetches":2,"skipped":{"depth":0,"error":0,"limit":0,"binary":0,"robots":0,"unsafe":0,"offsite":1,"pattern":0},"elapsedMs":420,"fetchedAt":"2026-08-22T00:00:00.000Z","robotsTxt":"not readable","truncated":false,"untrustedContent":true}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.02","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.02/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.02","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.02","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_WauIOW8T6vfYY2lr_u9Cv","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.02","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Crawls a website breadth-first up to a configurable depth and page limit, returning page content, links, titles, and crawl metadata as structured JSON","exampleAgentPrompt":"Crawl https://docs.example.com starting from the homepage, go 2 levels deep, fetch up to 15 pages, and return everything in markdown format — stay on the same host and skip any URLs containing /changelog","exampleUseCases":[{"title":"Competitor site content audit","prompt":"Crawl https://competitor.io starting from their homepage, fetch up to 20 pages, go 1 level deep, and return the content as markdown so I can see what they're covering on their main site"},{"title":"Documentation ingestion for RAG","prompt":"Scrape https://docs.mylib.dev up to 2 levels deep, limit to 15 pages, return as markdown, and only follow links that contain /guide or /reference — I want to build a knowledge base from it"},{"title":"Link and structure mapping","prompt":"Crawl https://myblog.com starting at the root, fetch up to 10 pages at depth 1, stay on the same host, and give me the titles, links, and HTTP status for each page so I can audit the site structure"}],"resultDescription":"A JSON object containing an array of crawled pages (each with URL, depth, HTTP status, title, content in markdown or text, character count, and outbound links), plus summary stats: total crawled, fetched, queued, skipped counts broken down by reason (depth, error, limit, binary, robots, unsafe, offsite, pattern), elapsed time in ms, fetch timestamp, robots.txt status, format used, and a truncated flag.","failureModes":["Invalid or unreachable start URL returns an error or zero crawled pages","Limit reached before all pages are fetched — skipped.limit counter increments","robots.txt disallows Agent402Bot on target pages — those pages are skipped and counted under skipped.robots","Requested depth of 0-2 only — values outside range may be rejected or clamped","Binary resources (PDFs, images) are skipped and counted under skipped.binary","Off-site links skipped when sameHost is true (default) — counted under skipped.offsite","Pattern filters excluding all found links result in minimal crawl output"],"whenToPreferThis":"Choose this endpoint when you need to programmatically crawl multiple pages of a website breadth-first without setting up your own scraping infrastructure. It is especially useful when you want robots.txt compliance baked in, need content in clean markdown or plain text, and want structured metadata (titles, links, status codes) alongside the content. Prefer it over single-page fetch tools when you need 2–20 pages from a site in one call.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-09-14T06:49:23.075Z","isFirstParty":false}