{"uid":"cap_FAu6-Ik8d8o4aiSE5rE0C","slug":"agentdata-web-crawl-aa0247f3","name":"AgentData Web Crawl","description":"Crawl one site in one paid call: start URL + max_pages (2-20); we follow same-host links breadth-first, returning each page as clean markdown (same extraction as /web/contents). Priced at $0.001 per requested page, quoted as max_pages up front — finding fewer pages is still a complete delivery. Failed pages are skipped and listed in skipped[]; the call still settles. Only an unreadable start page is a 400, no charge. USDC on Base, no account, no API key.","url":"https://agentdata-api.sander-van-aard.workers.dev/web/crawl","method":"POST","headers":{},"bodySchema":{"type":"object","$schema":"https://json-schema.org/draft/2020-12/schema","required":["input"],"properties":{"input":{"type":"object","required":["type","method","bodyType","body"],"properties":{"body":{"type":"object"},"type":{"type":"string","const":"http"},"method":{"enum":["POST","PUT","PATCH"],"type":"string"},"headers":{"type":"object","additionalProperties":{"type":"string"}},"bodyType":{"enum":["json","form-data","text"],"type":"string"},"queryParams":{"type":"object","additionalProperties":{"type":"string"}}},"additionalProperties":false},"output":{"type":"object","required":["type"],"properties":{"type":{"type":"string"},"example":{"type":"object"}}}}},"responseSchema":null,"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.01","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.01/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.01","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.01","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_NaW_T3j9eLTr-eQqrGbUK","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.01","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Breadth-first crawl of a website starting from a given URL, returning up to 20 pages as clean markdown, paid per page via USDC on Base with no account required.","exampleAgentPrompt":"Crawl the blog at https://example.com/blog and grab up to 10 pages, returning each one as clean markdown so I can index them.","exampleUseCases":[{"title":"RAG knowledge base ingestion","prompt":"Crawl https://docs.myproduct.com starting from the homepage and fetch up to 15 pages as markdown — I want to load all that documentation into my vector database."},{"title":"Competitor content audit","prompt":"Can you crawl https://competitor.com/features and follow links across up to 8 pages on that site, giving me the text of each page as clean markdown?"},{"title":"Small site archiving","prompt":"I need to archive the content of https://old-site.org — please crawl it breadth-first, up to 20 pages, and return everything as markdown so I have a readable snapshot."}],"resultDescription":"A structured response containing each successfully crawled page as clean markdown text (same extraction quality as the /web/contents endpoint), plus a list of any pages that were skipped due to fetch or parse errors. The caller pays for up to max_pages; fewer found pages is still a complete successful delivery. Only an unreadable start URL triggers a 400 error with no charge.","failureModes":["Start URL is unreachable or returns unreadable content — results in 400 error, no charge","Individual pages within the crawl fail to load — silently skipped and listed in the skipped[] array, call still settles","max_pages outside the accepted range of 2-20 — likely validation error","Domain does not resolve or TLS error on start URL — 400 error","Non-same-host links are not followed — content from external domains will not appear in results"],"whenToPreferThis":"Choose this endpoint when you need to extract content from multiple pages of a single website in one paid call, rather than making individual page requests. It is ideal for documentation indexing, RAG pipeline ingestion, site archiving, and competitive content gathering where you know a starting URL and want breadth-first link traversal up to 20 pages. It requires no account or API key and settles payment in USDC on Base via x402, making it suitable for autonomous agent workflows without pre-registration.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-09-15T12:48:57.536Z","isFirstParty":false}