{"uid":"cap_k-LSKVHzZnz8rbv4PhiK9","slug":"agentx402-site-crawler-c73379e4","name":"AgentX402 Site Crawler","description":"Agent-native site crawler over x402 — send a URL and a page budget, get back clean markdown for every page plus a manifest: no signup, no API key, no SDK. Settles the WHOLE max_pages budget UPFRONT, quoted as max_pages × price (a crawl outruns the ~10-min payment authorization, so it cannot pay per page); unused pages are REFUNDED AS CREDITS on completion. Crawl $0.004/page; prepay $1 = 10,000 credits at 20% off. Docs: https://agentx402.ai","url":"https://api.agentx402.ai/v1/scout/crawl","method":"POST","headers":{},"bodySchema":{"type":"object","$schema":"https://json-schema.org/draft/2020-12/schema","required":["input"],"properties":{"input":{"type":"object","required":["type","method","bodyType","body"],"properties":{"body":{"properties":{}},"type":{"type":"string","const":"http"},"method":{"enum":["POST","PUT","PATCH"],"type":"string"},"bodyType":{"enum":["json","form-data","text"],"type":"string"},"queryParams":{"type":"object","properties":{"max_pages":{"type":"number"}}}},"additionalProperties":false},"output":{"type":"object","required":["type"],"properties":{"type":{"type":"string"},"example":{"type":"object"}}}}},"responseSchema":null,"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.004","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.004/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.004","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.004","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_SUNd4O8yWjpl1A6S4is-P","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.004","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Crawls a website up to a specified page budget and returns clean markdown content for every page plus a manifest, settling payment upfront via x402 with credits refunded for unused pages.","exampleAgentPrompt":"Crawl the AgentX402 docs site starting at https://agentx402.ai and fetch up to 50 pages, giving me each page's content as clean markdown along with a manifest of what was found.","exampleUseCases":[{"title":"Competitor documentation indexing","prompt":"Crawl https://docs.competitor.com up to 100 pages and give me every page as markdown so I can analyze their product documentation."},{"title":"Training data collection from a blog","prompt":"Scrape the entire blog at https://blog.example.com — up to 200 pages — and return all the articles as clean markdown so I can build a fine-tuning dataset."},{"title":"Offline site archiving for research","prompt":"Crawl https://research-institute.org/publications up to 75 pages and return each page's content as readable markdown with a full manifest of URLs discovered."}],"resultDescription":"A structured response containing clean markdown text for each crawled page plus a manifest listing all discovered URLs. Payment is settled upfront for the full max_pages budget; any pages not crawled result in credits refunded to the caller's account upon completion.","failureModes":["Target URL unreachable or returns non-200 responses — crawl may return partial results or fail","Page budget exhausted before full site is covered — only pages within budget are returned","Payment authorization insufficient for the requested max_pages budget — crawl rejected upfront","Dynamic JavaScript-heavy pages may not render correctly, resulting in incomplete markdown","Rate limiting or bot detection by the target site may interrupt the crawl mid-way"],"whenToPreferThis":"Choose this endpoint when you need to crawl an entire site or large section of a site (multiple pages) and want results as clean markdown without managing authentication, SDKs, or API keys. It is ideal for agentic workflows that require bulk web content — documentation sites, blogs, research archives — paid seamlessly via x402 with no pre-registration. Prefer it over single-URL scrapers when you need a full multi-page crawl with a manifest of discovered pages.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-09-14T00:44:39.968Z","isFirstParty":false}