{"uid":"cap_kUJGpK9OqZvXxXuJb4KGW","slug":"agentsvc-io-site-crawl-a7472a33","name":"agentsvc.io Site Crawl","description":"86 APIs for AI agents: web scraping and crawling to Markdown, AI summaries, data extraction and cited answers, screenshots and PDF, PDF, OCR and document conversion, news, paper and developer search and more, paid per call with USDC on Base, Solana, Polygon or Arbitrum via x402. No account, no API key, free sample on every service.","url":"https://agentsvc.io/api/v1/proxy/site-crawl?utm_source=zero.xyz","method":"POST","headers":{},"bodySchema":{"type":"object","properties":{"url":{"type":"string"},"max_chars":{"type":"integer","default":4000,"maximum":20000,"minimum":500},"max_pages":{"type":"integer","default":5,"maximum":20,"minimum":1},"path_prefix":{"type":"string","description":"Only follow links whose path starts with this, e.g. /docs"}}},"responseSchema":{"type":"json","example":{"data":{"site":"https://agentsvc.io","pages":[{"ok":true,"url":"https://agentsvc.io/docs","title":"Docs: how agents pay per call with x402 · agentsvc.io","status":200,"markdown":"Documentation\n\n# Pay per call. Nothing else to set up.\n\nEvery service is a JS...","word_count":104,"description":"Quickstart, payment protocol (x402 v1 and v2), free trial, error codes and th..."}],"took_ms":125,"ok_count":2,"crawled_at":"2026-10-06T14:17:50.682Z","pages_crawled":2,"discovered_urls":37,"skipped_non_html":2},"success":true}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.005","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.005/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.005","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.005","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_PJLGiJxLPcN7jwJAZ3erY","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.005","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Crawls a website starting from a given URL, following internal links and returning page content as markdown with metadata for up to 20 pages.","exampleAgentPrompt":"Can you crawl https://docs.example.com for me, following up to 10 pages but only under the /api path, and give me the full text content of each page?","exampleUseCases":[{"title":"Index competitor documentation site","prompt":"Crawl https://stripe.com/docs starting from the root, follow up to 15 pages only under the /docs path, and give me the markdown content of each page so I can compare their API docs to ours."},{"title":"Gather all pages of a small business site","prompt":"Can you crawl https://www.morningbloomcoffee.com and collect the text from every page you find — up to 20 pages — so I can summarize what the business offers?"},{"title":"Extract content from a blog section","prompt":"Scrape https://blog.example.com, only follow links under /blog, grab up to 8 pages, and return the full text so I can find all their recent articles about AI."}],"resultDescription":"Returns a JSON object containing a list of crawled pages, each with its URL, HTTP status, title, meta description, markdown content, and word count. Also includes overall stats: number of pages crawled, discovered URLs, skipped non-HTML resources, total time taken, and the crawl timestamp.","failureModes":["Target site blocks crawlers or returns 403/429 — affected pages marked ok:false with status code","URL is unreachable or DNS fails — page entry with ok:false returned","max_pages or max_chars limits truncate content before full site is covered","path_prefix is too restrictive and no matching links are found — returns fewer pages than expected","Site uses JavaScript rendering and content is not in raw HTML — markdown may be incomplete or empty","Payment fails or USDC balance insufficient — HTTP 402 returned before crawl starts"],"whenToPreferThis":"Use this endpoint when you need to extract content from multiple pages of a site in a single call, especially when you want to follow internal links automatically under a specific path prefix. Prefer it over a single-page scraper when the target content spans multiple pages (e.g. documentation, blogs, product catalogs). It is ideal for indexing, summarization, or competitive research tasks where breadth of coverage matters and you can specify a path scope to stay focused.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-10T12:39:55.736Z","isFirstParty":false,"canonicalSlug":"agentsvc-io-site-crawl-a7472a33"}