{"uid":"cap_DUy96zXO0atnfTmuQ4AvL","slug":"grey-ridge-web-crawl-api-x402-f005427d","name":"Grey Ridge Web Crawl API (x402)","description":"A keyless web-retrieval and data API for AI agents. Retrieve live web pages (clean markdown/text, metadata, links, JSON-LD; recovers JS-rendered/SPA content), detect when a page changes, enumerate a site’s URLs, enrich a company from its domain, verify email deliverability, and read a maintained normalized jobs feed across 10 applicant-tracking systems (Greenhouse, Lever, Ashby, SmartRecruiters, Workday, Recruitee, Workable, BambooHR, Breezy, Personio) with company-domain auto-detection. Every endpoint reads live at request time. Pay per call in USDC on Base via x402 (HTTP 402) — no account, no API key, no subscription. Sources are drift-monitored and maintained so the data does not rot.","url":"https://assetfactory-jobs-x402.sigrunner.workers.dev/web/crawl?utm_source=zero.xyz","method":"HEAD","headers":{},"bodySchema":{"type":"object","$schema":"https://json-schema.org/draft/2020-12/schema","required":["input"],"properties":{"input":{"type":"object","required":["type","method"],"properties":{"type":{"type":"string","const":"http"},"method":{"enum":["GET"],"type":"string"},"queryParams":{"type":"object","required":["url"],"properties":{"url":{"type":"string","description":"Seed page URL. The crawl starts here, follows same-domain links breadth-first and returns clean content for every page it reaches. The buyer does not need to know the page list in advance."},"limit":{"type":"integer","default":15,"maximum":50,"description":"Hard ceiling on pages crawled (and billed as one call). Bounded downside."},"exclude":{"type":"string","description":"Skip URLs whose path/query contains this substring (e.g. /author/)."},"include":{"type":"string","description":"Only crawl URLs whose path/query contains this case-insensitive substring (e.g. /docs)."},"maxChars":{"type":"integer","default":20000,"maximum":200000,"description":"Per-page ceiling on returned text / markdown length."},"maxDepth":{"type":"integer","default":2,"maximum":4,"description":"Link-follow depth from the seed (0 = seed only)."},"sameHost":{"type":"boolean","default":false,"description":"true restricts the crawl to the seed's exact host; default follows the whole registrable domain (e.g. jobs.stripe.com from stripe.com)."},"useSitemap":{"type":"boolean","default":false,"description":"Also seed the frontier from the site's sitemap so a link-sparse SPA still reaches real pages."},"includeLinks":{"type":"boolean","default":false,"description":"Include each page's outbound-link graph in the output (always used internally for discovery)."},"includeMarkdown":{"type":"boolean","default":true,"description":"Return a light Markdown rendering per page."}}}},"additionalProperties":false},"output":{"type":"object","required":["type"],"properties":{"type":{"type":"string"},"example":{"type":"object"}}}}},"responseSchema":{"type":"json","example":{"ok":true,"seed":"https://blog.cloudflare.com/","pages":[{"ok":true,"url":"https://blog.cloudflare.com/","kind":"html","text":"clean main-content prose ...","depth":0,"title":"The Cloudflare Blog","status":200,"markdown":"## ...","char_count":6248,"word_count":858,"recovered_from":null,"injection_suspected":false},{"ok":true,"url":"https://blog.cloudflare.com/dns-targeted-zone-builds/","kind":"html","text":"...","depth":1,"title":"How we halved DNS build delay","status":200,"char_count":14000,"word_count":2100}],"stats":{"ok":15,"limit":15,"scope":"same_domain","failed":0,"crawled":15,"max_depth":2,"seed_host":"blog.cloudflare.com","discovered":55,"max_depth_reached":2,"frontier_truncated":true},"generated_at":"2026-11-04T00:00:00.000Z"}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.004","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.004/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.004","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.004","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_qUNRHSW380LwoKmwB8Ect","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.004","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Crawls a seed URL breadth-first, following same-domain links up to a configurable depth, and returns clean markdown, text, metadata, and links for every page reached — including JS-rendered/SPA content.","exampleAgentPrompt":"Crawl https://docs.example.com starting from the homepage, follow links up to 2 levels deep, limit to 20 pages, only include URLs containing /docs, and give me clean markdown for every page you find.","exampleUseCases":[{"title":"Docs site bulk content extraction","prompt":"Crawl https://docs.stripe.com, follow links 2 levels deep, cap at 30 pages, only include paths with /docs, and return clean markdown for each page so I can build a knowledge base."},{"title":"Competitor blog research sweep","prompt":"Spider https://blog.openai.com breadth-first, up to depth 2, limit 25 pages, and give me the clean text and titles for every post you find — I want to analyze what topics they've been covering."},{"title":"SPA job board content retrieval","prompt":"Crawl https://jobs.acme.com — it's a JavaScript-rendered single-page app — follow links up to depth 1, limit 15 pages, use the sitemap to seed the frontier, and return markdown content for every listing page you find."}],"resultDescription":"A JSON object containing an array of page results (each with URL, title, clean text, markdown, HTTP status, word/char counts, depth, JS-recovery flag, injection suspicion flag) plus aggregate crawl stats (pages crawled, discovered, failed, depth reached, frontier truncation flag) and a generated_at timestamp.","failureModes":["Seed URL unreachable or returns non-2xx — page entry marked ok:false with status code","JS rendering timeout on heavily dynamic pages — recovered_from field may indicate fallback","Crawl hits page limit before exhausting frontier — frontier_truncated:true in stats","include/exclude filters too restrictive — fewer pages returned than limit","Site blocks crawlers via robots.txt or rate limiting — individual pages fail silently with error status","Invalid or malformed seed URL — request rejected before crawl begins"],"whenToPreferThis":"Choose this endpoint when you need to retrieve content from multiple pages of a site in a single paid call, especially when link-following, depth control, or JS/SPA rendering is required. It is superior to single-page fetch endpoints when you don't know the full URL list in advance and want breadth-first discovery. The per-call pricing (not per-page) makes it cost-effective for bulk extraction up to 50 pages. Prefer it over manual URL enumeration when the site structure is unknown or when sitemap-seeding is needed for SPAs.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-11-18T00:36:18.810Z","isFirstParty":false,"canonicalSlug":"grey-ridge-web-crawl-api-x402-f005427d"}