{"uid":"cap_gRY2slaMJSQMrdlaElva_","slug":"agentutility-scrape-website-498fbfd7","name":"AgentUtility Scrape Website","description":"Scrapes any webpage and pulls title, description, canonical URL, OpenGraph + Twitter card metadata, headings, and outbound links from a single URL. Server-side rendering; body content rendered as text, raw HTML, or clean markdown, with optional link extraction. Cheerio-based with no headless browser, so it is fast and cheap, ideal for static pages and SSR sites. For JS-heavy SPAs that need a real browser, see website-screenshot. Use it as a web scraping API or webpage metadata extractor.","url":"https://x402.agentutility.ai/scrape-website","method":"POST","headers":{},"bodySchema":{"type":"object","$schema":"https://json-schema.org/draft/2020-12/schema","required":["input"],"properties":{"input":{"type":"object","required":["type","method","bodyType","body"],"properties":{"body":{"required":["url"],"properties":{"url":{"type":"string","description":"Public URL to fetch and parse. Must include scheme (http/https). Follows redirects."},"format":{"enum":["text","html","markdown"],"type":"string","description":"Body output format. 'text' (default, plain text), 'html' (raw HTML body), or 'markdown' (clean markdown — best for LLM ingestion)."},"user_agent":{"type":"string","description":"Custom User-Agent header. Defaults to a modern desktop Chrome UA."},"include_links":{"type":"boolean","description":"If true, also returns an array of all <a href> links found on the page. Default false."}}},"type":{"type":"string","const":"http"},"method":{"enum":["POST"],"type":"string"},"bodyType":{"enum":["json","form-data","text"],"type":"string"}},"additionalProperties":false},"output":{"type":"object","required":["type"],"properties":{"type":{"type":"string"},"example":{"type":"object","properties":{"h1":{"type":"string"},"og":{"type":"object","properties":{}},"url":{"type":"string"},"lang":{"type":"string"},"text":{"type":"string"},"title":{"type":"string"},"format":{"type":"string"},"twitter":{"type":"object","properties":{}},"canonical":{"type":"null"},"final_url":{"type":"string"},"body_chars":{"type":"integer"},"description":{"type":"string"},"status_code":{"type":"integer"}}}}}}},"responseSchema":{"type":"json","example":{"h1":"Example Domain","og":{},"url":"https://example.com","lang":"en","text":"Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\nLearn more","title":"Example Domain","format":"text","twitter":{},"canonical":null,"final_url":"https://example.com/","body_chars":128,"description":"","status_code":200}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.04","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"registry","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.04/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.04","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.04","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_oQQqeNBcdHy4dlqbKcO0n","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.04","costPer":"request","priority":0,"asset":null,"unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Scrapes a single webpage and extracts title, description, canonical URL, OpenGraph/Twitter metadata, headings, body content, and outbound links using Cheerio (no headless browser)","exampleAgentPrompt":"Can you scrape https://example.com/blog/post-1 and give me the page title, meta description, OpenGraph tags, and all the outbound links — return the body content as clean markdown?","exampleUseCases":[{"title":"SEO metadata audit for a page","prompt":"Scrape https://acme.com/products/widget and pull back the title, meta description, canonical URL, and any OpenGraph or Twitter card tags so I can audit their SEO metadata."},{"title":"Extract article body for summarization","prompt":"Fetch the content of https://techcrunch.com/2024/05/01/ai-agents-rise and give me the body as clean markdown so I can summarize it."},{"title":"Harvest outbound links from a resource page","prompt":"Scrape https://awesome-llm-resources.github.io and extract all the outbound links so I can build a list of referenced tools and sites."}],"resultDescription":"Returns structured data including page title, meta description, canonical URL, OpenGraph and Twitter card metadata fields, H1-H6 headings, body content in text/HTML/or markdown format, and an array of outbound links found on the page.","failureModes":["URL is unreachable or returns non-200 status — endpoint returns an error with HTTP status","Page is a JavaScript-heavy SPA that requires a real browser — content may be empty or incomplete since Cheerio does not execute JS","Invalid or malformed URL input — returns validation error","Rate limit or network timeout on the target server — returns timeout error","Paywalled or bot-blocked pages — may return partial or no content"],"whenToPreferThis":"Choose this endpoint when you need fast, cheap webpage scraping for static pages or server-side-rendered sites and you want structured metadata (title, OG tags, headings, links) alongside body content in multiple formats. It is significantly faster and cheaper than headless browser alternatives. Avoid it for JavaScript-heavy SPAs that require browser execution — use a screenshot or browser-rendering service instead.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-09-15T00:50:48.829Z","isFirstParty":false}