{"uid":"cap_rkm7O2BjBqapMnB4diaor","slug":"402utils-site-profile-d8c605d5","name":"402utils Site Profile","description":"Full profile of a web page in one paid call: <head> metadata (title, description, canonical, OpenGraph, favicon, feeds), DNS records (A/AAAA/MX/NS), robots.txt crawl permission for your user-agent, and domain registration/expiry (RDAP) — fanned out in parallel, each section with its own status. Pass ?userAgent= to test robots for a specific crawler. Page fetches are SSRF-checked. If every section fails technically you get a 503 and no charge.","url":"https://402utils.com/v1/site-profile?utm_source=zero.xyz","method":"GET","headers":{},"bodySchema":{"type":"object","$schema":"https://json-schema.org/draft/2020-12/schema","required":["input"],"properties":{"input":{"type":"object","required":["type","method"],"properties":{"type":{"type":"string","const":"http"},"method":{"enum":["GET"],"type":"string"},"queryParams":{"type":"object","required":["url"],"properties":{"url":{"type":"string","format":"uri","description":"Public http(s) page URL to profile (required)."},"userAgent":{"type":"string","default":"*","description":"Crawler product token for the robots check, e.g. \"GPTBot\"."}},"description":"Query string only."}},"additionalProperties":false},"output":{"type":"object","required":["type"],"properties":{"type":{"type":"string"},"example":{"type":"object","required":["url","meta","dns","robots","domain","llmsTxt","summary"],"properties":{"dns":{"type":"object","description":"dns-lookup result with `status` (ok | unavailable | skipped)."},"url":{"type":"string"},"meta":{"type":"object","description":"meta-extract result with a `status` (ok | unavailable)."},"domain":{"type":"object","description":"domain-check result with `status` (ok | unavailable | skipped)."},"robots":{"type":"object","description":"robots-check result with `status`."},"llmsTxt":{"type":"object","description":"llms-txt result with `status` and `found` (the agent-readiness signal)."},"summary":{"type":"object","required":["ok","checkedAt"],"properties":{"ok":{"type":"integer","description":"How many of the five sections produced a result."},"checkedAt":{"type":"string","format":"date-time"}}}}}}}}},"responseSchema":{"type":"json","example":{"dns":{"domain":"blog.cloudflare.com","status":"ok","records":[{"ttl":300,"type":"A","value":"104.16.0.1"}]},"url":"https://blog.cloudflare.com/","meta":{"title":"The Cloudflare Blog","status":"ok","favicon":"https://blog.cloudflare.com/favicon.ico"},"domain":{"status":"ok","registrar":"Cloudflare, Inc.","supported":true,"registered":true},"robots":{"status":"ok","allowed":true,"sitemaps":[],"robotsUrl":"https://blog.cloudflare.com/robots.txt","robotsStatus":200},"summary":{"ok":4,"checkedAt":"2026-07-05T12:00:00.000Z"}}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.007","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":{"p10Cents":"0.7000","medianCents":"0.7000","p90Cents":"0.7000","minCents":"0.7000","maxCents":"0.7000","p95Cents":"0.7000","sampleCount":1,"varies":false,"failureChargeRate":1},"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.007/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.007","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.007","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_X8fikwMqEpitW_PDS76U6","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.007","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Returns a comprehensive profile of a web page including HTML metadata, DNS records, robots.txt crawl permissions, RDAP domain registration, and llms.txt agent-readiness, all fetched in parallel in a single paid call.","exampleAgentPrompt":"Can you pull a full site profile for https://openai.com — I want the HTML metadata, DNS records, domain registration info, whether GPTBot is allowed by robots.txt, and whether they have an llms.txt file?","exampleUseCases":[{"title":"Verify competitor site crawlability before scraping","prompt":"I'm thinking about scraping competitor websites for market research. Can you pull a full profile for https://example-competitor.com and let me know if they block my user-agent in robots.txt, plus when their domain registration expires? I want to make sure I'm not wasting time on a site that's about to go down."},{"title":"Check if startup is AI-agent ready","prompt":"We're evaluating whether to integrate with https://startup-platform.com. Can you grab their full site profile — especially whether they have an llms.txt file, what their DNS setup looks like, and what metadata they expose? I want to understand how agent-friendly their infrastructure is."},{"title":"Do quick domain due diligence for acquisition","prompt":"We're looking at acquiring https://acquisition-target.io. Can you pull a comprehensive snapshot — their DNS records, domain registration details including expiry date, OpenGraph and meta tags, and whether they've set up any robots.txt restrictions? I need the full picture for the legal and technical teams."}],"resultDescription":"A JSON object with five sections — meta (title, description, OpenGraph, favicon, feeds), dns (A/AAAA/MX/NS records), robots (crawl permission for the specified user-agent), domain (RDAP registration and expiry), and llmsTxt (presence and content of llms.txt) — each with its own status field, plus a summary showing how many sections succeeded and the timestamp.","failureModes":["URL fails SSRF safety check — request rejected before any fetch","All five sections fail (e.g. unreachable host) — returns 503 and no charge is applied","Individual sections return status 'unavailable' or 'skipped' if that data source is inaccessible","Invalid or malformed URL parameter returns a 4xx error","Target site rate-limits or blocks the fetcher — affected section marked unavailable"],"whenToPreferThis":"Choose this endpoint when you need a broad reconnaissance snapshot of a website in a single call rather than making separate requests for DNS, WHOIS, metadata, and robots checks. It is especially useful for agents doing site discovery, crawl planning, or domain due diligence where latency matters and you want parallel fan-out rather than sequential lookups. Prefer it over the sibling DNS-only or tech-stack endpoints when you need the combined picture including domain registration and agent-readiness signals.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-10T06:50:20.263Z","isFirstParty":false,"canonicalSlug":"402utils-site-profile-d8c605d5"}