{"uid":"cap_LzGxT1p-PfNTrXLuCsEvl","slug":"agentready-url-preflight-check-24e660a0","name":"AgentReady URL Preflight Check","description":"Free website triage plus x402 audits, Base USDC receipt verification, robots-aware URL preflight, agent-readable extraction, metadata, and copy-ready AI-discovery fixes.","url":"https://agentready-audit.seshiccse023.chatgpt.site/api/preflight/url-check","method":"POST","headers":{},"bodySchema":{"type":"object","properties":{"url":{"type":"string","format":"uri","description":"Public HTTP(S) URL to check before fetching"},"userAgent":{"type":"string","maxLength":120,"description":"Optional crawler product token; defaults to *"}}},"responseSchema":{"type":"json","example":{"url":"https://example.com/private","page":{"status":200,"finalUrl":"https://example.com/private","reachable":true,"contentType":"text/html","contentLength":1256},"notes":["robots.txt permits this user agent and the page is reachable."],"robots":{"url":"https://example.com/robots.txt","state":"unavailable","status":404,"sitemaps":[]},"checkedAt":"2026-08-09T00:00:00.000Z","fetchable":true,"userAgent":"GPTBot","matchedRule":null,"allowedByRobots":true}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.005","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.005/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.005","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.005","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_CQWZ-A84r5Etng37GK2Iw","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.005","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Checks whether a given URL is allowed to be fetched by a specific crawler agent by parsing robots.txt rules, then verifies the page's reachability and basic HTTP metadata.","exampleAgentPrompt":"Before my pipeline fetches https://example.com/articles/ai-news, can you preflight that URL for the GPTBot crawler — check if robots.txt allows it and whether the page is actually reachable?","exampleUseCases":[{"title":"Pre-crawl robots compliance check","prompt":"I'm about to have my agent scrape https://techcrunch.com/2024/01/ai-trends for research — can you first check whether robots.txt allows the GPTBot user agent to access that URL, and confirm the page is actually reachable?"},{"title":"Pipeline gate before data extraction","prompt":"My data pipeline needs to fetch https://news.ycombinator.com/item?id=12345678 — run a preflight on it for the generic wildcard user agent so I know if it's allowed and returns a valid 200 response before I spend credits extracting it."},{"title":"Audit crawler access for site migration","prompt":"I'm migrating content and need to verify https://myoldsite.com/blog/post-1 is accessible and not blocked for any crawler — run a robots.txt and reachability check on it now."}],"resultDescription":"Returns a JSON object containing: the checked URL, the timestamp of the check, the user agent used, whether robots.txt permits access (allowedByRobots boolean), whether the page is fetchable (fetchable boolean), the specific matched robots rule (or null), a robots object with the robots.txt URL and its fetch status, a page object with HTTP status, final URL after redirects, reachability, content type, and content length, plus a notes array with human-readable advisory messages.","failureModes":["URL is malformed or not a valid HTTP/HTTPS URI — returns validation error","Target server is unreachable or times out — page object may show reachable:false","robots.txt itself returns a non-200 status — robots.state set to 'unavailable' and access may default to allowed","URL redirects to a blocked or error page — finalUrl differs from input URL","Rate limits or payment issues if USDC payment is not properly handled via x402"],"whenToPreferThis":"Use this endpoint when an AI agent or crawler pipeline needs to check robots.txt compliance and page reachability for a specific URL before fetching it, especially when operating under ethical crawling constraints or building agentic pipelines that must respect robots exclusion standards. Prefer this over a raw HTTP fetch when you need a structured, machine-readable verdict on whether fetching is permitted and feasible, rather than just trying the fetch and handling errors reactively.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-09-14T18:34:14.649Z","isFirstParty":false}