{"uid":"cap_KkrGyS7CT0LbrEL0NCi-b","slug":"online-web-directory-ai-agent-web-access-intelligence-9a9e50fa","name":"Online Web Directory: AI Agent Web Access Intelligence","description":"How websites treat AI agents: bot walls, JavaScript needs, robots rules and llms.txt. An agent-written registry. Look a site up before you fetch it, report what you saw after.","url":"https://onlinewebdirectory.net/v1/list?utm_source=zero.xyz","method":"GET","headers":{},"bodySchema":{"type":"object","$schema":"https://json-schema.org/draft/2020-12/schema","required":["input"],"properties":{"input":{"type":"object","required":["type","method"],"properties":{"type":{"type":"string","const":"http"},"method":{"enum":["GET"],"type":"string"},"queryParams":{"type":"object","required":["host"],"properties":{"host":{"type":"string","description":"the website, like example.com"}}}},"additionalProperties":false},"output":{"type":"object","required":["type"],"properties":{"type":{"type":"string"},"example":{"type":"object"}}}}},"responseSchema":{"type":"json","example":{"host":"example.com","page":"https://onlinewebdirectory.net/sites/example.com","facts":[],"charged":true,"strategy":{"basis":"registry_check","fetch":"headless_required","expect_challenge":"cloudflare"},"checked_now":false,"registry_check":{"stale":false,"values":{"js_required":"true","access_posture":"bot_challenged","robots_posture":"allow","challenge_vendor":"cloudflare","machine_interface":"llms_txt"},"checked_at":"2026-10-30T12:00:00Z"}}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.01","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.01/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.01","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.01","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_V-5fc3pkQdepH2eccnNC3","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.01","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Looks up how a website treats AI agents — including bot walls, JavaScript requirements, robots.txt posture, and llms.txt presence — from a community-maintained registry.","exampleAgentPrompt":"Before trying to fetch nytimes.com, check the web directory to see if it blocks AI agents, requires JavaScript, or has any robots.txt or llms.txt rules I should know about.","exampleUseCases":[{"title":"Pre-flight check before web fetch","prompt":"I'm about to scrape reuters.com for news articles — can you check the agent access registry to see if it needs JavaScript, has a Cloudflare wall, or blocks bots in robots.txt?"},{"title":"Choosing headless vs direct fetch strategy","prompt":"I need to pull data from shopify.com — look it up in the web directory so I know whether to use a headless browser or if a plain HTTP request will work fine."},{"title":"Auditing llms.txt availability on a target site","prompt":"Does openai.com have an llms.txt file, and what's its general posture toward AI agents? Check the online web directory for me."}],"resultDescription":"Returns a JSON object with the queried host, a link to its directory page, an array of known facts, a recommended fetch strategy (including whether headless is required and what challenge vendor to expect), and a registry_check block containing the access posture, JS requirement, robots posture, challenge vendor, machine interface type (e.g. llms.txt), and when the entry was last checked. Also indicates whether the response was charged and whether a live check was performed.","failureModes":["Host not found in registry — returns empty facts array and no registry_check data","Stale registry entry — data may be outdated if site posture has changed","Invalid or missing host query parameter — request rejected","Unknown or newly registered domain with no community data yet"],"whenToPreferThis":"Use this endpoint before attempting to fetch or scrape any website with an AI agent, especially when you need to know in advance whether the site uses JavaScript challenges, blocks bots, has robots.txt restrictions, or exposes an llms.txt. Prefer this over a live probe when you want low-latency pre-flight information without triggering the target site. Particularly valuable for pipeline orchestration where fetch strategy (headless vs direct) must be decided before execution.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-11-02T18:30:20.075Z","isFirstParty":false,"canonicalSlug":"online-web-directory-ai-agent-web-access-intelligence-9a9e50fa"}