{"uid":"cap_jDO6b9M2PRbLIfoJ1enI5","slug":"forgemesh-content-signals-lookup-2f2ba9d0","name":"ForgeMesh Content Signals Lookup","description":"Content Signals lookup API: does this domain permit AI use of its content? Fetches robots.txt and parses Content-Signal / aipref declarations (search=, ai-input=, ai-train=) into structured JSON plus a plain-English interpretation, including related AI directives. The essential pre-crawl compliance check for scraping, RAG ingestion, and dataset agents as the web gates against bots (Cloudflare defaults change Sept 15, 2026). First mover — nobody else sells this check.","url":"https://x402.forgemesh.io/domain-content-signals","method":"POST","headers":{},"bodySchema":{"type":"object","properties":{"domain":{"type":"string","description":"Domain to check, e.g. example.com"}}},"responseSchema":{"type":"json","example":{"domain":"theverge.com","signals":{"search":"yes","ai-input":"no","ai-train":"no"},"declared":true,"interpretation":"site declares: search=yes, ai-input=no, ai-train=no"}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.005","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.005/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.005","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.005","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_H2tQjcZSgBiqpyMKdRRbe","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.005","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Checks whether a domain permits AI use of its content by fetching and parsing robots.txt Content-Signal / aipref declarations into structured JSON with plain-English interpretation.","exampleAgentPrompt":"Before I scrape nytimes.com for my RAG pipeline, can you check whether that domain actually permits AI use of its content — look at the robots.txt and any Content-Signal or aipref declarations and tell me what's allowed?","exampleUseCases":null,"resultDescription":"Returns structured JSON containing parsed Content-Signal and aipref directive values (search=, ai-input=, ai-train=) found in the domain's robots.txt, plus a plain-English interpretation of what AI uses are permitted or restricted, and any related AI-specific directives.","failureModes":["Domain is unreachable or does not exist — returns error or empty policy","robots.txt is absent — interpretation defaults to permissive with note","Malformed robots.txt with no parseable Content-Signal blocks — returns empty directives","Network timeout fetching the remote robots.txt","Domain string is malformed or missing — returns validation error"],"whenToPreferThis":"Use this endpoint as a pre-crawl compliance gate before scraping, RAG ingestion, or dataset collection from any external domain. Prefer it when your agent needs to verify AI content permissions specifically (ai-input, ai-train, search signals) rather than just generic robot exclusion rules. Essential for pipelines that must respect the emerging web standard for AI content gating, especially ahead of the September 2026 Cloudflare default changes.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-09-15T13:04:00.240Z","isFirstParty":false}