{"uid":"cap_riqbYUZm3DAy5J4zSUKdk","slug":"robots-directive-merge-a76bc933","name":"Robots Directive Merge","description":"Robots Directive Merge: Robots Directive Merge merges generic and named-agent robots directives with specific precedence from bounded caller-supplied values without an external provider. Call Robots Directive Merge before accepting, caching, redirecting, or retrying a caller-supplied web response. Returns normalized web evidence, the computed finding, and an explicit pass or advisory status for Robots Directive Merge as versioned deterministic JSON. Price: $0.001 USDC via x402 on Base. First-pa…","url":"https://api.delx.ai/api/v1/x402/robots-directive-merge?utm_source=zero.xyz","method":"POST","headers":{},"bodySchema":{"type":"object","properties":{"generic":{"type":"string","maxLength":8192,"description":"Generic supplied to Robots Directive Merge; used only for this bounded calculation and processed in memory without retention."},"agent_specific":{"type":"string","maxLength":8192,"description":"Agent Specific supplied to Robots Directive Merge; used only for this bounded calculation and processed in memory without retention."}}},"responseSchema":{"type":"json","example":{"result":{"index":false,"follow":false,"directives":{"index":true,"follow":true,"noindex":true,"nofollow":true}},"schema":"delx/util-robots-directive-merge/v1","status":"pass","evidence":{"retained":false,"input_sha256":"b9eeb9550ac99b4dea8c9b0875b76b73921e1fa3b567beb37542161b59db80ab","external_calls":0},"operation":"web_reliability:robots_directive_merge"}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.001","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.001/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.001","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.001","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_CTCvlHHcNjgmldHrCPq2Z","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.001","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Merges generic and named-agent robots.txt directives with explicit precedence rules to produce a normalized, deterministic pass/advisory crawl decision.","exampleAgentPrompt":"Before I cache this page response, can you merge the generic robots directive 'User-agent: *\\nDisallow: /private/' with the agent-specific directive 'User-agent: MyBot\\nAllow: /private/' and tell me whether my bot has a pass or advisory status to proceed?","exampleUseCases":[{"title":"Pre-crawl robots compliance check","prompt":"I'm about to fetch and cache a page — merge these two robots.txt sections for me: the generic one says 'User-agent: *\\nDisallow: /admin/' and the agent-specific one says 'User-agent: GPTBot\\nAllow: /admin/' — does my agent get a pass or an advisory?"},{"title":"Redirect handling robots validation","prompt":"Before I follow this redirect, check whether my crawler is allowed: generic robots says 'User-agent: *\\nDisallow: /' and the named-agent block says 'User-agent: MyAgent\\nAllow: /public/' — what does the merged directive say about access?"},{"title":"Retry decision based on robots directives","prompt":"My agent is deciding whether to retry a blocked response — can you merge the generic robots rules 'User-agent: *\\nCrawl-delay: 10\\nDisallow: /api/' with my agent's specific rules 'User-agent: DelxBot\\nAllow: /api/' and give me the normalized finding and pass/advisory status?"}],"resultDescription":"Returns a versioned deterministic JSON object containing: the normalized web evidence (merged directive representation), the computed finding (effective rule after precedence resolution), and an explicit pass or advisory status indicating whether the caller's agent may proceed with the associated web action.","failureModes":["Malformed or non-UTF-8 robots directive text returns a validation error","Inputs exceeding 8192 characters are rejected","Missing or empty both inputs may return an indeterminate or default advisory status","Payment failure (x402) prevents the call from being processed","Network timeout on the API endpoint"],"whenToPreferThis":"Use this endpoint when an agent needs a deterministic, precedence-aware merge of generic (User-agent: *) and named-agent robots.txt blocks before accepting, caching, redirecting, or retrying a web response. Prefer it over manual parsing when you need a normalized, versioned JSON finding with an explicit pass/advisory signal rather than raw text interpretation. It is especially useful in automated pipelines where consistent, auditable crawl decisions are required without external dependencies.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-21T18:55:32.905Z","isFirstParty":false,"canonicalSlug":"robots-directive-merge-a76bc933"}