{"uid":"cap_VhR_xu_FBE8lvLUPMPK3l","slug":"delx-robots-path-budget-01ef5878","name":"Delx Robots Path Budget","description":"Count allow/disallow rules from robots.txt text. Call when crawl budget planning from a robots body you already fetched. Returns allow_n, disallow_n, sitemaps as deterministic JSON for $0.001 USDC via x402 on Base. First-party local math only—no RPC, live market feed, API keys, storage, or mediagen. Advisory only; the caller owns capital, risk, and production controls.","url":"https://api.delx.ai/api/v1/x402/robots-path-budget","method":"POST","headers":{},"bodySchema":{"type":"object","properties":{"robots":{"type":"string","description":"robots.txt body text"}}},"responseSchema":{"type":"json","example":{"schema":"delx/util-robots-path-budget/v1","allow_n":1,"sitemaps":["https://ex.com/sitemap.xml"],"disallow_n":1}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.001","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.001/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.001","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.001","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_PSTKACVsAsYWRXw4Jj1DO","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.001","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Parses a robots.txt body and returns a structured budget summary of allowed paths, disallowed paths, and sitemap URLs.","exampleAgentPrompt":"Here's the raw robots.txt content I fetched from the target site — can you parse it and tell me how many paths are allowed vs disallowed, and list any sitemap URLs it declares?","exampleUseCases":[{"title":"Pre-crawl robots.txt budget check","prompt":"Before my agent starts crawling example.com, I fetched its robots.txt — can you parse this text and give me a count of allowed and disallowed paths plus any sitemaps listed, so I know what's safe to crawl?"},{"title":"Sitemap discovery from robots.txt","prompt":"I have the raw robots.txt for a news site I want to index. Extract all the sitemap URLs declared in it so I can feed them into my sitemap crawler next."},{"title":"Compliance audit before scraping","prompt":"My scraping pipeline needs to respect robots.txt rules. Here's the robots.txt body from the target — summarize how many routes are blocked versus permitted so I can configure my agent's path budget accordingly."}],"resultDescription":"A JSON object containing the schema version ('delx/util-robots-path-budget/v1'), the count of allowed directives (allow_n), the count of disallowed directives (disallow_n), and an array of sitemap URLs found in the robots.txt.","failureModes":["Empty or null robots.txt body returns zero counts and empty sitemaps array","Malformed robots.txt with non-standard formatting may undercount directives","Very large robots.txt bodies may be truncated or cause timeout","Missing or unparseable Sitemap: lines result in empty sitemaps list"],"whenToPreferThis":"Use this endpoint when your agent has already fetched a raw robots.txt string and needs a quick structured summary of crawl permissions and sitemap pointers — especially before launching a web crawl, building a site index, or checking compliance. It is a lightweight, single-call utility that avoids writing custom robots.txt parsing logic in your agent pipeline.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-09-14T18:32:37.368Z","isFirstParty":false}