{"uid":"cap_CjnfmBkWoa2qgfQBu-WWh","slug":"sitecheck-web-intel-robots-txt-security-txt-fetcher-204e3913","name":"SiteCheck Web Intel – robots.txt & security.txt Fetcher","description":"Reads a site's public robots.txt (crawl rules, sitemaps, AI-crawler rules) and security.txt (RFC 9116: contact, expiry, policy), and says whether security.txt is present, current or expired. Missing files are reported as absent, not as errors.","url":"https://api.webintel-api.workers.dev/api/security/robots-security-txt?utm_source=zero.xyz","method":"GET","headers":{},"bodySchema":{"type":"object","$schema":"https://json-schema.org/draft/2020-12/schema","required":["input"],"properties":{"input":{"type":"object","required":["type","method"],"properties":{"type":{"type":"string","const":"http"},"method":{"enum":["GET"],"type":"string"},"queryParams":{"type":"object","required":["domain"],"properties":{"domain":{"type":"string","description":"Domain name such as github.com (required)"}}}},"additionalProperties":false},"output":{"type":"object","required":["type"],"properties":{"type":{"type":"string"},"example":{"type":"object"}}}}},"responseSchema":{"type":"json","example":{"domain":"github.com","source":"https://github.com/robots.txt","checkedAt":"2026-10-23T09:38:35.803Z","robotsTxt":{"groups":5,"present":true,"sitemaps":["https://github.com/sitemap_marketing.xml"],"userAgentsNamed":["GPTBot","OAI-SearchBot"],"wildcardDisallow":["/account-login","/copilot/"]},"securityTxt":{"policy":"https://bounty.github.com","contact":["https://hackerone.com/github"],"expired":false,"expires":"2026-11-22T09:38:35z","present":true,"location":"https://github.com/.well-known/security.txt","canonical":["https://github.com/.well-known/security.txt"],"hasEncryption":false}}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.002","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.002/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.002","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.002","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_6CMN6w2vorIORWawd16FQ","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.002","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Fetches and parses a domain's robots.txt and security.txt files, returning structured data about crawl rules, named user-agents, sitemaps, security contact/policy, and certificate expiry.","exampleAgentPrompt":"Can you check github.com's robots.txt and security.txt — I want to know which bots they explicitly name, what paths are disallowed, where to report a vulnerability, and whether their security.txt is still valid?","exampleUseCases":[{"title":"AI crawler access audit","prompt":"Check openai.com's robots.txt and tell me whether they explicitly block GPTBot or any other AI crawlers, and what paths are off-limits."},{"title":"Bug bounty program lookup","prompt":"I need to find the security contact and vulnerability disclosure policy for stripe.com — can you pull their security.txt and tell me if it's expired?"},{"title":"SEO sitemap discovery","prompt":"Fetch the robots.txt for shopify.com and tell me what sitemaps are listed there so I can start an SEO audit."}],"resultDescription":"A JSON object containing the domain checked, the source URL, and two sub-objects: robotsTxt (presence flag, number of groups, named user-agents, wildcard disallow paths, sitemap URLs) and securityTxt (presence flag, contact URLs, policy URL, canonical URLs, expiry timestamp, expired boolean, encryption key flag, and file location URL).","failureModes":["Domain not found or unreachable — may return empty presence flags or an error","robots.txt or security.txt absent on the target domain — fields will indicate not present","Malformed or non-standard files may result in incomplete parsing","Rate limits or blocks on the target domain may prevent retrieval","Invalid domain parameter format returns a validation error"],"whenToPreferThis":"Choose this endpoint when you need structured, parsed data from both robots.txt and security.txt in a single call without writing your own HTTP fetcher and parser. It is ideal for security audits, responsible-disclosure lookups, AI-crawler policy checks, or SEO sitemap discovery. Prefer it over generic web scraping when you specifically want crawl rules and security disclosure metadata in a normalized JSON format, and when pay-per-call pricing without API-key signup is acceptable.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-25T12:31:32.849Z","isFirstParty":false,"canonicalSlug":"sitecheck-web-intel-robots-txt-security-txt-fetcher-204e3913"}