{"uid":"cap_Pg3YcCA8rqwT5JeqEc6xk","slug":"robots-txt-parser-change-watch-fa5e08a9","name":"Robots.txt Parser Change Watch","description":"Change watch for robots txt parser: runs the check and compares it with the result from your previous call for the same url (kept 90 days), returning the current answer, changed true/false, exactly which fields changed with before/after values, and when it was last seen. Call it on a schedule to be told when robots changes. $0.01 per check.","url":"https://intel.rallylive.ca/watch/site/robots","method":"GET","headers":{},"bodySchema":{"type":"object","$schema":"https://json-schema.org/draft/2020-12/schema","required":["input"],"properties":{"input":{"type":"object","required":["type","method"],"properties":{"type":{"type":"string","const":"http"},"method":{"enum":["GET"],"type":"string"},"queryParams":{"type":"object","properties":{}}},"additionalProperties":false},"output":{"type":"object","required":["type"],"properties":{"type":{"type":"string"},"example":{"type":"object"}}}}},"responseSchema":null,"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.02","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.02/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.02","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.02","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_RuRK2mS7hTvLjhzWpygjF","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.02","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Monitors a URL's robots.txt file for changes by comparing the current parse result against the previously stored result, returning a diff with before/after field values.","exampleAgentPrompt":"Check if the robots.txt for example.com has changed since the last time you checked — tell me exactly which fields changed and what the before and after values were.","exampleUseCases":[{"title":"SEO audit change alerting","prompt":"Keep an eye on competitor.com's robots.txt and let me know the moment anything changes — I want to see exactly which directives were added, removed, or modified."},{"title":"Automated site compliance monitoring","prompt":"I need to make sure our partner site's robots.txt still allows our crawler — check it now and tell me if anything has changed since we last looked, including the specific before and after values for any modified fields."},{"title":"Scheduled robots.txt drift detection","prompt":"Run a scheduled check on myblog.com's robots.txt every day and flag it if the file changes — I want to catch any new Disallow rules before they affect our indexing."}],"resultDescription":"Returns the current parsed robots.txt result, a boolean 'changed' flag indicating whether it differs from the previous call, a field-level diff showing exactly which fields changed with their before and after values, and a timestamp of when the previous result was last seen. Previous results are retained for 90 days.","failureModes":["URL is unreachable or returns a non-200 status — endpoint may return an error or empty parse result","No previous result stored (first call for a URL) — 'changed' may be null or false with no diff","robots.txt file is malformed — parser may return partial or empty results","90-day retention window expired — previous result no longer available, treated as first call","Rate limiting or payment failure — call blocked before executing"],"whenToPreferThis":"Choose this endpoint when you need to detect changes in a site's robots.txt over time on a polling/scheduled basis, rather than just parsing the current robots.txt once. It is especially useful for SEO monitoring, compliance checks, and competitive intelligence workflows where knowing what changed (and exactly how) matters more than just the current state.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-09-14T07:21:49.242Z","isFirstParty":false}