{"uid":"cap_JZXmfWQnAgA1mSv3a3-LH","slug":"a-b-test-readout-check-ea776208","name":"A/B Test Readout Check","description":"A/B Test Readout Check — Checks the written readout of ONE A/B test (a paragraph, an email, a table or a slide transcript) for the five statistical mistakes that turn a no-effect test into a \"win\", and lists only those, one line each with a fixed code, the place, the evidence with the arithmetic and the fix - or exactly No findings. It catches stopping the moment a result looks significant, many metrics or variants judged without correction, a split of users that differs from the planned split …","url":"https://api.aiskills402.com/v1/skills/ab-test-readout-check/file?utm_source=zero.xyz","method":"GET","headers":{},"bodySchema":{"type":"object","$schema":"https://json-schema.org/draft/2020-12/schema","required":["input"],"properties":{"input":{"type":"object","required":["type","method"],"properties":{"type":{"type":"string","const":"http"},"method":{"enum":["GET","HEAD"],"type":"string"},"headers":{"type":"object","additionalProperties":{"type":"string"}}},"additionalProperties":false},"output":{"type":"object","required":["type"],"properties":{"type":{"type":"string"},"example":{"type":"object"}}}}},"responseSchema":null,"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.01","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.01/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.01","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.01","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_AehrioNltAa_Ht-Ha1_9W","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.01","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Reviews a written A/B test readout (paragraph, email, table, or slide transcript) and flags up to five specific statistical mistakes that can falsely declare a no-effect test a win.","exampleAgentPrompt":"Can you check this A/B test readout for statistical mistakes — like peeking, multiple comparisons without correction, or a skewed user split — and tell me exactly what's wrong and how to fix it? Here's the paragraph: 'We ran the test for 3 days and stopped when variant B hit p=0.04 across 12 metrics, with a 52/48 user split instead of the planned 50/50.'","exampleUseCases":[{"title":"Audit experiment email before sharing","prompt":"Before I send this A/B test results email to leadership, can you check it for the classic statistical mistakes — peeking, undeclared multiple comparisons, sample ratio mismatch — and list any problems with the evidence and the fix? Here's the email text: 'After 5 days, Variant B improved checkout rate by 4% (p=0.03) and also improved add-to-cart and session length, so we're calling it a win.'"},{"title":"Review slide deck test summary","prompt":"I have a slide transcript from our weekly experiment review — can you scan it for statistical errors like stopping early when it looked significant or judging too many variants without correction, and give me a one-line finding per issue with the error code and how to fix it?"},{"title":"Validate CRO agency test report","prompt":"Our CRO agency sent over a table summarising three A/B test variants and declared two winners. Can you review the table text for mistakes like multiple variant comparisons without a correction, a sample split that doesn't match the planned 50/50, or any peeking behaviour, and list only the actual problems found?"}],"resultDescription":"A list of statistical mistakes found in the readout, each on one line with a fixed error code, the location in the text, arithmetic evidence, and a concrete fix — or exactly 'No findings' if the readout is clean. The service checks for: early stopping/peeking, multiple metrics or variants without correction, sample ratio mismatch versus planned split, and related statistical errors.","failureModes":["Input text is too short or lacks enough context to assess statistical validity","The readout does not contain quantitative results, making statistical checks impossible","API returns an error if the request payload or headers are malformed","Service may not detect mistakes in highly ambiguous or incomplete descriptions","Edge case: if the readout is in a language other than English, detection quality may degrade"],"whenToPreferThis":"Choose this endpoint when you need a focused, codified statistical audit of a single A/B test write-up — particularly to catch peeking, multiple comparison issues, or sample ratio mismatch before results are shared with stakeholders. Prefer it over general-purpose LLM prompting when you need consistent error codes and structured one-line findings rather than free-form commentary.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-20T12:36:46.877Z","isFirstParty":false,"canonicalSlug":"a-b-test-readout-check-ea776208"}