{"uid":"cap_hv_18GfwVwOJKu3bwl-Ph","slug":"ai-calibration-error-calculator-beef1bac","name":"AI Calibration Error Calculator","description":"Compares scalar confidence with binary correctness, calculates mean squared Brier loss, partitions confidence into caller-selected equal-width bins and reports expected and maximum calibration error. It does not validate labels, correct class imbalance or prove calibration beyond the supplied sample.","url":"https://aialign.halowerk.com/v1/confidence-calib","method":"POST","headers":{},"bodySchema":{"type":"object","properties":{"bin_count":{"type":"integer","maximum":100,"minimum":2},"predictions":{"type":"array","items":{"type":"object","required":["prediction_id","confidence","correct"],"properties":{"correct":{"type":"boolean"},"confidence":{"type":"number","maximum":1,"minimum":0},"prediction_id":{"type":"string","maxLength":128,"minLength":1}},"additionalProperties":false},"maxItems":100000,"minItems":1}}},"responseSchema":null,"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.003","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.003/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.003","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.003","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_Lgx0hdhaSp3QXok8TUWeA","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.003","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Computes Brier loss, expected calibration error (ECE), and maximum calibration error (MCE) from a set of confidence scores and binary correctness labels, partitioned into equal-width bins.","exampleAgentPrompt":"Check how well-calibrated these 500 model predictions are — I'll give you each prediction's confidence score (between 0 and 1) and whether it was correct (true/false). Split them into 10 equal-width bins and give me the Brier loss, expected calibration error, and maximum calibration error.","exampleUseCases":[{"title":"LLM confidence calibration audit","prompt":"I have 2,000 predictions from our language model with their confidence scores and correctness labels. Can you compute the Brier loss, ECE, and MCE using 20 bins so I can see how well-calibrated it is before we ship it?"},{"title":"Classifier overconfidence diagnosis","prompt":"Our image classifier seems overconfident — it says 90%+ confidence on many wrong answers. Run calibration metrics on these 1,500 labeled predictions using 10 bins and tell me the expected and maximum calibration errors."},{"title":"A/B comparison of model calibration","prompt":"I want to compare two models' calibration. Here's the first batch of 800 predictions with confidence scores and correctness flags — use 15 bins and give me the Brier score and ECE so I can compare it against the second model's results."}],"resultDescription":"Returns the mean squared Brier loss across all predictions, the expected calibration error (ECE) as the weighted average gap between confidence and accuracy per bin, the maximum calibration error (MCE) as the worst-case bin gap, and per-bin statistics including confidence range, mean confidence, observed accuracy, and sample count.","failureModes":["Empty predictions array returns a validation error","Confidence values outside [0,1] are rejected","bin_count below 2 or above 100 is rejected","Prediction IDs exceeding 128 characters fail schema validation","Payload exceeding 100,000 predictions is rejected with a size error","Missing required fields (prediction_id, confidence, correct) cause a 400 error"],"whenToPreferThis":"Choose this endpoint when you need fast, stateless calibration metrics (Brier, ECE, MCE) for a batch of binary predictions with scalar confidences. It is ideal for post-hoc model evaluation pipelines, CI/CD regression checks on model calibration, or alignment audits where you have ground-truth correctness labels. It does not perform label validation, class-imbalance correction, or multi-class calibration — for those needs, use a more specialized tool.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-09-14T00:37:57.942Z","isFirstParty":false}