{"uid":"cap_WlG3Aw65vvwJNKTb4LMhe","slug":"manylives-label-agreement-calculator-a1e8bf55","name":"ManyLives Label Agreement Calculator","description":"Pay-per-call tools for people building AI agents: live web search and page contents, point-in-time crypto market data and news for backtests (no lookahead), verifiable track records of 1,000 crypto strategies locked on the Base network, company profiles, quick AI and batch prices. Ask us to turn a public website into a JSON endpoint. Pay per call, no account, free trial.","url":"https://api.manylives.me/paid/data/label-agreement?utm_source=zero.xyz","method":"POST","headers":{},"bodySchema":{"type":"object","properties":{"csv":{"type":"string","examples":["item,annotator,label\n1,a1,cat\n1,a2,cat\n1,a3,dog\n2,a1,dog\n2,a2,dog\n2,a3,dog\n3,a1,cat\n3,a2,bird\n3,a3,cat"],"description":"or CSV text with item,annotator,label columns"},"rows":{"type":"string","description":"list of {item, annotator, label}"},"allowed_labels":{"type":"string","examples":[["cat","dog"]],"description":"optional list of valid labels"},"case_sensitive":{"type":"string","description":"optional, default false"}}},"responseSchema":{"type":"json","example":{"rows":42,"items":12,"pairwise":[{"a":"A","b":"C","items":8,"kappa":0.478,"agreement":0.625}],"problems":["1 item with no majority label (tied)"],"annotators":4,"disputed_items":[{"item":"u6","labels":{"1":1,"2":1,"3":1,"4":1}}],"krippendorff_alpha":0.743,"annotators_vs_majority":[{"labels":10,"compared":9,"annotator":"C","agrees_with_others_majority":0.778}],"items_with_2plus_labels":11}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.003","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.003/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.003","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.003","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_TGVjwOKWIykc3i55pJk_G","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.003","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Computes inter-annotator agreement statistics (Cohen's kappa, Krippendorff's alpha, pairwise agreement) from a CSV or row list of item/annotator/label triples","exampleAgentPrompt":"Check how much my annotators agree — here's the CSV with columns item, annotator, label: 'item,annotator,label\\n1,a1,cat\\n1,a2,cat\\n1,a3,dog\\n2,a1,dog\\n2,a2,dog\\n2,a3,dog' — the valid labels are cat and dog, case-insensitive.","exampleUseCases":[{"title":"QA check on NLP labeling batch","prompt":"I just got back 500 annotations from three contractors labeling sentences as positive, negative, or neutral — can you run inter-annotator agreement on this CSV and tell me the Krippendorff's alpha and which sentences they kept disagreeing on?"},{"title":"Spotting unreliable annotators on image dataset","prompt":"I have annotation data where four people labeled 200 images as cat, dog, or bird — run the label agreement stats and show me the pairwise kappa between each pair of annotators so I can see who's the outlier."},{"title":"Validating crowdsourced medical labels","prompt":"Before I train on this crowdsourced dataset, check the label agreement — here's the CSV with allowed labels 'yes' and 'no', and flag any items where there's a tie with no clear majority."}],"resultDescription":"Returns a JSON object with: total row and item counts, pairwise Cohen's kappa and agreement percentage for every annotator pair, Krippendorff's alpha across all annotators, per-annotator agreement vs majority, list of disputed items with label distribution, and a problems array flagging ties or anomalies.","failureModes":["Malformed CSV (missing required columns item/annotator/label) returns a parse error","Labels outside the allowed_labels list are flagged or rejected","Fewer than 2 annotators per item makes kappa undefined","Very small datasets may produce unreliable alpha estimates","Non-UTF-8 or binary content in CSV body causes parse failure"],"whenToPreferThis":"Use this endpoint when you need fast, pay-per-call inter-annotator agreement computation without setting up an account or running your own statistics library. Ideal for ML pipelines that periodically validate annotation quality, for agents auditing crowdsourced labeling jobs, or when you need both pairwise kappa and Krippendorff's alpha in a single call. Prefer it over rolling your own when you need immediate results on small-to-medium annotation batches and want itemized dispute lists alongside summary statistics.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-17T12:30:03.132Z","isFirstParty":false,"canonicalSlug":"manylives-label-agreement-calculator-a1e8bf55"}