{"uid":"cap_ewJpn6-_BKN9YqiImqUpL","slug":"preplayer-ai-near-duplicate-document-detection-195e7a3d","name":"PrepLayer.ai Near-Duplicate Document Detection","description":"Near-duplicate detection across a batch of documents. Exact Jaccard similarity over word-3-shingle sets decides duplication; a 64-bit SimHash is returned per document as a compact content fingerprint. Up to 200 documents per call.","url":"https://api.maximusmediahouse.com/v1/dedupe?utm_source=zero.xyz","method":"POST","headers":{},"bodySchema":{"type":"object","properties":{"documents":{"type":"array","items":{"type":"string"},"maxItems":200,"minItems":1,"description":"Documents to compare."},"threshold":{"type":"integer","default":60,"maximum":100,"minimum":0,"description":"Minimum Jaccard similarity percent to count as a duplicate."}}},"responseSchema":null,"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.01","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.01/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.01","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.01","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_8hP_V6GY8Pcyflo7c-eCG","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.01","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Detects near-duplicate documents in a batch using Jaccard similarity over word 3-shingles and returns a 64-bit SimHash fingerprint per document","exampleAgentPrompt":"Can you run near-duplicate detection on these 50 articles and flag any pairs that share at least 70% Jaccard similarity — I also want the SimHash fingerprint for each one so I can track them later?","exampleUseCases":[{"title":"Deduplicate training corpus before fine-tuning","prompt":"I have a batch of 150 scraped web articles I want to use for fine-tuning — can you check for near-duplicates using a 75% similarity threshold and remove any redundant ones before I proceed?"},{"title":"Flag recycled content across submitted blog posts","prompt":"We received 80 blog post submissions and I'm worried some are copy-pasted or lightly reworded — can you run duplicate detection on all of them at a 60% Jaccard threshold and tell me which pairs are too similar?"},{"title":"Audit legal document set for repeated clauses","prompt":"I've got 40 contract documents and I need to know if any sections are near-identical across them — run duplicate detection with a threshold of 80% and give me the SimHash fingerprint for each document too."}],"resultDescription":"For each document in the batch, the endpoint returns a 64-bit SimHash content fingerprint. It also returns pairs of documents identified as near-duplicates where their Jaccard similarity over word 3-shingle sets meets or exceeds the specified threshold percentage.","failureModes":["Batch exceeds 200 documents — returns a validation error","Threshold out of range (not 0–100) — returns a schema validation error","Empty documents array — rejected with minItems constraint error","Very short documents may produce unreliable shingle-based similarity scores","Non-text or binary content passed as strings may yield meaningless SimHash values"],"whenToPreferThis":"Choose this endpoint when you need fast, deterministic near-duplicate detection over a batch of text documents using a mathematically rigorous Jaccard similarity metric. It is ideal for data cleaning pipelines, corpus deduplication before ML training, content audits, and plagiarism screening where you need both a similarity decision and a compact fingerprint (SimHash) per document for downstream indexing. Prefer it over semantic similarity approaches when you want exact lexical overlap measurement rather than meaning-based comparison, and when batch processing up to 200 documents per call fits your workflow.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-29T00:30:16.967Z","isFirstParty":false,"canonicalSlug":"preplayer-ai-near-duplicate-document-detection-195e7a3d"}