{"uid":"cap_QZJleklogRNo_FtM5tXco","slug":"machine-shop-url-deduplication-c37b50d7","name":"Machine Shop – URL Deduplication","description":"Turn Apify product scrape results into safe catalog updates with Catalog Update Gate. Find evidence in supplied documents, compare snapshots, extract typed JSON fields, inspect x402 quotes and assess API migrations. Preview inputs, evidence and cost for free. No live web search, scheduled monitoring or third-party purchasing.","url":"https://api.pay-forge.online/v1/deduplicate-urls?utm_source=zero.xyz","method":"POST","headers":{},"bodySchema":{"type":"object","properties":{"items":{"type":"array","items":{"type":"string"},"title":"Items","maxItems":100}}},"responseSchema":{"type":"json","example":{"unique":["https://example.com/"],"duplicates":[{"index":1,"duplicate_of":0}]}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.005","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.005/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.005","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.005","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_jYPA9FyeB1wKejpHtYy_n","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.005","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Accepts a list of up to 100 URLs and returns the unique set along with indices identifying which items are duplicates and what they duplicate.","exampleAgentPrompt":"I have this list of URLs from my crawler and I need you to deduplicate them — send them to Machine Shop and give me back just the unique ones along with which items were duplicates: ['https://example.com/', 'https://example.com/', 'https://foo.com/page', 'https://foo.com/page', 'https://bar.com/'].","exampleUseCases":[{"title":"Crawler output deduplication","prompt":"I just ran a web crawler and collected 80 URLs but there are a bunch of repeats — can you deduplicate this list and tell me which indices were duplicates and what they duplicated?"},{"title":"Link list cleanup before storage","prompt":"Before I save these links to my database, run them through Machine Shop's deduplication endpoint and give me only the unique URLs so I don't store redundant records."},{"title":"Sitemap URL deduplication","prompt":"I exported all the URLs from my sitemap and there are some duplicates mixed in — deduplicate this array of up to 100 URLs and tell me exactly which ones were repeated."}],"resultDescription":"Returns a JSON object with a 'unique' array containing the deduplicated URLs, and a 'duplicates' array of objects each containing the index of the duplicate item and the index of the original item it duplicates (duplicate_of).","failureModes":["Input array exceeds 100 items — request rejected","Non-string entries in the items array — validation error","Malformed JSON body — 400 bad request","Payment not provided or insufficient — 402 Payment Required","Empty items array — may return empty unique list with no duplicates"],"whenToPreferThis":"Choose this endpoint when you need a fast, deterministic deduplication of a URL list (up to 100 items) with exact duplicate detection and index-level traceability. It is ideal when you need to know not just the unique set but also which positions in the original array were duplicates and what they duplicated. Prefer it over client-side deduplication when you want an auditable, pay-per-call microservice with a structured response.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-20T12:34:55.596Z","isFirstParty":false,"canonicalSlug":"machine-shop-url-deduplication-c37b50d7"}