{"uid":"cap_ff36Q3O23D9yER5sNyX4d","slug":"minifetch-proxy-url-link-extractor-00c90e64","name":"Minifetch Proxy URL Link Extractor","description":"For pages that block the native Minifetch proxy that obeys robots.txt: extracts the same categorized links as /extract/url-links (internal, external, anchor) with rel attributes, anchor text and summary stats, fetched through an external rotating proxy. No JavaScript rendering; use /proxy/render for that. Optional ?country= (2-letter code) to fetch from a specific country. Not cached. Does not obey robots.txt. Supports GET & POST","url":"https://minifetch.com/api/v1/x402/proxy/extract/url-links?utm_source=zero.xyz","method":"GET","headers":{},"bodySchema":{"type":"object","$schema":"https://json-schema.org/draft/2020-12/schema","required":["input"],"properties":{"input":{"type":"object","required":["type","method"],"properties":{"type":{"type":"string","const":"http"},"method":{"enum":["GET"],"type":"string"},"queryParams":{"type":"object","required":["url"],"properties":{"url":{"type":"string","description":"The target URL from which to extract the links."},"country":{"type":"string","description":"Optional. 2-letter ISO 3166-1 country code (ex: de, mx) to fetch from. Invalid code returns 400."}}}},"additionalProperties":false},"output":{"type":"object","required":["type"],"properties":{"type":{"type":"string"},"example":{"type":"object","properties":{"results":{"type":"array","items":{"type":"object","properties":{"data":{"type":"object","properties":{"url":{"type":"string","description":"Final URL after redirects"},"links":{"type":"object","properties":{"anchors":{"type":"array","description":"In-page #fragment links, each { href, text }"},"summary":{"type":"object","properties":{"totalLinks":{"type":"integer"},"anchorCount":{"type":"integer"},"externalCount":{"type":"integer"},"internalCount":{"type":"integer"},"nofollowCount":{"type":"integer"},"emptyTextCount":{"type":"integer"},"topExternalDomains":{"type":"array","description":"Busiest external domains, each { domain, count }"},"topInternalTargets":{"type":"array","description":"Most-linked internal pages, each { url, count, anchorTexts[] }"},"uniqueExternalDomains":{"type":"integer"},"imageLinksMissingAltCount":{"type":"integer"}},"description":"Aggregate link stats for SEO / site-structure analysis"},"external":{"type":"array","description":"Cross-host links, each { href, text, rel[] }"},"internal":{"type":"array","description":"Same-host links, each { href, text, rel[] }"}}},"proxy":{"type":"object","properties":{"render":{"type":"boolean","description":"true if JS executed"},"country":{"type":["string","null"]},"provider":{"type":"string"},"proxyUrl":{"type":["string","null"]},"proxyType":{"type":"string","description":"Always \"managed\" on proxy routes"},"robotsTxt":{"type":"object","properties":{"allowed":{"type":["boolean","null"]}}},"userAgent":{"type":"string"}},"description":"Fetch provenance. External proxy routes do not check robots.txt"},"requestUrl":{"type":"string","description":"URL as requested, before redirects"},"minifetchCache":{"type":"object","properties":{"hit":{"type":"boolean"}},"description":"External proxy routes not cached: always { hit: false }"}}}}}},"success":{"type":"boolean"}}}}}}},"responseSchema":null,"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.003","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.003/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.003","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.003","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_SUFiu8gWC5KDKUFMcAKFc","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.003","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Extracts categorized links (internal, external, anchor) with rel attributes, anchor text, and summary stats from pages that block standard proxies, using a rotating external proxy that bypasses robots.txt restrictions.","exampleAgentPrompt":"That competitor's site at https://example.com/blog blocks normal crawlers — can you extract all the internal and external links from that page anyway, including anchor text and nofollow stats, by routing through an external proxy?","exampleUseCases":[{"title":"Competitor backlink profile audit","prompt":"I need to see all the outbound links from https://competitor.com/resources — the site blocks standard scrapers, so go through the rotating proxy and give me the external links with their anchor text and which domains they link to most."},{"title":"Geo-specific link extraction","prompt":"Fetch all internal and external links from https://example.de/shop using a German proxy — the site only serves full content to visitors in Germany — and give me a summary of link counts and top linked pages."},{"title":"Bot-blocked SEO link audit","prompt":"Pull all the links from https://blockingsite.com/sitemap — internal, external, and anchor links — using the bypass proxy since that site ignores robots.txt crawlers. I want the rel attributes and anchor text for each link too."}],"resultDescription":"Returns a structured object containing the final URL after redirects, categorized link arrays (internal links, external links, anchor/fragment links — each with href, text, and rel attributes), and a summary object with total link count, internal count, external count, anchor count, nofollow count, empty text count, unique external domains count, image links missing alt count, top external domains by frequency, and top internal target pages. Also includes proxy metadata (country used, provider, whether JS was rendered).","failureModes":["Invalid or malformed URL returns 400 error","Invalid 2-letter country code returns 400 error","Target page is unreachable or returns 5xx — propagated as error","Target page has no links — returns empty arrays with zero summary stats","Page requires JavaScript rendering — static content only returned, no JS execution (use /proxy/render instead)","External proxy rotation failure — may return 502 or timeout"],"whenToPreferThis":"Use this endpoint when the target page actively blocks Minifetch's standard proxy (which obeys robots.txt) and you need link extraction specifically. Choose this over the standard /extract/url-links when robots.txt compliance is a blocker. Choose this over /proxy/render when the page does not require JavaScript execution — this endpoint is faster and cheaper since it skips JS rendering. Use the optional country parameter when content is geo-restricted.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-14T06:41:37.354Z","isFirstParty":false,"canonicalSlug":"minifetch-proxy-url-link-extractor-00c90e64"}