{"uid":"cap_lmDQSPIMoO0AWXAOxyPOC","slug":"402utils-text-chunker-token-aware-rag-splitter-67654517","name":"402utils Text Chunker (Token-Aware RAG Splitter)","description":"Split a long text into token-sized chunks for RAG ingestion. Chunks are measured in REAL tokens (tiktoken o200k_base/cl100k_base), honour a token overlap, and cut on sentence, paragraph or word boundaries (strategy). Returns each chunk text, its exact token count and char offsets. The natural composite of token-count; counts are OpenAI-style estimates (no Claude tokenizer exists). Provided text only; nothing is fetched. Over 1 MB is truncated.","url":"https://402utils.com/v1/chunk","method":"POST","headers":{},"bodySchema":{"type":"object","properties":{"text":{"type":"string","description":"The text to chunk (max 1 MB UTF-8; larger is truncated)."},"overlap":{"type":"integer","default":50,"minimum":0,"description":"Token overlap between consecutive chunks (must be < maxTokens)."},"encoding":{"enum":["o200k_base","cl100k_base"],"type":"string","default":"o200k_base","description":"tiktoken encoding used to size chunks."},"strategy":{"enum":["sentence","paragraph","token"],"type":"string","default":"sentence","description":"Boundary chunks cut on: sentence (semantic), paragraph (blank-line), or token (word)."},"maxTokens":{"type":"integer","default":512,"maximum":8192,"minimum":1,"description":"Maximum tokens per chunk."}}},"responseSchema":null,"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"0.002","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$0.002/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.002","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"0.002","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_DRZLMj5-vXsepsxak-Igm","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"0.002","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Splits input text into token-measured chunks with configurable size, overlap, and boundary strategy (sentence/paragraph/token) for RAG ingestion, returning each chunk's text, token count, and character offsets.","exampleAgentPrompt":"Split this 10,000-word article into sentence-boundary chunks of at most 512 tokens each, with a 50-token overlap, using the o200k_base encoding — I need the chunk text, exact token counts, and character offsets for each piece so I can ingest them into my vector store.","exampleUseCases":null,"resultDescription":"A list of chunk objects, each containing: the chunk text string, its exact token count (measured via tiktoken), and the start/end character offsets within the original text. Input over 1 MB is silently truncated before processing.","failureModes":["Text over 1 MB is truncated without error — agent may receive incomplete chunking","overlap >= maxTokens causes a validation error","Invalid encoding value returns 422 unprocessable entity","Empty text body returns empty chunks array","Very short text with large maxTokens returns a single chunk covering entire input"],"whenToPreferThis":"Choose this endpoint when you need real, tiktoken-measured token counts (not character estimates) for RAG pipelines targeting OpenAI-compatible embedding or completion models. Prefer it over regex-based splitters when semantic boundary respect (sentence or paragraph cuts) and accurate overlap are required. Best for preprocessing before vector DB ingestion. Not suitable if you need Claude/Anthropic tokenizer estimates.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-09-14T18:42:21.527Z","isFirstParty":false}