{"uid":"cap_nyMasyhJyjLKIfthmamwz","slug":"latitude-10-agent-eval-kit-12e3c018","name":"Latitude 10 Agent Eval Kit","description":"I help B2B software get onto the agent platforms: a fixed-price sprint that turns your API into a production MCP server and submits it to AWS Marketplace, the Claude directory, the official MCP Registry and the other directories where agents find tools.","url":"https://latitude10.tech/api/agent-products/agent-eval-kit?utm_source=zero.xyz","method":"GET","headers":{},"bodySchema":{"type":"object","properties":{"properties":{"type":"string"}}},"responseSchema":{"type":"json","example":{"payment":{"network":"eip155:8453","transaction":"0x…"},"product":{"name":"Latitude 10 Agent Eval Kit for Node.js","slug":"agent-eval-kit","content":"https://latitude10.tech/api/kit/download?t=…","version":"1.0.3","delivery":"kit","description":"For Node.js: tests for what your agent does, not just what it says: a Vitest library and a CLI for TypeScript agents. Each run records a trajectory (model calls, tool calls with arguments and results, payments, tokens, cost, latency, outcome; the Agent Trace Kit format) and 25 matchers check it: which tools were called, in what order and with what arguments, cost, token, latency and step budgets, retries, payments, leaks, approvals and refusals. Mocked, failing, replayed and prompt-injected tools (a dated corpus of 24 original payloads) and a scripted model, so most suites run in CI with no API key. An LLM judge (Anthropic by default, or OpenAI or any compatible endpoint) with rubrics, votes and a cache; skipped loudly without a key. Trials with pass@k and pass^k, YAML suites, baselines that fail CI on regressions, JUnit and a job summary, and agent-eval import to turn a Trace Kit session into a regression test. Recording adapters for the Anthropic SDK, Claude Agent SDK, OpenAI SDK, OpenAI Agents SDK, Vercel AI SDK and MCP. Includes a setup wizard (node setup.mjs) running 85 example cases in seven folders (tool selection, hallucination, permissions, retries, failures, prompt injection, regression), agent-eval init for an existing project, and a GitHub Actions workflow.","contentExpiresAt":"2026-10-02T12:00:00.000Z"}}},"example":null,"exampleRequest":null,"tags":["x402"],"displayCostAmount":"49","displayCostAsset":"USDC","priceDynamic":false,"priceHint":null,"priceStatus":"priced","priceSource":"probe","requiresHandshake":false,"reviewCount":0,"rating":{"score":"0.00","successRate":"0.00","reviews":0,"stars":null,"state":"unrated"},"availabilityStatus":"unknown","priceObserved":null,"sessionDeposit":null,"pricing":{"kind":"static","summary":"$49/call","primary":{"kind":"static","protocol":"x402","network":"base","amountUsd":"49","per":"call","confidence":"exact"},"accepted":[{"kind":"static","protocol":"x402","network":"base","amountUsd":"49","per":"call","confidence":"exact"}]},"paymentMethods":[{"uid":"pm_PVaIOlTix5m5krlmi771G","protocol":"x402","methodType":"crypto","chain":"base","mode":"charge","costAmount":"49","costPer":"request","priority":0,"asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","unit":"request","depositMicros":null,"planRef":null}],"brandName":null,"brandSlug":null,"brandBaseUrl":null,"brandDocsUrl":null,"whatItDoes":"Downloads a TypeScript/Vitest testing library and CLI for evaluating AI agent behavior via trajectory recording and 25 behavioral matchers","exampleAgentPrompt":"Can you get me the Latitude 10 Agent Eval Kit so I can set up trajectory-based testing for my TypeScript agent — I want to check tool call order, cost budgets, and prompt injection resistance in CI without needing live API keys?","exampleUseCases":[{"title":"CI pipeline for agent tool validation","prompt":"I want to add automated tests to my CI pipeline that verify my TypeScript agent calls the right tools in the right order and stays within cost and latency budgets — can you download the Agent Eval Kit for me so I can get started?"},{"title":"Prompt injection regression suite","prompt":"I'm worried about prompt injection attacks on my agent — can you grab the Agent Eval Kit from Latitude 10 so I can run its corpus of 24 injection payloads against my agent and set up a baseline that fails CI if it regresses?"},{"title":"LLM judge evaluation for agent outputs","prompt":"I need to evaluate my agent's outputs using an LLM judge with rubrics and voting — can you download the Agent Eval Kit and point me to the judge setup so I can configure it with my Anthropic key?"}],"resultDescription":"Returns a signed download URL for the Agent Eval Kit (a ZIP containing a Vitest library, CLI, 85 example test cases across 7 folders, recording adapters for Anthropic/OpenAI/Vercel AI/MCP SDKs, a GitHub Actions workflow, and setup wizard), along with product metadata including name, version, delivery type, description, content expiry timestamp, and payment/transaction details for the x402 micropayment.","failureModes":["Payment not completed or incorrect USDC amount — returns 402 Payment Required","Download URL expired (contentExpiresAt exceeded) — link no longer valid","Invalid or malformed properties parameter — returns 400 Bad Request","Network or server error — returns 5xx with no kit delivered","Payment transaction not confirmed on Base network — delivery blocked"],"whenToPreferThis":"Choose this endpoint when you need a self-contained, offline-capable agent testing framework for TypeScript agents that records full execution trajectories (tool calls, model calls, costs, latencies, payments) and provides deterministic behavioral assertions — especially when you want to run most tests in CI without live API keys, need prompt injection testing with a real corpus, or want LLM-judge evaluation with caching. Prefer this over generic unit testing libraries or manual agent QA when you need structured regression baselines, pass@k trials, and JUnit-compatible CI output.","instructions":null,"reviewSummary":null,"reviewSummaryHighlights":null,"reviewSummaryConcerns":null,"reviewSummaryGeneratedAt":null,"activationCount":0,"lastUsedAt":null,"lastSuccessfullyRanAt":null,"lastHealthCheckAt":"2026-10-11T00:45:07.271Z","isFirstParty":false,"canonicalSlug":"latitude-10-agent-eval-kit-12e3c018"}