Generate eval cases from the suite's tools

Discovers the suite’s server tools over a live MCP connection, generates cases against them, and persists them — the only edit route that connects to a server, and the only one that SPENDS ORG CREDITS. Synchronous: connect, generate, persist, disconnect, respond.

An environment-based suite generates against that environment’s closed server set, so the cases match the tools its runs will actually see.

Pass x-mcpjam-idempotency-key to make a retry safe: drafts are recorded backend-side BEFORE any case is persisted, so a replay reuses them instead of spending credits again, and each case is persisted under a derived per-item key so the loop is resumable.

POST /projects/{projectId}/eval-suites/{suiteId}/cases/generate

Generate eval cases from the suite's tools

cURL

curl --request POST \
  --url https://app.mcpjam.com/api/v1/projects/{projectId}/eval-suites/{suiteId}/cases/generate \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "servers": [
    "<string>"
  ],
  "environmentId": "<string>",
  "caseModels": [
    {
      "model": "<string>",
      "provider": "<string>"
    }
  ],
  "caseMix": {
    "simple": 5,
    "multiTool": 5,
    "multiTurn": 5,
    "complex": 5,
    "negative": 5
  },
  "varyUserStyles": true
}
'

200

Example

{ "generationModel": "<string>", "created": [ { "id": "<string>", "title": "<string>", "steps": [ { "id": "<string>", "prompt": "<string>", "serverName": "<string>", "toolName": "<string>", "arguments": {}, "action": {}, "assertion": {} } ], "iterations": 5, "isNegative": true, "models": [ { "model": "<string>", "provider": "<string>" } ], "expectedOutput": "<string>", "scenario": "<string>", "matchOptions": {}, "checks": { "list": [ {} ] }, "createdAt": 123, "updatedAt": 123 } ], "counts": { "normal": 123, "negative": 123 }, "skipped": [ {} ] }

Authorizations

Authorization · string · header · required

MCPJam API key (sk_…). Create one at Settings → API keys. Guest sessions cannot use the API, and API keys cannot manage other API keys.

Headers

x-mcpjam-idempotency-key · string

Makes a retry replay the recorded drafts instead of spending credits again.

Path Parameters

projectId · string · required

ID of the hosted project that contains the server.

suiteId · string · required

Eval suite ID, as returned by POST /eval-runs.

Body

application/json

AI-generate cases from the suite's server tools and persist them. SPENDS ORG CREDITS.

mode · enum<string>

Superseded by caseMix when that is present.

Available options: normal, negative

servers · string[]

Server ids or names to discover tools from. Ignored when the suite is environment-based.

Minimum string length: 1

environmentId · string

Discover tools from this attached environment's closed server set, so generated cases are written against the tools the suite's runs will actually see.

Minimum string length: 1

caseModels · object[]

Show child attributes

caseMix · object

Per-bucket case counts. Omitted buckets inherit the default mix; the backend bounds each bucket and the total.

Show child attributes

varyUserStyles · boolean

Condition generated cases on a range of user styles so the queries read like different users wrote them.

Response

200

application/json

The generated cases.

generationModel · string · required

created · object[] · required

Show child attributes

counts · object · required

Show child attributes

skipped · object[]

Drafts that were generated but failed to persist. Surfaced rather than silently dropped.