Skip to main content
POST /grader/{episode_id} is the evaluation endpoint that scores your agent’s submitted pipeline output against the hidden ground truth snapshot that was locked in server memory at the time /reset was called. The grader performs a left-join on customer_id, checks every non-key column for exact match (case-insensitive, whitespace-stripped), and computes a normalized accuracy score clamped to the range [0.01, 0.99]. A penalty is applied for excess rows beyond the truth row count. Call this endpoint manually after your pipeline is complete — once /reset has been called and a SUBMIT_PIPELINE action has been executed with done = true.

Path Parameters

string
required
The UUID of the episode to grade. This is the value returned in observation.last_action_feedback after /reset (format: "Environment reset natively for task t1. Episode UUID: {uuid}"), and also available from GET /state as episode_id.

Query Parameters

string
required
The name of the source DataFrame to grade — the same value you passed as final_source in your SUBMIT_PIPELINE action. For tasks t1, this is typically "web_forms". For tasks t2 and t3, this is typically "merged_output".
string
required
The task identifier. Must be one of "t1", "t2", or "t3". This controls which truth key the grader loads from the stored snapshot: "web_forms" for t1, or "merged_output" for t2 and t3.

Request

All parameters are passed as URL query strings. No request body is needed.
For Task 2 or Task 3:

Response

Response Fields

string
The episode UUID echoed back from the path parameter. Use this to correlate grader results with specific reset calls in multi-episode evaluation runs.
string
The task identifier echoed back from the query parameter (t1, t2, or t3). Confirms which truth schema was used during evaluation.
float
The normalized accuracy score for the agent’s submitted DataFrame. Range is always [0.01, 0.99]. See the Scoring Formula and Score Interpretation sections below for how this value is computed and what it means.
integer
The number of rows in the agent’s submitted DataFrame at the time of grading. Compare this against the truth row count to detect whether your pipeline produced extra rows (bots not removed, duplicates retained) or fewer rows than expected (over-aggressive dropping).

Error Responses

Returned when episode_id does not match any cached ground truth snapshot. This occurs when /reset was never called, the server was restarted (clearing in-memory state), or you supplied an incorrect UUID.
Returned when task_id is not one of the recognized values (t1, t2, t3).
Returned when the agent has not yet executed a SUBMIT_PIPELINE action that set done = true. The env.final_df pointer is None.
An internal state error indicating the truth snapshot exists but does not contain the expected truth key for the given task_id. This should not occur in normal operation.

Scoring Formula

The grader executes the following algorithm in evaluate_dataframes() from server/graders.py: 1. Deduplicate agent output on customer_id Duplicate customer_id values in the agent’s DataFrame are dropped before comparison to prevent Cartesian explosion during the merge. 2. Left-join truth against agent on customer_id
Every truth row is preserved. Agent rows without a matching customer_id in the truth are excluded. 3. Column-by-column exact match For every non-key column in the truth schema, both the truth value and the agent value are lowercased and whitespace-stripped, then compared for exact equality:
A row is marked correct only if all column comparisons are True. 4. Compute base score
5. Apply excess-row penalty Each row in the agent’s output beyond the truth row count incurs a -0.15 penalty per extra row:
6. Clamp to valid range
Because the grader uses case-insensitive, stripped exact-string matching, your pipeline must produce values that match the ground truth format precisely after normalization. Dates must be YYYY-MM-DDTHH:MM:SS (ISO 8601 via TO_DATETIME_ISO). Phones must be +{digits} with no separators (E.164 via EXTRACT_NUMBERS). Emails must be fully lowercase with no surrounding whitespace.

Score Interpretation

Calling the Grader Programmatically

Ground truth snapshots are held in server process memory and are not persisted to disk. If the server process restarts between /reset and your grader call, the snapshot is lost and the grader will return a 404. Always complete a full episode (reset → step → submit → grade) within a single uninterrupted server session.