POST /grader/{episode_id} is the evaluation endpoint that scores your agent’s submitted pipeline output against the hidden ground truth snapshot that was locked in server memory at the time /reset was called. The grader performs a left-join on customer_id, checks every non-key column for exact match (case-insensitive, whitespace-stripped), and computes a normalized accuracy score clamped to the range [0.01, 0.99]. A penalty is applied for excess rows beyond the truth row count.
Call this endpoint manually after your pipeline is complete — once /reset has been called and a SUBMIT_PIPELINE action has been executed with done = true.
Path Parameters
string
required
The UUID of the episode to grade. This is the value returned in
observation.last_action_feedback after /reset (format:
"Environment reset natively for task t1. Episode UUID: {uuid}"),
and also available from GET /state as episode_id.Query Parameters
string
required
The name of the source DataFrame to grade — the same value you passed as
final_source in your SUBMIT_PIPELINE action. For tasks t1, this is
typically "web_forms". For tasks t2 and t3, this is typically
"merged_output".string
required
The task identifier. Must be one of
"t1", "t2", or "t3". This
controls which truth key the grader loads from the stored snapshot:
"web_forms" for t1, or "merged_output" for t2 and t3.Request
All parameters are passed as URL query strings. No request body is needed.Response
Response Fields
string
The episode UUID echoed back from the path parameter. Use this to correlate
grader results with specific reset calls in multi-episode evaluation runs.
string
The task identifier echoed back from the query parameter (
t1, t2, or
t3). Confirms which truth schema was used during evaluation.float
The normalized accuracy score for the agent’s submitted DataFrame. Range is
always
[0.01, 0.99]. See the Scoring Formula and Score Interpretation
sections below for how this value is computed and what it means.integer
The number of rows in the agent’s submitted DataFrame at the time of
grading. Compare this against the truth row count to detect whether your
pipeline produced extra rows (bots not removed, duplicates retained) or
fewer rows than expected (over-aggressive dropping).
Error Responses
Returned when
episode_id does not match any cached ground truth snapshot.
This occurs when /reset was never called, the server was restarted
(clearing in-memory state), or you supplied an incorrect UUID.Returned when
task_id is not one of the recognized values (t1, t2, t3).Returned when the agent has not yet executed a
SUBMIT_PIPELINE action
that set done = true. The env.final_df pointer is None.An internal state error indicating the truth snapshot exists but does not
contain the expected truth key for the given
task_id. This should not
occur in normal operation.Scoring Formula
The grader executes the following algorithm inevaluate_dataframes() from server/graders.py:
1. Deduplicate agent output on customer_id
Duplicate customer_id values in the agent’s DataFrame are dropped before comparison to prevent Cartesian explosion during the merge.
2. Left-join truth against agent on customer_id
customer_id in the truth are excluded.
3. Column-by-column exact match
For every non-key column in the truth schema, both the truth value and the agent value are lowercased and whitespace-stripped, then compared for exact equality:
True.
4. Compute base score
-0.15 penalty per extra row:
Because the grader uses case-insensitive, stripped exact-string matching,
your pipeline must produce values that match the ground truth format
precisely after normalization. Dates must be
YYYY-MM-DDTHH:MM:SS (ISO 8601
via TO_DATETIME_ISO). Phones must be +{digits} with no separators
(E.164 via EXTRACT_NUMBERS). Emails must be fully lowercase with no
surrounding whitespace.