Why Dense Rewards Matter for Data Engineering Agents
In a sparse reward setting, the agent receives a signal only at the very end of the episode — when it submits the pipeline and the grader compares its output against the hidden truth. For a typical Task 1 episode with 40+ rows and 5 columns, the space of possible cleaning strategies is enormous. An agent operating under sparse rewards would need to randomly explore this space for hundreds of episodes before receiving any useful signal. This is computationally intractable for LLM-based agents operating with API budget constraints. Dense rewards solve this by providing immediate feedback after every action. When your agent correctly identifies that an email column contains uppercase values and appliesLOWERCASE_STRIP, it receives a small positive reward proportional to how many values were actually changed. When it makes a mistake (referencing a nonexistent column, issuing a malformed SQL query), it receives a negative reward immediately. This creates a continuous learning signal that guides the agent toward productive sequences of actions.
The key design insight is that the dense rewards are heuristic proxies — they measure likely improvement without consulting the hidden ground truth. They are designed to be aligned with true quality improvement in expectation, but they are not the true score.
The step rewards you accumulate during an episode are completely separate from the grader score (0.01–0.99) produced at
SUBMIT_PIPELINE time. A high cumulative step reward does not guarantee a high grader score. A low cumulative step reward does not preclude a high grader score. The grader is the true and final measure of pipeline quality.Complete Step Reward Schedule
Progressive Rewards and Systematic Cleaning
The reward structure is intentionally designed to encourage a systematic cleaning sequence rather than random action selection. Notice the progression:-
Profiling earns
+0.02— rewarding the agent for gathering information before acting. An agent that blindly standardizes without profiling misses this reward. -
Standardization scales with impact — the reward for
STANDARDIZE_COLUMNis computed asmin(0.05, 0.01 + improvement_ratio × 0.1), whereimprovement_ratiois the fraction of rows that actually changed. A column that was already clean earns only+0.01; a heavily dirty column earns up to+0.05. This encourages the agent to standardize the most problematic columns first. -
Missing value handling rewards proportionally — the reward for
HANDLE_MISSINGscales with the null density of the column. Fixing a column with 11 nulls out of 40 rows earns more than fixing a column with 1 null. This encourages the agent to prioritize high-null columns. -
Deduplication reward is proportional to yield —
(removed / start_len) × 0.2means a dataset with many duplicates yields high reward when cleaned. If a source has few duplicates, deduplication earns little reward, implicitly signaling it may not be the priority. -
SQL execution earns
+0.1— the highest individual step reward (excluding the terminal submission reward), reflecting the high value of complex multi-source operations that SQL enables.
The Premature Submission Penalty
AttemptingSUBMIT_PIPELINE when step_count < MIN_STEPS_BEFORE_SUBMIT (where MIN_STEPS_BEFORE_SUBMIT = 3) results in a -0.15 penalty and done=False. The episode continues.
This is the largest per-step penalty in the system, designed to strongly discourage agents from submitting immediately after reset. The -0.15 penalty costs more than the combined reward of roughly 7 successful PROFILE_SOURCE calls, creating a significant incentive to perform meaningful cleaning work.
Terminal Reward: Schema Match Heuristic
WhenSUBMIT_PIPELINE succeeds (i.e., step_count >= 3), the environment immediately computes a fast heuristic reward based solely on column presence:
phone column:
The Grader: True Terminal Scoring
The grader inserver/graders.py computes the real evaluation score. It is triggered server-side when SUBMIT_PIPELINE succeeds, and it compares your final DataFrame against the hidden ground truth using row-level, column-level exact matching.
Grader Algorithm (Step by Step)
Step 1: Deduplication guard The grader enforces uniqueness on the join key before evaluating, to prevent a Cartesian explosion if the agent failed to deduplicate:len(final_df) is measured before this deduplication).
Step 2: Left join truth onto agent output
- Every row in
truth_dfis represented in the merged result - If a
customer_idexists in truth but not in your output, that row has all_agentcolumns as NaN - If a
customer_idexists in your output but not in truth, it does not appear in the merged result (but counts toward the row count penalty)
truth_df:
.fillna("").astype(str).str.strip().str.lower() before comparing. This means:
- NaN values in both truth and agent become
""and match each other - Trailing whitespace is stripped on both sides
- Case differences are ignored at the comparison stage
0.15 per extra row. This is the mechanism that punishes agents for failing to remove bot rows and duplicate entries.
For example: if truth has 40 rows but your final DataFrame has 48 rows (8 bot rows not removed), your penalty is 8 × 0.15 = 1.2. This can devastate an otherwise good score.
Step 6: Final score clamping
[0.01, 0.99]. A perfect clean pipeline scores 0.99 (never 1.0 by design). An empty or severely broken submission scores 0.01.
Complete Grader Formula
Reward Accumulation: A Sample Episode
The following table traces the reward accumulation for a well-executed Task 1 episode, illustrating how step rewards build up alongside the terminal grader score.
In this scenario, the step reward (
+0.61) and the grader score (0.94) are high and aligned — because the systematic cleaning approach that earns step rewards genuinely improves data quality. However, the two numbers are computed completely independently.
Why Extra Rows Are Penalized So Heavily
The0.15 penalty per extra row is deliberately steep. Consider a scenario where an agent submits a final DataFrame with all 40 truth rows correctly cleaned plus 8 bot rows that were not removed:
Step Rewards vs. Grader Score: Key Differences
In research settings, cumulative step reward is useful as a proxy metric during training — it provides a signal after every episode without requiring ground truth access. The grader score is the authoritative number for benchmarking and comparison across agent implementations.