data_generator.py module is the core of the benchmark’s difficulty system. Every time the environment resets, it procedurally generates a fresh batch of messy CRM data by following a strict four-phase pipeline: it creates clean ground-truth records, feeds them through a chaos engine that corrupts every column, injects realistic-looking bot rows, and then shuffles the combined result into a random order. Because this pipeline runs at reset time with unseeded randomness (Faker.seed(None), random.seed(None)), no two episodes are identical — your agent must handle the full distribution of corruption, not a fixed dataset.
Generation Pipeline Overview
Every episode follows the same four-phase pipeline. The diagram below shows how clean ground-truth rows are corrupted, augmented with bots, and exposed to the agent — while the original clean rows are stored separately for grading. The agent never has direct access toGLOBAL_TRUTH_STORE. It can only interact with the messy DataFrames exposed through available_sources in the observation. The grader compares your submitted merged_output (or final source) against the hidden truth snapshot to compute accuracy.
Phase 1 — create_base_truth(size)
create_base_truth(size) generates size clean, fully-populated rows using the Faker library. This is the pristine ground truth before any corruption is applied.
+ E.164 prefix. Dates are valid ISO 8601. This is the ideal target schema your pipeline should converge toward.
Base truth row counts by task
Phase 2 — correlate_truth() — The Chaos Engine
correlate_truth(truth_list) iterates over every clean row and applies five independent corruption steps. Each step is probabilistic and operates on a different column family. Understanding the exact probability thresholds and mutation shapes is essential for writing the correct cleaning strategy.
Step 1 — Missing Data Chaos
Two independent coin flips remove the email and phone values.Step 2 — String Chaos
Name and email receive casing and whitespace mutations separately:"Jane Doe" becomes either " Jane Doe " (whitespace) or "JANE DOE" (uppercase). An email like "jane.doe@gmail.com" becomes "JANE.DOE@GMAIL.COM" with 50 % probability.
Cleaning prescription: Apply LOWERCASE_STRIP to email and name before any join operation.
Step 3 — Date Chaos (Five Branches)
signup_date is replaced according to a single random draw against four nested thresholds:
The
TO_DATETIME_ISO standardization strategy handles all four cases automatically. Parseable slash-separated dates are converted to ISO 8601. The three invalid/missing cases are mapped to an empty string "".Step 4 — Phone Chaos (Four Branches)
Phone mutation only applies to rows wheredirty["phone"] is not None after Step 1. The raw digits are first extracted by stripping the + prefix, then one of three mutations is applied:
Cleaning prescription: Use
EXTRACT_NUMBERS first (strips ext. and parenthetical formatting, but note the environment’s implementation also handles the ext pattern), then apply PHONE_E164 to re-attach the + prefix.
Step 5 — Deduplication Chaos (Fuzzy Duplicate Injection)
After appending the corrupted row tomessy, the engine has a 25 % chance of also appending a second, fuzzy-duplicate row:
email = None, which makes EXACT_EMAIL deduplication alone insufficient. The duplicate is detectable by matching the last name plus phone number — exactly what FUZZY_NAME_PHONE is designed for. Across a 40-row truth list, you can expect roughly 10 extra fuzzy-duplicate rows after this step.
Phase 3 — generate_bot_row()
Bot rows are injected aftercorrelate_truth() completes. Each bot row is constructed with fixed structural anomalies:
Bot constants
correlate_truth(). They are appended directly to the messy list before the shuffle.
Phase 4 — Shuffle and DataFrame Construction
After all rows (corrupted real + bots) are assembled, the list is shuffled uniformly at random and converted to a Pandas DataFrame:Task-Specific Generation Flows
- Task 1 — Easy
- Task 2 — Medium
- Task 3 — Hard
{"web_forms": DataFrame(original 40 clean rows)}Expected messy DataFrame size before cleaning: ~58 rows (40 base + ~10 fuzzy duplicates + 8 bots).Hidden Truth vs. Messy Input
The ground truth is captured beforecorrelate_truth() runs and stored in GLOBAL_TRUTH_STORE keyed by the episode UUID. Your agent has no direct access to it. The only signal you receive is the reward feedback from the grader when you submit.
SUBMIT_PIPELINE, the environment retrieves your designated final_source DataFrame and compares it row-by-row against the hidden truth. Your cleaning pipeline must reconstruct the truth as closely as possible purely from the observable messy data.