Skip to main content
The data_generator.py module is the core of the benchmark’s difficulty system. Every time the environment resets, it procedurally generates a fresh batch of messy CRM data by following a strict four-phase pipeline: it creates clean ground-truth records, feeds them through a chaos engine that corrupts every column, injects realistic-looking bot rows, and then shuffles the combined result into a random order. Because this pipeline runs at reset time with unseeded randomness (Faker.seed(None), random.seed(None)), no two episodes are identical — your agent must handle the full distribution of corruption, not a fixed dataset.

Generation Pipeline Overview

Every episode follows the same four-phase pipeline. The diagram below shows how clean ground-truth rows are corrupted, augmented with bots, and exposed to the agent — while the original clean rows are stored separately for grading. The agent never has direct access to GLOBAL_TRUTH_STORE. It can only interact with the messy DataFrames exposed through available_sources in the observation. The grader compares your submitted merged_output (or final source) against the hidden truth snapshot to compute accuracy.

Phase 1 — create_base_truth(size)

create_base_truth(size) generates size clean, fully-populated rows using the Faker library. This is the pristine ground truth before any corruption is applied.
Every field is fully populated. Emails are already lowercased. Phones already carry the + E.164 prefix. Dates are valid ISO 8601. This is the ideal target schema your pipeline should converge toward.

Base truth row counts by task


Phase 2 — correlate_truth() — The Chaos Engine

correlate_truth(truth_list) iterates over every clean row and applies five independent corruption steps. Each step is probabilistic and operates on a different column family. Understanding the exact probability thresholds and mutation shapes is essential for writing the correct cleaning strategy.

Step 1 — Missing Data Chaos

Two independent coin flips remove the email and phone values.
On average, 1 in 5 rows will have no email and 1 in 5 will have no phone. These two events are independent, so roughly 4 % of rows lose both. Rows that lose their email become effectively un-joinable unless a fuzzy name/phone strategy is used.

Step 2 — String Chaos

Name and email receive casing and whitespace mutations separately:
A name like "Jane Doe" becomes either " Jane Doe " (whitespace) or "JANE DOE" (uppercase). An email like "jane.doe@gmail.com" becomes "JANE.DOE@GMAIL.COM" with 50 % probability. Cleaning prescription: Apply LOWERCASE_STRIP to email and name before any join operation.

Step 3 — Date Chaos (Five Branches)

signup_date is replaced according to a single random draw against four nested thresholds:
The probability breakdown across all rows in a source:
The TO_DATETIME_ISO standardization strategy handles all four cases automatically. Parseable slash-separated dates are converted to ISO 8601. The three invalid/missing cases are mapped to an empty string "".

Step 4 — Phone Chaos (Four Branches)

Phone mutation only applies to rows where dirty["phone"] is not None after Step 1. The raw digits are first extracted by stripping the + prefix, then one of three mutations is applied:
Cleaning prescription: Use EXTRACT_NUMBERS first (strips ext. and parenthetical formatting, but note the environment’s implementation also handles the ext pattern), then apply PHONE_E164 to re-attach the + prefix.

Step 5 — Deduplication Chaos (Fuzzy Duplicate Injection)

After appending the corrupted row to messy, the engine has a 25 % chance of also appending a second, fuzzy-duplicate row:
The duplicate row deliberately has email = None, which makes EXACT_EMAIL deduplication alone insufficient. The duplicate is detectable by matching the last name plus phone number — exactly what FUZZY_NAME_PHONE is designed for. Across a 40-row truth list, you can expect roughly 10 extra fuzzy-duplicate rows after this step.

Phase 3 — generate_bot_row()

Bot rows are injected after correlate_truth() completes. Each bot row is constructed with fixed structural anomalies:

Bot constants

Bot rows do not go through correlate_truth(). They are appended directly to the messy list before the shuffle.

Phase 4 — Shuffle and DataFrame Construction

After all rows (corrupted real + bots) are assembled, the list is shuffled uniformly at random and converted to a Pandas DataFrame:
The shuffle ensures that bot rows and duplicate rows are not conveniently positioned at the end of the DataFrame — your agent must handle them appearing anywhere.

Task-Specific Generation Flows

Hidden truth: {"web_forms": DataFrame(original 40 clean rows)}Expected messy DataFrame size before cleaning: ~58 rows (40 base + ~10 fuzzy duplicates + 8 bots).

Hidden Truth vs. Messy Input

The ground truth is captured before correlate_truth() runs and stored in GLOBAL_TRUTH_STORE keyed by the episode UUID. Your agent has no direct access to it. The only signal you receive is the reward feedback from the grader when you submit.
When you call SUBMIT_PIPELINE, the environment retrieves your designated final_source DataFrame and compares it row-by-row against the hidden truth. Your cleaning pipeline must reconstruct the truth as closely as possible purely from the observable messy data.
Because randomness is unseeded at the start of each episode, you cannot hard-code expected row counts or specific corruption values. Your pipeline logic must be robust to the full probability distribution described above.