available_sources field of the observation. Each source is a distinct in-memory Pandas DataFrame with its own column schema, data quality characteristics, and corruption patterns introduced by the chaos engine. Understanding which sources appear in which tasks — and exactly how their data is distorted — is the prerequisite for building a reliable cleaning pipeline.
Source Availability by Task
The three task tiers progressively introduce more sources and more complex inter-source conflicts. Use this table as a quick reference when deciding which sources to load and clean for a given episode.At the start of each episode the environment sets
available_sources in the observation to exactly these source names. Pass the exact name string (e.g., "web_forms") as the source field in every action payload.web_forms
web_forms is the primary source in Tasks 1 and 2. It is produced by running create_base_truth() through the full correlate_truth() chaos engine, then appending synthetic bot rows. Expect severe corruption across every column.
Schema
Date Chaos — signup_date
The chaos engine applies one of four mutations to everysignup_date value based on a uniform random draw:
Phone Chaos — phone
When a phone value is present (80 % of rows), the engine applies one of three mutations:salesforce
salesforce appears only in Task 3. It is the canonical source for email and customer_id according to the task’s CONFLICT_RULES. The data generator samples 90 % of the base truth rows into Salesforce, so up to 10 % of real customers may be absent from this source.
Schema
salesforce does not go through correlate_truth(). Its data is cleaner than web_forms, but it still contains bot rows and can have None phone values.legacy_db
legacy_db appears in both Task 2 and Task 3, but its schema differs between the two tasks.
Task 2 Schema
In Task 2,legacy_db is generated from a second independent call to create_base_truth(20) followed by the full correlate_truth() chaos engine. Its schema is identical to web_forms:
Task 3 Schema
In Task 3,legacy_db is generated directly from the base truth using a different mapping that preserves old enterprise conventions. It is sampled at 85 % of the 60 base rows.
web_leads
web_leads appears only in Task 3. It contains fresher phone data than Salesforce, making it the preferred source for the phone column per CONFLICT_RULES. The generator samples 80 % of base truth rows, so 20 % of customers may be absent.
Schema
Bot Injection Patterns
Both the easy/medium tasks and the hard task inject synthetic bot rows into every source. Bots are designed to look plausible at a glance — valid email format, reasonable-looking names — but carry several identifying signals.Bot Identifiers
Bot Counts per Task
Filtering Bots in SQL
Use the known bot domains in aWHERE clause to exclude bot rows during SQL-based merges:
legacy_db in Task 3 where emails are uppercase, apply LOWER() first:
You can also filter bots using
customer_id LIKE 'BOT%' or phone = '+00000000000' or signup_date = '1970-01-01'. Combining multiple signals in a single WHERE clause produces the most robust removal.