Skip to main content
Deduplication is one of the highest-impact steps in your CRM pipeline. The chaos engine’s correlate_truth() function appends a fuzzy-duplicate row with a 25 % probability for every base truth record — which means a 40-row truth list can produce roughly 10 additional duplicate rows before bot injection even begins. These duplicates have subtly different names but the same last name, and deliberately have email = None to resist naive email-matching. If you submit without deduplicating, the grader penalizes you proportionally for every extra row beyond the expected truth count. Eliminating duplicates early and cleanly is essential to maximizing your episode score.

Why Duplicates Are Hard to Remove

The chaos engine constructs duplicates specifically to defeat simple deduplication:
  1. The duplicate row has a different first name (generated by fake.first_name()), so exact-name matching fails.
  2. The duplicate row has email = None, so email-based deduplication cannot match it to the original.
  3. The duplicate inherits all other corruption from the original row — same uppercase casing, same whitespace, same phone chaos — making it visually indistinguishable from a legitimate record.
The only reliable signals available for matching these duplicates are:
  • Last name (shared between original and duplicate)
  • Phone number (shared when both rows had a non-None phone after Step 1 chaos)
This is precisely why the FUZZY_NAME_PHONE strategy exists.

The DEDUPLICATE Action

Send a DEDUPLICATE action with a source and deduplication_strategy:
The environment mutates the source DataFrame in place, removing duplicate rows, and returns a feedback string and a proportional reward. After deduplication, VIEW_SOURCE on the same source will reflect the reduced row count.

EXACT_EMAIL

EXACT_EMAIL calls DataFrame.drop_duplicates(subset=["email"]) on the target source. It keeps the first occurrence of each email value and drops all subsequent rows with the same email.

Behavior

  • Match criterion: Two rows are duplicates if and only if their email values are identical strings.
  • Null handling: Rows with None or NaN in the email column are not considered duplicates of each other — Pandas’ drop_duplicates treats NaN as unique. All None-email rows are kept.
  • Order sensitivity: Keeps the first occurrence (lowest integer index in the DataFrame). Because the DataFrame is shuffled at reset time, “first” is effectively random with respect to the original truth order.

JSON Action

Expected Feedback

where N is the number of rows dropped.

Reward

The reward is proportional to how many rows were removed:
A source with 10 duplicates out of 50 rows returns (10/50) × 0.2 = 0.04.

When to Use It

EXACT_EMAIL is your primary deduplication tool in Task 1. After standardizing email with LOWERCASE_STRIP, all casing inconsistencies are resolved, making exact matching reliable. Run the sequence:
  1. STANDARDIZE_COLUMN → email → LOWERCASE_STRIP
  2. DEDUPLICATE → EXACT_EMAIL
  3. HANDLE_MISSING → email → DROP_ROW (remove bot rows and None-email duplicates)
The 25 % fuzzy duplicates have email = None, so EXACT_EMAIL alone does not remove them. You will still have those duplicate rows after this step. Follow up with FUZZY_NAME_PHONE to catch the remaining ones.

FUZZY_NAME_PHONE

FUZZY_NAME_PHONE is a name-and-phone-based deduplication strategy designed to catch the chaos engine’s fuzzy duplicates that evade EXACT_EMAIL. The environment implements it by:
  1. Creating a temporary _tmp_name column: name.astype(str).str.lower().str.strip()
  2. Creating a temporary _tmp_phone column: phone.astype(str).str.replace(r'\D+', '', regex=True) (digits only)
  3. Calling drop_duplicates(subset=["_tmp_name", "_tmp_phone"])
  4. Dropping the two temporary columns

Behavior

  • Match criterion: Two rows are duplicates if their normalized (lowercase, stripped) name and their digit-only phone are both identical.
  • Name normalization: Lowercases and strips whitespace. This means " JANE DOE " and "jane doe" are treated as the same name.
  • Phone normalization: Strips all non-digits. "+15551234567", "15551234567", and "(155) 512-34567" all normalize to "15551234567".
  • Limitation: Because duplicates have a different first name, they will not match on full normalized name unless the last-name-only signal happens to match after normalization. For best results, apply LOWERCASE_STRIP to name before running FUZZY_NAME_PHONE.

JSON Action

When to Use It

Use FUZZY_NAME_PHONE as a second deduplication pass after EXACT_EMAIL, targeting rows that had email = None in the chaos engine’s fuzzy duplicates. It is also useful when email data is heavily missing (>30 % null) and you need an alternative matching signal.
FUZZY_NAME_PHONE can produce false positives if two distinct customers happen to share the same normalized name and the same phone number (e.g., a father/son with the same last name and shared household phone). Use it selectively on sources where you have strong reason to believe duplicates exist, and verify the row count afterward with VIEW_SOURCE.

1

Standardize email first

Always run LOWERCASE_STRIP on email before any deduplication. Without it, "JANE.DOE@GMAIL.COM" and "jane.doe@gmail.com" are treated as different emails, and EXACT_EMAIL misses the duplicate entirely.
2

Run EXACT_EMAIL deduplication

Remove all rows where the normalized email is an exact duplicate of an earlier row.
3

Standardize name before fuzzy dedup

Normalize names to lowercase/stripped before running fuzzy matching so the temporary _tmp_name comparison works correctly.
4

Run FUZZY_NAME_PHONE for residual duplicates

Catch the chaos engine’s fuzzy duplicates (different first name, email = None) that survived the EXACT_EMAIL pass.
5

Drop remaining None-email rows

After deduplication, rows with email = None that were not matched as duplicates are likely either bot rows (which should be removed by other means) or legitimate customers with missing emails. Decide whether to drop or fill based on your task’s schema requirements.

Grader Penalty for Extra Rows

The grader measures the accuracy of your submitted DataFrame against the hidden truth. Extra rows — rows in your submission that have no corresponding truth record — subtract from your score. The penalty formula is:
This means every single extra row beyond the expected truth count costs 0.15 points. With 10 fuzzy duplicates and 8 bot rows in a Task 1 dataset (18 total extra rows), failing to deduplicate and remove bots costs 18 × 0.15 = 2.7 points — a substantial fraction of the total possible score.
The extra-row penalty is asymmetric: submitting too many rows is penalized at 0.15 per row, but removing too many real rows (under-counting) affects your recall metric separately. The optimal strategy is precision-first deduplication followed by careful bot removal, not aggressive DROP_ROW on all uncertain records.

Deduplication Strategy Quick Reference