Skip to main content
The CRM Data Pipeline Environment is a production-grade, OpenEnv-compliant benchmark that puts autonomous AI agents to work on one of the most persistent pain points in enterprise software: dirty, inconsistent, and duplicated customer data. Instead of toy puzzles or synthetic game worlds, this environment replicates the exact multi-step data engineering workflow a human Data Engineer would execute against a real CRM system — profile sources, identify corruption, apply targeted transformations, reconcile conflicting records across systems, and produce a pristine, analytics-ready master table. It exists to answer a concrete research question: can an LLM-driven agent reason about data quality, plan a multi-step cleaning pipeline, and produce output that survives row-exact comparison against a hidden ground truth?
All agent interactions happen through a strongly-typed REST API. Every action your agent sends is validated against a Pydantic schema before it touches any data. Every response your agent receives is a structured CRMPipelineObservation JSON object — there is no unstructured text stream, no scraping, and no ambiguity about the interface contract.

The Core Challenge

Real enterprise CRM systems accumulate data from many origins simultaneously. A Salesforce integration writes records one way; a web sign-up form writes them another; a legacy database migrated from a decade-old Oracle system writes them a third way. By the time these three streams need to be unified for a marketing campaign, a customer-success dashboard, or a churn-prediction model, the data is a mess: emails are uppercase in one source and lowercase in another, phone numbers mix E.164 notation with (xxx) xxx-xxxx formats, dates appear as ??/??/???? or 2023-13-12, and 20% of rows are missing critical fields entirely. On top of that, bot registrations have been injected into every source — rows with suspicious domains like promo-alerts.net or all-zero phone numbers that must be filtered before any join. Your agent must navigate this landscape with only partial observability. It cannot see the ground truth. It must actively profile each source to understand its shape, standardize columns one at a time, handle missing values explicitly, deduplicate by semantic key, and ultimately merge everything into a single unified table that matches the hidden truth schema exactly.

The Three Data Sources

Salesforce

The canonical source for customer_id and email. Contains ~54 real rows (90% sample of 60) plus 5 injected bot rows. Schema: customer_id, email, phone. Phone data is present ~70% of the time. Email is already lowercase — this source is the authoritative reference for email identity.

Web Leads / Web Forms

The freshest source for phone numbers. Contains ~48 real rows (80% sample) plus 5 bots. Schema mirrors salesforce (customer_id, email, phone) but customer_id is nullable ~20% of the time. Phone data, when present, is more current than Salesforce — making this the preferred phone source under the conflict resolution rules.

Legacy DB

The oldest source, with a divergent schema: columns are named legacy_id, contact_email, and home_phone. legacy_id uses an OLD- prefix instead of CUST_. contact_email is stored in UPPERCASE and must be lowercased before any join. Contains ~51 real rows (85% sample) plus 5 bots.
Each source is generated fresh at episode reset using Faker with a randomized seed, so no two episodes are identical. Corruption is applied deterministically by the correlate_truth function in server/data_generator.py.

OpenEnv Specification Compliance

This environment is built on the OpenEnv specification, which defines a standard protocol for agent-environment interaction modelled on the POMDP (Partially Observable Markov Decision Process) formalism. Compliance means:
  • /reset initializes a fresh episode, generates corrupted source data, snapshots the hidden ground truth keyed by a UUID, and returns the first observation.
  • /step accepts a CRMPipelineAction payload, executes the corresponding Pandas transformation, computes a dense reward signal, and returns a CRMPipelineObservation.
  • /state returns the current CRMPipelineState including episode_id, step_count, and task_id.
  • /grader/{episode_id} triggers the final evaluation engine, comparing the agent’s submitted DataFrame against the hidden truth snapshot and returning a score between 0.0 and 1.0.
The openenv.yaml manifest at the repository root declares all three tasks:
The active task is injected into the environment via the TASK_ID environment variable before each /reset call.

The Agent Interaction Loop

Your agent operates in a tight observe → act → observe loop. After each action, the server returns a fresh observation that includes the updated data view, a quality report (if you have profiled the source), and feedback from the last action. The loop ends when your agent sends SUBMIT_PIPELINE with a final_source pointer — or when it has exceeded the step budget.

Key Capabilities

Dense Reward Shaping

Every successful transformation earns an immediate reward (+0.02 to +0.05), solving the sparse-reward problem common in code-generation environments. Bad actions incur penalties (-0.05 to -0.15) that discourage hallucination and premature submission.

Stochastic Data Generation

The Faker library with seed(None) ensures no two episodes share the same data. Agents must generalize their cleaning strategy, not memorize a fixed dataset. Every reset spawns fresh corruption patterns.

SQL Execution Engine

The EXECUTE_SQL action spins up an in-memory SQLite connection, registers all current DataFrames as tables, executes your SELECT query, and writes the result to a named output table — enabling complex multi-source JOINs without leaving the environment.

Conflict Resolution Rules

Task 3 exposes a conflict_rules dictionary in every observation, telling your agent which source wins on each column. These rules (prefer_salesforce, prefer_web_leads) are derived from real enterprise data-governance patterns and must be honoured during the merge.

Bot Injection & Filtering

Every source contains injected bot rows using domains like promo-alerts.net and all-zero phone numbers. Your agent must identify and exclude these rows — bots left in the final submission incur a 0.15 × excess_rows penalty in the grader.

Reproducible Baseline

inference.py ships a RuleBasedBaseline that follows a hardcoded deterministic pipeline for all three tasks. It achieves a perfect score, giving researchers a clear target delta to measure LLM agent improvement against.