> ## Documentation Index
> Fetch the complete documentation index at: https://ritika-fa61c43f.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# CRM Data Pipeline: OpenEnv Benchmark for AI Agents

> The CRM Data Pipeline Environment simulates real enterprise data engineering workloads for evaluating autonomous AI agent capabilities.

The CRM Data Pipeline Environment is a production-grade, OpenEnv-compliant benchmark that puts autonomous AI agents to work on one of the most persistent pain points in enterprise software: dirty, inconsistent, and duplicated customer data. Instead of toy puzzles or synthetic game worlds, this environment replicates the exact multi-step data engineering workflow a human Data Engineer would execute against a real CRM system — profile sources, identify corruption, apply targeted transformations, reconcile conflicting records across systems, and produce a pristine, analytics-ready master table. It exists to answer a concrete research question: **can an LLM-driven agent reason about data quality, plan a multi-step cleaning pipeline, and produce output that survives row-exact comparison against a hidden ground truth?**

> [!NOTE]
>
> All agent interactions happen through a strongly-typed REST API. Every action your agent sends is validated against a Pydantic schema before it touches any data. Every response your agent receives is a structured `CRMPipelineObservation` JSON object — there is no unstructured text stream, no scraping, and no ambiguity about the interface contract.

## The Core Challenge

Real enterprise CRM systems accumulate data from many origins simultaneously. A Salesforce integration writes records one way; a web sign-up form writes them another; a legacy database migrated from a decade-old Oracle system writes them a third way. By the time these three streams need to be unified for a marketing campaign, a customer-success dashboard, or a churn-prediction model, the data is a mess: emails are uppercase in one source and lowercase in another, phone numbers mix E.164 notation with `(xxx) xxx-xxxx` formats, dates appear as `??/??/????` or `2023-13-12`, and 20% of rows are missing critical fields entirely. On top of that, bot registrations have been injected into every source — rows with suspicious domains like `promo-alerts.net` or all-zero phone numbers that must be filtered before any join.

Your agent must navigate this landscape with only partial observability. It cannot see the ground truth. It must actively **profile** each source to understand its shape, **standardize** columns one at a time, **handle** missing values explicitly, **deduplicate** by semantic key, and ultimately **merge** everything into a single unified table that matches the hidden truth schema exactly.

## The Three Data Sources

> [card] Salesforce
>
> The canonical source for `customer_id` and `email`. Contains \~54 real rows (90% sample of 60) plus 5 injected bot rows. Schema: `customer_id`, `email`, `phone`. Phone data is present \~70% of the time. Email is already lowercase — this source is the authoritative reference for email identity.

> [card] Web Leads / Web Forms
>
> The freshest source for phone numbers. Contains \~48 real rows (80% sample) plus 5 bots. Schema mirrors `salesforce` (`customer_id`, `email`, `phone`) but `customer_id` is nullable \~20% of the time. Phone data, when present, is more current than Salesforce — making this the preferred phone source under the conflict resolution rules.

> [card] Legacy DB
>
> The oldest source, with a divergent schema: columns are named `legacy_id`, `contact_email`, and `home_phone`. `legacy_id` uses an `OLD-` prefix instead of `CUST_`. `contact_email` is stored in **UPPERCASE** and must be lowercased before any join. Contains \~51 real rows (85% sample) plus 5 bots.

Each source is generated fresh at episode reset using [Faker](https://faker.readthedocs.io/en/stable/) with a randomized seed, so no two episodes are identical. Corruption is applied deterministically by the `correlate_truth` function in `server/data_generator.py`.

## OpenEnv Specification Compliance

This environment is built on the [OpenEnv specification](https://openenv.dev), which defines a standard protocol for agent-environment interaction modelled on the POMDP (Partially Observable Markov Decision Process) formalism. Compliance means:

* **`/reset`** initializes a fresh episode, generates corrupted source data, snapshots the hidden ground truth keyed by a UUID, and returns the first observation.
* **`/step`** accepts a `CRMPipelineAction` payload, executes the corresponding Pandas transformation, computes a dense reward signal, and returns a `CRMPipelineObservation`.
* **`/state`** returns the current `CRMPipelineState` including `episode_id`, `step_count`, and `task_id`.
* **`/grader/{episode_id}`** triggers the final evaluation engine, comparing the agent's submitted DataFrame against the hidden truth snapshot and returning a score between `0.0` and `1.0`.

The `openenv.yaml` manifest at the repository root declares all three tasks:

```yaml theme={null}
id: crm-data-pipeline
name: CRM Data Pipeline Agent
version: 1.0.0
tasks:
  - id: t1
    description: "Task 1 (Easy): Normalize the single web_forms dataset. Fix date formats, lowercase emails, strip spaces."
    difficulty: easy
  - id: t2
    description: "Task 2 (Medium): Deduplicate the legacy_db dataset. Identify matching customers with messy info and merge them."
    difficulty: medium
  - id: t3
    description: "Task 3 (Hard): Resolve a 3-way merge conflict. Standardize and merge Salesforce, Web Leads, and Legacy DBs."
    difficulty: hard
```

The active task is injected into the environment via the `TASK_ID` environment variable before each `/reset` call.

## The Agent Interaction Loop

Your agent operates in a tight observe → act → observe loop. After each action, the server returns a fresh observation that includes the updated data view, a quality report (if you have profiled the source), and feedback from the last action. The loop ends when your agent sends `SUBMIT_PIPELINE` with a `final_source` pointer — or when it has exceeded the step budget.

```mermaid theme={null}
graph LR
    A([LLM Agent]) -->|POST /step\nCRMPipelineAction JSON| B[FastAPI Server\nserver/app.py]
    B -->|dispatch action| C[CRMDataPipelineEnv\nserver/environment.py]
    C -->|mutate| D[(Pandas DataFrames\nin-memory sources)]
    D -->|measure delta| E[Dense Reward\nFunction]
    E -->|CRMPipelineObservation| A
    A -->|SUBMIT_PIPELINE| F[Grader Engine\nserver/graders.py]
    G[(Hidden Ground Truth\nGLOBAL_TRUTH_STORE)] -.->|row-exact compare| F
    F -->|score 0.0–1.0| A
```

## Key Capabilities

> [card] Dense Reward Shaping
>
> Every successful transformation earns an immediate reward (+0.02 to +0.05), solving the sparse-reward problem common in code-generation environments. Bad actions incur penalties (-0.05 to -0.15) that discourage hallucination and premature submission.

> [card] Stochastic Data Generation
>
> The `Faker` library with `seed(None)` ensures no two episodes share the same data. Agents must generalize their cleaning strategy, not memorize a fixed dataset. Every reset spawns fresh corruption patterns.

> [card] SQL Execution Engine
>
> The `EXECUTE_SQL` action spins up an in-memory SQLite connection, registers all current DataFrames as tables, executes your SELECT query, and writes the result to a named output table — enabling complex multi-source JOINs without leaving the environment.

> [card] Conflict Resolution Rules
>
> Task 3 exposes a `conflict_rules` dictionary in every observation, telling your agent which source wins on each column. These rules (`prefer_salesforce`, `prefer_web_leads`) are derived from real enterprise data-governance patterns and must be honoured during the merge.

> [card] Bot Injection & Filtering
>
> Every source contains injected bot rows using domains like `promo-alerts.net` and all-zero phone numbers. Your agent must identify and exclude these rows — bots left in the final submission incur a `0.15 × excess_rows` penalty in the grader.

> [card] Reproducible Baseline
>
> `inference.py` ships a `RuleBasedBaseline` that follows a hardcoded deterministic pipeline for all three tasks. It achieves a perfect score, giving researchers a clear target delta to measure LLM agent improvement against.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.