> ## Documentation Index
> Fetch the complete documentation index at: https://ritika-fa61c43f.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# CRM Data Pipeline Environment: Get Started in 5 Minutes

> Spin up the CRM Data Pipeline environment server and run your first agent episode loop with real data cleaning actions in under 5 minutes.

The CRM Data Pipeline Environment is an OpenEnv-compliant FastAPI benchmark server that gives autonomous AI agents a realistic, production-grade CRM data engineering challenge. In under five minutes you will have the server running locally, your first episode loop executing, and an agent making real data cleaning decisions against messy, procedurally-generated enterprise CRM datasets sourced from Salesforce exports, web lead forms, and legacy database dumps.

## Clone the Repository

Pull the source directly from GitHub. The repository contains the FastAPI server, procedural data generator, Pydantic model definitions, typed client library, and an out-of-the-box baseline inference script.

```bash theme={null}
git clone https://github.com/GaurRitika/CRM-Datapipeline-env.git
cd CRM-Datapipeline-env
```

## Install Dependencies

The project supports two installation paths. `uv` is the recommended package manager — it resolves the lock file deterministically and is significantly faster than `pip`. Either path installs the same set of packages: `openenv-core`, `pandas`, `pydantic`, `faker`, `openai`, `fastapi`, `uvicorn`, `python-dotenv`, and `aiofiles`.

```bash uv (Recommended) theme={null}
# Install uv if you haven't already
curl -LsSf https://astral.sh/uv/install.sh | sh

# Sync all dependencies from the lock file
uv sync
```

```bash pip theme={null}
pip install -r requirements.txt
```

## Configure Environment Variables

The server reads configuration exclusively from environment variables — never from hardcoded values. At minimum you need an API key to use the LLM-powered inference script, and optionally a `TASK_ID` to control which task loads at reset time.

Create a `.env` file in the project root and set the following variables:

```bash .env theme={null}
# Your HuggingFace token (used by inference.py for LLM calls via the HF Router)
HF_TOKEN=hf_your_token_here

# Alternatively, any OpenAI-compatible API key
# API_KEY=sk-your-key-here

# The task to load when the server receives a /reset call (t1, t2, or t3)
TASK_ID=t1

# LLM provider base URL (defaults to HuggingFace Router if not set)
API_BASE_URL=https://router.huggingface.co/v1

# Model to use during inference (defaults to Qwen2.5-72B-Instruct if not set)
MODEL_NAME=Qwen/Qwen2.5-72B-Instruct

# The environment server URL used by the client
OPENENV_BASE_URL=http://localhost:8080
```

> [!NOTE]
>
> The `TASK_ID` environment variable controls which task loads every time `POST /reset` is called. Valid values are `t1` (Easy — single-source web forms normalization), `t2` (Medium — two-source deduplication), and `t3` (Hard — three-way enterprise merge). If `TASK_ID` is not set, the server defaults to `t1`. You can change the active task at any time by updating this variable and restarting the server, or by passing `-e TASK_ID=t2` when using Docker.

## Start the Server

The FastAPI server exposes the full OpenEnv protocol — `/reset`, `/step`, `/state` — plus a custom `/grader` endpoint and a root health check. It binds to `0.0.0.0:8080` by default.

```bash uv (Recommended) theme={null}
uv run -m server.app
```

```bash Python directly theme={null}
python -m server.app
```

```bash Docker theme={null}
docker build -t crm-pipeline-env .
docker run -p 8080:8080 \
  -e TASK_ID=t1 \
  -e HF_TOKEN=hf_your_token_here \
  crm-pipeline-env
```

You should see uvicorn startup output similar to:

```
INFO:     Started server process [12345]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
```

## Verify the Server is Running

With the server up, confirm it is responding correctly using the root health check endpoint. This endpoint returns a static JSON payload and requires no authentication.

```bash theme={null}
curl http://localhost:8080/
```

Expected response:

```json theme={null}
{
  "status": "ok",
  "message": "CRM Pipeline Environment is actively running."
}
```

If you see this response, the server is fully initialized and ready to accept agent connections. If the request hangs or returns a connection error, double-check that port 8080 is not already in use on your machine (`lsof -i :8080` on macOS/Linux).

## Run Your First Episode

Now you can connect the typed Python client, reset the environment to start a fresh episode, and execute your first data cleaning action. The code below uses `CRMDataPipelineEnvClient` from `client.py` and the `CRMPipelineAction` / `PipelineActionType` models from `models.py`. The `.sync()` context manager opens a persistent HTTP connection and closes it cleanly when the block exits.

```python theme={null}
from client import CRMDataPipelineEnvClient
from models import CRMPipelineAction, PipelineActionType

# Open a synchronous connection to the running server
with CRMDataPipelineEnvClient(base_url="http://localhost:8080").sync() as env:

    # Start a new episode — the server loads the task specified by TASK_ID,
    # generates fresh procedurally-messy data, and returns the initial observation.
    result = env.reset()

    # Inspect the task objective and which data sources are available this episode
    print(result.observation.current_task_objective)
    # → "Normalize web_forms dataset"

    print(result.observation.available_sources)
    # → ['web_forms']

    # Peek at the raw data in the web_forms source (top 3 rows rendered as markdown table)
    step = env.step(CRMPipelineAction(
        action_type=PipelineActionType.VIEW_SOURCE,
        source="web_forms"
    ))
    print(step.observation.current_view)
    # →
    # |    | customer_id   | name             | email                  | phone            | signup_date   |
    # |---:|:--------------|:-----------------|:-----------------------|:-----------------|:--------------:|
    # |  0 | CUST_001      | Alice Johnson    | Alice.Johnson@Mail.COM | (555) 123-4567   | 01/15/2023    |
    # |  1 | CUST_002      | bob smith        | bob.smith@example.com  | 555.987.6543 x2  | 2023-02-20    |
    # |  2 | ???           | Charlie Brown    | Charlie@BROWN.net      | +1-800-555-0199  | March 3 2023  |

    print(f"Reward: {step.reward}, Done: {step.done}")
    # → Reward: 0.0, Done: False
```

> [!TIP]
>
> After `VIEW_SOURCE`, your next action should be `PROFILE_SOURCE` on the same source. Profiling generates a statistical data quality report (null percentages, inferred data types per column) and earns a small reward of `+0.02`. The observation's `data_quality_report` field will then guide which columns need `STANDARDIZE_COLUMN` or `HANDLE_MISSING` actions next.

## Run the Built-In Baseline Evaluator

The repository ships with a complete end-to-end inference script in `inference.py`. It runs all three tasks sequentially using an LLM-driven agent (falling back to a deterministic rule-based pipeline if the LLM fails or rate-limits). This script reproduces the published baseline and is a useful starting reference for building your own agent.

```bash theme={null}
uv run python inference.py
```

Expected terminal output (abbreviated):

```
[START] task=t1 env=crm_data_pipeline model=Qwen/Qwen2.5-72B-Instruct
[STEP] step=1 action={"action_type":"PROFILE_SOURCE","source":"web_forms"} reward=0.02 done=false error=null
[STEP] step=2 action={"action_type":"STANDARDIZE_COLUMN","source":"web_forms","column":"email","standardization_strategy":"LOWERCASE_STRIP"} reward=0.02 done=false error=null
[STEP] step=3 action={"action_type":"STANDARDIZE_COLUMN","source":"web_forms","column":"phone","standardization_strategy":"EXTRACT_NUMBERS"} reward=0.02 done=false error=null
...
[STEP] step=8 action={"action_type":"SUBMIT_PIPELINE","final_source":"web_forms_clean"} reward=0.20 done=true error=null
[END] success=true steps=8 score=0.990 rewards=0.02,0.02,0.02,0.02,0.02,0.10,0.20
[START] task=t2 env=crm_data_pipeline model=Qwen/Qwen2.5-72B-Instruct
...
[DEBUG] === Final Scores ===
[DEBUG] {"t1": 0.99, "t2": 0.99, "t3": 0.99, "average": 0.99}
```

> [!NOTE]
>
> The built-in rule-based fallback baseline achieves a perfect normalized score of `1.0` across all three tasks. Your custom agent's performance is measured as the delta from this deterministic ceiling. The final score for each task is computed by the grader engine, which compares your submitted DataFrame row-by-row against a hidden ground truth snapshot using exact string matching on all non-key columns after normalization.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.