> ## Documentation Index
> Fetch the complete documentation index at: https://ritika-fa61c43f.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# CRM Data Pipeline: Local and Docker Installation Guide

> Install the CRM Data Pipeline Environment locally with uv or pip, or run it as a containerized service using the included Dockerfile.

The CRM Data Pipeline Environment can be installed in three ways: locally using the `uv` package manager (recommended for development and research), locally using `pip` with the provided `requirements.txt`, or as a fully self-contained Docker container suitable for CI pipelines, HuggingFace Spaces deployments, or any environment where you want strict process isolation. All three paths yield an identical runtime — a FastAPI server bound to port `8080` that speaks the OpenEnv REST protocol.

## Prerequisites

Before installing, ensure the following are available on your machine:

* **Python 3.11+** — The Dockerfile uses `python:3.11-slim` as its base image. The `pyproject.toml` does not declare a hard upper bound, so Python 3.12 works equally well.
* **`uv` package manager** — Strongly recommended. `uv` resolves the pinned `uv.lock` file for fully reproducible installs and is 10–100× faster than `pip`. Install it with `curl -LsSf https://astral.sh/uv/install.sh | sh`.
* **`pip`** — An alternative to `uv`, available with any standard Python installation.
* **Docker** — Optional. Required only if you want to run the containerized variant. Any version of Docker Engine that supports multi-stage builds is compatible.
* **`openenv-core>=0.1.0`** — The OpenEnv protocol library that provides `EnvClient`, `Environment`, and associated base types. This is installed automatically by either `uv sync` or `pip install -r requirements.txt`.

> [!NOTE]
>
> The `openenv-core` package provides graceful fallback stubs for all its types — if it is unavailable in your environment, the server and client will still import and run correctly using the inline stub classes defined in `client.py` and `models.py`. This makes the codebase portable even in sandboxes where the package registry is restricted.

## Local Installation with uv (Recommended)

`uv sync` reads the `uv.lock` file to install a fully pinned, reproducible dependency graph. This is the fastest and most reliable installation method for local development and baseline evaluation runs.

```bash theme={null}
git clone https://github.com/GaurRitika/CRM-Datapipeline-env.git
cd CRM-Datapipeline-env
uv sync
```

After `uv sync` completes, all dependencies are installed into a project-local virtual environment (typically `.venv/`). You do not need to activate it manually — `uv run` handles activation transparently.

**What `uv sync` installs** (sourced from `pyproject.toml`):

| Package | Minimum Version | Role |
| - | - | - |
| `openenv-core` | `>=0.1.0` | OpenEnv protocol base types and REST scaffolding |
| `pandas` | `>=2.0.0` | Core in-memory DataFrame engine for all data transformations |
| `pydantic` | `>=2.0.0` | Typed action/observation/state model validation |
| `faker` | `>=20.0.0` | Procedural generation of realistic, messy CRM data |
| `openai` | `>=1.0.0` | LLM API client used by `inference.py` (OpenAI SDK, compatible with any provider) |
| `fastapi` | `>=0.100.0` | ASGI web framework powering the OpenEnv REST server |
| `uvicorn` | `>=0.20.0` | ASGI server that serves the FastAPI application |

To start the server with `uv`:

```bash theme={null}
uv run -m server.app
```

## Local Installation with pip

If you prefer `pip`, the `requirements.txt` file lists the same packages with identical version constraints, plus `python-dotenv` (for `.env` file loading) and `aiofiles` (for async file I/O). This file is also what the Dockerfile uses to build the container image.

```bash theme={null}
git clone https://github.com/GaurRitika/CRM-Datapipeline-env.git
cd CRM-Datapipeline-env
pip install -r requirements.txt
```

**Complete `requirements.txt` package list:**

```text theme={null}
openenv-core>=0.1.0
pandas>=2.0.0
pydantic>=2.0.0
faker>=20.0.0
openai>=1.0.0
fastapi>=0.100.0
uvicorn>=0.20.0
python-dotenv>=1.0.0
aiofiles>=23.0.0
```

> [!NOTE]
>
> `python-dotenv` and `aiofiles` are listed in `requirements.txt` but not in `pyproject.toml`. The `inference.py` script calls `load_dotenv()` at startup to automatically source your `.env` file, and `aiofiles` is used for non-blocking I/O in certain async server paths. Both are needed for production-quality operation.

To start the server with a plain `pip` install:

```bash theme={null}
python -m server.app
```

## Docker Installation

The repository ships a production-ready `Dockerfile` designed for compatibility with both local Docker runtimes and HuggingFace Spaces free tiers. The image is built from `python:3.11-slim`, installs GCC and build tools (required by pandas' C extensions on slim images), and runs the server as a non-root user (UID 1000) — a hard requirement for HuggingFace Spaces.

**Build the image:**

```bash theme={null}
docker build -t crm-pipeline-env .
```

**Run the container:**

```bash theme={null}
docker run -p 8080:8080 \
  -e TASK_ID=t1 \
  -e HF_TOKEN=hf_your_token_here \
  crm-pipeline-env
```

**Run with all configuration options:**

```bash theme={null}
docker run -p 8080:8080 \
  -e TASK_ID=t3 \
  -e HF_TOKEN=hf_your_token_here \
  -e API_BASE_URL=https://router.huggingface.co/v1 \
  -e MODEL_NAME=Qwen/Qwen2.5-72B-Instruct \
  -e OPENENV_BASE_URL=http://localhost:8080 \
  crm-pipeline-env
```

**Dockerfile explained, line by line:**

```dockerfile theme={null}
FROM python:3.11-slim
# Lean Python 3.11 base — minimal attack surface, fast pulls.

WORKDIR /app

# GCC and build-essential are required so pandas can compile its C extensions
# on a slim image that ships without a compiler toolchain by default.
RUN apt-get update && apt-get install -y gcc build-essential && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Create a non-root user at UID 1000 — mandatory for HuggingFace Spaces compatibility.
# Running as root on HF Spaces causes permission errors and container rejections.
RUN useradd -m -u 1000 user
USER user
ENV HOME=/home/user \
    PATH=/home/user/.local/bin:$PATH

WORKDIR $HOME/app
COPY --chown=user . $HOME/app

# HuggingFace Spaces default port is 8080 — do not change this.
EXPOSE 8080

# Launch uvicorn directly, binding to all interfaces on port 8080.
CMD ["uvicorn", "server.app:app", "--host", "0.0.0.0", "--port", "8080"]
```

> [!WARNING]
>
> Do not change the exposed port from `8080`. HuggingFace Spaces enforces this port in its reverse proxy configuration, and the OpenEnv client defaults to `http://localhost:8080` as well. If you need a different host port, use Docker's port mapping (`-p 9000:8080`) rather than changing the `EXPOSE` or `CMD` directives inside the Dockerfile.

## Verify the Installation

Regardless of which installation method you used, confirm the server is responding correctly:

```bash theme={null}
curl http://localhost:8080/
```

Expected output:

```json theme={null}
{"status": "ok", "message": "CRM Pipeline Environment is actively running."}
```

You can also hit the health check from a browser at [http://localhost:8080/](http://localhost:8080/) or view the auto-generated OpenAPI documentation at [http://localhost:8080/docs](http://localhost:8080/docs).

> [!TIP]
>
> The FastAPI server exposes interactive Swagger UI at `/docs` and a ReDoc interface at `/redoc`. Both are generated automatically from the Pydantic models in `models.py` and are extremely useful for exploring the full action and observation schemas without writing any code.

## Project Structure

Once cloned, the repository has the following layout. Understanding each file's role will help you navigate the codebase, extend the environment, or build custom evaluation harnesses.

```
CRM-Datapipeline-env/
├── server/
│   ├── app.py              # FastAPI application factory: registers the OpenEnv REST
│   │                       # endpoints (/reset, /step, /state) via create_fastapi_app(),
│   │                       # adds the health-check GET / route, and exposes a custom
│   │                       # POST /grader/{episode_id} endpoint for end-of-episode scoring.
│   │
│   ├── environment.py      # Core POMDP engine (CRMDataPipelineEnv). Implements reset(),
│   │                       # step(), and all action handlers (view, profile, standardize,
│   │                       # handle_missing, deduplicate, merge, execute_sql, submit).
│   │                       # Manages GLOBAL_TRUTH_STORE and GLOBAL_ENV_STORE for
│   │                       # episode isolation in concurrent deployments.
│   │
│   ├── data_generator.py   # Procedural messy data generation using Faker. Produces
│   │                       # realistic CRM records with intentional formatting errors:
│   │                       # mixed-case emails, inconsistent phone formats, null gaps,
│   │                       # duplicate rows, and bot-injected outlier entries. Also
│   │                       # generates the hidden ground truth DataFrames used by the grader.
│   │
│   └── graders.py          # Scoring engine. evaluate_dataframes() joins the agent's
│                           # submitted DataFrame against the hidden truth snapshot using
│                           # customer_id as the primary key, then computes exact string
│                           # match ratios per column. Applies a -0.15 penalty per extra
│                           # row beyond the truth row count (junk/bot rows not filtered).
│
├── client.py               # CRMDataPipelineEnvClient — a typed Python REST client that
│                           # extends openenv.core.EnvClient. Handles JSON serialization of
│                           # CRMPipelineAction payloads and deserialization of server
│                           # responses back into typed CRMPipelineObservation objects.
│
├── inference.py            # Baseline inference script. Runs all three tasks (t1, t2, t3)
│                           # sequentially using an LLM agent (OpenAI SDK) with exponential
│                           # backoff retry logic. Falls back to a hardcoded deterministic
│                           # rule-based pipeline if the LLM fails. Logs step-by-step
│                           # rewards and prints final scores in OpenEnv format.
│
├── models.py               # All Pydantic v2 models: CRMPipelineAction (action payload),
│                           # CRMPipelineObservation (server response), CRMPipelineState
│                           # (episode metadata), and all Enum types (PipelineActionType,
│                           # StandardizationStrategy, MissingStrategy, DeduplicationStrategy,
│                           # ConflictRule).
│
├── scripts/
│   └── validate-submission.sh  # Pre-submission validation script. Run this before
│                               # submitting to verify your agent implementation meets
│                               # the OpenEnv specification requirements.
│
├── Dockerfile              # Production container definition. python:3.11-slim base,
│                           # GCC for pandas C extensions, non-root UID 1000 user for
│                           # HuggingFace Spaces compatibility, port 8080.
│
├── pyproject.toml          # Project metadata and core dependency declarations for uv/pip.
│                           # Defines the `server` entry point script and py-modules list.
│
├── requirements.txt        # Pip-installable flat dependency list. Used by the Dockerfile
│                           # and pip-based local installs. Includes python-dotenv and
│                           # aiofiles in addition to the pyproject.toml dependencies.
│
└── openenv.yaml            # OpenEnv task catalogue. Declares the environment ID, version,
                            # tags, and the three task definitions (t1/t2/t3) with their
                            # difficulty levels. Read by OpenEnv tooling for registration
                            # and benchmarking infrastructure.
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.