Skip to main content
The inference.py script is the out-of-the-box evaluator bundled with the CRM Data Pipeline Environment. It drives an LLM-powered agent (or falls back to a deterministic rule-based baseline) through all three CRM cleaning tasks — t1, t2, and t3 — in sequence, then prints a final score for each task plus a computed average. You do not need to write any agent code to run it; it works immediately once your environment variables are set and the server is running.

Prerequisites

Before running inference, confirm the following are installed and available in your shell:
  • Python 3.12+ with uv (the project uses uv as the package runner)
  • openenv-core: pip install openenv-core
  • python-dotenv, openai, requests, pandas — all installed automatically via uv
The script reads credentials exclusively from environment variables. It never accepts hardcoded keys. Three variables control the LLM backend:
If neither HF_TOKEN nor API_KEY is set, the openai.OpenAI client is initialized with api_key=None. The script will still run — every LLM call will fail and the rule-based fallback will handle all steps automatically. This is intentional design: the fallback achieves the maximum grader score of 0.99 on all three tasks without any LLM.

Running Inference

1

Set environment variables

Export your credentials in the terminal where you will run the script. The API_BASE_URL and MODEL_NAME values shown below are the defaults; you may omit them if you want to use Qwen2.5-72B via the HuggingFace Router.
Alternatively, create a .env file in the repo root — the script calls load_dotenv() at startup and will pick it up automatically without overriding any already-exported variables:
2

Start the FastAPI server

Open a second terminal and launch the environment server. The server must be running before you start inference — it binds to http://127.0.0.1:8080 by default.
You should see Uvicorn startup output confirming the server is listening. Keep this terminal open for the entire inference run. If you need a different port, set OPENENV_BASE_URL in the inference terminal:
3

Run the inference script

In your first terminal (with credentials exported), run:
The script iterates over ["t1", "t2", "t3"], runs one full episode for each task, then prints aggregate results. Total runtime with the rule-based fallback is typically under 10 seconds. With a live LLM, runtime depends on API latency and the number of retries.
4

Inspect the output

Inference emits structured log lines to stdout. A typical successful run looks like this:
The final JSON line is your benchmark result. A score of 0.99 is the maximum the grader awards — it means every row in your submitted DataFrame matched the hidden ground truth exactly.

Understanding the Log Format

Every line emitted by inference.py follows one of four structured prefixes: Step rewards are the intermediate reward signal from the server’s dense heuristic — they are useful for debugging but are not your final grade. Your real score is the terminal grader output, which you see in the [END] line’s score field and the [DEBUG] Final Score line.

How the Episode Loop Works

The run_task(task_id, use_llm=True) function implements the full POMDP interaction loop. Understanding it helps you debug unexpected behavior and write better custom agents.
The MAX_STEPS_PER_TASK constant is set to 15, which is sufficient headroom even for the complex three-way SQL join in Task 3. If your LLM agent exceeds 15 steps without submitting, the loop terminates and the episode is scored based on whatever state the pipeline was left in.

The Fallback Mechanism

call_gpt_with_retry uses exponential backoff across up to 3 attempts. After each failed attempt the code sleeps 2 ** attempt seconds before retrying:
  • Attempt 1 (attempt index 0) — immediate call; on RateLimitError sleeps 1 s (2^0)
  • Attempt 2 (attempt index 1) — on RateLimitError sleeps 2 s (2^1)
  • Attempt 3 (attempt index 2) — on RateLimitError sleeps 4 s (2^2)
  • If all three fail — returns None
When validate_action receives None or an invalid payload, the code immediately falls back to build_smart_fallback(obs, step, task_id), which returns the next action from a fully hardcoded, per-task action plan:
The RuleBasedBaseline class is a thin wrapper around build_smart_fallback — it holds the task_id and calls the fallback on every act(obs, step) invocation. It achieves the maximum grader score of 0.99 on all three tasks deterministically.

Using a Different LLM Model

inference.py is compatible with any OpenAI API-compatible provider. Change the target by updating three environment variables before launching the script.
The script uses response_format={"type": "json_object"} in the LLM call, which requires the model to support JSON mode. Not all models or providers honour this parameter. If your model does not support it, json.JSONDecodeError will be caught and the fallback will engage automatically — but you will lose LLM-driven actions for every step.

Running Only a Single Task

By default the script runs all three tasks. To run only one task, restrict the task list in the __main__ block — run_task sets TASK_ID programmatically before calling reset(), so no server restart is needed:
Alternatively, call run_task directly from a Python script or REPL:

Running in Pure Rule-Based Mode

Pass use_llm=False to bypass all LLM calls entirely. This is the fastest way to establish a 0.99 baseline and verify that your server setup is working correctly:
Use rule-based mode (use_llm=False) as a sanity check whenever you change the server code or Docker image. If the rule-based baseline drops below 0.99, something in the environment is broken — not your LLM.