> ## Documentation Index
> Fetch the complete documentation index at: https://ritika-fa61c43f.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# POST /step — Execute a Pipeline Data Engineering Action

> POST /step executes a CRMPipelineAction on the active episode, mutates in-memory DataFrames, and returns an observation with reward and feedback.

`POST /step` is the primary interaction endpoint. Each call deserializes the request body into a `CRMPipelineAction` Pydantic model, increments the internal step counter, dispatches to the matching handler function, mutates the in-memory Pandas DataFrames in place, and returns a `CRMStepResult` carrying the updated `CRMPipelineObservation`, a scalar `reward`, and a `done` flag. You call this endpoint repeatedly — profiling, cleaning, merging — until you are ready to submit.

## Request Format

```json theme={null}
{
  "action_type": "<PipelineActionType>",
  "source": "optional_source_name",
  "column": "optional_column_name"
}
```

Only `action_type` is universally required. All other fields are conditionally required depending on the action type, as documented below.

## Action Types and Examples

### `VIEW_SOURCE`

Retrieves a markdown-formatted preview of the top 3 rows of a source DataFrame. Returns `reward = 0.0`. Use this to inspect raw data before deciding which columns need cleaning.

**Request:**

```json theme={null}
{
  "action_type": "VIEW_SOURCE",
  "source": "web_forms"
}
```

**Response:**

```json theme={null}
{
  "observation": {
    "done": false,
    "reward": 0.0,
    "current_task_objective": "Normalize web_forms dataset",
    "schema_target": {
      "customer_id": "string",
      "name": "string (stripped)",
      "email": "string (lowercase, stripped)",
      "signup_date": "string ISO 8601 or empty",
      "phone": "string E.164 format (+1...) or empty"
    },
    "available_sources": ["web_forms"],
    "current_view": "| customer_id | name              | email               | signup_date | phone            |\n|-------------|-------------------|---------------------|-------------|------------------|\n| CUST_1000   |   Alice Johnson   | ALICE@EXAMPLE.COM   | 2023/04/12  | (415) 555-0192   |\n| CUST_1001   | BOB SMITH         | bob.smith@email.com | ??/??/???? | +14155550173 ext. 42 |\n| BOT12345    | Newsletter Bot    | promo@bulk-mail.co  | 1970-01-01  | +00000000000     |",
    "data_quality_report": "",
    "last_action_feedback": "Viewing top 3 rows of web_forms",
    "conflict_rules": null
  },
  "reward": 0.0,
  "done": false
}
```

***

### `PROFILE_SOURCE`

Generates a per-column data quality report showing the dtype and null count of every column. Returns `reward = +0.02`. Always profile before cleaning — the report is written to `data_quality_report` in the observation.

**Request:**

```json theme={null}
{
  "action_type": "PROFILE_SOURCE",
  "source": "web_forms"
}
```

**Response:**

```json theme={null}
{
  "observation": {
    "done": false,
    "reward": 0.02,
    "current_task_objective": "Normalize web_forms dataset",
    "schema_target": { "...": "..." },
    "available_sources": ["web_forms"],
    "current_view": "Select a source to view.",
    "data_quality_report": "- **customer_id**: type=object, nulls=0\n- **name**: type=object, nulls=0\n- **email**: type=object, nulls=9\n- **signup_date**: type=object, nulls=6\n- **phone**: type=object, nulls=7",
    "last_action_feedback": "Profiled web_forms",
    "conflict_rules": null
  },
  "reward": 0.02,
  "done": false
}
```

***

### `STANDARDIZE_COLUMN`

Applies a deterministic transformation to a single column in a source DataFrame. The mutation is performed **in place** — the source DataFrame is permanently updated in server memory. Returns `reward` between `+0.01` and `+0.05`, scaled by the fraction of rows that were actually changed.

Available strategies are defined by the `StandardizationStrategy` enum:

| Strategy | Effect | Best For |
| - | - | - |
| `LOWERCASE_STRIP` | `.str.lower().str.strip()` | Email, name normalization |
| `EXTRACT_NUMBERS` | Extracts digits, formats as `+{digits}`, strips extensions | Phone E.164 conversion |
| `TO_DATETIME_ISO` | Parses dates with `pd.to_datetime(errors='coerce')`, formats as `YYYY-MM-DDTHH:MM:SS`, replaces unparseable values with `""` | `signup_date` ISO 8601 |
| `PHONE_E164` | Alias for phone-number extraction | Phone normalization |

**Request (lowercase email):**

```json theme={null}
{
  "action_type": "STANDARDIZE_COLUMN",
  "source": "web_forms",
  "column": "email",
  "standardization_strategy": "LOWERCASE_STRIP"
}
```

**Response:**

```json theme={null}
{
  "observation": {
    "done": false,
    "reward": 0.04,
    "current_task_objective": "Normalize web_forms dataset",
    "schema_target": { "...": "..." },
    "available_sources": ["web_forms"],
    "current_view": "Select a source to view.",
    "data_quality_report": "- **customer_id**: type=object, nulls=0\n- **name**: type=object, nulls=0\n- **email**: type=object, nulls=9\n- **signup_date**: type=object, nulls=6\n- **phone**: type=object, nulls=7",
    "last_action_feedback": "Standardized web_forms.email using LOWERCASE_STRIP",
    "conflict_rules": null
  },
  "reward": 0.04,
  "done": false
}
```

**Request (phone E.164):**

```json theme={null}
{
  "action_type": "STANDARDIZE_COLUMN",
  "source": "web_forms",
  "column": "phone",
  "standardization_strategy": "EXTRACT_NUMBERS"
}
```

**Request (date ISO 8601):**

```json theme={null}
{
  "action_type": "STANDARDIZE_COLUMN",
  "source": "web_forms",
  "column": "signup_date",
  "standardization_strategy": "TO_DATETIME_ISO"
}
```

***

### `HANDLE_MISSING`

Resolves null values in a single column using one of the `MissingStrategy` options. Returns `reward` proportional to the fraction of nulls resolved (minimum `+0.01`).

| Strategy | Behavior |
| - | - |
| `DROP_ROW` | Drops all rows where the column is null. Reward scales with `null_count / original_rows × 0.1` (minimum `+0.01`) |
| `FILL_VALUE` | Fills nulls with the literal `fallback_value` string. Reward scales with `null_count / total_rows × 0.05` (minimum `+0.01`) |
| `FILL_MEAN` | Accepted by the API but not yet implemented — no transformation is applied |
| `FILL_MODE` | Accepted by the API but not yet implemented — no transformation is applied |

**Request:**

```json theme={null}
{
  "action_type": "HANDLE_MISSING",
  "source": "web_forms",
  "column": "email",
  "missing_strategy": "DROP_ROW"
}
```

**Response:**

```json theme={null}
{
  "observation": {
    "done": false,
    "reward": 0.03,
    "current_task_objective": "Normalize web_forms dataset",
    "schema_target": { "...": "..." },
    "available_sources": ["web_forms"],
    "current_view": "Select a source to view.",
    "data_quality_report": "",
    "last_action_feedback": "Handled 9 missing values in web_forms.email",
    "conflict_rules": null
  },
  "reward": 0.03,
  "done": false
}
```

**Request (fill with empty string):**

```json theme={null}
{
  "action_type": "HANDLE_MISSING",
  "source": "web_forms",
  "column": "phone",
  "missing_strategy": "FILL_VALUE",
  "fallback_value": ""
}
```

***

### `DEDUPLICATE`

Removes duplicate rows from a source DataFrame based on the chosen `DeduplicationStrategy`. The mutation is saved back to the source. Returns `reward = removed_rows / original_rows × 0.2`.

| Strategy | Logic |
| - | - |
| `EXACT_EMAIL` | `df.drop_duplicates(subset=["email"])` |
| `FUZZY_NAME_PHONE` | Lowercases and strips name, strips non-digits from phone, then deduplicates on that pair |

**Request:**

```json theme={null}
{
  "action_type": "DEDUPLICATE",
  "source": "web_forms",
  "deduplication_strategy": "EXACT_EMAIL"
}
```

**Response:**

```json theme={null}
{
  "observation": {
    "done": false,
    "reward": 0.05,
    "current_task_objective": "Normalize web_forms dataset",
    "schema_target": { "...": "..." },
    "available_sources": ["web_forms"],
    "current_view": "Select a source to view.",
    "data_quality_report": "",
    "last_action_feedback": "Deduplicated web_forms, removed 12 rows.",
    "conflict_rules": null
  },
  "reward": 0.05,
  "done": false
}
```

***

### `MERGE_SOURCES`

Performs an outer join of two source DataFrames on a shared key column. The result is always stored under the fixed name `merged_output`, which then appears in `available_sources`. Returns `reward = +0.05` on success or `-0.05` if the join key is missing from either source.

Conflict resolution for overlapping columns is controlled by the `ConflictRule` enum:

| Rule | Behavior |
| - | - |
| `PREFER_S1` | Uses `source` column value when both sides are non-null |
| `PREFER_S2` | Uses `source2` column value when both sides are non-null |
| `COALESCE` *(default)* | Uses the first non-null value between `source` and `source2` |

> [!NOTE]
>
> For Task 3, read the `conflict_rules` dictionary from the reset observation
> to determine which `ConflictRule` to pass per column. The hidden truth was
> built using those exact priority rules — mismatched rules will reduce your
> grader score.

**Request:**

```json theme={null}
{
  "action_type": "MERGE_SOURCES",
  "source": "salesforce",
  "source2": "web_leads",
  "join_key": "customer_id",
  "conflict_rule": "PREFER_S1",
  "output_table": "merged_output"
}
```

**Response:**

```json theme={null}
{
  "observation": {
    "done": false,
    "reward": 0.05,
    "current_task_objective": "Merge Salesforce, Web Leads, Legacy databases",
    "schema_target": {
      "customer_id": "string",
      "email": "string (lowercase, canonical from Salesforce)",
      "phone": "string (E.164, prefer Web Leads when available)"
    },
    "available_sources": ["salesforce", "web_leads", "legacy_db", "merged_output"],
    "current_view": "Select a source to view.",
    "data_quality_report": "",
    "last_action_feedback": "Merged salesforce and web_leads into 'merged_output' using customer_id.",
    "conflict_rules": {
      "email": "prefer_salesforce",
      "phone": "prefer_web_leads",
      "customer_id": "prefer_salesforce"
    }
  },
  "reward": 0.05,
  "done": false
}
```

***

### `EXECUTE_SQL`

Executes a `SELECT` SQL query across all currently loaded sources using an in-memory SQLite connection. Every source DataFrame is registered as a table under its source name. The result is stored as a new source under `output_table` (defaulting to `"sql_output"`). Returns `reward = +0.1` on success.

> [!WARNING]
>
> Only `SELECT` queries are permitted. Any query beginning with `DROP`, `DELETE`,
> `UPDATE`, `INSERT`, `ALTER`, `CREATE`, or `TRUNCATE` is rejected with
> `reward = -0.1` and a blocked-action feedback message. This is enforced
> server-side and cannot be bypassed.

**Request:**

```json theme={null}
{
  "action_type": "EXECUTE_SQL",
  "query": "SELECT * FROM web_forms WHERE email IS NOT NULL AND customer_id NOT LIKE 'BOT%'",
  "output_table": "web_forms_clean"
}
```

**Response:**

```json theme={null}
{
  "observation": {
    "done": false,
    "reward": 0.1,
    "current_task_objective": "Normalize web_forms dataset",
    "schema_target": { "...": "..." },
    "available_sources": ["web_forms", "web_forms_clean"],
    "current_view": "Select a source to view.",
    "data_quality_report": "",
    "last_action_feedback": "Executed SQL successfully. Wrote 38 rows to 'web_forms_clean'.",
    "conflict_rules": null
  },
  "reward": 0.1,
  "done": false
}
```

**Multi-table join example:**

```json theme={null}
{
  "action_type": "EXECUTE_SQL",
  "query": "SELECT s.customer_id, s.email, w.phone FROM salesforce s LEFT JOIN web_leads w ON s.customer_id = w.customer_id",
  "output_table": "merged_output"
}
```

***

### `SUBMIT_PIPELINE`

Marks the end of the episode. The `final_source` DataFrame is frozen as the agent's submission. The server computes a preliminary `schema_match_ratio` — the fraction of `schema_target` columns that are present in the submitted DataFrame — and returns `reward = 0.2 × schema_match_ratio`. The terminal grader score is computed separately via `/grader/{episode_id}`.

> [!WARNING]
>
> You must execute **at least 3 steps** (any combination of profile, standardize,
> handle missing, deduplicate, merge, SQL) before submitting. Submitting with
> `step_count < 3` returns `reward = -0.15` and `done = false` — the episode
> continues and you are penalized.

**Request:**

```json theme={null}
{
  "action_type": "SUBMIT_PIPELINE",
  "final_source": "web_forms"
}
```

**Response (`step_count >= 3`, success):**

```json theme={null}
{
  "observation": {
    "done": true,
    "reward": 0.2,
    "current_task_objective": "Normalize web_forms dataset",
    "schema_target": {
      "customer_id": "string",
      "name": "string (stripped)",
      "email": "string (lowercase, stripped)",
      "signup_date": "string ISO 8601 or empty",
      "phone": "string E.164 format (+1...) or empty"
    },
    "available_sources": ["web_forms"],
    "current_view": "Select a source to view.",
    "data_quality_report": "",
    "last_action_feedback": "Pipeline submitted with final source: web_forms.",
    "conflict_rules": null
  },
  "reward": 0.2,
  "done": true
}
```

**Response (early submission, `step_count < 3`):**

```json theme={null}
{
  "observation": {
    "done": false,
    "reward": -0.15,
    "current_task_objective": "Normalize web_forms dataset",
    "schema_target": { "...": "..." },
    "available_sources": ["web_forms"],
    "current_view": "Select a source to view.",
    "data_quality_report": "",
    "last_action_feedback": "Early submission blocked. You must perform at least 3 cleaning steps (Profile, Standardize, etc.) before submitting.",
    "conflict_rules": null
  },
  "reward": -0.15,
  "done": false
}
```

***

## Error Response Pattern

When any action raises an exception — invalid source name, missing column, malformed SQL, incorrect join key — the server catches the exception, sets `reward = -0.05`, and continues the episode with `done = false`.

```json theme={null}
{
  "observation": {
    "done": false,
    "reward": -0.05,
    "current_task_objective": "Normalize web_forms dataset",
    "schema_target": { "...": "..." },
    "available_sources": ["web_forms"],
    "current_view": "Select a source to view.",
    "data_quality_report": "",
    "last_action_feedback": "Error: Column 'email_address' not found. Available: ['customer_id', 'name', 'email', 'signup_date', 'phone']",
    "conflict_rules": null
  },
  "reward": -0.05,
  "done": false
}
```

Always check `last_action_feedback` for the human-readable error string when `reward` is negative.

***

## Reward Reference

| Action | Success Reward | Error Reward |
| - | - | - |
| `VIEW_SOURCE` | `0.0` | `-0.05` |
| `PROFILE_SOURCE` | `+0.02` (fixed) | `-0.05` |
| `STANDARDIZE_COLUMN` | `+0.01` to `+0.05` (proportional to rows changed) | `-0.05` |
| `HANDLE_MISSING` | `+0.01` to `+0.10` (proportional to nulls resolved; max `+0.10` for `DROP_ROW`, max `+0.05` for `FILL_VALUE`) | `-0.05` |
| `DEDUPLICATE` | `removed/total × 0.2` | `-0.05` |
| `MERGE_SOURCES` | `+0.05` | `-0.05` |
| `EXECUTE_SQL` | `+0.1` (or `-0.1` for forbidden keywords) | `-0.1` |
| `SUBMIT_PIPELINE` (step \< 3) | `−0.15`, `done=false` | — |
| `SUBMIT_PIPELINE` (step ≥ 3) | `0.2 × schema_match_ratio`, `done=true` | `-0.05` |
| Unknown `action_type` | `-0.05` | — |

***

## Request Field Reference

**action\_type** `PipelineActionType` *required*

The operation to execute. Must be one of the `PipelineActionType` enum
values: `VIEW_SOURCE`, `PROFILE_SOURCE`, `STANDARDIZE_COLUMN`,
`HANDLE_MISSING`, `DEDUPLICATE`, `MERGE_SOURCES`, `EXECUTE_SQL`,
`SUBMIT_PIPELINE`.

**source** `string`

Name of the primary source DataFrame to operate on. Must be a value
currently present in `available_sources`. Required for `VIEW_SOURCE`,
`PROFILE_SOURCE`, `STANDARDIZE_COLUMN`, `HANDLE_MISSING`, `DEDUPLICATE`,
and `MERGE_SOURCES`.

**source2** `string`

Name of the secondary source DataFrame. Required only for `MERGE_SOURCES`
as the right-hand table in the outer join.

**column** `string`

The column name within `source` to mutate. Required for
`STANDARDIZE_COLUMN` and `HANDLE_MISSING`. Must be a column that exists
in the DataFrame or the action returns `reward = -0.05`.

**standardization\_strategy** `StandardizationStrategy`

The transformation to apply to `column`. Required for `STANDARDIZE_COLUMN`.
Enum values: `LOWERCASE_STRIP`, `EXTRACT_NUMBERS`, `TO_DATETIME_ISO`,
`PHONE_E164`.

**missing\_strategy** `MissingStrategy`

The strategy for handling nulls in `column`. Required for `HANDLE_MISSING`.
Enum values: `DROP_ROW`, `FILL_VALUE`, `FILL_MEAN`, `FILL_MODE`.

**deduplication\_strategy** `DeduplicationStrategy`

The heuristic for identifying duplicate rows. Required for `DEDUPLICATE`.
Enum values: `EXACT_EMAIL`, `FUZZY_NAME_PHONE`.

**fallback\_value** `string`

The literal string value used to fill nulls when `missing_strategy` is
`FILL_VALUE`. Pass `""` to fill with an empty string. Ignored for other
missing strategies.

**join\_key** `string`

The column name used as the join predicate for `MERGE_SOURCES`. Must exist
in both `source` and `source2` after the join-key normalization step
(both sides are lowercased and stripped before matching).

**conflict\_rule** `ConflictRule`

Governs how overlapping columns are resolved after the merge. Used only
in `MERGE_SOURCES`. Enum values: `PREFER_S1`, `PREFER_S2`, `COALESCE`.
Defaults to `COALESCE` if omitted.

**final\_source** `string`

The source name whose DataFrame is submitted as the agent's final output.
Required for `SUBMIT_PIPELINE`. Must be a value present in `available_sources`.

**query** `string`

A raw SQL `SELECT` statement to execute against the in-memory SQLite
database populated with all current sources as tables. Required for
`EXECUTE_SQL`. Must begin with `SELECT` — any other statement verb is
blocked and penalized.

**output\_table** `string`

The name under which the result DataFrame of an `EXECUTE_SQL` operation is
stored and surfaced in `available_sources`. Defaults to `"sql_output"` if
omitted. Note: `MERGE_SOURCES` always writes its output to the fixed name
`"merged_output"` regardless of this field.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.