Transformations

Transformations, also called standardizers, convert validated vendor data into a clean, predictable output schema. They are where you map columns, clean values, apply business logic, and enrich data from reference datasets.

When to use a transformation

Use a transformation when vendor files do not match the schema your downstream systems expect. Common reasons include:

• Column names differ between vendors.
• Date or number formats are inconsistent.
• Multiple vendor fields need to be combined into one.
• Values need to be looked up or normalized.

Building a standardizer

A standardizer is built around output columns. For each output column, you choose how the value is produced:

• Simple source field: copy a value directly from an input column.
• Transform steps: apply a sequence of operations such as source_field, uppercase, lowercase, trim, strip_non_numeric, prepend, append, default, regex_extract, json_extract, lookup, decimal_round, expression, or date_reformat.
• Expression: write a formula that references input columns, functions, and lookup datasets.
• LLM mapping: send a prompt to a provider-owned LLM and parse the returned value.

Inputs and aliases

A standardizer can read from one or more pipeline inputs. Each input is given an alias, such as orders or suppliers. You reference fields in expressions using the alias and column name, for example orders.quantity.

When you need to combine two inputs, you can configure a join on matching keys using inner, left, right, outer, or cross joins. Join keys can be single columns or composite lists.

Dependencies

A standardizer can declare dependencies on other pipeline or transformation runs. Dependencies let you require that another run succeeded within a freshness window (max_age_hours) before this standardizer can run.

Expressions

Expressions support arithmetic, comparisons, boolean logic, and helper functions such as:

• upper(value), lower(value), trim(value): string case and whitespace helpers.
• coalesce(value, fallback), ifnull(value, fallback): return the first non-null value.
• round(value, decimals), abs(value), min(...), max(...), clamp(value, lo, hi): numeric helpers.
• lookup(dataset_alias, match_column, return_column, value): look up a value from a reference dataset or input alias.
• regex_extract(value, pattern, group): extract a regex capture group.
• json_extract(value, path): extract a value from JSON.
• if_else(cond, true_val, false_val), case(...): conditional logic.
• is_blank(value), is_null(value), nullif(value, sentinel): null handling.
• parse_money(value), normalize_phone(value), normalize_email(value): vendor-data normalization.

See the Expressions page for the full function reference.

Lookup datasets

Lookup datasets let you enrich or validate data without hard-coding values in your standardizer. For example, you can map raw supplier codes to canonical names by loading a reference file and using the lookup() function. Datasets can be created manually, imported from a file, or populated from a pipeline output.

Output and delivery

Once the standardizer runs, the output file contains only the columns you defined, with values transformed according to your rules. You can deliver the output to any supported destination, either directly from the standardizer or through a pipeline.

Example

A vendor sends an invoice file with columns inv_id, vendor_code, and inv_amt. Your target schema expects invoice_id, supplier_name, and amount. A standardizer can:

• Map inv_id to invoice_id.
• Look up vendor_code in a supplier dataset to produce supplier_name.
• Copy and format inv_amt as amount.