Lookup datasets

A lookup dataset is a reference table that a standardizer can use to enrich or validate data. Common uses include mapping vendor codes to internal names, validating country codes, or looking up product categories.

Creating a lookup dataset

1. Go to Lookup Datasets.
2. Click New Dataset.
3. Upload a CSV or Excel file, enter rows manually, or populate the dataset from a pipeline output.
4. Define the key column and any value columns.
5. Save.

Using a lookup dataset in a standardizer

Reference the dataset by its alias in the standardizer configuration. For example:

reference_datasets:
  - alias: brand_map
    dataset_id: "<dataset-id>"
    key_column: vendor_brand_code
    value_columns:
      - internal_brand_name

Then use it in a mapping:

output_column: brand_name
mode: steps
steps:
  - type: source_field
    params:
      alias: products
      field: brand_code
  - type: lookup
    params:
      dataset: brand_map
      match_column: vendor_brand_code
      return_column: internal_brand_name

Or in an expression:

output_column: brand_name
mode: expression
expression: 'lookup("brand_map", "vendor_brand_code", "internal_brand_name", brand_code)'

Same-dataset lookups

You can also look up values within the same input by using the input alias as the dataset. This is useful when a row needs to reference another row in the same file.

Performance

Lookup datasets are loaded into memory for the duration of a standardizer run. Keep datasets reasonably sized. If a lookup key is missing, the function returns an empty value.