Creating a pipeline

A pipeline moves data from a source through optional validation and transformation to a destination. This guide walks through the basic creation flow.

Choose a pipeline type

VendorGate supports two pipeline types:

• Single: validates one file against a single contract.
• Workbook: validates an Excel workbook with one contract per sheet. Workbook pipelines can use a decision policy such as all_must_pass or any_failure_fails, and individual sheets can be marked as optional with required: false.

For most vendors, start with a single pipeline.

Define the source

1. Select the source type: SFTP, FTP, S3-compatible storage, HTTP API, or Database.
2. Choose the credential that grants access.
3. Enter the source details:
   • For SFTP/FTP: the folder path to poll.
   • For S3: the bucket and key prefix.
   • For HTTP API: the endpoint, method, pagination strategy, and output mode.
   • For Database: the query to run.

Set the schedule

Pipelines can run:

• On demand: you trigger runs manually.
• On a cron schedule: for example, every hour at 15 minutes past the hour.
• By polling: at a fixed interval such as every 5 minutes.
• By event: triggered by another pipeline's validated output.

Add a contract

For single pipelines, select the contract that validates the incoming file. For workbook pipelines, assign a contract to each sheet.

Configure delivery

Choose where VendorGate should send the validated file. Options include SFTP, FTP, S3, HTTP API, Database, and Lookup dataset. You can configure different destinations for pass, review, and fail outcomes.

Add alerts

Pipeline alerts can notify your team by email or webhook when something happens, such as:

• A run fails.
• A run is marked for review.
• A run passes.
• The source is unavailable.
• Delivery fails.

Set execution options

Optionally enable fail-fast execution. When fail-fast is enabled, the pipeline stops processing after it reaches the configured maximum number of errors. This saves compute on files that are clearly invalid and surfaces the first set of problems quickly.

Save and test

After saving the pipeline, run it once manually to confirm the source, contract, and delivery all work together. Inspect the run details for any errors before enabling the schedule.