Pipelines

Pipelines are the core mechanism for moving data in VendorGate. A pipeline connects a source, one or more contracts, and a delivery destination. A single pipeline uses one contract; a workbook pipeline uses one contract per sheet. Pipelines can run on a schedule or on demand.

Pipeline components

Every pipeline has at least three parts:

• Source: where the data comes from.
• Contract: the rules the file must satisfy before delivery.
• Delivery: where the data goes after processing.

You can also configure:

• Transformation: to standardize or reshape the file before delivery.
• Execution options: such as fail-fast, which stops processing after a configured number of errors.

Supported sources

VendorGate supports several source types:

• SFTP / FTP: poll a remote folder for new files.
• HTTP API: fetch data from a REST endpoint.
• S3-compatible storage: connect to buckets on Amazon S3, MinIO, or similar services.
• Database: run a query against a supported database.

Supported delivery destinations

After processing, data can be delivered to:

• SFTP / FTP: write to a remote folder.
• HTTP API: post to an endpoint.
• Database: insert into a table.
• S3-compatible storage: write to a bucket on Amazon S3, MinIO, or similar services.
• Lookup dataset: load the output into a reference dataset that standardizers can use.

Creating a pipeline

1. Go to Pipelines.
2. Click New Pipeline.
3. Choose a source type and select the credential that grants access.
4. Configure the source details, such as folder path or endpoint URL.
5. Select the contract that validates the file. For workbook pipelines, select one contract per sheet.
6. Choose a delivery destination and credential.
7. Set the schedule or leave it manual.
8. Save.

Running a pipeline

You can run a pipeline manually from the pipeline detail page or let the schedule trigger it. Each run produces a log of events, including how many rows were processed and any validation errors.

Pipeline runs

The run history shows every execution, its status, and a link to detailed logs. Use this view to debug failures, monitor throughput, and confirm that files are arriving on time.