Using the Kestra Plugin
Kestra has an official plugin for Airbyte, including support for self-hosted Airbyte and Airbyte Cloud. This plugin allows you to trigger data replication jobs (
Syncs) and wait for their completion before proceeding with any downstream tasks. Alternatively, you may also run those syncs in a fire-and-forget way by setting the
wait argument to
After Airbyte tasks successfully ingest raw data, you can easily start running downstream data transformations with dbt, Python, SQL, Spark, and many more, using a variety of available plugins. Check the plugin documentation for a list of all supported integrations.
These are the two main tasks to orchestrate Airbyte syncs:
io.kestra.plugin.airbyte.connections.Sync task will sync connections for a self-hosted Airbyte instance
io.kestra.plugin.airbyte.cloud.jobs.Sync task will sync connections for Airbyte Cloud
1. Set up the tools
First, make sure you have Docker installed. We'll be using the
docker-compose command, so your installation should contain
docker-compose. When you use Docker Desktop, Docker Compose is already included.
If this is your first time using Airbyte, we suggest following the Quickstart Guide. When creating Airbyte connections intended to be orchestrated with Kestra, set your Connection's sync frequency to manual. Kestra will automate triggering Airbyte jobs in response to external events or based on a schedule you’ll provide.
If you haven’t started Kestra yet, download the Docker Compose file:
curl -o docker-compose.yml https://raw.githubusercontent.com/kestra-io/kestra/develop/docker-compose.yml
2. Create a flow from the UI
Kestra UI provides a wide range of Blueprints to help you get started.
Navigate to Blueprints. Then type "Airbyte" in the search bar to find the desired integration. This way, you can easily accomplish fairly standardized data orchestration tasks, such as the following:
- Run a single Airbyte sync on a schedule
- Run multiple Airbyte syncs in parallel
- Run multiple Airbyte syncs in parallel, then clone a Git repository with dbt code and trigger dbt CLI commands
- Run a single Airbyte Cloud sync on a schedule
- Run multiple Airbyte Cloud syncs in parallel
- Run multiple Airbyte Cloud syncs in parallel, then clone a Git repository with dbt code and trigger dbt CLI commands
- Run multiple Airbyte Cloud syncs in parallel, then run a dbt Cloud job
Select a blueprint matching your use case and click "Use".
Then, within the editor, adjust the connection ID and task names and click "Save". Finally, trigger your flow.
3. Simple demo
Here is an example flow that triggers multiple Airbyte connections in parallel to sync data for multiple Pokémon.
description: Gotta catch ‘em all!
- id: data-ingestion
- id: charizard
- id: pikachu
- id: psyduck
- type: io.kestra.plugin.airbyte.connections.Sync
- id: everyMinute
cron: "*/1 * * * *"
If you liked that demo, check out the blog post about using Airbyte and Kestra Terraform providers together to manage Everything as Code.