Managing Connector Updates in Airbyte

CoreStandardProEnterprise FlexSelf-Managed Enterprise Compare

While maintaining an Airbyte instance, you'll need to manage connector updates. These are essential for improved functionality, reliability, and compatibility over time. Our team and community contributors are dedicated to maintaining up-to-date connectors. Reasons for updates can be broadly categorized into bug fixes, new features, and changes that impact the user experience or the functionality of the connector.

As various APIs and databases you're syncing with Airbyte evolve over time, we must review the connectors for compatibility with those changes. Sometimes, this kind of update introduces a notable change that may cause a connection to stop succeeding or your schema to change. Without your review of the changes, your connection may eventually stop working because it will become incompatible with the source or destination API.

This guide helps you understand the types of updates you may see, their impact on your Airbyte environment, and the actions you may need to take in response to certain types of updates.

Understanding Connector Versions

To manage connection updates effectively, it's important to understand versioning and how to interpret the changelog entries.

Connectors in Airbyte's catalog generally follow semantic versioning (semver) Major.Minor.Patch (e.g., 1.2.5)

A connector reaching version 1.0 is considered mature and comes with semver guarantees.

• Patch updates (1.0.x): These typically contain bug fixes and small improvements that won't affect your existing configurations.
• Minor updates (1.x.0): These might introduce new features like streams or properties, but they are designed to be fully backward compatible with your existing setup.
• Major updates (x.0.0): These are significant changes that may require your review. We'll discuss this in more detail below.

Each connector's changelog details its update history. You can find it in the connector catalog at the end of each individual connector's entry.

How Airbyte Handles Connector Updates

Airbyte Cloud

Minor and Patch Versions: These are applied automatically and immediately to your instance. You don't need to take any action.

Major Versions: A major version will include notable changes that affect your schema or sync success. We will notify you ahead of time to give you a window to prepare for the change. At the end of the window, we will automatically upgrade your connector to ensure you receive the latest updates. Examples of major version changes are shared in our breaking change documentation.

Airbyte Open Source (OSS) and Self-Managed Enterprise (SME)

Airbyte recommends using the latest version of Airbyte when updating connections.

All Versions (Major, Minor, Patch): These are opt-in via the settings page. You'll see a badge in the sidebar indicating available updates.

Minor and Patch Versions: Once you opt-in, these are applied immediately and globally to all connectors of that type in your instance.

Major Versions: These require a two-step opt-in process:

1. Opt-in to the update on the settings page.

2. While there is an option available to upgrade all connections at once, we recommend that you accept the update for each individual connection using the affected connector. This allows you to review and potentially adjust the connection settings before applying the update.

When new major versions are released, we recommended you update promptly to avoid potential compatibility issues.

Actions to Take in Response to Connector Updates

Review the following information to prepare for and execute your upgrade.

Review the changelog

Before updating a connector, review the changelog to understand the changes and their potential impact on your existing connections. Find the changelog for any connector by navigating to the bottom of the documentation for that connector. Major version releases also include a migration guide.

Plan for major updates

Major updates may require you to adjust connection settings or even make changes to your data pipelines. Allocate enough time and resources for this. Use the migration guide to ensure your transition process goes smoothly.

Airbyte provides tooling that guarantees safe connector version bumps and enforces automated version bumps for minor and patch updates. You always need to manually update for major version bumps.

Self-managed plans: pin a specific version if you can't update

If you're unable to upgrade to the new version of a connector, you can pin that connector to a specific version.

1. In the navigation bar:

   • If you're on the Self-Managed Enterprise plan, click Organization settings > Sources/Destinations.

   • If you're on any other plan, click Workspace settings > Sources/Destinations.

2. Edit the entry for the connector you want to pin.

3. Set the Default Version to the version you want to use.

Self-managed plans: update the local connector image

If you self-manage Airbyte, you must manually update the connector image in your local registry before proceeding with the migration. Follow the steps below.

1. In the navigation bar:

   • If you're on the Self-Managed Enterprise plan, click Organization settings > Sources/Destinations.

   • If you're on any other plan, click Workspace settings > Sources/Destinations.

2. Find the connector you want to update in the list of connectors.

   note

   Airbyte lists two versions, the current in-use version and the latest version available.

3. Click Change to update your OSS version to the latest available version.

Update the connector version

Update each instance of the connector separately. If you have multiple instances of a connector, updating one doesn't affect the others.

1. In the navigation bar:

   • If you're on the Self-Managed Enterprise plan, click Organization settings > Sources/Destinations.

   • If you're on any other plan, click Workspace settings > Sources/Destinations.

2. Select the instance of the connector you wish to upgrade.

3. Select Upgrade.

4. Follow the prompt to confirm you are ready to upgrade to the new version.

Clear data from affected streams

After upgrading a connector with a breaking change, you must refresh affected schemas and clear your data.

1. In the nav bar, click Connections.

2. Find the connection affected by the upgrade.

3. Click the Schema tab.

4. Click Refresh source schema (looks like ). When Airbyte finishes, it shows you any detected schema changes.

5. Click OK.

6. Click Save changes

7. Clear the data for the streams affected by this upgrade.

Once the clear is complete, you can begin syncing your data again as usual.