Skip to main content

ClickHouse Migration Guide

Availability
Available CoreAvailable StandardAvailable PlusAvailable ProAvailable Enterprise FlexAvailable PyAirbyte
Support Level
Marketplace
Connector Version
0.3.1 (last updated a month ago)
Sync Success Rate
Low
Usage Rate
High
Data workers used
0.5 per sync (Learn more)
Definition ID
bad83517-5e54-4a3d-9b53-63e85fbd4d7c

Upgrading to 0.4.0​

Version 0.4.0 formally declares the temporal column typing introduced in version 0.3.1 as a breaking change. Before version 0.3.1, Date, Date32, DateTime, and DateTime64 columns were emitted as unformatted strings. These columns now emit Airbyte date or timestamp types.

Who is affected​

This change affects connections that sync at least one stream containing a Date, Date32, DateTime, or DateTime64 column. Typed destinations may reject the schema change because an existing string column can't always evolve to a date or timestamp column.

Connections whose selected streams don't contain these column types require no migration action.

Use your connection's current status to choose a migration path:

  • If the connection syncs successfully on version 0.3.1, upgrade to version 0.4.0. Don't refresh the schema or clear data. The destination already uses the new temporal types.
  • If the connection fails with a Schema evolution ... between string and timestamp is not allowed error, follow the steps below.
  • If the connection is still on version 0.3.0 or earlier, follow the steps below when you're ready to upgrade.

Resolve a schema evolution failure​

For connections that fail with the string-to-timestamp schema evolution error:

  1. Upgrade the ClickHouse source to version 0.4.0.
  2. Open the connection and select the Schema tab.
  3. Select Refresh source schema, then select OK. If no new changes appear because the connection already uses version 0.3.1, continue to the next step.
  4. Select Save changes and ensure Clear affected streams is selected for the failing streams.
  5. After the clear completes, run a new sync.

Clearing an affected stream removes its existing destination data and reads the stream again from the beginning.

If you can't clear the data:

  • A connection that has never successfully synced on version 0.3.1 or later can remain temporarily on version 0.3.0. Contact Airbyte Support before upgrading.
  • Don't downgrade a connection that has successfully synced on version 0.3.1 or later. Its destination already contains temporal columns, so downgrading can cause a timestamp-to-string schema evolution failure.

Connector upgrade guide​

Review the following information to prepare for and execute your upgrade.

Review the changelog​

Before updating a connector, review the changelog to understand the changes and their potential impact on your existing connections. Find the changelog for any connector by navigating to the bottom of the documentation for that connector. Major version releases also include a migration guide.

Plan for major updates​

Major updates may require you to adjust connection settings or even make changes to your data pipelines. Allocate enough time and resources for this. Use the migration guide to ensure your transition process goes smoothly.

Airbyte provides tooling that guarantees safe connector version bumps and enforces automated version bumps for minor and patch updates. You always need to manually update for major version bumps.

Self-managed plans: pin a specific version if you can't update​

If you're unable to upgrade to the new version of a connector, you can pin that connector to a specific version.

  1. In the navigation bar:

    • If you're on the Self-Managed Enterprise plan, click Organization settings > Sources/Destinations.

    • If you're on any other plan, click Workspace settings > Sources/Destinations.

  2. Edit the entry for the connector you want to pin.

  3. Set the Default Version to the version you want to use.

Self-managed plans: update the local connector image​

If you self-manage Airbyte, you must manually update the connector image in your local registry before proceeding with the migration. Follow the steps below.

  1. In the navigation bar:

    • If you're on the Self-Managed Enterprise plan, click Organization settings > Sources/Destinations.

    • If you're on any other plan, click Workspace settings > Sources/Destinations.

  2. Find the connector you want to update in the list of connectors.

    note

    Airbyte lists two versions, the current in-use version and the latest version available.

  3. Click Change to update your OSS version to the latest available version.

Update the connector version​

Update each instance of the connector separately. If you have multiple instances of a connector, updating one doesn't affect the others.

  1. In the navigation bar:

    • If you're on the Self-Managed Enterprise plan, click Organization settings > Sources/Destinations.

    • If you're on any other plan, click Workspace settings > Sources/Destinations.

  2. Select the instance of the connector you wish to upgrade.

  3. Select Upgrade.

  4. Follow the prompt to confirm you are ready to upgrade to the new version.

Clear data from affected streams​

After upgrading a connector with a breaking change, you must refresh affected schemas and clear your data.

  1. In the nav bar, click Connections.

  2. Find the connection affected by the upgrade.

  3. Click the Schema tab.

  4. Click Refresh source schema (looks like ). When Airbyte finishes, it shows you any detected schema changes.

  5. Click OK.

  6. Click Save changes

  7. Clear the data for the streams affected by this upgrade.

Once the clear is complete, you can begin syncing your data again as usual.

Was this page helpful?