Skip to main content

Customer.io Migration Guide

Availability
Available CoreAvailable StandardAvailable PlusAvailable ProAvailable Enterprise FlexAvailable PyAirbyte
Connector type (beta)
Available Data replicationAvailable Agent(learn more)
Support Level
Marketplace
Connector Version
0.8.0 (last updated a day ago)
Sync Success Rate
Medium
Usage Rate
High
Data workers used
0.2 per sync (Learn more)
Definition ID
34f697bc-b989-4eda-b06f-d0f39b88825b

Upgrading to 1.0.0​

Risk of permanent data loss

Clearing a stream empties its destination table; the next sync reloads it from Customer.io. Customer.io doesn't return deleted automations, actions or one-time sends, so their rows aren't restored, and in Incremental | Append and Full refresh | Append mode the clear also removes every earlier version of each record that previous syncs kept. Back up the affected tables before you clear them if you need that history.

Version 1.0.0 changes the declared type of seven fields whose type didn't match Customer.io's API reference. The values Customer.io returns are unchanged.

Retyped fields
  • campaigns_actions (automation actions): from_id and reply_to_id change from string to integer.
  • newsletters (one-time sends): sent_at changes from array to integer, the Unix time in seconds of the last send. Most typed destinations wrote it as null under the old type; MotherDuck and destinations that store raw JSON kept the value.
  • newsletters: tags becomes an array of strings and content_ids an array of integers.
  • campaigns (automations): tags becomes an array of strings and trigger_segment_ids an array of integers.

Under the old types, typed destinations already stored the right values for the other six fields: from_id and reply_to_id as text, and the arrays as JSON. What the upgrade does depends on the destination:

  • BigQuery, Redshift, Databricks, ClickHouse and Postgres 11 or later convert sent_at, from_id and reply_to_id to integers in place and keep the same array columns.
  • Snowflake, SQL Server and Postgres 10 or older convert from_id and reply_to_id in place but can't convert sent_at; step 1 below handles it.
  • MotherDuck keeps the existing column types until a Full refresh | Overwrite sync or a clear recreates the table: until then sent_at stays JSON, and from_id and reply_to_id stay text.
  • S3 Data Lake and GCS Data Lake can't convert these fields in place; they store integers and lists after a clear (step 4) or a Full refresh | Overwrite sync.
  • Avro and Parquet files written after the upgrade hold integers and lists where earlier files hold text, JSON strings and a null sent_at. JSONL and CSV files and DuckDB don't use the declared types and don't change.

If you don't sync campaigns, campaigns_actions or newsletters, you don't need to take any action.

To upgrade:

  1. If your destination is Snowflake, SQL Server or Postgres 10 or older and you sync newsletters, drop the sent_at column from its destination table before the first sync on 1.0.0. The column holds only nulls, and these destinations can't convert it to the new type, so every sync fails until it's gone; the next sync adds it back as an integer column. On Snowflake, also drop any SENT_AT_ column with a suffix that a failed sync left behind.

  2. If your destination is Postgres or Redshift, drop every view that reads newsletters.sent_at, campaigns_actions.from_id or campaigns_actions.reply_to_id, including SELECT * views, before the first sync on 1.0.0, and recreate the views after it. Neither database changes the type of a column that a view depends on, so the sync fails. The Drop tables with CASCADE option doesn't help on Postgres; on Redshift, Drop tables and columns with CASCADE drops those views instead. On Redshift you can also recreate the views WITH NO SCHEMA BINDING.

  3. Open the connection, go to Schema and click Refresh source schema.

  4. Clear only the streams that need it. When Airbyte offers to clear the streams whose schema changed, clear only these:

    • On S3 Data Lake and GCS Data Lake, clear campaigns, campaigns_actions and newsletters if you sync them in Incremental or Full refresh | Append mode; until you do, every sync of the connection fails.
    • On other destinations, clear newsletters if you sync it in an Incremental mode and want sent_at on the rows synced before the upgrade. In Full refresh | Append mode, the first sync on 1.0.0 already appends a full copy with sent_at, and a clear only deletes the earlier copies. MotherDuck, DuckDB, and JSONL or CSV files already hold sent_at, so don't clear for it there. campaigns and campaigns_actions need no clear.
    • Streams in Full refresh | Overwrite mode are rebuilt on every sync and need no clear.

    If you moved your Start Date later after the first sync, move it back before clearing; the re-sync only reads records last updated on or after it.

  5. Run a sync.

If you skip the newsletters clear on a destination that wrote sent_at as null, rows synced before the upgrade keep a null sent_at: in Incremental | Append + Deduped mode until the one-time send changes in Customer.io, and permanently in Incremental | Append and Full refresh | Append mode.

Update downstream consumers​

Update casts and joins on the retyped columns: campaigns_actions.from_id and campaigns_actions.reply_to_id are integers that join to sender_identities.id; newsletters.sent_at is a Unix timestamp in seconds; campaigns.trigger_segment_ids holds segments.id values and newsletters.content_ids holds newsletter_variants.id values. Queries that read the four arrays as JSON strings from S3 Data Lake, GCS Data Lake, or Avro or Parquet files must read them as lists. On MotherDuck, cast sent_at, from_id and reply_to_id to BIGINT where you need integers until the table is recreated.

Connector upgrade guide​

Review the following information to prepare for and execute your upgrade.

Review the changelog​

Before updating a connector, review the changelog to understand the changes and their potential impact on your existing connections. Find the changelog for any connector by navigating to the bottom of the documentation for that connector. Major version releases also include a migration guide.

Plan for major updates​

Major updates may require you to adjust connection settings or even make changes to your data pipelines. Allocate enough time and resources for this. Use the migration guide to ensure your transition process goes smoothly.

Airbyte provides tooling that guarantees safe connector version bumps and enforces automated version bumps for minor and patch updates. You always need to manually update for major version bumps.

Self-managed plans: pin a specific version if you can't update​

If you're unable to upgrade to the new version of a connector, you can pin that connector to a specific version.

  1. In the navigation bar:

    • If you're on the Self-Managed Enterprise plan, click Organization settings > Sources/Destinations.

    • If you're on any other plan, click Workspace settings > Sources/Destinations.

  2. Edit the entry for the connector you want to pin.

  3. Set the Default Version to the version you want to use.

Self-managed plans: update the local connector image​

If you self-manage Airbyte, you must manually update the connector image in your local registry before proceeding with the migration. Follow the steps below.

  1. In the navigation bar:

    • If you're on the Self-Managed Enterprise plan, click Organization settings > Sources/Destinations.

    • If you're on any other plan, click Workspace settings > Sources/Destinations.

  2. Find the connector you want to update in the list of connectors.

    note

    Airbyte lists two versions, the current in-use version and the latest version available.

  3. Click Change to update your OSS version to the latest available version.

Update the connector version​

Update each instance of the connector separately. If you have multiple instances of a connector, updating one doesn't affect the others.

  1. In the navigation bar:

    • If you're on the Self-Managed Enterprise plan, click Organization settings > Sources/Destinations.

    • If you're on any other plan, click Workspace settings > Sources/Destinations.

  2. Select the instance of the connector you wish to upgrade.

  3. Select Upgrade.

  4. Follow the prompt to confirm you are ready to upgrade to the new version.

Clear data from affected streams​

After upgrading a connector with a breaking change, you must refresh affected schemas and clear your data.

  1. In the nav bar, click Connections.

  2. Find the connection affected by the upgrade.

  3. Click the Schema tab.

  4. Click Refresh source schema (looks like ). When Airbyte finishes, it shows you any detected schema changes.

  5. Click OK.

  6. Click Save changes

  7. Clear the data for the streams affected by this upgrade.

Once the clear is complete, you can begin syncing your data again as usual.

Was this page helpful?