Skip to main content

Gmail Migration Guide

Availability
Available CoreAvailable StandardAvailable PlusAvailable ProAvailable Enterprise FlexAvailable PyAirbyte
Connector type
Available Data replicationAvailable Agent(learn more)
Support Level
Marketplace
Connector Version
0.3.0 (last updated 37 minutes ago)
Sync Success Rate
Low
Usage Rate
High
Data workers used
0.2 per sync (Learn more)
Definition ID
f7833dac-fc18-4feb-a2a9-94b22001edc6

Upgrading to 1.0.0​

Risk of permanent data loss

Only run Refresh stream and remove records if step 3 below tells you to. It deletes the existing destination rows before re-syncing, and Gmail can only return messages that still exist in the mailbox and are allowed by the connection's start date and spam/trash settings. Messages that were deleted, trashed or marked as spam since the original sync don't come back. Back up the affected tables before removing records if you need that data.

Version 1.0.0 declares the count and size fields of four streams as integers, and gives the profile stream a primary key. The values themselves don't change: Gmail documents these fields as int32 integers, and the connector already emitted whole numbers. Only the declared type changes, so typed destinations change the matching columns from a decimal or float type to an integer type.

What changes​

The profile stream now uses emailAddress as its primary key. The following fields change from number to integer; they stay nullable:

  • profile: messagesTotal, threadsTotal
  • labels_details: messagesTotal, messagesUnread, threadsTotal, threadsUnread
  • messages_details: sizeEstimate, payload.body.size, payload.parts[].body.size, payload.parts[].parts[].body.size, and payload.parts[].parts[].parts[].body.size
  • threads_details: messages[].sizeEstimate, messages[].payload.body.size, messages[].payload.parts[].body.size, messages[].payload.parts[].parts[].body.size, and messages[].payload.parts[].parts[].parts[].body.size

Why​

The Gmail API documents these fields as int32 integers in its users.getProfile, users.labels and users.messages references. The profile endpoint returns one record for the authenticated mailbox, and emailAddress identifies that record, so you can now select a deduplicated sync mode for profile.

Who is affected​

Connections that sync at least one of profile, labels_details, messages_details or threads_details. Connections that sync only drafts, labels, messages or threads aren't affected and need no action.

Among affected connections:

  • Destinations that type their columns (BigQuery, Snowflake, Postgres, Redshift, MSSQL, ClickHouse and similar) change messagesTotal, messagesUnread, threadsTotal, threadsUnread and sizeEstimate from a decimal or float column to an integer column. The nested payload and messages objects are stored as JSON, so their size fields don't change a column type. Most of these destinations convert the existing column in place on the next sync.
  • S3 Data Lake (Iceberg) types nested fields too, and can't convert a double column to a long column in place. All four streams in an Append or Append + Deduped mode fail with Schema evolution for column "..." between double and long is not allowed until you refresh them; see step 3. Refresh stream and remove records recreates the table with the new column types.
  • Object storage and file destinations that write JSON or CSV aren't affected: the values are written the same way as before.
  • profile in Full Refresh | Append keeps one row per sync, as before. Connections that set their own primary key for profile now use emailAddress.

Upgrade steps​

  1. Upgrade the Gmail source to version 1.0.0.
  2. Refresh the source schema for each affected connection and review the detected changes before saving.
  3. Run a sync. Most destinations convert the columns automatically and no further action is needed. If the sync fails with a schema-evolution error, for example on S3 Data Lake in an Append or Append + Deduped mode, refresh only that stream with Refresh stream → Refresh stream and remove records. Read the warning at the top of this guide first: a messages_details re-sync reads every message from your start date again, one messages.get call per message, so large mailboxes take a long time and messages deleted from the mailbox don't come back. Streams in Full Refresh | Overwrite are rebuilt by every sync and never need a separate refresh.
  4. Update downstream queries, models and exports that depend on the changed columns or on the profile key.

Update downstream queries​

On destinations that type their columns, these seven columns become integer columns (for example INT64 in BigQuery, NUMBER(38,0) in Snowflake, BIGINT in Postgres): profile.messagesTotal, profile.threadsTotal, labels_details.messagesTotal, labels_details.messagesUnread, labels_details.threadsTotal, labels_details.threadsUnread and messages_details.sizeEstimate. Check:

  • Divisions between two of these columns. In Postgres, messagesUnread / messagesTotal on integer columns truncates to 0; cast one side to a decimal type first.
  • Casts and type checks that expected a decimal or float column, such as CAST(messagesTotal AS INTEGER), which are no longer needed.
  • Models or dashboards that deduplicate profile on a key of their own; the stream's key is now emailAddress.

For more information about refresh options, see Refresh data.

Connector upgrade guide​

Review the following information to prepare for and execute your upgrade.

Review the changelog​

Before updating a connector, review the changelog to understand the changes and their potential impact on your existing connections. Find the changelog for any connector by navigating to the bottom of the documentation for that connector. Major version releases also include a migration guide.

Plan for major updates​

Major updates may require you to adjust connection settings or even make changes to your data pipelines. Allocate enough time and resources for this. Use the migration guide to ensure your transition process goes smoothly.

Airbyte provides tooling that guarantees safe connector version bumps and enforces automated version bumps for minor and patch updates. You always need to manually update for major version bumps.

Self-managed plans: pin a specific version if you can't update​

If you're unable to upgrade to the new version of a connector, you can pin that connector to a specific version.

  1. In the navigation bar:

    • If you're on the Self-Managed Enterprise plan, click Organization settings > Sources/Destinations.

    • If you're on any other plan, click Workspace settings > Sources/Destinations.

  2. Edit the entry for the connector you want to pin.

  3. Set the Default Version to the version you want to use.

Self-managed plans: update the local connector image​

If you self-manage Airbyte, you must manually update the connector image in your local registry before proceeding with the migration. Follow the steps below.

  1. In the navigation bar:

    • If you're on the Self-Managed Enterprise plan, click Organization settings > Sources/Destinations.

    • If you're on any other plan, click Workspace settings > Sources/Destinations.

  2. Find the connector you want to update in the list of connectors.

    note

    Airbyte lists two versions, the current in-use version and the latest version available.

  3. Click Change to update your OSS version to the latest available version.

Update the connector version​

Update each instance of the connector separately. If you have multiple instances of a connector, updating one doesn't affect the others.

  1. In the navigation bar:

    • If you're on the Self-Managed Enterprise plan, click Organization settings > Sources/Destinations.

    • If you're on any other plan, click Workspace settings > Sources/Destinations.

  2. Select the instance of the connector you wish to upgrade.

  3. Select Upgrade.

  4. Follow the prompt to confirm you are ready to upgrade to the new version.

Clear data from affected streams​

After upgrading a connector with a breaking change, you must refresh affected schemas and clear your data.

  1. In the nav bar, click Connections.

  2. Find the connection affected by the upgrade.

  3. Click the Schema tab.

  4. Click Refresh source schema (looks like ). When Airbyte finishes, it shows you any detected schema changes.

  5. Click OK.

  6. Click Save changes

  7. Clear the data for the streams affected by this upgrade.

Once the clear is complete, you can begin syncing your data again as usual.

Was this page helpful?