Youtube Data Migration Guide

Availability

Core Standard Plus Pro Enterprise Flex PyAirbyte

Support Level

Marketplace

Connector Version

0.0.66 (last updated 3 days ago)

Sync Success Rate

Medium

Usage Rate

High

Data workers used

0.2 per sync (Learn more)

Definition ID

743a2a44-fd13-4109-a8fe-fb0e68f467f5

Upgrading to 1.0.0

note

This change affects all streams. The primary-key and result-set changes are limited to comments, videos, and channel_comments; the timestamp typing affects every stream.

Version 1.0.0 makes three changes that alter what lands in your destination:

• Primary keys declared. videos is now keyed by videoId, and comments / channel_comments by composite keys ([videoId, id] / [channelId, id]), where id is a new top-level field carrying the comment-thread id. Declaring primary keys does not change existing syncs by itself; it lets you select a deduplicating sync mode for these streams if your destination supports one.
• Timestamp fields typed. Nine fields (publishedAt, updatedAt, publishAt, timeLinked, datetime, and their nested occurrences) now declare format: date-time, so destinations that map JSON-schema formats will change these columns from plain strings to timestamp types. The connector-synthesized video.datetime field also changed from Python's space-separated form to ISO-8601.
• videos returns only videos. The stream's search request now pins type=video; channel and playlist id records that previously appeared (with a null videoId) are no longer returned.

danger

If any stream uses Full Refresh | Append, clearing it permanently deletes history the source cannot re-supply: the per-sync video.statistics snapshots, videos beyond YouTube's 500-result search limit for a channel, and comments since deleted on YouTube. Snapshot or rename those tables before clearing, or skip the reset and accept mixed datetime formats in existing rows.

Migration Steps

Refresh affected schemas and reset data where needed

1. Select Connections in the main nav bar.
   1. Select the connection affected by the update.
2. Select the Schema tab.
   1. Select Refresh source schema.
   2. Select OK.

note

Any detected schema changes will be listed for your review.

1. Select Save changes at the top right of the page.
   1. If your streams use Full Refresh | Overwrite, or your destination fails to alter the timestamp columns in place, check Reset affected streams. Otherwise leave it unchecked.

note

Depending on destination type you may not be prompted to reset your data.

1. Select Save connection.

note

If you checked Reset affected streams, this clears the data in your destination and initiates a fresh sync. Otherwise the next sync continues with the updated schema.

Data-lake destinations (for example Iceberg-based ones) do not recreate the physical table on a stream reset. If a sync fails after upgrading with a schema-evolution error on a timestamp column, drop or recreate the affected destination tables, then run a fresh sync. For Full Refresh | Append streams this permanently deletes the accumulated history, so snapshot those tables first.

For more information on resetting your data in Airbyte, see this page

Connector upgrade guide

Review the following information to prepare for and execute your upgrade.

Review the changelog

Before updating a connector, review the changelog to understand the changes and their potential impact on your existing connections. Find the changelog for any connector by navigating to the bottom of the documentation for that connector. Major version releases also include a migration guide.

Plan for major updates

Major updates may require you to adjust connection settings or even make changes to your data pipelines. Allocate enough time and resources for this. Use the migration guide to ensure your transition process goes smoothly.

Airbyte provides tooling that guarantees safe connector version bumps and enforces automated version bumps for minor and patch updates. You always need to manually update for major version bumps.

Self-managed plans: pin a specific version if you can't update

If you're unable to upgrade to the new version of a connector, you can pin that connector to a specific version.

1. In the navigation bar:

   • If you're on the Self-Managed Enterprise plan, click Organization settings > Sources/Destinations.

   • If you're on any other plan, click Workspace settings > Sources/Destinations.

2. Edit the entry for the connector you want to pin.

3. Set the Default Version to the version you want to use.

Self-managed plans: update the local connector image

If you self-manage Airbyte, you must manually update the connector image in your local registry before proceeding with the migration. Follow the steps below.

1. In the navigation bar:

   • If you're on the Self-Managed Enterprise plan, click Organization settings > Sources/Destinations.

   • If you're on any other plan, click Workspace settings > Sources/Destinations.

2. Find the connector you want to update in the list of connectors.

   note

   Airbyte lists two versions, the current in-use version and the latest version available.

3. Click Change to update your OSS version to the latest available version.

Update the connector version

Update each instance of the connector separately. If you have multiple instances of a connector, updating one doesn't affect the others.

1. In the navigation bar:

   • If you're on the Self-Managed Enterprise plan, click Organization settings > Sources/Destinations.

   • If you're on any other plan, click Workspace settings > Sources/Destinations.

2. Select the instance of the connector you wish to upgrade.

3. Select Upgrade.

4. Follow the prompt to confirm you are ready to upgrade to the new version.

Clear data from affected streams

After upgrading a connector with a breaking change, you must refresh affected schemas and clear your data.

1. In the nav bar, click Connections.

2. Find the connection affected by the upgrade.

3. Click the Schema tab.

4. Click Refresh source schema (looks like ). When Airbyte finishes, it shows you any detected schema changes.

5. Click OK.

6. Click Save changes

7. Clear the data for the streams affected by this upgrade.

Once the clear is complete, you can begin syncing your data again as usual.