Skip to main content

Goldcast Migration Guide

Availability
Available CoreAvailable StandardAvailable PlusAvailable ProAvailable Enterprise FlexAvailable PyAirbyte
Support Level
Marketplace
Connector Version
0.2.26 (last updated a month ago)
Sync Success Rate
Medium
Usage Rate
Low
Data workers used
0.2 per sync (Learn more)
Definition ID
c2c25d04-9bb1-4171-8a7a-bb7cead8add0

Upgrading to 1.0.0​

note

This change only affects you if you sync the event_members stream to S3 or GCS in Avro or Parquet format and read fields inside props. Database and data-lake destinations (BigQuery, Snowflake, Redshift, Postgres, ClickHouse, MSSQL, S3 Data Lake, GCS Data Lake) already store props as a complete JSON value and are not affected - no action is needed there.

props carries Goldcast registration form fields, which every workspace defines for itself. Until now the schema enumerated a fixed list of eleven of them (city, solutions, tag_source, utm_source, utm_medium, utm_campaign, utm_content, tag_country, tag_form_type, revenue_type, contact_job_title). On S3 and GCS in Avro or Parquet format, props was written as a record with exactly those eleven fields, and any other registration field was dropped silently - it did not even appear in _airbyte_meta.changes[]. Destinations that store objects as JSON (databases and data lakes) always kept every field.

props is now a schemaless object. Avro and Parquet files carry it as a JSON string, so every field every workspace defines is preserved, and no schema change is needed when a workspace adds one. The trade-off is that props is delivered as a JSON string rather than a set of nested columns.

What to do​

  • Database or data-lake destination: nothing. The props column keeps the same type and content.
  • S3 or GCS in Avro/Parquet, Full Refresh | Overwrite: refresh the source schema (steps below). The next sync rewrites event_members with props as a JSON string; a reset is optional.
  • S3 or GCS in Avro/Parquet, Full Refresh | Append: refresh the source schema. Then either keep the existing files and handle both shapes of props in your readers, or reset the stream to get one consistent shape - read the warning first.
Risk of permanent data loss

Resetting an event_members stream synced in Full Refresh | Append deletes every earlier snapshot in the destination. Goldcast only returns the current registrants, so those point-in-time snapshots cannot be re-fetched. Copy or archive the stream's prefix before resetting.

Update downstream consumers​

Readers of the Avro or Parquet files that access props.<field> as a nested column must extract the value from the JSON string instead:

  • Athena / Trino: json_extract_scalar(props, '$.city')
  • Spark: get_json_object(props, '$.city')
  • DuckDB: props->>'$.city'

Glue or Athena table definitions that declare props as a struct need the column redefined as string.

Refresh affected schemas and reset data​

  1. Select Connections in the main nav bar.
    1. Select the connection affected by the update.
  2. Select the Schema tab.
    1. Select Refresh source schema.
    2. Select OK.
  3. Select Save changes at the bottom of the page.
    1. Check Reset affected streams only if you decided to reset the stream (see above).
  4. Select Save connection.

For more information on resetting your data in Airbyte, see this page.

Connector upgrade guide​

Review the following information to prepare for and execute your upgrade.

Review the changelog​

Before updating a connector, review the changelog to understand the changes and their potential impact on your existing connections. Find the changelog for any connector by navigating to the bottom of the documentation for that connector. Major version releases also include a migration guide.

Plan for major updates​

Major updates may require you to adjust connection settings or even make changes to your data pipelines. Allocate enough time and resources for this. Use the migration guide to ensure your transition process goes smoothly.

Airbyte provides tooling that guarantees safe connector version bumps and enforces automated version bumps for minor and patch updates. You always need to manually update for major version bumps.

Self-managed plans: pin a specific version if you can't update​

If you're unable to upgrade to the new version of a connector, you can pin that connector to a specific version.

  1. In the navigation bar:

    • If you're on the Self-Managed Enterprise plan, click Organization settings > Sources/Destinations.

    • If you're on any other plan, click Workspace settings > Sources/Destinations.

  2. Edit the entry for the connector you want to pin.

  3. Set the Default Version to the version you want to use.

Self-managed plans: update the local connector image​

If you self-manage Airbyte, you must manually update the connector image in your local registry before proceeding with the migration. Follow the steps below.

  1. In the navigation bar:

    • If you're on the Self-Managed Enterprise plan, click Organization settings > Sources/Destinations.

    • If you're on any other plan, click Workspace settings > Sources/Destinations.

  2. Find the connector you want to update in the list of connectors.

    note

    Airbyte lists two versions, the current in-use version and the latest version available.

  3. Click Change to update your OSS version to the latest available version.

Update the connector version​

Update each instance of the connector separately. If you have multiple instances of a connector, updating one doesn't affect the others.

  1. In the navigation bar:

    • If you're on the Self-Managed Enterprise plan, click Organization settings > Sources/Destinations.

    • If you're on any other plan, click Workspace settings > Sources/Destinations.

  2. Select the instance of the connector you wish to upgrade.

  3. Select Upgrade.

  4. Follow the prompt to confirm you are ready to upgrade to the new version.

Clear data from affected streams​

After upgrading a connector with a breaking change, you must refresh affected schemas and clear your data.

  1. In the nav bar, click Connections.

  2. Find the connection affected by the upgrade.

  3. Click the Schema tab.

  4. Click Refresh source schema (looks like ). When Airbyte finishes, it shows you any detected schema changes.

  5. Click OK.

  6. Click Save changes

  7. Clear the data for the streams affected by this upgrade.

Once the clear is complete, you can begin syncing your data again as usual.

Was this page helpful?