Skip to main content
Version: 1.8

Upgrade to Helm chart V2 (Self-Managed Community)

Not availableCloud AvailableSelf-Managed CommunityNot availableSelf-Managed Enterprise Compare

Airbyte has upgraded its Helm chart to a new version called "V2." Using Helm chart V2 is currently optional. At some future date the V2 Helm chart will become the standard, so we advise that you upgrade your existing deployment to use the new chart before the transition. If you're a new Airbyte user, you can skip the upgrade altogether and start with the new chart.

Enterprise customers

Follow the Self-Managed Enterprise guide instead.

Why you should upgrade​

Upgrading to the new Helm chart now has the following benefits.

  1. By upgrading in advance, you can schedule this upgrade for a convenient time. Avoid blocking yourself from upgrading Airbyte to a future version when the new chart is mandatory and you're busy.

  2. The new Helm chart is more aligned with Helm's best practices for chart design.

  3. The new Helm chart has broader and more detailed options to customize your deployment. In most cases, it's no longer necessary to specify environment variables in your values.yaml file because the chart offers a more detailed interface for customization. If you do need to use environment variables, you can use fewer of them.

Which versions can upgrade to Helm chart V2​

The following versions of Airbyte can use Helm chart V2:

  • Airbyte version 1.6.0 and later, if installed and managed with Helm

The following versions of Airbyte can't use Helm chart V2:

  • Airbyte versions before 1.6.0

  • Airbyte versions installed and managed with abctl

How to upgrade​

In most cases, upgrading is straightforward. To upgrade to Helm chart V2, you complete the following steps.

  1. Ensure you have configured Airbyte to use an external database and external bucket storage. If you haven't, backup Airbyte's internal Postgres database.

  2. Prepare to deploy a fresh installation of Airbyte in a new namespace, but only if you're using an external database. If you're using Airbyte's internal Postgres database, you'll reuse your existing namespace.

  3. Create a new values.yaml file.

  4. Deploy a new version of Airbyte using your new values.yaml file and the new Helm chart version.

Backup Airbyte's internal Postgres database​

If you have configured an external database, skip this step. However, it is a good idea to backup your database on a regular cadence.

Prepare a new namespace for Airbyte​

When moving to Helm chart V2, deploy Airbyte with a new namespace and use a fresh values and secrets file. It is possible to do a straight upgrade, but different Airbyte users have different and sometimes complex configurations that could produce unique and unexpected situations during the upgrade. By doing a fresh install, you create a separate environment that's easier to troubleshoot if something in your values or secrets files acts unexpectedly.

kubectl create namespace airbyte-v2

Add V2 chart repo​

Helm chart V2 uses a different repo URL than V1 did. In your command line tool, add this repo and index it.

# Helm chart V1
# helm repo add airbyte https://airbytehq.github.io/helm-charts

# Helm chart V2
helm repo add airbyte-v2 https://airbytehq.github.io/charts
helm repo update

You can browse all charts uploaded to your repository.

helm search repo airbyte-v2

Update your values.yaml file​

In most cases, the adjustments to values.yaml are small and involve changing keys and moving sections. This section walks you through the main updates you need to make. If you already know what to do, see Values.yaml reference for the full V1 and V2 interfaces.

Airbyte recommends approaching this project in this way:

  1. Note the customizations in your V1 values.yaml file to ensure you don't forget anything.

  2. Start with a basic V2 values.yaml to verify that it works. Map your V1 settings to V2, transferring one set of configurations at a time.

  3. Don't test in production.

Follow the steps below to start generating values.yaml.

Create a values.yaml file and a global configuration

Create a new values.yaml file on your machine. In that file, create your basic global configuration.

values.yaml
global:
  edition: community

  airbyteUrl: "" # The URL where Airbyte will be reached; This should match your Ingress host

Add your database (if applicable)

If you're not using an external database, skip this step.

If you are using an external database, disable Airbyte's default Postgres database and add your own. The main difference in Helm chart V2 is the global.database.database key has changed to global.database.name.

values.yaml
global: 
  database:
    # -- Secret name where database credentials are stored
    secretName: "" # e.g. "airbyte-config-secrets"
    # -- The database host
    host: ""
    # -- The database port
    port:
    # -- The database name - this key used to be "database" in Helm chart 1.0
    name: ""

    # Use EITHER user or userSecretKey, but not both
    # -- The database user
    user: ""
    # -- The key within `secretName` where the user is stored
    userSecretKey: "" # e.g. "database-user"

    # Use EITHER password or passwordSecretKey, but not both
    # -- The database password
    password: ""
    # -- The key within `secretName` where the password is stored
    passwordSecretKey: "" # e.g."database-password"

postgresql:
  enabled: false

Add external log storage (if applicable)

If you're using external log storage, implement it. If you aren't, skip this step.

global:
  storage:
    secretName: ""
    type: minio # default storage is minio. Set to s3, gcs, or azure, according to what you use.

    bucket:
      log: airbyte-bucket
      state: airbyte-bucket
      workloadOutput: airbyte-bucket
      activityPayload: airbyte-bucket

    # Set ONE OF the following storage types, according to your specification above

    # S3
    s3:
      region: "" ## e.g. us-east-1
      authenticationType: credentials ## Use "credentials" or "instanceProfile"
      accessKeyId: ""
      secretAccessKey: ""

    # GCS
    gcs:
      projectId: <project-id>
      credentialsJson:  <base64-encoded>
      credentialsJsonPath: /secrets/gcs-log-creds/gcp.json

    # Azure
    azure:
      # one of the following: connectionString, connectionStringSecretKey
      connectionString: <azure storage connection string>
      connectionStringSecretKey: <secret coordinate containing an existing connection-string secret>

Add external connector secret management (if applicable)

If you're using external secret management, implement it. If you aren't, skip this step.

global:
  secretsManager:
    enabled: false
    type: "" # one of: VAULT, GOOGLE_SECRET_MANAGER, AWS_SECRET_MANAGER, AZURE_KEY_VAULT, TESTING_CONFIG_DB_TABLE
    secretName: "airbyte-config-secrets"

    # Set ONE OF the following groups of configurations, based on your configuration in global.secretsManager.type.

    awsSecretManager:
      region: <aws-region>
      authenticationType: credentials ## Use "credentials" or "instanceProfile"
      tags: ## Optional - You may add tags to new secrets created by Airbyte.
      - key: ## e.g. team
          value: ## e.g. deployments
        - key: business-unit
          value: engineering
      kms: ## Optional - ARN for KMS Decryption.

    # OR

    googleSecretManager:
      projectId: <project-id>
      credentialsSecretKey: gcp.json

    # OR

    azureKeyVault:
      tenantId: ""
      vaultUrl: ""
      clientId: ""
      clientIdSecretKey: ""
      clientSecret: ""
      clientSecretSecretKey: ""
      tags: ""

    # OR

    vault:
      address: ""
      prefix: ""
      authToken: ""
      authTokenSecretKey: ""

Update syntax for other customizatons

If you have further customizations in your V1 values.yaml file, move those over to your new values.yaml file, and update key names where appropriate.

  • Change hyphenated V1 keys keys to camel case in V2. For example, when copying over workload-launcher, change it to workloadLauncher.

  • Some keys have different names. For example, orchestrator is containerOrchestrator in V2.

Here is the full list of changes.

Helm chart V1Helm chart V2
global.database.databaseglobal.database.name
workload-launcherworkloadLauncher
airbyte-bootloaderairbyteBootloader
orchestratorcontainerOrchestrator
workload-launcher.extraEnvs[JOB_KUBE_NODE_SELECTORS]global.jobs.kube.nodeSelector
workload-launcher.extraEnvs[CHECK_JOB_KUBE_NODE_SELECTORS]global.jobs.kube.scheduling.check.nodeSelectors
workload-launcher.extraEnvs[DISCOVER_JOB_KUBE_NODE_SELECTORS]global.jobs.kube.scheduling.discover.nodeSelectors
worker.extraEnvs[MAX_SYNC_WORKERS]worker.maxSyncWorkers
worker.extraEnvs[MAX_CHECK_WORKERS]worker.maxCheckWorkers
server.extraEnvs[HTTP_IDLE_TIMEOUT]server.httpIdleTimeout
global.env_vars[TRACKING_STRATEGY]global.tracking.strategy
server.env_vars[AUDIT_LOGGING_ENABLED]server.auditLoggingEnabled
global.env_vars[STORAGE_BUCKET_AUDIT_LOGGING]global.storage.bucket.auditLogging
global.env_vars[JOB_MAIN_CONTAINER_CPU_REQUEST]global.workloads.resources.mainContainer.cpu.request
orchestrator.nodeSelectorglobal.jobs.kube.nodeSelector
Individual bucket env vars (S3_LOG_BUCKET, GCS_LOG_BUCKET, etc.)global.storage.bucket.log
STORAGE_BUCKET_STATEglobal.storage.bucket.state
STORAGE_BUCKET_WORKLOAD_OUTPUTglobal.storage.bucket.workloadOutput
STORAGE_BUCKET_ACTIVITY_PAYLOADglobal.storage.bucket.activityPayload

Convert extraEnv variables

In previous versions of your values.yaml file, you might have specified a number of environment variables through extraEnv. Many (but not all) of these variables have a dedicated interface in Helm chart V2. For example, look at the following configuration, which tells workload-launcher to run pods in the jobs node group.

values.yaml using Helm chart V1
workload-launcher:
  nodeSelector:
    type: static
  ## Pods spun up by the workload launcher will run in the 'jobs' node group.
  extraEnv:
    - name: JOB_KUBE_NODE_SELECTORS
      value: type=jobs
    - name: SPEC_JOB_KUBE_NODE_SELECTORS
      value: type=jobs
    - name: CHECK_JOB_KUBE_NODE_SELECTORS
      value: type=jobs
    - name: DISCOVER_JOB_KUBE_NODE_SELECTORS
      value: type=jobs

You can specify these values directly without using environment variables, achieving the same effect.

values.yaml using Helm chart V2
global:
  jobs:
    kube:
      nodeSelector:
        type: jobs
      scheduling:
        check:
          nodeSelectors:
            type: jobs
        discover:
          nodeSelectors:
            type: jobs
        spec:
          nodeSelectors:
            type: jobs

workloadLauncher:
  nodeSelector:
    type: static

Here is a complete list of environment variables with their Helm chart V2 equivalent. Some environment variables don't have direct V2 equivalents, so you can set these using the extraEnv configuration in the appropriate service section.

Environment variableHelm chart V2 equivalent
Core
AIRBYTE_VERSIONglobal.version
AIRBYTE_EDITIONglobal.edition
AIRBYTE_CLUSTER_TYPEglobal.cluster.type
AIRBYTE_CLUSTER_NAMEglobal.cluster.name
AIRBYTE_URLglobal.airbyteUrl
AIRBYTE_API_HOSTglobal.api.host
AIRBYTE_API_AUTH_HEADER_NAMEglobal.api.authHeaderName
AIRBYTE_API_AUTH_HEADER_VALUEglobal.api.authHeaderValue
AIRBYTE_SERVER_HOSTglobal.server.host
API_AUTHORIZATION_ENABLEDglobal.auth.enabled
CONNECTOR_BUILDER_SERVER_API_HOSTglobal.connectorBuilderServer.apiHost
DEPLOYMENT_ENVglobal.deploymentEnv
INTERNAL_API_HOSTglobal.api.internalHost
LOCALglobal.local
WEBAPP_URLglobal.webapp.url
SPEC_CACHE_BUCKETUse extraEnvs
Secrets
SECRET_PERSISTENCEglobal.secretsManager.type
SECRET_STORE_GCP_PROJECT_IDglobal.secretsManager.googleSecretManager.projectId
SECRET_STORE_GCP_CREDENTIALSglobal.secretsManager.googleSecretManager.credentials
VAULT_ADDRESSglobal.secretsManager.vault.address
VAULT_PREFIXglobal.secretsManager.vault.prefix
VAULT_AUTH_TOKENglobal.secretsManager.vault.token
VAULT_AUTH_METHODglobal.secretsManager.vault.authMethod
AWS_ACCESS_KEYglobal.aws.accessKeyId
AWS_SECRET_ACCESS_KEYglobal.aws.secretAccessKey
AWS_KMS_KEY_ARNglobal.secretsManager.awsSecretManager.kmsKeyArn
AWS_SECRET_MANAGER_SECRET_TAGSglobal.secretsManager.awsSecretManager.tags
AWS_ASSUME_ROLE_ACCESS_KEY_IDglobal.aws.assumeRole.accessKeyId
Database
DATABASE_USERglobal.database.user
DATABASE_PASSWORDglobal.database.password
DATABASE_URLglobal.database.url
DATABASE_HOSTglobal.database.host
DATABASE_PORTglobal.database.port
DATABASE_DBglobal.database.database
JOBS_DATABASE_INITIALIZATION_TIMEOUT_MSglobal.database.initializationTimeoutMs
CONFIG_DATABASE_USERglobal.database.user
CONFIG_DATABASE_PASSWORDglobal.database.password
CONFIG_DATABASE_URLglobal.database.url
CONFIG_DATABASE_INITIALIZATION_TIMEOUT_MSglobal.database.initializationTimeoutMs
RUN_DATABASE_MIGRATION_ON_STARTUPglobal.migrations.runAtStartup
USE_CLOUD_SQL_PROXYglobal.cloudSqlProxy.enabled
Airbyte Services
TEMPORAL_HOSTtemporal.host
Jobs
SYNC_JOB_MAX_ATTEMPTSUse extraEnvs
SYNC_JOB_RETRIES_COMPLETE_FAILURES_MAX_SUCCESSIVEUse extraEnvs
SYNC_JOB_RETRIES_COMPLETE_FAILURES_MAX_TOTALUse extraEnvs
SYNC_JOB_RETRIES_COMPLETE_FAILURES_BACKOFF_MIN_INTERVAL_SUse extraEnvs
SYNC_JOB_RETRIES_COMPLETE_FAILURES_BACKOFF_MAX_INTERVAL_SUse extraEnvs
SYNC_JOB_RETRIES_COMPLETE_FAILURES_BACKOFF_BASEUse extraEnvs
SYNC_JOB_RETRIES_PARTIAL_FAILURES_MAX_SUCCESSIVEUse extraEnvs
SYNC_JOB_RETRIES_PARTIAL_FAILURES_MAX_TOTALUse extraEnvs
SYNC_JOB_MAX_TIMEOUT_DAYSUse extraEnvs
JOB_MAIN_CONTAINER_CPU_REQUESTglobal.workloads.resources.mainContainer.cpu.request
JOB_MAIN_CONTAINER_CPU_LIMITglobal.workloads.resources.mainContainer.cpu.limit
JOB_MAIN_CONTAINER_MEMORY_REQUESTglobal.workloads.resources.mainContainer.memory.request
JOB_MAIN_CONTAINER_MEMORY_LIMITglobal.workloads.resources.mainContainer.memory.limit
JOB_KUBE_TOLERATIONSglobal.jobs.kube.tolerations
JOB_KUBE_NODE_SELECTORSglobal.jobs.kube.nodeSelector
JOB_KUBE_ANNOTATIONSglobal.jobs.kube.annotations
JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_POLICYglobal.jobs.kube.mainContainerImagePullPolicy
JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_SECRETglobal.jobs.kube.mainContainerImagePullSecret
JOB_KUBE_SIDECAR_CONTAINER_IMAGE_PULL_POLICYglobal.jobs.kube.sidecarContainerImagePullPolicy
JOB_KUBE_SOCAT_IMAGEglobal.jobs.kube.images.socat
JOB_KUBE_BUSYBOX_IMAGEglobal.jobs.kube.images.busybox
JOB_KUBE_CURL_IMAGEglobal.jobs.kube.images.curl
JOB_KUBE_NAMESPACEglobal.jobs.kube.namespace
JOB_KUBE_SERVICEACCOUNTglobal.jobs.kube.serviceAccount
Jobs-specific
SPEC_JOB_KUBE_NODE_SELECTORSglobal.jobs.kube.scheduling.spec.nodeSelectors
CHECK_JOB_KUBE_NODE_SELECTORSglobal.jobs.kube.scheduling.check.nodeSelectors
DISCOVER_JOB_KUBE_NODE_SELECTORSglobal.jobs.kube.scheduling.discover.nodeSelectors
SPEC_JOB_KUBE_ANNOTATIONSglobal.jobs.kube.scheduling.spec.annotations
CHECK_JOB_KUBE_ANNOTATIONSglobal.jobs.kube.scheduling.check.annotations
DISCOVER_JOB_KUBE_ANNOTATIONSglobal.jobs.kube.scheduling.discover.annotations
Connections
MAX_FIELDS_PER_CONNECTIONUse extraEnvs
MAX_DAYS_OF_ONLY_FAILED_JOBS_BEFORE_CONNECTION_DISABLEUse extraEnvs
MAX_FAILED_JOBS_IN_A_ROW_BEFORE_CONNECTION_DISABLEUse extraEnvs
Logging
LOG_LEVELglobal.logging.level
GCS_LOG_BUCKETglobal.storage.gcs.bucket
S3_BUCKETglobal.storage.s3.bucket
S3_REGIONglobal.storage.s3.region
S3_AWS_KEYglobal.storage.s3.accessKeyId
S3_AWS_SECRETglobal.storage.s3.secretAccessKey
S3_MINIO_ENDPOINTglobal.storage.minio.endpoint
S3_PATH_STYLE_ACCESSglobal.storage.s3.pathStyleAccess
Monitoring
PUBLISH_METRICSglobal.metrics.enabled
METRIC_CLIENTglobal.metrics.client
DD_AGENT_HOSTglobal.datadog.agentHost
DD_AGENT_PORTglobal.datadog.agentPort
OTEL_COLLECTOR_ENDPOINTglobal.metrics.otel.exporter.endpoint
MICROMETER_METRICS_ENABLEDglobal.metrics.enabled
Worker
MAX_CHECK_WORKERSworker.maxCheckWorkers
MAX_SYNC_WORKERSworker.maxSyncWorkers
TEMPORAL_WORKER_PORTSworker.temporalWorkerPorts
DISCOVER_REFRESH_WINDOW_MINUTESUse extraEnvs
Launcher
WORKLOAD_LAUNCHER_PARALLELISMworkloadLauncher.parallelism
Data Retention
TEMPORAL_HISTORY_RETENTION_IN_DAYSUse extraEnvs
Server
AUDIT_LOGGING_ENABLEDserver.auditLoggingEnabled
STORAGE_BUCKET_AUDIT_LOGGINGserver.auditLoggingBucket
HTTP_IDLE_TIMEOUTserver.httpIdleTimeout
READ_TIMEOUTUse extraEnvs
Authentication
AB_INSTANCE_ADMIN_PASSWORDglobal.auth.instanceAdmin.password
AB_AUTH_SECRET_CREATION_ENABLEDglobal.auth.secretCreationEnabled
AB_KUBERNETES_SECRET_NAMEglobal.auth.managedSecretName
AB_INSTANCE_ADMIN_CLIENT_IDglobal.auth.instanceAdmin.clientId
AB_INSTANCE_ADMIN_CLIENT_SECRETglobal.auth.instanceAdmin.clientSecret
AB_JWT_SIGNATURE_SECRETglobal.auth.security.jwtSignatureSecret
AB_COOKIE_SECUREglobal.auth.security.cookieSecureSetting
INITIAL_USER_FIRST_NAMEglobal.auth.instanceAdmin.firstName
INITIAL_USER_LAST_NAMEglobal.auth.instanceAdmin.lastName
INITIAL_USER_EMAILglobal.auth.instanceAdmin.email
INITIAL_USER_PASSWORDglobal.auth.instanceAdmin.password
Tracking
TRACKING_ENABLEDglobal.tracking.enabled
TRACKING_STRATEGYglobal.tracking.strategy
Enterprise
AIRBYTE_LICENSE_KEYglobal.enterprise.licenseKey
Feature Flags
FEATURE_FLAG_CLIENTglobal.featureFlags.client
LAUNCHDARKLY_KEYglobal.featureFlags.launchDarkly.sdkKey
Java
JAVA_TOOL_OPTIONSglobal.java.opts
Temporal
AUTO_SETUPtemporal.autoSetup
TEMPORAL_CLI_ADDRESSglobal.temporal.cli.address
TEMPORAL_CLOUD_ENABLEDglobal.temporal.cloud.enabled
TEMPORAL_CLOUD_HOSTglobal.temporal.cloud.host
TEMPORAL_CLOUD_NAMESPACEglobal.temporal.cloud.namespace
TEMPORAL_CLOUD_CLIENT_CERTglobal.temporal.cloud.clientCert
TEMPORAL_CLOUD_CLIENT_KEYglobal.temporal.cloud.clientKey
Container Orchestrator
CONTAINER_ORCHESTRATOR_SECRET_NAMEglobal.workloads.containerOrchestrator.secretName
CONTAINER_ORCHESTRATOR_SECRET_MOUNT_PATHglobal.workloads.containerOrchestrator.secretMountPath
CONTAINER_ORCHESTRATOR_DATA_PLANE_CREDS_SECRET_NAMEglobal.workloads.containerOrchestrator.dataPlane.credentialsSecretName
CONTAINER_ORCHESTRATOR_IMAGEglobal.workloads.containerOrchestrator.image
Workload Launcher
WORKLOAD_LAUNCHER_PARALLELISMworkloadLauncher.parallelism
CONNECTOR_PROFILER_IMAGEworkloadLauncher.connectorProfiler.image
WORKLOAD_INIT_IMAGEworkloadLauncher.workloadInit.image
Connector Registry
CONNECTOR_REGISTRY_SEED_PROVIDERglobal.connectorRegistry.seedProvider
CONNECTOR_REGISTRY_BASE_URLglobal.connectorRegistry.baseUrl
AI Assist
AI_ASSIST_URL_BASEconnectorBuilderServer.aiAssistUrlBase
AI_ASSIST_API_KEYconnectorBuilderServer.aiAssistApiKey
Connector Rollout
CONNECTOR_ROLLOUT_EXPIRATION_SECONDSglobal.connectorRollout.expirationSeconds
CONNECTOR_ROLLOUT_PARALLELISMglobal.connectorRollout.parallelism
CONNECTOR_ROLLOUT_GITHUB_AIRBYTE_PATconnectorRolloutWorker.githubToken
Customer.io
CUSTOMERIO_API_KEYglobal.customerio.apiKey
Shopify
SHOPIFY_CLIENT_IDglobal.shopify.clientId
SHOPIFY_CLIENT_SECRETglobal.shopify.clientSecret
Keycloak
KEYCLOAK_ADMIN_USERkeycloak.auth.adminUsername
KEYCLOAK_ADMIN_PASSWORDkeycloak.auth.adminPassword
KEYCLOAK_ADMIN_REALMkeycloak.auth.adminRealm
KEYCLOAK_INTERNAL_REALM_ISSUERkeycloak.realmIssuer
MinIO
MINIO_ROOT_USERminio.rootUser
MINIO_ROOT_PASSWORDminio.rootPassword
Micronaut
MICRONAUT_ENVIRONMENTSglobal.micronaut.environments
Topology
NODE_SELECTOR_LABELglobal.topology.nodeSelectorLabel
QUICK_JOBS_NODE_SELECTOR_LABELglobal.topology.quickJobsNodeSelectorLabel
Workloads
CONNECTOR_SPECIFIC_RESOURCE_DEFAULTS_ENABLEDglobal.workloads.resources.useConnectorResourceDefaults
DATA_CHECK_TASK_QUEUESglobal.workloads.queues.check

Deploy Airbyte​

Deploy Airbyte. Here is an example of how to deploy version 1.7.0 of Airbyte using the latest Helm chart V2 values. Normally the Helm chart version is identical to the Airbyte version. Since using this chart version is optional, the Helm chart and Airbyte have different, but compatible, versions.

helm install airbyte airbyte-v2/airbyte \
  --namespace airbyte-v2 \       # Target Kubernetes namespace
  --values ./values.yaml \       # Custom configuration values
  --version 2.0.3 \              # Helm chart version to use
  --set global.image.tag=1.7.0   # Airbyte version to use
Was this page helpful?