Skip to main content

External Secret Management

Not availableCoreNot availableStandardNot availablePlusAvailableProAvailableEnterprise FlexNot availableSelf-Managed Enterprise Compare

This guide provides step-by-step instructions for configuring external secrets management with Airbyte. External secrets management allows Airbyte to securely store and manage connection credentials in your cloud provider's secrets manager (AWS Secrets Manager, Azure Key Vault, or Google Cloud Secret Manager) instead of storing them in Airbyte's internal database.

info

External secrets management is available for Airbyte Pro and Enterprise Flex customers.

Prerequisites​

  • Airbyte organization on a Pro or Enterprise Flex plan
  • Active account with your chosen cloud provider (AWS, Azure, or GCP)
  • Appropriate permissions to create and manage IAM roles/policies or service principals
  • Access to your cloud provider's secrets management service

Step 1: Configure Cloud Provider Permissions​

First, you'll need to create the appropriate permissions in your cloud provider to allow Airbyte to manage secrets.

Follow the AWS Secret Manager Policy documentation to create the required IAM policy. This policy ensures Airbyte can create, read, update, and manage secrets while restricting access to only Airbyte-managed secrets.

Step 2: Choose Authentication Method​

The authentication method varies by cloud provider:

AWS supports two authentication methods:

MethodUse CaseSecurity Note
IAM Role (Recommended)Best for EKS deployments or when using IRSA (IAM Roles for Service Accounts)More secure - no static credentials, uses temporary credentials
Access KeyFor EC2 instances or non-Kubernetes deploymentsRequires managing static credentials securely

Step 3: Set Up Cloud Provider Authentication​

Follow the instructions for your chosen cloud provider:

Choose one of the following authentication methods:

IAM Role (Recommended)

  1. Create an IAM role with a trust relationship to your EKS cluster or service account
    • This requires additional coordination with Airbyte and will enable the use of IAM authentication for some AWS-hosted sources
  2. Attach the AirbyteSecretsManagerPolicy (created in Step 1) to this role
  3. Note the Role ARN (e.g., arn:aws:iam::123456789012:role/AirbyteSecretsRole)

IAM User with Access Keys

  1. Create an IAM user
  2. Attach the AirbyteSecretsManagerPolicy (created in Step 1) to this user
  3. Generate an access key and secret access key for this user
  4. Store these credentials securely in a way they can be safely shared with Airbyte

Step 4: Provide Configuration to Airbyte​

Provide Airbyte with the configuration details using the appropriate JSON format for your cloud provider:

IAM Role Example:

{
  "auth_type": "IAM_ROLE",
  "roleArn": "arn:aws:iam::123456789012:role/AirbyteSecretsRole",
  "awsRegion": "us-east-1",
  "tagKey": "AirbyteManaged"
}

Access Key Example:

{
  "auth_type": "ACCESS_KEY",
  "awsAccessKey": "AKIAIOSFODNN7EXAMPLE",
  "awsSecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  "awsRegion": "us-west-2",
  "tagKey": "AirbyteManaged"
}

In the Airbyte API, these secret storage configuration endpoints use the internal configuration API path, such as POST /v1/secret_storage/create, POST /v1/secret_storage/list, POST /v1/secret_storage/get, POST /v1/secret_storage/delete, and POST /v1/secret_storage/migrate.

Using External Secret References in Connector Configurations​

Once your secret storage is configured, you can reference secrets stored in your external secret manager when creating or updating connectors via the API or Terraform. Instead of passing a raw secret value, use the secret_coordinate:: prefix followed by the key or path of the secret in your configured secret manager.

This is useful when:

  • You manage secrets externally (for example, via Terraform, CI/CD pipelines, or a secrets team) and want Airbyte connectors to reference them without Airbyte ever seeing the plain text value.
  • You want to rotate secrets in your secret manager without updating Airbyte connector configurations.
  • Compliance requirements prevent you from sending secret values through the Airbyte API.

How It Works​

Any field in a connector spec marked airbyte_secret: true can accept a value prefixed with secret_coordinate::. When Airbyte sees this prefix, it strips the prefix and stores the remaining text as an external reference to the secret in your configured secret manager rather than writing the value to Airbyte's own secret persistence.

At runtime, Airbyte resolves the reference by reading the secret value from your configured secret manager.

Supported APIs​

External secret references work with source and destination create and update endpoints.

For the public API, use the configuration field:

  • Create Source (POST /public/v1/sources)
  • Update Source (PUT /public/v1/sources/{sourceId})
  • Patch Source (PATCH /public/v1/sources/{sourceId})
  • Create Destination (POST /public/v1/destinations)
  • Update Destination (PUT /public/v1/destinations/{destinationId})
  • Patch Destination (PATCH /public/v1/destinations/{destinationId})

For the internal configuration API, use the connectionConfiguration field:

  • Create Source (POST /v1/sources/create)
  • Update Source (POST /v1/sources/update)
  • Partial Update Source (POST /v1/sources/partial_update)
  • Create Destination (POST /v1/destinations/create)
  • Update Destination (POST /v1/destinations/update)
  • Partial Update Destination (POST /v1/destinations/partial_update)

API Example​

curl --request POST \
  --url https://api.airbyte.com/v1/sources \
  --header 'Authorization: Bearer <YOUR_ACCESS_TOKEN>' \
  --header 'Content-Type: application/json' \
  --data '{
    "name": "My Postgres Source",
    "workspaceId": "<YOUR_WORKSPACE_ID>",
    "definitionId": "<POSTGRES_SOURCE_DEFINITION_ID>",
    "configuration": {
      "host": "db.example.com",
      "port": 5432,
      "database": "mydb",
      "username": "airbyte_user",
      "password": "secret_coordinate::production/database/password"
    }
  }'

In this example, password is an airbyte_secret field. Instead of providing the actual password, you pass secret_coordinate::production/database/password, and Airbyte resolves it from your configured secret manager. For AWS Secrets Manager, production/database/password is an example path-style secret name; Airbyte uses the text after secret_coordinate:: as the AWS Secrets Manager SecretId.

Terraform Example​

The same approach works with the Airbyte Terraform Provider:

resource "airbyte_source" "postgres" {
  name          = "My Postgres Source"
  workspace_id  = var.workspace_id
  definition_id = var.postgres_source_definition_id

  configuration = jsonencode({
    host     = "db.example.com"
    port     = 5432
    database = "mydb"
    username = "airbyte_user"
    password = "secret_coordinate::production/database/password"
  })
}
tip

Secret fields are masked by default when you read a source or destination. To read back stored secret coordinates instead, pass includeSecretCoordinates=true to the supported API read endpoints or set include_secret_coordinates = true on the airbyte_source or airbyte_destination Terraform data sources.

Supported API read endpoints for returning coordinates:

  • Internal source read (POST /v1/sources/get_with_metadata) with includeSecretCoordinates: true in the request body
  • Public source read (GET /public/v1/sources/{sourceId}?includeSecretCoordinates=true)
  • Public destination read (GET /public/v1/destinations/{destinationId}?includeSecretCoordinates=true)

The internal POST /v1/sources/get and POST /v1/destinations/get endpoints always mask secrets. Airbyte doesn't provide an internal POST /v1/destinations/get_with_metadata endpoint.

Limitations​

  • The secret coordinate you pass after the secret_coordinate:: prefix must match the key, path, or name of a secret that exists (or will exist) in the secret manager configured for your workspace or organization.
  • If no custom secret storage is configured for your workspace/organization, the secret_coordinate:: prefix will not resolve correctly at runtime.

Verification Steps​

After providing your configuration to Airbyte:

  1. Airbyte will create a test secret in your secrets manager
  2. Verify the secret appears in your cloud provider's console:
    • AWS: Check AWS Secrets Manager in your specified region
    • Azure: Check your Azure Key Vault secrets
    • GCP: Check Google Cloud Secret Manager
  3. Confirm the secret has the appropriate tags or identifiers (e.g., AirbyteManaged=true for AWS)
  4. All future connection credentials will be stored as secrets in your external secrets manager

Best Practices​

General Best Practices​

  • Use the most secure authentication method available for your cloud provider and deployment type
  • Use specific tags or naming conventions to differentiate Airbyte-managed secrets from other secrets
  • Monitor access logs for your secrets manager (CloudTrail for AWS, Azure Monitor for Azure, Cloud Audit Logs for GCP)
  • Implement least privilege access - only grant the minimum permissions required
  • Choose a region close to your Airbyte deployment for better performance and lower latency

AWS-Specific Best Practices​

  • Prefer IAM Roles over Access Keys for enhanced security
  • Rotate access keys regularly if using ACCESS_KEY authentication
  • Restrict policy conditions to only allow operations on tagged secrets

Azure-Specific Best Practices​

  • Rotate client secrets regularly before they expire
  • Use Azure Key Vault's built-in logging to monitor access patterns
  • Consider using managed identities where possible for additional security

Troubleshooting​

Issue: "Access Denied" errors when Airbyte tries to access Secrets Manager

Solution: Verify the IAM policy is correctly attached and the condition for the AirbyteManaged tag is properly configured

Issue: Secrets not appearing in AWS Console

Solution: Check that the correct AWS region is specified in the configuration

Issue: IAM Role authentication not working

Solution: Verify the trust relationship is configured correctly and the service account has the proper annotations

Security Considerations​

  • Never commit credentials to version control - always use secure methods to share configuration details with Airbyte
  • Regularly review and rotate credentials - especially client secrets and access keys
  • Enable audit logging - monitor all access to your secrets manager for suspicious activity
  • Use encryption - ensure data is encrypted both in transit and at rest (most cloud providers enable this by default)
  • Implement network restrictions where possible - limit access to your secrets manager to known IP ranges or VPCs
  • Set appropriate secret expiration policies - configure automatic rotation where supported

Additional Resources​

AWS​

Azure​

GCP​

Support​

If you encounter issues not covered in this guide, please contact your Airbyte Customer Success representative or reach out to [email protected].

Was this page helpful?