Skip to main content
Version: Cloud

External Secret Management

Not availableCoreNot availableStandardNot availablePlusNot availableProAvailableEnterprise FlexNot availableSelf-Managed Enterprise Compare

This guide provides step-by-step instructions for configuring external secrets management with Airbyte Enterprise Flex. External secrets management allows Airbyte to securely store and manage connection credentials in your cloud provider's secrets manager (AWS Secrets Manager, Azure Key Vault, or Google Cloud Secret Manager) instead of storing them in Airbyte's internal database.

info

External secrets management is available for Airbyte Enterprise Flex customers.

Prerequisites​

  • Airbyte organization on an Enterprise Flex plan
  • Active account with your chosen cloud provider (AWS, Azure, or GCP)
  • Appropriate permissions to create and manage IAM roles/policies or service principals
  • Access to your cloud provider's secrets management service

Step 1: Configure Cloud Provider Permissions​

First, you'll need to create the appropriate permissions in your cloud provider to allow Airbyte to manage secrets.

Follow the AWS Secret Manager Policy documentation to create the required IAM policy. This policy ensures Airbyte can create, read, update, and manage secrets while restricting access to only Airbyte-managed secrets.

Step 2: Choose Authentication Method​

The authentication method varies by cloud provider:

AWS supports two authentication methods:

MethodUse CaseSecurity Note
IAM Role (Recommended)Best for EKS deployments or when using IRSA (IAM Roles for Service Accounts)More secure - no static credentials, uses temporary credentials
Access KeyFor EC2 instances or non-Kubernetes deploymentsRequires managing static credentials securely

Step 3: Set Up Cloud Provider Authentication​

Follow the instructions for your chosen cloud provider:

Choose one of the following authentication methods:

IAM Role (Recommended)

  1. Create an IAM role with a trust relationship to your EKS cluster or service account
    • This requires additional coordination with Airbyte and will enable the use of IAM authentication for some AWS-hosted sources
  2. Attach the AirbyteSecretsManagerPolicy (created in Step 1) to this role
  3. Note the Role ARN (e.g., arn:aws:iam::123456789012:role/AirbyteSecretsRole)

IAM User with Access Keys

  1. Create an IAM user
  2. Attach the AirbyteSecretsManagerPolicy (created in Step 1) to this user
  3. Generate an access key and secret access key for this user
  4. Store these credentials securely in a way they can be safely shared with Airbyte

Step 4: Provide Configuration to Airbyte​

Provide Airbyte with the configuration details using the appropriate JSON format for your cloud provider:

IAM Role Example:

{
  "auth_type": "IAM_ROLE",
  "roleArn": "arn:aws:iam::123456789012:role/AirbyteSecretsRole",
  "awsRegion": "us-east-1",
  "tagKey": "AirbyteManaged"
}

Access Key Example:

{
  "auth_type": "ACCESS_KEY",
  "awsAccessKey": "AKIAIOSFODNN7EXAMPLE",
  "awsSecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  "awsRegion": "us-west-2",
  "tagKey": "AirbyteManaged"
}

Verification Steps​

After providing your configuration to Airbyte:

  1. Airbyte will create a test secret in your secrets manager
  2. Verify the secret appears in your cloud provider's console:
    • AWS: Check AWS Secrets Manager in your specified region
    • Azure: Check your Azure Key Vault secrets
    • GCP: Check Google Cloud Secret Manager
  3. Confirm the secret has the appropriate tags or identifiers (e.g., AirbyteManaged=true for AWS)
  4. All future connection credentials will be stored as secrets in your external secrets manager

Best Practices​

General Best Practices​

  • Use the most secure authentication method available for your cloud provider and deployment type
  • Use specific tags or naming conventions to differentiate Airbyte-managed secrets from other secrets
  • Monitor access logs for your secrets manager (CloudTrail for AWS, Azure Monitor for Azure, Cloud Audit Logs for GCP)
  • Implement least privilege access - only grant the minimum permissions required
  • Choose a region close to your Airbyte deployment for better performance and lower latency

AWS-Specific Best Practices​

  • Prefer IAM Roles over Access Keys for enhanced security
  • Rotate access keys regularly if using ACCESS_KEY authentication
  • Restrict policy conditions to only allow operations on tagged secrets

Azure-Specific Best Practices​

  • Rotate client secrets regularly before they expire
  • Use Azure Key Vault's built-in logging to monitor access patterns
  • Consider using managed identities where possible for additional security

Troubleshooting​

Issue: "Access Denied" errors when Airbyte tries to access Secrets Manager

Solution: Verify the IAM policy is correctly attached and the condition for the AirbyteManaged tag is properly configured

Issue: Secrets not appearing in AWS Console

Solution: Check that the correct AWS region is specified in the configuration

Issue: IAM Role authentication not working

Solution: Verify the trust relationship is configured correctly and the service account has the proper annotations

Security Considerations​

  • Never commit credentials to version control - always use secure methods to share configuration details with Airbyte
  • Regularly review and rotate credentials - especially client secrets and access keys
  • Enable audit logging - monitor all access to your secrets manager for suspicious activity
  • Use encryption - ensure data is encrypted both in transit and at rest (most cloud providers enable this by default)
  • Implement network restrictions where possible - limit access to your secrets manager to known IP ranges or VPCs
  • Set appropriate secret expiration policies - configure automatic rotation where supported

Additional Resources​

AWS​

Azure​

GCP​

Support​

If you encounter issues not covered in this guide, please contact your Airbyte Customer Success representative or reach out to [email protected].

Was this page helpful?