Getting started

Enterprise FlexSelf-Managed CommunitySelf-Managed Enterprise Compare

Any Airbyte Cloud enviornment can be easily upgraded to Enterprise Flex. To learn more about upgrading to Enterprise Flex, talk to sales.

You may choose to run a self-managed data plane while using Airbyte Enterprise Flex.

If you are not using any self-managed data planes, then no additional infrastructure is required to begin creating connections and running syncs.

The following diagram illustrates a typical Airbyte Enterpris Flex deployment running a self-managed data plane:

Infrastructure prerequisites

For a production-ready deployment of self-managed data planes, various infrastructure components are required. We recommend deploying to Amazon EKS, Google Kubernetes Engine,

Amazon EKS

Component	Recommendation
Kubernetes Cluster	Amazon EKS cluster running on EC2 instances in 2 or more availability zones.
External Secrets Manager	Amazon Secrets Manager for storing connector secrets.
Object Storage (Optional)	Amazon S3 bucket with a directory for log storage.

A few notes on Kubernetes cluster provisioning for Airbyte Self-Managed Enterprise:

• We support Amazon Elastic Kubernetes Service (EKS) on EC2, Google Kubernetes Engine (GKE) on Google Compute Engine (GCE), or Azure Kubernetes Service (AKS) on Azure.
• While we support GKE Autopilot, we do not support Amazon EKS on Fargate.

We require you to install and configure the following Kubernetes tooling:

1. Install helm by following these instructions
2. Install kubectl by following these instructions.
3. Configure kubectl to connect to your cluster by using kubectl use-context my-cluster-name:

Configure kubectl to connect to your cluster

Amazon EKS

1. Configure your AWS CLI to connect to your project.
2. Install eksctl.
3. Run eksctl utils write-kubeconfig --cluster=$CLUSTER_NAME to make the context available to kubectl.
4. Use kubectl config get-contexts to show the available contexts.
5. Run kubectl config use-context $EKS_CONTEXT to access the cluster with kubectl.

GKE

1. Configure gcloud with gcloud auth login.
2. On the Google Cloud Console, the cluster page will have a "Connect" button, with a command to run locally: gcloud container clusters get-credentials $CLUSTER_NAME --zone $ZONE_NAME --project $PROJECT_NAME.
3. Use kubectl config get-contexts to show the available contexts.
4. Run kubectl config use-context $EKS_CONTEXT to access the cluster with kubectl.

We also require you to create a Kubernetes namespace for your Airbyte deployment:

kubectl create namespace airbyte

Limitations and considerations

• While data planes process data in their respective regions, some metadata remains in the control plane.
  • Airbyte stores Cursor and Primary Key data in the control plane regardless of data plane location. If you have data that you can't store in the control plane, don't use it as a cursor or primary key.
• The Connector Builder processes all data through the control plane, regardless of workspace settings. This limitation applies to the development and testing phase only; published connectors respect workspace data residency settings during syncs.
• If you want to run multiple data planes in the same region for higher availability, both must be part of the same region in Airbyte and use the same secrets manager to ensure connection credentials are the same.
• Data planes must be able to communicate with the control plane.
• Data planes will only ever send requests to the control plane and never require inbound requests.