OpenTelemetry metrics monitoring

Cloud Self-Managed CommunitySelf-Managed Enterprise Compare

Airbyte Self-Managed Enterprise generates a number of crucial metrics about syncs and volumes of data moved. You can configure Airbyte to send telemetry data to an OpenTelemetry collector endpoint so you can consume these metrics in your downstream monitoring tool of choice. Airbyte doesn't send traces and logs.

Airbyte sends specific metrics to provide you with health insight in the following areas.

• Resource provisioning: Monitor API requests and sync attempts to ensure your deployment has adequate resources

• Sync performance: Track sync duration and data volume moved to understand performance

• System health: Monitor sync status and completion rates to ensure system stability

Example dashboard

Here's an example of a dashboard in Datadog using Airbyte's OTEL metrics.

Configure OpenTelemetry metrics

1. Deploy an OpenTelemetry collector if you don't already have one. See the OpenTelemetry documentation for help doing this. If you use Datadog as your monitoring tool, they have an excellent guide to set up a collector and exporter.

2. Update Airbyte's values.yaml file to enable OpenTelemetry.

   values.yaml

   global:
       edition: enterprise # This is an enterprise-only feature
       metrics:
           enabled: true
           otlp:
               enabled: true
               collectorEndpoint: "YOUR_ENDPOINT" # The OTel collector endpoint Airbyte sends metrics to. You configure this endpoint outside of Airbyte as part of your OTel deployment.

3. Redeploy Airbyte with the updated values.

Airbyte sends metrics to the collector you specified in your configuration.

Available metrics

The following metrics are available. They're published every minute.

Metric	Tag	Example Value
airbyte.syncs	connection_id	653a067e-cd0b-4cab-96b5-5e5cb03f159b
	workspace_id	bed3b473-1518-4461-a37f-730ea3d3a848
	job_id	23642492
	status	success, failed
	attempt_count	3
	source_connector_id	82c7fb2d-7de1-4d4e-b12e-510b0d61e374
	destination_connector_id	3cb42982-755b-4644-9ed4-19651b53ebdd
	version	1.5.0
	service	airbyte-worker
airbyte.gb_moved	connection_id	653a067e-cd0b-4cab-96b5-5e5cb03f159b
	workspace_id	bed3b473-1518-4461-a37f-730ea3d3a848
	job_id	23642492
	source_connector_id	82c7fb2d-7de1-4d4e-b12e-510b0d61e374
	destination_connector_id	3cb42982-755b-4644-9ed4-19651b53ebdd
	version	1.5.0
	service	airbyte-worker
airbyte.sync_duration	connection_id	653a067e-cd0b-4cab-96b5-5e5cb03f159b
	workspace_id	bed3b473-1518-4461-a37f-730ea3d3a848
	job_id	23642492
	source_connector_id	82c7fb2d-7de1-4d4e-b12e-510b0d61e374
	destination_connector_id	3cb42982-755b-4644-9ed4-19651b53ebdd
	version	1.5.0
	service	airbyte-worker
airbyte.api_requests	uri	/v1/applications/create
	status	200
	exception	NullPointerException
	method	GET, POST
	version	1.5.0
	service	airbyte-server