Collecting Metrics
Airbyte supports two ways to collect metrics - using datadog or open telemetry.
Fill in METRIC_CLIENT
field in .env
file to get started!
Prerequisite: In order to get metrics from airbyte we need to deploy a container / pod called metrics-reporter like below
airbyte-metrics:
image: airbyte/metrics-reporter:${VERSION}
container_name: airbyte-metrics
environment:
- METRIC_CLIENT=${METRIC_CLIENT}
- OTEL_COLLECTOR_ENDPOINT=${OTEL_COLLECTOR_ENDPOINT}
Open Telemetry
- In
.env
changeMETRIC_CLIENT
tootel
. - Similarly, configure
OTEL_COLLECTOR_ENDPOINT
to tell Airbyte where to send metrics RPC to.
Example
Run Opentelemetry and Airbyte locally
In this example we will run Airbyte locally along with an Open Telemetry Collector. The Open telemetry collector will expose port 4317 to the localhost as the receiving endpoint.
Steps:
- Setting up Open telemetry. In this example we will use the repository from
opentelemetry-java-docs
. Run the following commands to have it up and running.
git clone https://github.com/open-telemetry/opentelemetry-java-docs
cd opentelemetry-java-docs/otlp/docker
docker-compose up
- Configure Airbyte
.env
file.- Change
METRIC_CLIENT
tootel
to indicate Airbyte to use Open telemetry to emit metric data. - Change
OTEL_COLLECTOR_ENDPOINT
to"http://host.docker.internal:4317"
because Open Telemetry Collector has enabled port forward from localhost:4317 to container port 4317. To send data to Collector container port 4317, we want to need to export data to physical machine's localhost:4317, which in docker will be represented ashttp://host.docker.internal:4317
.Do not use
localhost:4317
or you will send data to the same container where Airbyte Worker is running. - Start Airbyte server by running
docker compose up
under airbyte repository. Go tolocalhost:8000
to visit Airbyte and start a sync, then go tolocalhost:9090
to access Prometheus - you should be able to see the metrics there. Alternatively,
- Change
Run Opentelemetry and Airbyte on kubernetes
Prerequisite: Read https://github.com/airbytehq/airbyte/blob/master/docs/deploying-airbyte/on-kubernetes.md to understand how to start Airbyte on Kubernetes
We will use stable
in this example.
Steps:
- Run open telemetry collector in the same Kubernetes context. Here we follow example in OpenTelemetry doc
- edit
kube/overlays/stable/.env
and add the following lines:
METRIC_CLIENT=otel
OTEL_COLLECTOR_ENDPOINT=<address>
If you started open telemetry collector in the link above, the address should be http://otel-collector:4317
.
Note the format - unlike the base .env
, there is no quote in .env
file under kubernetes.
Tutorial
Deploy the airbyte metric pod :
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Release.Name }}-metrics
namespace: {{ .Release.Namespace }}
labels: {{ set . "component" "metrics" | include "labels" | nindent 4 }}
spec:
selector:
matchLabels: {{ set . "component" "metrics" | include "labels" | nindent 6 }}
template:
metadata:
labels: {{ set . "component" "metrics" | include "labels" | nindent 8 }}
spec:
containers:
- name: airbyte-metrics
image: "airbyte/metrics-reporter:latest"
imagePullPolicy: IfNotPresent
env:
- name: AIRBYTE_VERSION
value: latest
- name: DATABASE_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "airbyte.database.secret.name" . }}
key: DATABASE_PASSWORD
- name: DATABASE_URL
value: {{ include "airbyte.database.url" . | quote }}
- name: DATABASE_USER
valueFrom:
secretKeyRef:
name: {{ .Release.Name }}-secrets
key: DATABASE_USER
- name: CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION
value: 0.35.15.001
- name: METRIC_CLIENT
value: otel
- name: OTEL_COLLECTOR_ENDPOINT
value: http://otel-collector:4317
Deploy an Open telemetry pod like below :
apiVersion: apps/v1
kind: Deployment
metadata:
name: otel-collector
namespace: {{ .Release.Namespace }}
labels: {{ set . "component" "otel-collector" | include "labels" | nindent 4 }}
spec:
selector:
matchLabels: {{ set . "component" "otel-collector" | include "labels" | nindent 6 }}
replicas: 1
template:
metadata:
labels: {{ set . "component" "otel-collector" | include "labels" | nindent 8 }}
spec:
containers:
- command:
- "/otelcol"
- "--config=/conf/otel-collector-config.yaml"
image: "otel/opentelemetry-collector:latest"
name: otel-collector
ports:
- containerPort: 4317 # Default endpoint for OpenTelemetry receiver.
- containerPort: 8889 # Port for Prometheus instance
volumeMounts:
- name: config
mountPath: /conf
volumes:
- configMap:
name: otel-collector-conf
items:
- key: otel-collector-config
path: otel-collector-config.yaml
name: config
WIth this Config Map :
apiVersion: v1
kind: ConfigMap
metadata:
name: otel-collector-conf
namespace: {{ .Release.Namespace }}
labels: {{ set . "component" "otel-collector" | include "labels" | nindent 4 }}
data:
otel-collector-config: |
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
processors:
batch:
memory_limiter:
limit_mib: 1500
spike_limit_mib: 512
check_interval: 5s
exporters:
prometheus:
endpoint: "0.0.0.0:8889"
namespace: "default"
service:
pipelines:
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [prometheus]
Then we need a service to be able to access both open telemetry GRPC and Prometheus
apiVersion: v1
kind: Service
metadata:
name: otel-collector
namespace: {{ .Release.Namespace }}
labels: {{ set . "component" "otel-collector" | include "labels" | nindent 4 }}
spec:
ports:
- name: otlp-grpc # Default endpoint for OpenTelemetry gRPC receiver.
port: 4317
protocol: TCP
targetPort: 4317
- name: prometheus
port: 8889
selector: {{ set . "component" "otel-collector" | include "labels" | nindent 6 }}
And finally We can add a service monitor to receive metrics in prometheus and optionally add some prometheus rules to generate alerts. You can replace with your prometheus name.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: {{ .Release.Name }}
namespace: {{ .Release.Namespace }}
labels:
{{ set . "component" "metrics" | include "labels" | nindent 4 }}
prometheus: <your_prometheus_name>
spec:
endpoints:
- interval: 30s
port: prometheus
path: /metrics
relabelings:
- action: labeldrop
regex: (service|endpoint|namespace|container)
selector:
matchLabels: {{ set . "component" "otel-collector" | include "labels" | nindent 6 }}
One rule example :
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: {{ .Release.Name }}-rules
namespace: {{ .Release.Namespace }}
labels:
{{ set . "component" "prometheus-rules" | include "labels" | nindent 4 }}
prometheus: <your_prometheus_name>
spec:
groups:
- name: airbyte
rules:
- alert: AirbyteJobFail
for: 0m
expr: min(airbyte_job_failed_by_release_stage) > 0
labels:
priority: P2
annotations:
summary: {{ `"An Airbyte Job has failed"` }}
Datadog
The set up for Datadog is pretty straightforward.
Set two env vars:
1) METRIC_CLIENT
to datadog
.
2) PUBLISH_METRICS
to true
.
Configure two additional env vars to the Datadog endpoint:
1) Set DD_AGENT_HOST
to the IP where the Datadog agent is running.
2) Set DD_DOGSTATSD_PORT
to the port the Datadog agent is using.
Metrics
Visit OssMetricsRegistry.java to get a complete list of metrics Airbyte is sending.
Additional information
Suppose you are looking for a non-production way of collecting metrics with dbt and Metabase, the tutorial Airbyte Monitoring with dbt and Metabase by accessing Airbyte's Postgres DB. The source code is open on airbytehq/open-data-stack. Think of it as an exploratory for data analysts and data engineers of building a dashboard on top of the existing Airbyte Postgres database versus the Prometheus more for DevOps engineers in production.