Connector Metadata.yaml File
metadata.yaml file is a new addition to Airbyte's connector folders. This file is created with the goal of simplifying and enhancing how we manage information related to each connector. It is designed to replace the previous
metadata.yaml file contains crucial information about the connector, including its type, definition ID, Docker image tag, Docker repository, and much more. It plays a key role in the way Airbyte handles connector data and improves the overall organization and accessibility of this data.
Below is an example of a
metadata.yaml file for the Postgres source:
registries section within the
metadata.yaml file plays a vital role in determining the contents of the
This section contains two subsections:
oss (Open Source Software). Each subsection contains details about the specific registry, such as the Docker repository associated with it and whether it's enabled or not.
Here's how the
registries section is structured in our previous
In this example, both
oss registries are enabled, and the Docker repository for the
cloud registry is overrode to
metadata.yaml file is updated, this data is automatically uploaded to Airbyte's metadata service. This service then generates the publicly available
cloud_registry.json registries based on the information provided in the
For instance, if a connector is listed as
enabled: true under the
oss section, it will be included in the
oss_registry.json file. Similarly, if it's listed as
enabled: true under the
cloud section, it will be included in the
registries section in the
metadata.yaml file provides a flexible and organized way to manage which connectors are included in each registry.
tags field is an optional part of the
metadata.yaml file. It is designed to provide additional context about a connector and improve the connector's discoverability. This field can contain any string, making it a flexible tool for adding additional details about a connector.
tags is a list that may contain any number of string elements. Each element in the list is a separate tag. For instance:
In the example above, the connector has three tags. Tags are used for two primary purposes in Airbyte:
Denoting the Programming Language(s): Tags that begin with language: are used to specify the programming languages that are utilized by the connector. This information is auto-generated by a script that scans the connector's files for recognized programming languages. In the example above, language:java means that the connector uses Java.
Keywords for Searching: Tags that begin with keyword: are used to make the connector more discoverable by adding searchable terms related to it. In the example above, the tags keyword:database and keyword:SQL can be used to find this connector when searching for
These are just examples of how tags can be used. As a free-form field, the tags list can be customized as required for each connector. This flexibility allows tags to be a powerful tool for managing and discovering connectors.
This denotes the name of the icon file for the connector. At this time the icon file is located in the
platform-internal repository. So please ensure that the icon file is present in the
platform-internal repository at oss/airbyte-config/init/src/main/resources/icons before adding the icon name to the
⚠️ This property is in the process of being refactored to be a file in the connector folder
You may notice a
icon.svg file in the connectors folder.
This is because we are transitioning away from icons being stored in the
platform-internal repository. Instead, we will be storing them in the connector folder itself. This will allow us to have a single source of truth for all connector-related information.
This transition is currently in progress. Once it is complete, the
icon field in the
metadata.yaml file will be removed, and the
icon.svg file will be used instead.
releases section contains extra information about certain types of releases. The current types of releases are:
breakingChanges section of
releases contains a dictionary of version numbers (usually major versions, i.e.
1.0.0) and information about
their associated breaking changes. Each entry must contain the following parameters:
message: A description of the breaking change, written in a user-friendly format. This message should briefly describe
- What the breaking change is, and which users it effects (e.g. all users of the source, or only those using a certain stream)
- Why the change is better for the user (fixed a bug, something got faster, etc)
- What the user should do to fix the issue (e.g. a full reset, run a SQL query in the destinaton, etc)
YYYY-MM-DD) The date by which the user should upgrade to the new version.
When considering what the
upgradeDeadline should be, target the amount of time which would be reasonable for the user to make the required changes described in the
message and upgrade giude. If the required changes are simple (e.g. "do a full reset"), 2 weeks is recommended. Note that you do not want to link the duration of
upgradeDeadline to an upstream API's deprecation date. While it is true that the older version of a connector will continue to work for that period of time, it means that users who are pinned to the older version of the connector will not benefit from future updates and fixes.
Without all 3 of these points, the breaking change message is not helpful to users.
Here is an example:
message: "This version changes the connector’s authentication by removing ApiKey authentication, which is now deprecated by the [upstream source](upsteam-docs-url.com). Users currently using ApiKey auth will need to reauthenticate with OAuth after upgrading to continue syncing."