This page guides you through the process of setting up the Kafka source connector.
Set up guide
Step 1: Set up Kafka
To use the Kafka source connector, you'll need:
- A Kafka cluster 1.0 or above
- Airbyte user should be allowed to read messages from topics, and these topics should be created before reading from Kafka.
Step 2: Setup the Kafka source in Airbyte
You'll need the following information to configure the Kafka source:
- Group ID - The Group ID is how you distinguish different consumer groups. (e.g. group.id)
- Protocol - The Protocol used to communicate with brokers.
- Client ID - An ID string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging. (e.g. airbyte-consumer)
- Test Topic - The Topic to test in case the Airbyte can consume messages. (e.g. test.topic)
- Subscription Method - You can choose to manually assign a list of partitions, or subscribe to all topics matching specified pattern to get dynamically assigned partitions.
- List of topic
- Bootstrap Servers - A list of host/port pairs to use for establishing the initial connection to the Kafka cluster.
For Airbyte Cloud:
- Log into your Airbyte Cloud account.
- In the left navigation bar, click Sources. In the top-right corner, click +new source.
- On the Set up the source page, enter the name for the Kafka connector and select Kafka from the Source type dropdown.
- Follow the Setup the Kafka source in Airbyte
Supported sync modes
The Kafka source connector supports the followingsync modes:
|Full Refresh Sync||Yes|
|Incremental - Append Sync||Yes|
|0.1.7||2022-06-17||13864||Updated stacktrace format for any trace message errors|
|0.1.6||2022-05-29||12903||Add Polling Time to Specification (default 100 ms)|
|0.1.5||2022-04-19||12134||Add PLAIN Auth|
|0.1.4||2022-02-15||10186||Add SCRAM-SHA-512 Auth|
|0.1.2||2021-12-21||8865||Fix SASL config read issue|
|0.1.1||2021-12-06||8524||Update connector fields title/description|