Skip to main content

Redpanda [ARCHIVED]

The Airbyte Redpanda destination connector allows you to sync data to Redpada. Each stream is written to the corresponding Redpanda topic.

Sync overview

Output schema

Each stream will be output into a Redpanda topic.

The Redpanda topic will be created with the following format {namespace}_{stream}

Currently, this connector only writes data with JSON format. More formats (e.g. Apache Avro) will be supported in the future.

Each record will contain in its key the uuid assigned by Airbyte, and in the value these 3 fields:

  • _airbyte_ab_id: a uuid assigned by Airbyte to each event that is processed.
  • _airbyte_emitted_at: a timestamp representing when the event was pulled from the data source.
  • _airbyte_data: a json blob representing with the event data.

Data type mapping

Integration TypeAirbyte TypeNotes

Features

This section should contain a table with the following format:

FeatureSupported?(Yes/No)Notes
Full Refresh SyncNo
Incremental - Append SyncYes
Incremental - Append + DedupedNo
NamespacesYes

Performance considerations

Granted you have enough Redpanda nodes/partitions the cluster should be able to handle any type of load you throw at it from the connector.

Getting started

Requirements

  • The connector should be able to create topics using the AdminClient
  • Configuration options
    • Bootstrap servers
    • Buffer Memory
    • Compression Type
    • Batch Size
    • Retries
    • Number of topic partitions
    • Topic replication factor
    • Socket Connection Setup Timeout
    • Socket Connection Setup Max Timeout

More info about this can be found in the Redpanda producer configs documentation site.

NOTE: Configurations for SSL are not available yet.

Changelog

Expand to review
VersionDatePull RequestSubject
0.1.02022-08-0518884Initial commit