updated_at), CDC allows you to sync your table incrementally.
LOGINpermissions. You can create a role with
CREATE ROLE <name> REPLICATION LOGIN;and grant that role to the user. You still need to make sure the user can connect to the database, use the schema, and to use
SELECTon tables (the same are required for non-CDC incremental syncs and all full refreshes).
pgoutputplugin as it is the standard logical decoding plugin in Postgres. In case the replication table contains a lot of big JSON blobs and table size exceeds 1 GB, we recommend using a
wal2jsoninstead. Please note that
wal2jsonmay require additional installation for Bare Metal, VMs (EC2/GCE/etc), Docker, etc. For more information read wal2json documentation.
wal2jsonplugin, please change
wal2jsonvalue in the above query.
ALTER TABLE tbl1 REPLICA IDENTITY DEFAULT;to use primary keys to distinguish between rows. After setting the replication identity, you will need to run
CREATE PUBLICATION airbyte_publication FOR TABLE <tbl1, tbl2, tbl3>;. This publication name is customizable. You must add the replication identity before creating the publication. Otherwise,
DELETEstatements may fail if Postgres cannot determine how to uniquely identify rows. Please refer to the Postgres docs if you need to add or remove tables from your publication in the future.
postgresql.conffile for your database. You can find the location of this file using
psql -U postgres -c 'SHOW config_file'withe the correct
psqlcredentials specified. Alternatively, a custom file can be specified when running postgres with the
-cflag. For example
postgres -c config_file=/etc/postgresql/postgresql.confruns Postgres with the config file at
postgresDocker image, you will need to mount a file and change the command to run Postgres with the set config file. If you're just testing CDC behavior, you may want to use a modified version of a sample
wal_levelis the type of coding used within the Postgres write-ahead log. This must be set to
logicalfor Airbyte CDC.
max_wal_sendersis the maximum number of processes used for handling WAL changes. This must be at least one.
max_replication_slotsis the maximum number of replication slots that are allowed to stream WAL changes. This must one if Airbyte will be the only service reading subscribing to WAL changes or more if other services are also reading from the WAL.
Configurationtab for your DB cluster.
rds.logical_replication. Select this row and click on the
Edit parametersbutton. Set this value to
SSH Tunnel Method.
SSH Tunnel Methoddefaults to
No Tunnel(meaning a direct connection). If you want to use an SSH Tunnel choose
SSH Key Authenticationor
Key Authenticationif you will be using an RSA Private as your secrets for establishing the SSH Tunnel (see below for more information on generating this key).
Password Authenticationif you will be using a password as your secret for establishing the SSH Tunnel.
SSH Tunnel Jump Server Hostrefers to the intermediate (bastion) server that Airbyte will connect to. This should be a hostname or an IP Address.
SSH Connection Portis the port on the bastion server with which to make the SSH connection. The default port for SSH connections is
22, so unless you have explicitly changed something, go with the default.
SSH Login Usernameis the username that Airbyte should use when connection to the bastion server. This is NOT the Postgres username.
Password Authentication, then
SSH Login Usernameshould be set to the password of the User from the previous step. If you are using
SSH Key Authenticationleave this blank. Again, this is not the Postgres password, but the password for the OS-user that Airbyte is using to perform commands on the bastion.
SSH Key Authentication, then
SSH Private Keyshould be set to the RSA Private Key that you are using to create the SSH connection. This should be the full contents of the key file starting with
-----BEGIN RSA PRIVATE KEY-----and ending with
-----END RSA PRIVATE KEY-----.
authorized_keysfile on your bastion host. The public key should be added to your bastion host to whichever user you want to use with Airbyte. The private key is provided via copy-and-paste to the Airbyte connector configuration screen, so it may log in to the bastion.
AIRBYTE_ENTRYPOINTfor Kubernetes support