The incoming Airbyte data is structured depending on the target Redis cache/data type. This connector maps an incoming data from a namespace and stream to a unique Redis key.
For the hash implementation as a Redis data type the keys and the hashes are structured in the following way:
_airbyte_ab_id: Sequential id for a given key generated by using the INCR Redis command.
_airbyte_emitted_at: a timestamp representing when the event was received from the data source.
_airbyte_data: a json text/object representing the data that was received from the data source.
|Full Refresh Sync||✅||Existing keys in the Redis cache are deleted and replaced with the new keys.|
|Incremental - Append Sync||✅||New keys are inserted in the same keyspace without touching the existing keys.|
|Incremental - Deduped History||❌|
|Namespaces||✅||Namespaces will be used to determine the correct Redis key.|
As long as you have the necessary memory capacity for your cache, Redis should be able to handle even millions of records without any issues since the data is stored in-memory with the option to save snapshots periodically on disk.
- The connector is fully compatible with redis 2.8.x, 3.x.x and above
- host: Hostname or address of the Redis server where to connect.
- port: Port of the Redis server where to connect.
- username: Username for authenticating with the Redis server.
- password: Password for authenticating with the Redis server.
- cache_type: Redis cache/data type to use when storing the incoming messages. i.e hash,set,list,stream,etc.
######TODO: more info, screenshots?, etc...