|Full Refresh Sync||No|
|Incremental - Append Sync||Yes|
|Incremental - Append + Deduped||No|
Uploading Vectors: Use the vectors configuration if you want to upload vectors from a source database into Weaviate. You can do this by specifying the stream name and vector field name in the following format:
For example, if you have a table named
my_table and the vector is stored using the column
you should use the following
Dynamic Schema: Weaviate will automatically create a schema for the stream if no class was defined unless you have disabled the Dynamic Schema feature in Weaviate. You can also create the class in Weaviate in advance if you need more control over the schema in Weaviate.
IDs: If your source table has an int based id stored as field name
id then the
ID will automatically be converted to a UUID. Weaviate only supports the ID to be a UUID.
For example, if the record has
id=1 then this would become a uuid of
Any field name starting with an upper case letter will be converted to lower case. For example,
if you have a field name
USD then that field will become
uSD. This is due to a limitation
in Weaviate, see this issue in Weaviate.
Airbyte Cloud only supports connecting to your Weaviate Instance instance with TLS encryption and with a username and password.
Getting Started (Airbyte Open-Source)
To use the Weaviate destination, you'll need:
- A Weaviate cluster version 220.127.116.11 or above
Configure Network Access
Make sure your Weaviate database can be accessed by Airbyte. If your database is within a VPC, you may need to allow access from the IP you're using to expose Airbyte.
You need a Weaviate user or use a Weaviate instance that's accessible to all
Setup the Weaviate Destination in Airbyte
You should now have all the requirements needed to configure Weaviate as a destination in the UI. You'll need the following information to configure the Weaviate destination:
- URL for example http://localhost:8080 or https://my-wcs.semi.network
- Username (Optional)
- Password (Optional)
- Batch Size (Optional, defaults to 100)
- Vectors a comma separated list of
<stream_name.vector_field_name>to specify the field
- ID Schema a comma separated list of
<stream_name.id_field_name>to specify the field name that contains the ID of a record
|0.1.1||2022-02-08||#22527||Multiple bug fixes: Support String based IDs, arrays of uknown type and additionalProperties of type object and array of objects|
|0.1.0||2022-12-06||#20094||Add Weaviate destination|