Skip to main content

Vectara

This page contains the setup guide and reference information for the Vectara destination connector.

Vectara is the trusted GenAI platform that provides Retrieval Augmented Generation or RAG as a service.

The Vectara destination connector allows you to connect any Airbyte source to Vectara and ingest data into Vectara for your RAG pipeline.

info

If you have issues, use the following public channels for support:

Overview​

The Vectara destination connector supports Full Refresh Overwrite, Full Refresh Append, and Incremental Append. It uses the Vectara REST API v2.

Output schema​

The connector writes all streams into a single Vectara corpus, which you specify by name in the connector configuration. Each record becomes one structured document in that corpus:

  • Sections: Each configured text field becomes one document section, formatted as field_name: value.
  • Title: The value of the configured title field, or Untitled if you don't set a title field or the record doesn't contain it.
  • Metadata: The configured metadata fields, plus an _ab_stream field that contains {namespace}_{stream_name} and identifies the source stream. Values that aren't strings, numbers, or booleans are stored as JSON-encoded strings.
  • Document ID: If the stream has a primary key, the document ID is built from the namespace, stream name, and primary key values. If the stream has no primary key, the connector generates a random ID for each record.

The connector looks up the corpus by its exact name:

  • If no corpus with that name exists, the connector creates one. It derives the corpus key from the name by replacing any character other than letters, digits, _, =, and - with _, and truncating the result to 50 characters. It also creates the _ab_stream filter attribute on the new corpus.
  • If exactly one corpus with that name exists, the connector uses it.
  • If more than one corpus has that name, the connection check fails because the connector can't determine which corpus to use.

Supported sync modes​

Sync modeSupported?
Full Refresh - OverwriteYes
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedNo
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedNo

How the sync modes behave in Vectara:

  • Full Refresh - Overwrite: At the start of the sync, the connector deletes every document in the corpus whose _ab_stream metadata matches the stream, then indexes the new records. The stream's documents are unavailable in the corpus until the sync re-indexes them.
  • Append modes and primary keys: If a stream has a primary key and a document with the same ID already exists in the corpus, Vectara rejects the new document and the connector keeps the existing one. Updated records with an existing primary key aren't re-indexed in append modes. To pick up changes to existing records, use Full Refresh - Overwrite. If a stream has no primary key, every synced record is indexed as a new document, so repeated syncs can create duplicates.

Getting started​

You need a Vectara account to use Vectara with Airbyte.

  1. Sign up for a Vectara account if you don't already have one.
  2. Optionally, create the corpus you want to load data into. If you skip this step, the connector creates the corpus for you during the first connection check or sync.
    • To create a corpus in the Vectara console, click Corpora and then Create corpus. Enter a name and a key for the corpus. For details, see Create a Corpus.
    • If you create the corpus yourself, you must add a document-level filter attribute named _ab_stream with the Text data type, and mark it as indexed. The connector uses this attribute to identify source streams and to delete a stream's documents during Full Refresh - Overwrite syncs.
    • Optionally, add filter attributes for any other metadata fields you want to filter on.
  3. Create an app client for OAuth 2.0 authentication. In the Vectara console, go to API access > App clients and click Create app client. Give the client roles that allow it to list and create corpora, and to index, list, and delete documents in your corpus. Copy the Client ID and Client Secret. For details, see App Clients for OAuth and OAuth 2.0.

Set up the Vectara destination in Airbyte​

Configure the Vectara destination with the following information:

  • (Required) OAuth Client ID and OAuth Client Secret: The credentials of the app client you created.
  • (Optional) Customer ID: Kept only for backward compatibility. Vectara API v2 authenticates with the OAuth 2.0 client credentials alone, so you can leave this field empty.
  • (Required) Corpus Name: The name of the corpus to load data into. If no corpus with this name exists, the connector creates it.
  • (Optional) Parallelize: Index documents into Vectara using multiple threads. Defaults to false.

You can also control how the connector maps record fields to Vectara documents. These settings apply to all streams. Fields that don't exist in a record are ignored. To reference nested fields, use dot notation, for example user.name. To reference a field in every element of an array, use a wildcard, for example users.*.name.

  • Text fields to index with Vectara (text_fields): The fields that become document sections, which Vectara uses for search and summarization. If you don't specify any, all fields are text fields. Each record must contain at least one configured text field, or the sync fails.
  • Text field to use as document title with Vectara (title_field): The field that populates each document's title.
  • Fields to store as metadata (metadata_fields): The fields to store as document metadata. If you don't specify any, all fields are stored as metadata. When a nested path matches multiple values, the connector stores them as an array in a metadata field named after the path.

Namespace support​

This destination supports namespaces. All namespaces write to the same corpus. The namespace is part of the _ab_stream metadata value, so streams with the same name in different namespaces remain distinguishable.

Reference​

Config fields reference

Field
Type
Property name
string
corpus_name
object
oauth2
string
customer_id
array<string>
metadata_fields
boolean
parallelize
array<string>
text_fields
string
title_field

Changelog​

Expand to review
VersionDatePull RequestSubject
0.3.02026-11-0390862Upgrade to Python 3.13 (python-connector-base 4.1.1) and airbyte-cdk 7
0.2.322026-10-0888339Migrate from the retired Vectara API v1 (and per-customer Cognito OAuth endpoint) to API v2 / auth.vectara.com
0.2.312024-11-2548659Update dependencies
0.2.302024-11-0448222Update dependencies
0.2.292024-10-2947744Update dependencies
0.2.282024-10-2347084Update dependencies
0.2.272024-10-1246812Update dependencies
0.2.262024-10-0546438Update dependencies
0.2.252024-09-2846114Update dependencies
0.2.242024-09-2145806Update dependencies
0.2.232024-09-1445481Update dependencies
0.2.222024-09-0745324Update dependencies
0.2.212024-08-3145021Update dependencies
0.2.202024-08-2444657Update dependencies
0.2.192024-08-2244530Update test dependencies
0.2.182024-08-1744310Update dependencies
0.2.172024-08-1243858Update dependencies
0.2.162024-08-1043494Update dependencies
0.2.152024-08-0343153Update dependencies
0.2.142024-07-2742705Update dependencies
0.2.132024-07-2042168Update dependencies
0.2.122024-07-1341829Update dependencies
0.2.112024-07-1041362Update dependencies
0.2.102024-07-0941140Update dependencies
0.2.92024-07-0640953Update dependencies
0.2.82024-06-2740215Replaced deprecated AirbyteLogger with logging.Logger
0.2.72024-06-2540321Update dependencies
0.2.62024-06-2239973Update dependencies
0.2.52024-06-0639193[autopull] Upgrade base image to v1.2.2
0.2.42024-05-2038432[autopull] base image + poetry + up_to_date
0.2.32024-03-22#37333Updated CDK & pytest version to fix security vulnerabilities
0.2.22024-03-22#36261Move project to Poetry
0.2.12024-03-05#35206Fix: improved title parsing
0.2.02024-01-29#34579Add document title file configuration
0.1.02023-11-10#31958🎉 New Destination: Vectara (Vector Database)