Skip to main content

Elasticsearch

Sync overview

Output schema

Elasticsearch is a Lucene based search engine that's a type of NoSql storage.
Documents are created in an index, similar to a tablein a relation database.

The output schema matches the input schema of a source. Each source stream becomes a destination index.
For example, in with a relational database source -
The DB table name is mapped to the destination index. The DB table columns become fields in the destination document.
Each row becomes a document in the destination index.

Data type mapping

See Elastic documentation for detailed information about the field types This section should contain a table mapping each of the connector's data types to Airbyte types. At the moment, Airbyte uses the same types used by JSONSchema. string, date-time, object, array, boolean, integer, and number are the most commonly used data types.

Integration TypeAirbyte TypeNotes
textstringmore info
datedate-timemore info
objectobjectmore info
arrayarraymore info
booleanbooleanmore info
numericintegermore info
numericnumbermore info

Features

This section should contain a table with the following format:

FeatureSupported?(Yes/No)Notes
Full Refresh Syncyes
Incremental Syncyes
Replicate Incremental Deletesno
SSL connectionyes
SSH Tunnel Support??

Performance considerations

Batch/bulk writes are performed. Large records may impact performance.
The connector should be enhanced to support variable batch sizes.

Getting started

Requirements

  • Elasticsearch >= 7.x
  • Configuration
    • Endpoint URL [ex. https://elasticsearch.savantly.net:9423]
    • Username [optional] (basic auth)
    • Password [optional] (basic auth)
    • Api key ID [optional]
    • Api key secret [optional]
  • If authentication is used, the user should have permission to create an index if it doesn't exist, and/or be able to create documents

Setup guide

Enter the hostname and/or other configuration information ...

CHANGELOG

VersionDatePull RequestSubject
0.1.22022-04-1911752Reduce batch size to 32Mb
0.1.12022-02-1010256Add ExitOnOutOfMemoryError connectors
0.1.02021-10-137005Initial release.