Keen [ARCHIVED]

Availability:

Airbyte Cloud Airbyte OSS PyAirbyte

Support Level:

Archived

Overview

The Airbyte Keen destination allows you to stream data from any Airbyte Source into Keen for storage, analysis, and visualization. Keen is a flexible, fully managed event streaming and analytics platform that empowers anyone to ship custom, embeddable dashboards in minutes, not months.

Prerequisites

• For Airbyte Open Source users using the Postgres source connector, upgrade your Airbyte platform to version v0.40.0-alpha or newer and upgrade your Keen connector to version 0.2.4 or newer

Sync overview

Output schema

Each replicated stream from Airbyte will output data into a corresponding event collection in Keen. Event collections store data in JSON format. Each collection will inherit the name from the stream with all non-alphanumeric characters removed, except for .’, ‘-’, ‘_’, and whitespace characters. When possible, the connector will try to infer the timestamp value for the record and override the special field keen.timestamp with it.

Features

Feature	Supported?(Yes/No)	Notes
Full Refresh Sync	Yes
Incremental - Append Sync	Yes
Incremental - Append + Deduped	No
Namespaces	No

Getting started

Requirements

To use the Keen destination, you'll first need to create a Keen account (if you don’t already have one).

Once you have a Keen account, you can use the following credentials to set up the connector

• A Keen Project ID
• A Keen Master API key associated with the project

See the setup guide for more information about how to get started.

Setup guide

Keen Project

If you haven’t set up a project to stream your data to:

Login to the Keen application and add a new project. To do this, click the ‘Add New’ link next to the Projects label on the left-hand, side ribbon. Then, give the project a name.

You can think of a project as a data silo. The data in a project is completely separate from data in other projects. We recommend that you create separate projects for each of your applications and separate projects for Dev and Prod environments.

Now, head to the ‘Access’ section and grab your Project ID and Master API Key.

If you already have a project set up:

Head to the ‘Access’ tab and grab your Project ID and Master API Key

API Key and Project ID

The Keen Connector uses the Keen Kafka Inbound Cluster to stream data. It requires your Project ID and Master Key for authentication. To get them, navigate to the Access tab from the left-hand, side panel and check the Project Details section. Important: This destination requires the Project's Master Key.

Timestamp Inference

The Infer Timestamp field lets you specify if you want the connector to infer the keen.timestamp field based on the data from the event that occurred in the source application. This feature allows for historical data synchronization enabling you to fully leverage the power of Keen's time series analytics. By default, this property is set to true. If toggled off, keen.timestamp will be set to the datetime when the data was recorded by Keen.

Setup the Keen destination in Airbyte

Now, you should have all the parameters needed to configure Keen destination.

• Project ID
• Master API Key
• Infer Timestamp

Connect your first source and then head to the Keen application. You can seamlessly run custom analysis on your data and build interactive dashboards for key stakeholders.

If you have any questions, please reach out to us at [email protected] and we’ll be happy to help!

Changelog

Expand to review

Version	Date	Pull Request	Subject
0.2.4	2022-08-04	15291	Update Keen destination to use outputRecordCollector to properly store state
0.2.3	2022-06-17	13864	Updated stacktrace format for any trace message errors
0.2.1	2021-12-30	8809	Update connector fields title/description
0.2.0	2021-09-10	5973	Fix timestamp inference for complex schemas
0.1.0	2021-08-18	5339	Keen Destination Release!