Skip to main content

Add a connection

AvailableCoreAvailableStandardAvailablePlusAvailableProAvailableEnterprise Flex Compare

After you add a source and a destination, add a new connection to start syncing data from your source to your destination.

ELT and data activation destinations are different​

You configure all connections in a similar way. However, the exact process is different for data activation destinations. This page explains how to set up both.

  • ELT databases, warehouses, and lakes: These destinations are more agnostic about schema. You can, for example, create tables with any number of columns and change those column types if you need to. This is typical of data warehouses and data lakes, and allows you more freedom to determine the structure of your data.

  • Data activation: These destinations have stricter schemas that your connection must adhere to. For example, if you're syncing data to HubSpot, your HubSpot records have pre-existing fields for name, email, company, phone number, etc. Some of those fields may be optional, some may be required, and all expect data in a certain format. This means you need to map data from your source to your destination to ensure it arrives in the necessary format and structure. Typically, you have less freedom over the structure of data in these destinations.

The unique needs of a destination account for why setting up some connections is different than others.

Create an ELT connection​

Follow these steps to create a connection to a database, warehouse, lake, or similar type of destination.

  1. Click Connections in the navigation.

  2. Click New connection.

  3. Click the source you want to use. If you don't have one yet, you can add one.

  4. Click the destination you want to use. If you don't have one yet, you can add one. Wait a moment while Airbyte fetches the schema of your data.

  5. Under Select sync mode, select your sync mode. You can either replicate sources, which maintains an up-to-date copy of your source data in the destination, or you can append historical changes, allowing you to track changes to your data over time in your destination. Airbyte automatically selects the most appropriate sync mode for each stream based on this selection. However, you can update specific streams later.

  6. In the Schema table, configure your schema.

    1. Choose the specific streams and fields you want to sync. You may have data you don't want to sync. For example, you might consider that data irrelevant or it might be subject to security or compliance rules. If you're unsure, you can always add it later. For help, see Configuring schemas.

    2. Choose the sync mode for each stream. For help, see sync modes.

    3. Set primary keys for each stream, if applicable. If possible, Airbyte sets this for you automatically.

  7. Click Next.

  8. Under Configure connection, finalize the settings for this connection.

    OptionDescription
    Connection nameGive your connection a more meaningful name if the default Source -> Destination isn't sufficient.
    TagsCreate and apply tags to organize your connections.
    Schedule typeChoose if and how you want to schedule this connection.
    Destination namespaceThe location in the destination where you want Airbyte to write this data.
    Stream prefixIf you want to prefix each stream name with a unique value.
    Advanced settingsIn most cases, you don't need to change these.

  9. Click Set up connection. Airbyte takes you to the page for that connection, where you can manage it and initiate syncs.

Create a data activation connection​

Follow these steps to create a connection to a data activation destination.

Step 1: Choose your connectors​

  1. Click Connections in the navigation.

  2. Click New connection.

  3. Click the source you want to use. If you don't have one yet, you can add one.

  4. Click the destination you want to use. If you don't have one yet, you can add one. Wait a moment while Airbyte fetches the schema of your data. When it's done, Airbyte asks you to set up mappings.

Step 2: Set up mappings, sync modes, and insertion methods​

Data activation destinations have stricter schemas that your connection must adhere to. Some of those fields may be optional, some may be required, and all expect data in a certain format. This means you need to map data from your source to your destination to ensure it arrives in the necessary format and structure.

  1. Select the first field you want to map from your source.

  2. Select the sync mode and, if necessary, the cursor. For help, see sync modes.

  3. Select the destination field to which you want to map that field.

  4. Select the insertion type. The insertion options available depend on the capabilities of the destination and its connector. Depending on your selection, Airbyte may require you to map other fields as well. In this case, it auto-populates those fields in the destination column.

  5. Continue mapping source fields to destination fields until you've mapped all required fields, plus any optional fields you want to include.

Step 3: Configure your connection​

The final step is to configure details of your connection.

OptionDescription
Connection nameGive your connection a more meaningful name if the default Source -> Destination isn't sufficient.
TagsCreate and apply tags to organize your connections.
Schedule typeChoose if and how you want to schedule this connection.
Destination namespaceThe location in the destination where you want Airbyte to write this data.
Stream prefixIf you want to prefix each stream name with a unique value.
Advanced settingsIn most cases, you don't need to change these.

When you're ready to create the connection, click Set up connection. Airbyte takes you to the page for that connection, where you can manage it and initiate syncs.

Step 4: Review and manage rejected records​

Destinations may reject records. See Rejected records to learn more.

Troubleshooting​

If you contact Airbyte Support about a problem setting up a connection, including a HAR file and the details of the relevant network request helps Airbyte diagnose your issue and turn around a resolution faster.

Capture a HAR file​

A HAR (HTTP Archive) file records the network activity between your browser and Airbyte. It's one of the most helpful things you can share with Airbyte Support, because it captures the exact requests your browser made, along with their responses and any errors.

  1. Open the Airbyte page where you encountered the problem.

  2. Open developer tools. Right-click anywhere on the page and click Inspect, or press F12 (Windows and Linux) or Option+Cmd+I (Mac).

  3. Click the Network tab.

  4. Select Preserve log so requests aren't cleared when the page reloads or redirects.

  5. Click Clear (the ⊘ icon) to remove any existing requests.

  6. Reproduce the problem while developer tools stays open. Airbyte records each network request as it happens.

  7. Right-click any request in the list, then click Save all as HAR with content.

  8. Save the file to your computer.

warning

HAR files can contain sensitive information, such as authentication tokens, cookies, and any data visible on the page. Only share HAR files with Airbyte Support, and let them know if there's anything you want to review or redact first.

Find the schema discovery request​

When you choose a source and destination, Airbyte fetches your source's schema with a discover_schema request. If connection setup fails while Airbyte is fetching the schema, this request usually contains the underlying error.

  1. Open developer tools and click the Network tab, following the steps above to capture requests.

  2. In the filter box, type discover_schema.

  3. Reproduce the problem. For example, select your source and destination so Airbyte fetches the schema.

  4. Click the discover_schema request.

  5. Review the Response and Preview tabs for error messages, and the Headers tab for the request details.

  6. Include this request, along with the HAR file, in your support submission.

Was this page helpful?