Skip to main content

Amazon SQS [ARCHIVED]

Overview

The Airbyte SQS destination allows you to sync data to Amazon SQS. It currently supports sending all streams to a single Queue.

Sync overview

Output schema

All streams will be output into a single SQS Queue.

Amazon SQS messages can only contain JSON, XML or text, and this connector supports writing messages in all three formats. See the Writing Text or XML messages section for more info.

Features

FeatureSupported?(Yes/No)Notes
Full Refresh SyncNo
Incremental - Append SyncYes
Incremental - Append + DedupedNo
NamespacesNo

Getting started

Requirements

  • AWS IAM Access Key
  • AWS IAM Secret Key
  • AWS SQS Queue

Permissions

If the target SQS Queue is not public, you will need the following permissions on the Queue:

  • sqs:SendMessage

Properties

Required properties are 'Queue URL' and 'AWS Region' as noted in bold below.

  • Queue URL (STRING)
    • The full AWS endpoint URL of the queue e.g.https://sqs.eu-west-1.amazonaws.com/1234567890/example-queue-url
  • AWS Region (STRING)
    • The region code for the SQS Queue e.g. eu-west-1
  • Message Delay (INT)
    • Time in seconds that this message should be hidden from consumers.
    • See the AWS SQS documentation for more detail.
  • AWS IAM Access Key ID (STRING)
    • The Access Key for the IAM User with permissions on this Queue
    • Permission sqs:SendMessage is required
  • AWS IAM Secret Key (STRING)
    • The Secret Key for the IAM User with permissions on this Queue
  • Message Body Key (STRING)
    • Rather than sending the entire Record as the Message Body, use this property to reference a Key in the Record to use as the message body. The value of this property should be the Key name in the input Record. The key must be at the top level of the Record, nested Keys are not supported.
  • Message Group Id (STRING)

Setup guide

Using the Message Body Key

This property allows you to reference a Key within the input Record as using that properties Value as the SQS Message Body.

For example, with the input Record:

{
"parent_with_child": {
"child": "child_value"
},
"parent": "parent_value"
}

To send only the parent_with_child object, we can set Message Body Key to parent_with_child. Giving an output SQS Message of:

{
"child": "child_value"
}

Writing Text or XML messages

To output Text or XML, the data must be contained within a String field in the input data, and then referenced by setting the Message Body Key property.

For example, with an input Record as:

{
"my_xml_field": "<something>value</something>"
}

To send a pure XML message, you would set the Message Body Key to my_xml_field.

The output SQS message would contain:

<something>value</something>

Changelog

Expand to review
VersionDatePull RequestSubject
0.1.142024-07-2042165Update dependencies
0.1.132024-07-1341733Update dependencies
0.1.122024-07-1041439Update dependencies
0.1.112024-07-0941237Update dependencies
0.1.102024-07-0640960Update dependencies
0.1.92024-06-2940622Update dependencies
0.1.82024-06-2740215Replaced deprecated AirbyteLogger with logging.Logger
0.1.72024-06-2640536Update dependencies
0.1.62024-06-2540461Update dependencies
0.1.52024-06-2240075Update dependencies
0.1.42024-06-0439070[autopull] Upgrade base image to v1.2.1
0.1.32024-05-2138493[autopull] base image + poetry + up_to_date
0.1.22024-03-05#35838Un-archive connector
0.1.12024-01-03#33924Add new ap-southeast-3 AWS region
0.1.02021-10-27#0000Initial version