ConcurrencyLevel

Defines the amount of parallelization for the streams that are being synced. The factor of parallelization is how many partitions or streams are synced at the same time. For example, with a concurrency_level of 10, ten streams or partitions of data will processed at the same time. Note that a value of 1 could create deadlock if a stream has a very high number of partitions.

• type: object
• required: ["default_concurrency"]

type

• Required: No
• type: string
• enum: ["ConcurrencyLevel"]

default_concurrency

The amount of concurrency that will applied during a sync. This value can be hardcoded or user-defined in the config if different users have varying volume thresholds in the target API.

• Required: Yes
• Available variables: config

Examples:

10

{{ config['num_workers'] or 10 }}

• anyOf:

  • type: integer

• anyOf:

  • type: string

max_concurrency

The maximum level of concurrency that will be used during a sync. This becomes a required field when the default_concurrency derives from the config, because it serves as a safeguard against a user-defined threshold that is too high.

• Required: No
• type: integer

Examples:

20

100

$parameters

• Required: No
• type: object