# YAML Reference
Copy Page
This page lists all components, interpolation variables and interpolation macros that can be used when defining a low code YAML file.
For the technical JSON schema definition that low code manifests are validated against, see [here](https://github.com/airbytehq/airbyte-python-cdk/blob/main/airbyte_cdk/sources/declarative/declarative_component_schema.yaml).
## Components[](#components "Direct link to Components")
### DeclarativeSource `object`[](#/definitions/DeclarativeSource "Direct link to /definitions/DeclarativeSource")
An API source that extracts data according to its declarative components.
Properties:
* #### check
Type:
* [`#/definitions/CheckStream`](#/definitions/CheckStream)
* [`#/definitions/CheckDynamicStream`](#/definitions/CheckDynamicStream)
* #### streams `array`
* #### dynamic\_streams `array` `#/definitions/DynamicDeclarativeStream`
* #### version `string`
The version of the Airbyte CDK used to build and test the source.
* #### schemas `#/definitions/Schemas`
* #### spec `#/definitions/Spec`
* #### concurrency\_level `#/definitions/ConcurrencyLevel`
* #### api\_budget `#/definitions/HTTPAPIBudget`
* #### stream\_groups `object`
Groups of streams that share a common resource and should not be read simultaneously. Each group defines a set of stream references and an action that controls how concurrent reads are managed. Only applies to ConcurrentDeclarativeSource.
* #### max\_concurrent\_async\_job\_count `integerstring`
Maximum number of concurrent asynchronous jobs to run. This property is only relevant for sources/streams that support asynchronous job execution through the AsyncRetriever (e.g. a report-based stream that initiates a job, polls the job status, and then fetches the job results). This is often set by the API's maximum number of concurrent jobs on the account level. Refer to the API's documentation for this information.
Examples:
```
3
```
```
{{ config['max_concurrent_async_job_count'] }}
```
* #### metadata `object`
For internal Airbyte use only - DO NOT modify manually. Used by consumers of declarative manifests for storing related metadata.
* #### description `string`
A description of the connector. It will be presented on the Source documentation page.
### AddedFieldDefinition `object`[](#/definitions/AddedFieldDefinition "Direct link to /definitions/AddedFieldDefinition")
Defines the field to add on a record.
Properties:
* #### path `array`
List of strings defining the path where to add the value on the record.
Examples:
```
[
"segment_id"
]
```
```
[
"metadata",
"segment_id"
]
```
* #### value `string`
Value of the new field. Use {{ record\['existing\_field'] }} syntax to refer to other fields in the record.
Available variables:
* [config](#/variables/config)
* [record](#/variables/record)
* [stream\_interval](#/variables/stream_interval)
* [stream\_partition](#/variables/stream_partition)
* [stream\_slice](#/variables/stream_slice)
Examples:
```
{{ record['updates'] }}
```
```
{{ record['MetaData']['LastUpdatedTime'] }}
```
```
{{ stream_partition['segment_id'] }}
```
* #### value\_type `#/definitions/ValueType`
Type of the value. If not specified, the type will be inferred from the value.
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### AddFields `object`[](#/definitions/AddFields "Direct link to /definitions/AddFields")
Transformation which adds field to an output record. The path of the added field can be nested.
Properties:
* #### fields `array` `#/definitions/AddedFieldDefinition`
List of transformations (path and corresponding value) that will be added to the record.
* #### condition `string`
Fields will be added if expression is evaluated to True.
Available variables:
* [config](#/variables/config)
* [property](#/variables/property)
* [parameters](#/variables/parameters)
Examples:
```
{{ property|string == '' }}
```
```
{{ property is integer }}
```
```
{{ property|length > 5 }}
```
```
{{ property == 'some_string_to_match' }}
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### ApiKeyAuthenticator `object`[](#/definitions/ApiKeyAuthenticator "Direct link to /definitions/ApiKeyAuthenticator")
Authenticator for requests authenticated with an API token injected as an HTTP request header.
Properties:
* #### api\_token `string`
The API key to inject in the request. Fill it in the user inputs.
Available variables:
* [config](#/variables/config)
Examples:
```
{{ config['api_key'] }}
```
```
Token token={{ config['api_key'] }}
```
* #### header `string`
The name of the HTTP header that will be set to the API key. This setting is deprecated, use inject\_into instead. Header and inject\_into can not be defined at the same time.
Available variables:
* [config](#/variables/config)
Examples:
```
Authorization
```
```
Api-Token
```
```
X-Auth-Token
```
* #### inject\_into `#/definitions/RequestOption`
Configure how the API Key will be sent in requests to the source API. Either inject\_into or header has to be defined.
Examples:
```
{
"inject_into": "header",
"field_name": "Authorization"
}
```
```
{
"inject_into": "request_parameter",
"field_name": "authKey"
}
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### AuthFlow `object`[](#/definitions/AuthFlow "Direct link to /definitions/AuthFlow")
Additional and optional specification object to describe what an 'advanced' Auth flow would need to function.
* A connector should be able to fully function with the configuration as described by the ConnectorSpecification in a 'basic' mode.
* The 'advanced' mode provides easier UX for the user with UI improvements and automations. However, this requires further setup on the server side by instance or workspace admins beforehand. The trade-off is that the user does not have to provide as many technical inputs anymore and the auth process is faster and easier to complete.
Properties:
* #### auth\_flow\_type `string`
The type of auth to use
* #### predicate\_key `array`
JSON path to a field in the connectorSpecification that should exist for the advanced auth to be applicable.
Example:
```
[
"credentials",
"auth_type"
]
```
* #### predicate\_value `string`
Value of the predicate\_key fields for the advanced auth to be applicable.
Example:
```
Oauth
```
* #### oauth\_config\_specification `#/definitions/OAuthConfigSpecification`
### BasicHttpAuthenticator `object`[](#/definitions/BasicHttpAuthenticator "Direct link to /definitions/BasicHttpAuthenticator")
Authenticator for requests authenticated with the Basic HTTP authentication scheme, which encodes a username and an optional password in the Authorization request header.
Properties:
* #### username `string`
The username that will be combined with the password, base64 encoded and used to make requests. Fill it in the user inputs.
Available variables:
* [config](#/variables/config)
Examples:
```
{{ config['username'] }}
```
```
{{ config['api_key'] }}
```
* #### password `string`
The password that will be combined with the username, base64 encoded and used to make requests. Fill it in the user inputs.
Available variables:
* [config](#/variables/config)
Examples:
```
{{ config['password'] }}
```
```
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### BearerAuthenticator `object`[](#/definitions/BearerAuthenticator "Direct link to /definitions/BearerAuthenticator")
Authenticator for requests authenticated with a bearer token injected as a request header of the form `Authorization: Bearer `.
Properties:
* #### api\_token `string`
Token to inject as request header for authenticating with the API.
Available variables:
* [config](#/variables/config)
Examples:
```
{{ config['api_key'] }}
```
```
{{ config['token'] }}
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### SelectiveAuthenticator `object`[](#/definitions/SelectiveAuthenticator "Direct link to /definitions/SelectiveAuthenticator")
Authenticator that selects concrete authenticator based on config property.
Properties:
* #### authenticator\_selection\_path `array`
Path of the field in config with selected authenticator name
Examples:
```
[
"auth"
]
```
```
[
"auth",
"type"
]
```
* #### authenticators `object`
Authenticators to select from.
Example:
```
{
"authenticators": {
"token": "#/definitions/ApiKeyAuthenticator",
"oauth": "#/definitions/OAuthAuthenticator",
"jwt": "#/definitions/JwtAuthenticator"
}
}
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### RateLimitedMultipleTokenAuthenticator `object`[](#/definitions/RateLimitedMultipleTokenAuthenticator "Direct link to /definitions/RateLimitedMultipleTokenAuthenticator")
Authenticator that rotates between multiple interchangeable tokens, tracking each token's remaining call quota. Outgoing requests are classified into quota pools using request matchers. When the active token's quota for the matched pool is exhausted, the authenticator rotates to the next token. When all tokens are exhausted, it waits until the earliest quota reset (bounded by `max_wait_time`) before resuming. Quota counters are seeded per token from `quota_status_source` at startup and refreshed after an exhaustion wait.
Properties:
* #### tokens
The tokens to rotate between. Either an explicit list of tokens, or a single string containing multiple tokens separated by `token_delimiter`.
Type:
* `string`
* `array`
Available variables:
* [config](#/variables/config)
Examples:
```
{{ config['credentials']['personal_access_token'] }}
```
```
[
"{{ config['token_1'] }}",
"{{ config['token_2'] }}"
]
```
* #### token\_delimiter `string`
Delimiter used to split a single token string into multiple tokens.
* #### auth\_method `string`
The prefix to prepend to the token in the auth header value (e.g. `Authorization: Bearer `).
Examples:
```
Bearer
```
```
token
```
* #### header `string`
The name of the HTTP header in which to inject the token.
* #### quota\_status\_source `#/definitions/QuotaStatusSource`
Defines where to fetch each token's current quota status. Called once per token at startup and after an exhaustion wait, not per data request.
* #### quotas `array` `#/definitions/TokenQuota`
Quota pools tracked per token. Each outgoing request is classified into the first pool whose matchers match the request; a pool with no matchers acts as the default. The `remaining_path` and `reset_path` locate each pool's values in the quota status response.
* #### max\_wait\_time `string`
ISO 8601 duration. When all tokens are exhausted, the maximum time to wait for a quota reset before raising a transient error.
Available variables:
* [config](#/variables/config)
Examples:
```
PT2H
```
```
PT30M
```
```
PT{{ config.get('max_waiting_time', 120) }}M
```
* #### budget\_reserve\_fraction `number`
Fraction of each token's quota to keep in reserve. When every token drops below its reserve, requests are proactively throttled to spread the remaining calls until the quota reset. Set to 0 (along with `budget_min_reserve`) to disable throttling.
* #### budget\_min\_reserve `integer`
Minimum number of calls to keep in reserve per token before proactive throttling kicks in.
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### QuotaStatusSource `object`[](#/definitions/QuotaStatusSource "Direct link to /definitions/QuotaStatusSource")
Describes the endpoint from which a token's current rate limit quota status can be fetched.
Properties:
* #### url `string`
The full URL of the quota status endpoint.
Available variables:
* [config](#/variables/config)
Examples:
```
https://api.github.com/rate_limit
```
```
{{ config.get('api_url', 'https://api.github.com') }}/rate_limit
```
* #### http\_method `string`
The HTTP method used to fetch the quota status.
* #### request\_headers `object`
Additional headers to send with the quota status request.
* #### unavailable\_status\_codes `array`
Status codes from the quota status endpoint that mean quota tracking is unavailable rather than broken, such as a self-hosted deployment with rate limiting turned off. Every pool of the token whose request returned that status is then treated as untracked, so the authenticator stops waiting for quota resets, stops throttling proactively and stops rotating on exhaustion for it, while still signing requests. A token untracked this way stays untracked for the rest of the sync, because the endpoint is never consulted for it again, so a status the endpoint can also return transiently costs quota tracking for the whole run. If only some tokens return that status the others stay tracked, but they are no longer refreshed either, because the authenticator stops waiting for quota resets as soon as one token is untracked; once their counters are locally spent all traffic moves onto the untracked tokens. Rate limiting reported by ordinary responses is still handled by the stream's error handler, so one that retries 429 or 403 keeps working, and a retry rotates onto the next token; it pays the backoff the response asks for rather than the shortened one a tracked pool would get, since an untracked pool has no counters with which to argue the rejection was about that credential. Any status not listed still fails the connection, and this field never excuses a quota path missing from a response the endpoint did answer, so list only the codes the endpoint uses to report that rate limiting is not enabled. Do not list authentication or authorization statuses, since a 401 or 403 from a revoked credential would then be read as quota tracking being unavailable rather than as a credentials failure.
Example:
```
[
404
]
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### TokenQuota `object`[](#/definitions/TokenQuota "Direct link to /definitions/TokenQuota")
A named per-token quota pool with matchers that classify outgoing requests into the pool.
Properties:
* #### name `string`
Name of the quota pool.
Examples:
```
rest
```
```
graphql
```
* #### remaining\_path `array`
Path to the remaining call count for this pool in the quota status response.
Example:
```
[
"resources",
"core",
"remaining"
]
```
* #### reset\_path `array`
Path to the quota reset timestamp for this pool in the quota status response.
Example:
```
[
"resources",
"core",
"reset"
]
```
* #### limit\_path `array`
Optional path to the total call limit for this pool in the quota status response. Used to compute the proactive throttling reserve; falls back to the initially observed remaining count when not set. Setting it on every pool is recommended so the reserve does not shrink when a sync starts with the pool already partially consumed.
Example:
```
[
"resources",
"core",
"limit"
]
```
* #### matchers `array` `#/definitions/HttpRequestRegexMatcher`
List of matchers that classify outgoing requests into this quota pool. The first pool whose matcher matches a request is used. A pool with no matchers acts as the default pool.
* #### remaining\_header `string`
Optional response header carrying the remaining call count for this pool. When set, the pool's counter is reconciled against this header on every response, which corrects drift caused by sharing the token with other clients, by requests in flight concurrently, or by a sync running long enough for the initial quota status read to go stale. Without it the pool is only ever seeded from the quota status endpoint.
Example:
```
X-RateLimit-Remaining
```
* #### reset\_header `string`
Optional response header carrying the quota reset timestamp for this pool. Parsed with the same rules as `reset_path`, so epoch seconds and ISO 8601 both work. Used to tell a rolled-over quota window from the current one; a response proving the window has rolled over restores the pool to its limit. Most useful alongside `remaining_header`.
Example:
```
X-RateLimit-Reset
```
* #### limit\_header `string`
Optional response header carrying the total call limit for this pool, used to keep the proactive throttling reserve accurate as the limit changes.
Example:
```
X-RateLimit-Limit
```
* #### exhaustion\_status\_codes `array`
Response status codes that mean this token's pool is spent. These have two effects. A response carrying one of them but no remaining count sets the pool to zero, so the next request rotates to another token instead of waiting out the reset window. They also mark which responses may report a zero for a quota window that has already elapsed, so a rate limit whose reset header trails the value being held still stops the token being used; a zero on any other response is treated as the last call of a finished window and ignored. Leaving this empty means such trailing rejections are ignored unless their reset is within the skew tolerance of the current window. Only list codes the API uses exclusively for rate limiting -- a code that also signals other failures would park a healthy token.
Example:
```
[
429
]
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### CheckStream `object`[](#/definitions/CheckStream "Direct link to /definitions/CheckStream")
Defines the streams to try reading when running a check operation.
Properties:
* #### stream\_names `array`
Names of the streams to try reading from when running a check operation.
Examples:
```
[
"users"
]
```
```
[
"users",
"contacts"
]
```
* #### dynamic\_streams\_check\_configs `array` `#/definitions/DynamicStreamCheckConfig`
* #### config\_overrides `object`
Values overlaid onto the connector config for the duration of the check operation only. Use this when a check must behave differently from a sync - for example a shorter rate limit wait budget, so that check fails fast with a clear message instead of sleeping until the quota resets. Keys should be fields declared in the connector's spec. Values are used as-is. They are not interpolated, a `$ref` inside them is not resolved, they replace a nested object rather than deep-merging into it, and they are applied after config migrations and transformations have run, so a field derived from an overridden field is not recomputed. Keys must be strings, and two combinations are rejected outright. Keys prefixed with `__airbyte` belong to the platform rather than to the connector's spec. And a manifest that declares a `refresh_token_updater` cannot use this field at all, because a token refresh during check emits the whole config it was handed as a CONNECTOR\_CONFIG control message, which the platform persists - so a check-only override would become the connection's saved config and apply to every later sync.
Examples:
```
{
"max_waiting_time": 0
}
```
```
{
"page_size": 1
}
```
### DynamicStreamCheckConfig `object`[](#/definitions/DynamicStreamCheckConfig "Direct link to /definitions/DynamicStreamCheckConfig")
Properties:
* #### dynamic\_stream\_name `string`
The dynamic stream name.
* #### stream\_count `integer`
The number of streams to attempt reading from during a check operation. If unset, all generated streams are checked. Must be a positive integer; if it exceeds the total number of available streams, all streams are checked.
### CheckDynamicStream `object`[](#/definitions/CheckDynamicStream "Direct link to /definitions/CheckDynamicStream")
(This component is experimental. Use at your own risk.) Defines the dynamic streams to try reading when running a check operation.
Properties:
* #### stream\_count `integer`
Numbers of the streams to try reading from when running a check operation.
* #### use\_check\_availability `boolean`
Enables stream check availability. This field is automatically set by the CDK.
* #### config\_overrides `object`
Values overlaid onto the connector config for the duration of the check operation only. Use this when a check must behave differently from a sync - for example a shorter rate limit wait budget, so that check fails fast with a clear message instead of sleeping until the quota resets. Keys should be fields declared in the connector's spec. Values are used as-is. They are not interpolated, a `$ref` inside them is not resolved, they replace a nested object rather than deep-merging into it, and they are applied after config migrations and transformations have run, so a field derived from an overridden field is not recomputed. Keys must be strings, and two combinations are rejected outright. Keys prefixed with `__airbyte` belong to the platform rather than to the connector's spec. And a manifest that declares a `refresh_token_updater` cannot use this field at all, because a token refresh during check emits the whole config it was handed as a CONNECTOR\_CONFIG control message, which the platform persists - so a check-only override would become the connection's saved config and apply to every later sync.
Examples:
```
{
"max_waiting_time": 0
}
```
```
{
"page_size": 1
}
```
### CompositeErrorHandler `object`[](#/definitions/CompositeErrorHandler "Direct link to /definitions/CompositeErrorHandler")
Error handler that sequentially iterates over a list of error handlers.
Properties:
* #### error\_handlers `array`
List of error handlers to iterate on to determine how to handle a failed response.
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### ConcurrencyLevel `object`[](#/definitions/ConcurrencyLevel "Direct link to /definitions/ConcurrencyLevel")
Defines the amount of parallelization for the streams that are being synced. The factor of parallelization is how many partitions or streams are synced at the same time. For example, with a concurrency\_level of 10, ten streams or partitions of data will processed at the same time. Note that a value of 1 could create deadlock if a stream has a very high number of partitions.
Properties:
* #### default\_concurrency
The amount of concurrency that will applied during a sync. This value can be hardcoded or user-defined in the config if different users have varying volume thresholds in the target API.
Type:
* `integer`
* `string`
Available variables:
* [config](#/variables/config)
Examples:
```
10
```
```
{{ config['num_workers'] or 10 }}
```
* #### max\_concurrency `integer`
The maximum level of concurrency that will be used during a sync. This becomes a required field when the default\_concurrency derives from the config, because it serves as a safeguard against a user-defined threshold that is too high.
Examples:
```
20
```
```
100
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### ConditionalStreams `object`[](#/definitions/ConditionalStreams "Direct link to /definitions/ConditionalStreams")
Streams that are only available while performing a connector operation when the condition is met.
Properties:
* #### condition `string`
Condition that will be evaluated to determine if a set of streams should be available.
Available variables:
* [config](#/variables/config)
* [parameters](#/variables/parameters)
Example:
```
{{ config['is_sandbox'] }}
```
* #### streams `array` `#/definitions/DeclarativeStream`
Streams that will be used during an operation based on the condition.
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### ConstantBackoffStrategy `object`[](#/definitions/ConstantBackoffStrategy "Direct link to /definitions/ConstantBackoffStrategy")
Backoff strategy with a constant backoff interval.
Properties:
* #### backoff\_time\_in\_seconds
Backoff time in seconds.
Type:
* `number`
* `string`
Available variables:
* [config](#/variables/config)
Examples:
```
30
```
```
30.5
```
```
{{ config['backoff_time'] }}
```
* #### jitter\_range\_in\_seconds `number`
Optional additive jitter range in seconds. When set, the backoff time is uniformly distributed between backoff\_time\_in\_seconds and backoff\_time\_in\_seconds + (jitter\_range\_in\_seconds \* 2), so jitter only increases the base backoff.
Example:
```
15
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### CursorPagination `object`[](#/definitions/CursorPagination "Direct link to /definitions/CursorPagination")
Pagination strategy that evaluates an interpolated string to define the next page to fetch.
Properties:
* #### cursor\_value `string`
Value of the cursor defining the next page to fetch.
Available variables:
* [config](#/variables/config)
* [headers](#/variables/headers)
* [last\_page\_size](#/variables/last_page_size)
* [last\_record](#/variables/last_record)
* [response](#/variables/response)
Examples:
```
{{ headers.link.next.cursor }}
```
```
{{ last_record['key'] }}
```
```
{{ response['nextPage'] }}
```
* #### page\_size
The number of records to include in each pages.
Type:
* `integer`
* `string`
Available variables:
* [config](#/variables/config)
Examples:
```
100
```
```
{{ config['page_size'] }}
```
* #### stop\_condition `string`
Template string evaluating when to stop paginating.
Available variables:
* [config](#/variables/config)
* [headers](#/variables/headers)
* [last\_record](#/variables/last_record)
* [response](#/variables/response)
Examples:
```
{{ response.data.has_more is false }}
```
```
{{ 'next' not in headers['link'] }}
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### CustomAuthenticator `object`[](#/definitions/CustomAuthenticator "Direct link to /definitions/CustomAuthenticator")
Authenticator component whose behavior is derived from a custom code implementation of the connector.
Properties:
* #### class\_name `string`
Fully-qualified name of the class that will be implementing the custom authentication strategy. Has to be a sub class of DeclarativeAuthenticator. The format is `source_..`.
Example:
```
source_railz.components.ShortLivedTokenAuthenticator
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### CustomBackoffStrategy `object`[](#/definitions/CustomBackoffStrategy "Direct link to /definitions/CustomBackoffStrategy")
Backoff strategy component whose behavior is derived from a custom code implementation of the connector.
Properties:
* #### class\_name `string`
Fully-qualified name of the class that will be implementing the custom backoff strategy. The format is `source_..`.
Example:
```
source_railz.components.MyCustomBackoffStrategy
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### CustomErrorHandler `object`[](#/definitions/CustomErrorHandler "Direct link to /definitions/CustomErrorHandler")
Error handler component whose behavior is derived from a custom code implementation of the connector.
Properties:
* #### class\_name `string`
Fully-qualified name of the class that will be implementing the custom error handler. The format is `source_..`.
Example:
```
source_railz.components.MyCustomErrorHandler
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### CustomPaginationStrategy `object`[](#/definitions/CustomPaginationStrategy "Direct link to /definitions/CustomPaginationStrategy")
Pagination strategy component whose behavior is derived from a custom code implementation of the connector.
Properties:
* #### class\_name `string`
Fully-qualified name of the class that will be implementing the custom pagination strategy. The format is `source_..`.
Example:
```
source_railz.components.MyCustomPaginationStrategy
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### CustomRecordExtractor `object`[](#/definitions/CustomRecordExtractor "Direct link to /definitions/CustomRecordExtractor")
Record extractor component whose behavior is derived from a custom code implementation of the connector.
Properties:
* #### class\_name `string`
Fully-qualified name of the class that will be implementing the custom record extraction strategy. The format is `source_..`.
Example:
```
source_railz.components.MyCustomRecordExtractor
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### CustomRecordFilter `object`[](#/definitions/CustomRecordFilter "Direct link to /definitions/CustomRecordFilter")
Record filter component whose behavior is derived from a custom code implementation of the connector.
Properties:
* #### class\_name `string`
Fully-qualified name of the class that will be implementing the custom record filter strategy. The format is `source_..`.
Example:
```
source_railz.components.MyCustomCustomRecordFilter
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### CustomRequester `object`[](#/definitions/CustomRequester "Direct link to /definitions/CustomRequester")
Requester component whose behavior is derived from a custom code implementation of the connector.
Properties:
* #### class\_name `string`
Fully-qualified name of the class that will be implementing the custom requester strategy. The format is `source_..`.
Example:
```
source_railz.components.MyCustomRecordExtractor
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### CustomRetriever `object`[](#/definitions/CustomRetriever "Direct link to /definitions/CustomRetriever")
Retriever component whose behavior is derived from a custom code implementation of the connector.
Properties:
* #### class\_name `string`
Fully-qualified name of the class that will be implementing the custom retriever strategy. The format is `source_..`.
Example:
```
source_railz.components.MyCustomRetriever
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### CustomPartitionRouter `object`[](#/definitions/CustomPartitionRouter "Direct link to /definitions/CustomPartitionRouter")
Partition router component whose behavior is derived from a custom code implementation of the connector.
Properties:
* #### class\_name `string`
Fully-qualified name of the class that will be implementing the custom partition router. The format is `source_..`.
Example:
```
source_railz.components.MyCustomPartitionRouter
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### CustomSchemaLoader `object`[](#/definitions/CustomSchemaLoader "Direct link to /definitions/CustomSchemaLoader")
Schema Loader component whose behavior is derived from a custom code implementation of the connector.
Properties:
* #### class\_name `string`
Fully-qualified name of the class that will be implementing the custom schema loader. The format is `source_..`.
Example:
```
source_railz.components.MyCustomSchemaLoader
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### CustomSchemaNormalization `object`[](#/definitions/CustomSchemaNormalization "Direct link to /definitions/CustomSchemaNormalization")
Schema normalization component whose behavior is derived from a custom code implementation of the connector.
Properties:
* #### class\_name `string`
Fully-qualified name of the class that will be implementing the custom normalization. The format is `source_..`.
Example:
```
source_amazon_seller_partner.components.LedgerDetailedViewReportsTypeTransformer
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### CustomStateMigration `object`[](#/definitions/CustomStateMigration "Direct link to /definitions/CustomStateMigration")
Apply a custom transformation on the input state.
Properties:
* #### class\_name `string`
Fully-qualified name of the class that will be implementing the custom state migration. The format is `source_..`.
Example:
```
source_railz.components.MyCustomStateMigration
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### CustomTransformation `object`[](#/definitions/CustomTransformation "Direct link to /definitions/CustomTransformation")
Transformation component whose behavior is derived from a custom code implementation of the connector.
Properties:
* #### class\_name `string`
Fully-qualified name of the class that will be implementing the custom transformation. The format is `source_..`.
Example:
```
source_railz.components.MyCustomTransformation
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### LegacyToPerPartitionStateMigration `object`[](#/definitions/LegacyToPerPartitionStateMigration "Direct link to /definitions/LegacyToPerPartitionStateMigration")
Transforms the input state for per-partitioned streams from the legacy format to the low-code format. The cursor field and partition ID fields are automatically extracted from the stream's DatetimebasedCursor and SubstreamPartitionRouter. Example input state: { "13506132": { "last\_changed": "2022-12-27T08:34:39+00:00" } Example output state: { "partition": {"id": "13506132"}, "cursor": {"last\_changed": "2022-12-27T08:34:39+00:00"} }
Properties:
### IncrementingCountCursor `object`[](#/definitions/IncrementingCountCursor "Direct link to /definitions/IncrementingCountCursor")
Cursor that allows for incremental sync according to a continuously increasing integer.
Properties:
* #### cursor\_field `string`
The location of the value on a record that will be used as a bookmark during sync. To ensure no data loss, the API must return records in ascending order based on the cursor field. Nested fields are not supported, so the field must be at the top level of the record. You can use a combination of Add Field and Remove Field transformations to move the nested field to the top.
Available variables:
* [config](#/variables/config)
Examples:
```
created_at
```
```
{{ config['record_cursor'] }}
```
* #### allow\_catalog\_defined\_cursor\_field `boolean`
Whether the cursor allows users to override the default cursor\_field when configuring their connection. The user defined cursor field will be specified from within the configured catalog.
* #### start\_value
The value that determines the earliest record that should be synced.
Type:
* `string`
* `integer`
Available variables:
* [config](#/variables/config)
Examples:
```
0
```
```
{{ config['start_value'] }}
```
* #### start\_value\_option `#/definitions/RequestOption`
Optionally configures how the start value will be sent in requests to the source API.
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### DatetimeBasedCursor `object`[](#/definitions/DatetimeBasedCursor "Direct link to /definitions/DatetimeBasedCursor")
Cursor to provide incremental capabilities over datetime.
Properties:
* #### clamping `object`
This option is used to adjust the upper and lower boundaries of each datetime window to beginning and end of the provided target period (day, week, month)
* #### cursor\_field `string`
The location of the value on a record that will be used as a bookmark during sync. To ensure no data loss, the API must return records in ascending order based on the cursor field. Nested fields are not supported, so the field must be at the top level of the record. You can use a combination of Add Field and Remove Field transformations to move the nested field to the top.
Available variables:
* [config](#/variables/config)
Examples:
```
created_at
```
```
{{ config['record_cursor'] }}
```
* #### allow\_catalog\_defined\_cursor\_field `boolean`
Whether the cursor allows users to override the default cursor\_field when configuring their connection. The user defined cursor field will be specified from within the configured catalog.
* #### cursor\_datetime\_formats `array`
The possible formats for the cursor field, in order of preference. The first format that matches the cursor field value will be used to parse it. If not provided, the Outgoing Datetime Format will be used. Use placeholders starting with "%" to describe the format the API is using. The following placeholders are available:
* **%s**: Epoch unix timestamp - `1686218963`
* **%s\_as\_float**: Epoch unix timestamp in seconds as float with microsecond precision - `1686218963.123456`
* **%ms**: Epoch unix timestamp - `1686218963123`
* **%a**: Weekday (abbreviated) - `Sun`
* **%A**: Weekday (full) - `Sunday`
* **%w**: Weekday (decimal) - `0` (Sunday), `6` (Saturday)
* **%d**: Day of the month (zero-padded) - `01`, `02`, ..., `31`
* **%b**: Month (abbreviated) - `Jan`
* **%B**: Month (full) - `January`
* **%m**: Month (zero-padded) - `01`, `02`, ..., `12`
* **%y**: Year (without century, zero-padded) - `00`, `01`, ..., `99`
* **%Y**: Year (with century) - `0001`, `0002`, ..., `9999`
* **%H**: Hour (24-hour, zero-padded) - `00`, `01`, ..., `23`
* **%I**: Hour (12-hour, zero-padded) - `01`, `02`, ..., `12`
* **%p**: AM/PM indicator
* **%M**: Minute (zero-padded) - `00`, `01`, ..., `59`
* **%S**: Second (zero-padded) - `00`, `01`, ..., `59`
* **%f**: Microsecond (zero-padded to 6 digits) - `000000`, `000001`, ..., `999999`
* **%\_ms**: Millisecond (zero-padded to 3 digits) - `000`, `001`, ..., `999`
* **%z**: UTC offset - `(empty)`, `+0000`, `-04:00`
* **%Z**: Time zone name - `(empty)`, `UTC`, `GMT`
* **%j**: Day of the year (zero-padded) - `001`, `002`, ..., `366`
* **%U**: Week number of the year (Sunday as first day) - `00`, `01`, ..., `53`
* **%W**: Week number of the year (Monday as first day) - `00`, `01`, ..., `53`
* **%c**: Date and time representation - `Tue Aug 16 21:30:00 1988`
* **%x**: Date representation - `08/16/1988`
* **%X**: Time representation - `21:30:00`
* **%%**: Literal '%' character
Some placeholders depend on the locale of the underlying system - in most cases this locale is configured as en/US. For more information see the [Python documentation](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes).
Examples:
```
%Y-%m-%d
```
```
%Y-%m-%d %H:%M:%S
```
```
%Y-%m-%dT%H:%M:%S
```
```
%Y-%m-%dT%H:%M:%SZ
```
```
%Y-%m-%dT%H:%M:%S%z
```
```
%Y-%m-%dT%H:%M:%S.%fZ
```
```
%Y-%m-%dT%H:%M:%S.%f%z
```
```
%Y-%m-%d %H:%M:%S.%f+00:00
```
```
%s
```
```
%ms
```
* #### start\_datetime
The datetime that determines the earliest record that should be synced.
Type:
* [`#/definitions/MinMaxDatetime`](#/definitions/MinMaxDatetime)
* `string`
Available variables:
* [config](#/variables/config)
Examples:
```
2020-01-1T00:00:00Z
```
```
{{ config['start_time'] }}
```
* #### start\_time\_option `#/definitions/RequestOption`
Optionally configures how the start datetime will be sent in requests to the source API.
* #### end\_datetime
The datetime that determines the last record that should be synced. If not provided, `{{ now_utc() }}` will be used.
Type:
* [`#/definitions/MinMaxDatetime`](#/definitions/MinMaxDatetime)
* `string`
Available variables:
* [config](#/variables/config)
Examples:
```
2021-01-1T00:00:00Z
```
```
{{ now_utc() }}
```
```
{{ day_delta(-1) }}
```
* #### end\_time\_option `#/definitions/RequestOption`
Optionally configures how the end datetime will be sent in requests to the source API.
* #### datetime\_format `string`
The datetime format used to format the datetime values that are sent in outgoing requests to the API. Use placeholders starting with "%" to describe the format the API is using. The following placeholders are available:
* **%s**: Epoch unix timestamp - `1686218963`
* **%s\_as\_float**: Epoch unix timestamp in seconds as float with microsecond precision - `1686218963.123456`
* **%ms**: Epoch unix timestamp (milliseconds) - `1686218963123`
* **%a**: Weekday (abbreviated) - `Sun`
* **%A**: Weekday (full) - `Sunday`
* **%w**: Weekday (decimal) - `0` (Sunday), `6` (Saturday)
* **%d**: Day of the month (zero-padded) - `01`, `02`, ..., `31`
* **%b**: Month (abbreviated) - `Jan`
* **%B**: Month (full) - `January`
* **%m**: Month (zero-padded) - `01`, `02`, ..., `12`
* **%y**: Year (without century, zero-padded) - `00`, `01`, ..., `99`
* **%Y**: Year (with century) - `0001`, `0002`, ..., `9999`
* **%H**: Hour (24-hour, zero-padded) - `00`, `01`, ..., `23`
* **%I**: Hour (12-hour, zero-padded) - `01`, `02`, ..., `12`
* **%p**: AM/PM indicator
* **%M**: Minute (zero-padded) - `00`, `01`, ..., `59`
* **%S**: Second (zero-padded) - `00`, `01`, ..., `59`
* **%f**: Microsecond (zero-padded to 6 digits) - `000000`
* **%\_ms**: Millisecond (zero-padded to 3 digits) - `000`
* **%z**: UTC offset - `(empty)`, `+0000`, `-04:00`
* **%Z**: Time zone name - `(empty)`, `UTC`, `GMT`
* **%j**: Day of the year (zero-padded) - `001`, `002`, ..., `366`
* **%U**: Week number of the year (starting Sunday) - `00`, ..., `53`
* **%W**: Week number of the year (starting Monday) - `00`, ..., `53`
* **%c**: Date and time - `Tue Aug 16 21:30:00 1988`
* **%x**: Date standard format - `08/16/1988`
* **%X**: Time standard format - `21:30:00`
* **%%**: Literal '%' character
Some placeholders depend on the locale of the underlying system - in most cases this locale is configured as en/US. For more information see the [Python documentation](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes).
Examples:
```
%Y-%m-%dT%H:%M:%S.%f%z
```
```
%Y-%m-%d
```
```
%s
```
```
%ms
```
```
%s_as_float
```
* #### cursor\_granularity `string`
Smallest increment the datetime\_format has (ISO 8601 duration) that is used to ensure the start of a slice does not overlap with the end of the previous one, e.g. for %Y-%m-%d the granularity should be P1D, for %Y-%m-%dT%H:%M:%SZ the granularity should be PT1S. Given this field is provided, `step` needs to be provided as well.
* **PT0.000001S**: 1 microsecond
* **PT0.001S**: 1 millisecond
* **PT1S**: 1 second
* **PT1M**: 1 minute
* **PT1H**: 1 hour
* **P1D**: 1 day
Example:
```
PT1S
```
* #### is\_data\_feed `boolean`
A data feed API is an API that does not allow filtering and paginates the content from the most recent to the least recent. Given this, the CDK needs to know when to stop paginating and this field will generate a stop condition for pagination. The last page fetched still holds records that fall outside the cursor window, and those are filtered out as well, so Client-side Incremental Filtering does not need to be enabled alongside this field. Records are kept when their cursor value is within the window that starts at the previous sync's cursor value (or the start date) and ends at the end date, defaulting to the current time, so records dated in the future are filtered out too.
* #### is\_client\_side\_incremental `boolean`
Set to True if the target API endpoint does not take cursor values to filter records and returns all records anyway. This will cause the connector to filter out records locally, keeping only the ones whose cursor value falls within the window that starts at the previous sync's cursor value (or the start date) and ends at the end date, defaulting to the current time. This means that all records would be read from the API, but only the records within that window will be emitted to the destination. This is not needed when Data Feed API is enabled, as a data feed already filters on the same window.
* #### is\_compare\_strictly `boolean`
Set to True if the target API does not accept queries where the start time equal the end time. This will cause those requests to be skipped.
* #### global\_substream\_cursor `boolean`
Setting to True causes the connector to store the cursor as one value, instead of per-partition. This setting optimizes performance when the parent stream has thousands of partitions. Notably, the substream state is updated only at the end of the sync, which helps prevent data loss in case of a sync failure. See more info in the [docs](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/incremental-syncs).
* #### lookback\_window `string`
Time interval (ISO8601 duration) before the start\_datetime to read data for, e.g. P1M for looking back one month.
* **PT1H**: 1 hour
* **P1D**: 1 day
* **P1W**: 1 week
* **P1M**: 1 month
* **P1Y**: 1 year
Available variables:
* [config](#/variables/config)
Examples:
```
P1D
```
```
P{{ config['lookback_days'] }}D
```
* #### partition\_field\_end `string`
Name of the partition start time field.
Example:
```
ending_time
```
* #### partition\_field\_start `string`
Name of the partition end time field.
Example:
```
starting_time
```
* #### step `string`
The size of the time window (ISO8601 duration). Given this field is provided, `cursor_granularity` needs to be provided as well.
* **PT1H**: 1 hour
* **P1D**: 1 day
* **P1W**: 1 week
* **P1M**: 1 month
* **P1Y**: 1 year
Examples:
```
P1W
```
```
{{ config['step_increment'] }}
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### JwtAuthenticator `object`[](#/definitions/JwtAuthenticator "Direct link to /definitions/JwtAuthenticator")
Authenticator for requests using JWT authentication flow.
Properties:
* #### secret\_key `string`
Secret used to sign the JSON web token.
Available variables:
* [config](#/variables/config)
Example:
```
{{ config['secret_key'] }}
```
* #### base64\_encode\_secret\_key `boolean`
When set to true, the secret key will be base64 encoded prior to being encoded as part of the JWT. Only set to "true" when required by the API.
* #### algorithm `string`
Algorithm used to sign the JSON web token.
Examples:
```
ES256
```
```
HS256
```
```
RS256
```
```
{{ config['algorithm'] }}
```
* #### token\_duration `integer`
The amount of time in seconds a JWT token can be valid after being issued.
Examples:
```
1200
```
```
3600
```
* #### header\_prefix `string`
The prefix to be used within the Authentication header.
Examples:
```
Bearer
```
```
Basic
```
* #### jwt\_headers `object`
JWT headers used when signing JSON web token.
* #### additional\_jwt\_headers `object`
Additional headers to be included with the JWT headers object.
* #### jwt\_payload `object`
JWT Payload used when signing JSON web token.
* #### additional\_jwt\_payload `object`
Additional properties to be added to the JWT payload.
* #### passphrase `string`
A passphrase/password used to encrypt the private key. Only provide a passphrase if required by the API for JWT authentication. The API will typically provide the passphrase when generating the public/private key pair.
Example:
```
{{ config['passphrase'] }}
```
* #### request\_option `#/definitions/RequestOption`
A request option describing where the signed JWT token that is generated should be injected into the outbound API request.
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### OAuthAuthenticator `object`[](#/definitions/OAuthAuthenticator "Direct link to /definitions/OAuthAuthenticator")
Authenticator for requests using OAuth 2.0 authorization flow.
Properties:
* #### client\_id\_name `string`
The name of the property to use to refresh the `access_token`.
Example:
```
custom_app_id
```
* #### client\_id `string`
The OAuth client ID. Fill it in the user inputs.
Available variables:
* [config](#/variables/config)
Examples:
```
{{ config['client_id'] }}
```
```
{{ config['credentials']['client_id }}
```
* #### client\_secret\_name `string`
The name of the property to use to refresh the `access_token`.
Example:
```
custom_app_secret
```
* #### client\_secret `string`
The OAuth client secret. Fill it in the user inputs.
Available variables:
* [config](#/variables/config)
Examples:
```
{{ config['client_secret'] }}
```
```
{{ config['credentials']['client_secret }}
```
* #### refresh\_token\_name `string`
The name of the property to use to refresh the `access_token`.
Example:
```
custom_app_refresh_value
```
* #### refresh\_token `string`
Credential artifact used to get a new access token.
Available variables:
* [config](#/variables/config)
Examples:
```
{{ config['refresh_token'] }}
```
```
{{ config['credentials]['refresh_token'] }}
```
* #### token\_refresh\_endpoint `string`
The full URL to call to obtain a new access token.
Example:
```
https://connect.squareup.com/oauth2/token
```
* #### access\_token\_name `string`
The name of the property which contains the access token in the response from the token refresh endpoint.
Example:
```
access_token
```
* #### access\_token\_value `string`
The value of the access\_token to bypass the token refreshing using `refresh_token`.
Available variables:
* [config](#/variables/config)
Example:
```
secret_access_token_value
```
* #### expires\_in\_name `string`
The name of the property which contains the expiry date in the response from the token refresh endpoint.
Example:
```
expires_in
```
* #### grant\_type\_name `string`
The name of the property to use to refresh the `access_token`.
Example:
```
custom_grant_type
```
* #### grant\_type `string`
Specifies the OAuth2 grant type. If set to refresh\_token, the refresh\_token needs to be provided as well. For client\_credentials, only client id and secret are required. Other grant types are not officially supported.
Examples:
```
refresh_token
```
```
client_credentials
```
* #### refresh\_request\_body `object`
Body of the request sent to get a new access token.
Example:
```
{
"applicationId": "{{ config['application_id'] }}",
"applicationSecret": "{{ config['application_secret'] }}",
"token": "{{ config['token'] }}"
}
```
* #### refresh\_request\_headers `object`
Headers of the request sent to get a new access token.
Example:
```
{
"Authorization": "",
"Content-Type": "application/x-www-form-urlencoded"
}
```
* #### send\_refresh\_request\_as\_query\_params `boolean`
When set to true, the standard OAuth refresh args (`grant_type`, `refresh_token`, client credentials when not in an `Authorization` header, scopes, plus any `refresh_request_body` extras) are sent on the URL query string and the request body is emitted empty. Use this for OAuth providers like Gong that document their refresh endpoint with refresh args on the URL query string.
Example:
```
true
```
* #### scopes `array`
List of scopes that should be granted to the access token.
Example:
```
[
"crm.list.read",
"crm.objects.contacts.read",
"crm.schema.contacts.read"
]
```
* #### token\_expiry\_date `string`
The access token expiry date.
Examples:
```
2023-04-06T07:12:10.421833+00:00
```
```
1680842386
```
* #### token\_expiry\_date\_format `string`
The format of the time to expiration datetime. Provide it if the time is returned as a date-time string instead of seconds.
Example:
```
%Y-%m-%d %H:%M:%S.%f+00:00
```
* #### refresh\_token\_error\_status\_codes `array`
Status Codes to Identify refresh token error in response (Refresh Token Error Key and Refresh Token Error Values should be also specified). Responses with one of the error status code and containing an error value will be flagged as a config error
Example:
```
[
400,
500
]
```
* #### refresh\_token\_error\_key `string`
Key to Identify refresh token error in response (Refresh Token Error Status Codes and Refresh Token Error Values should be also specified).
Example:
```
error
```
* #### refresh\_token\_error\_values `array`
List of values to check for exception during token refresh process. Used to check if the error found in the response matches the key from the Refresh Token Error Key field (e.g. response={"error": "invalid\_grant"}). Only responses with one of the error status code and containing an error value will be flagged as a config error
Example:
```
[
"invalid_grant",
"invalid_permissions"
]
```
* #### refresh\_token\_updater
When the refresh token updater is defined, new refresh tokens, access tokens and the access token expiry date are written back from the authentication response to the config object. This is important if the refresh token can only used once.
* #### profile\_assertion `#/definitions/JwtAuthenticator`
The authenticator being used to authenticate the client authenticator.
* #### use\_profile\_assertion `boolean`
Enable using profile assertion as a flow for OAuth authorization.
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### DeclarativeStream `object`[](#/definitions/DeclarativeStream "Direct link to /definitions/DeclarativeStream")
A stream whose behavior is described by a set of declarative low code components.
Properties:
* #### name `string`
The stream name.
* #### retriever
Component used to coordinate how records are extracted across stream slices and request pages.
Type:
* [`#/definitions/SimpleRetriever`](#/definitions/SimpleRetriever)
* [`#/definitions/AsyncRetriever`](#/definitions/AsyncRetriever)
* [`#/definitions/CustomRetriever`](#/definitions/CustomRetriever)
* #### incremental\_sync
Component used to fetch data incrementally based on a time field in the data.
Type:
* [`#/definitions/DatetimeBasedCursor`](#/definitions/DatetimeBasedCursor)
* [`#/definitions/IncrementingCountCursor`](#/definitions/IncrementingCountCursor)
* #### primary\_key `#/definitions/PrimaryKey`
* #### schema\_loader
One or many schema loaders can be used to retrieve the schema for the current stream. When multiple schema loaders are defined, schema properties will be merged together. Schema loaders defined first taking precedence in the event of a conflict.
Type:
* [`#/definitions/InlineSchemaLoader`](#/definitions/InlineSchemaLoader)
* [`#/definitions/DynamicSchemaLoader`](#/definitions/DynamicSchemaLoader)
* [`#/definitions/JsonFileSchemaLoader`](#/definitions/JsonFileSchemaLoader)
* `array`
* [`#/definitions/CustomSchemaLoader`](#/definitions/CustomSchemaLoader)
* #### transformations `array`
A list of transformations to be applied to each output record.
* #### state\_migrations `array`
Array of state migrations to be applied on the input state
* #### file\_uploader `object`
(experimental) Describes how to fetch a file
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### HTTPAPIBudget `object`[](#/definitions/HTTPAPIBudget "Direct link to /definitions/HTTPAPIBudget")
Defines how many requests can be made to the API in a given time frame. `HTTPAPIBudget` extracts the remaining call count and the reset time from HTTP response headers using the header names provided by `ratelimit_remaining_header` and `ratelimit_reset_header`. Only requests using `HttpRequester` are rate-limited; custom components that bypass `HttpRequester` are not covered by this budget.
Properties:
* #### policies `array`
List of call rate policies that define how many calls are allowed.
* #### ratelimit\_reset\_header `string`
The HTTP response header name that indicates when the rate limit resets.
* #### ratelimit\_remaining\_header `string`
The HTTP response header name that indicates the number of remaining allowed calls.
* #### status\_codes\_for\_ratelimit\_hit `array`
List of HTTP status codes that indicate a rate limit has been hit.
### FixedWindowCallRatePolicy `object`[](#/definitions/FixedWindowCallRatePolicy "Direct link to /definitions/FixedWindowCallRatePolicy")
A policy that allows a fixed number of calls within a specific time window.
Properties:
* #### period `string`
The time interval for the rate limit window.
* #### call\_limit `integer`
The maximum number of calls allowed within the period.
* #### matchers `array` `#/definitions/HttpRequestRegexMatcher`
List of matchers that define which requests this policy applies to.
### MovingWindowCallRatePolicy `object`[](#/definitions/MovingWindowCallRatePolicy "Direct link to /definitions/MovingWindowCallRatePolicy")
A policy that allows a fixed number of calls within a moving time window.
Properties:
* #### rates `array` `#/definitions/Rate`
List of rates that define the call limits for different time intervals.
* #### matchers `array` `#/definitions/HttpRequestRegexMatcher`
List of matchers that define which requests this policy applies to.
### UnlimitedCallRatePolicy `object`[](#/definitions/UnlimitedCallRatePolicy "Direct link to /definitions/UnlimitedCallRatePolicy")
A policy that allows unlimited calls for specific requests.
Properties:
* #### matchers `array` `#/definitions/HttpRequestRegexMatcher`
List of matchers that define which requests this policy applies to.
### Rate `object`[](#/definitions/Rate "Direct link to /definitions/Rate")
Defines a rate limit with a specific number of calls allowed within a time interval.
Properties:
* #### limit
The maximum number of calls allowed within the interval.
Type:
* `integer`
* `string`
Available variables:
* [config](#/variables/config)
* #### interval `string`
The time interval for the rate limit.
Examples:
```
PT1H
```
```
P1D
```
### HttpRequestRegexMatcher `object`[](#/definitions/HttpRequestRegexMatcher "Direct link to /definitions/HttpRequestRegexMatcher")
Matches HTTP requests based on method, base URL, URL path pattern, query parameters, and headers. Use `url_base` to specify the scheme and host (without trailing slash) and `url_path_pattern` to apply a regex to the request path.
Properties:
* #### method `string`
The HTTP method to match (e.g., GET, POST).
* #### url\_base `string`
The base URL (scheme and host, e.g. "https\://api.example.com") to match.
* #### url\_path\_pattern `string`
A regular expression pattern to match the URL path.
* #### params `object`
The query parameters to match.
* #### headers `object`
The headers to match.
* #### weight
The weight of a request matching this matcher when acquiring a call from the rate limiter. Different endpoints can consume different amounts from a shared budget by specifying different weights. If not set, each request counts as 1.
Type:
* `integer`
* `string`
### DefaultErrorHandler `object`[](#/definitions/DefaultErrorHandler "Direct link to /definitions/DefaultErrorHandler")
Component defining how to handle errors. Default behavior includes only retrying server errors (HTTP 5XX) and too many requests (HTTP 429) with an exponential backoff.
Properties:
* #### backoff\_strategies `array`
List of backoff strategies to use to determine how long to wait before retrying a retryable request.
* #### max\_retries
The maximum number of times to retry a retryable request before giving up and failing. Can be a hardcoded integer or a string interpolated from the connector config.
Type:
* `integer`
* `string`
Available variables:
* [config](#/variables/config)
Examples:
```
5
```
```
0
```
```
10
```
```
{{ config['max_retries_on_throttle'] }}
```
* #### response\_filters `array` `#/definitions/HttpResponseFilter`
List of response filters to iterate on when deciding how to handle an error. When using an array of multiple filters, the filters will be applied sequentially and the response will be selected if it matches any of the filter's predicate.
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### DefaultPaginator `object`[](#/definitions/DefaultPaginator "Direct link to /definitions/DefaultPaginator")
Default pagination implementation to request pages of results with a fixed size until the pagination strategy no longer returns a next\_page\_token.
Properties:
* #### pagination\_strategy
Strategy defining how records are paginated.
Type:
* [`#/definitions/PageIncrement`](#/definitions/PageIncrement)
* [`#/definitions/OffsetIncrement`](#/definitions/OffsetIncrement)
* [`#/definitions/CursorPagination`](#/definitions/CursorPagination)
* [`#/definitions/CustomPaginationStrategy`](#/definitions/CustomPaginationStrategy)
* #### page\_size\_option `#/definitions/RequestOption`
* #### page\_token\_option
Inject the page token into the outgoing HTTP requests by inserting it into either the request URL path or a field on the request.
Type:
* [`#/definitions/RequestOption`](#/definitions/RequestOption)
* [`#/definitions/RequestPath`](#/definitions/RequestPath)
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### DpathExtractor `object`[](#/definitions/DpathExtractor "Direct link to /definitions/DpathExtractor")
Record extractor that searches a decoded response over a path defined as an array of fields.
Properties:
* #### field\_path `array`
List of potentially nested fields describing the full path of the field to extract. Use "\*" to extract all values from an array. See more info in the [docs](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/record-selector).
Available variables:
* [config](#/variables/config)
Examples:
```
[
"data"
]
```
```
[
"data",
"records"
]
```
```
[
"data",
"{{ parameters.name }}"
]
```
```
[
"data",
"*",
"record"
]
```
* #### record\_expander `#/definitions/RecordExpander`
Optional component to expand records by extracting items from nested array fields.
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### ResponseToFileExtractor `object`[](#/definitions/ResponseToFileExtractor "Direct link to /definitions/ResponseToFileExtractor")
A record extractor designed for handling large responses that may exceed memory limits (to prevent OOM issues). It downloads a CSV file to disk, reads the data from disk, and deletes the file once it has been fully processed.
Properties:
* #### preserve\_na\_values `boolean`
When enabled, string values such as "NA", "N/A", "NULL", "None" and "NaN" are kept as-is instead of being interpreted as missing and converted to null. Empty cells are still treated as null. Defaults to false to preserve historical behavior.
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### RecordExpander `object`[](#/definitions/RecordExpander "Direct link to /definitions/RecordExpander")
Expands records by extracting items from a nested array field. When configured, this component extracts items from a specified nested array path within each record and emits each item as a separate record. Optionally, the original parent record can be embedded in each expanded item for context preservation. Supports wildcards (\*) for matching multiple arrays.
Properties:
* #### expand\_records\_from\_field `array`
Path to a nested array field within each record. Items from this array will be extracted and emitted as separate records. Supports wildcards (\*) for matching multiple arrays.
Available variables:
* [config](#/variables/config)
Examples:
```
[
"lines",
"data"
]
```
```
[
"items"
]
```
```
[
"nested",
"array"
]
```
```
[
"sections",
"*",
"items"
]
```
* #### remain\_original\_record `boolean`
If true, each expanded record will include the original parent record in an "original\_record" field. Defaults to false.
* #### on\_no\_records `string`
Behavior when the expansion path is missing, not a list, or an empty list. "skip" (default) emits nothing. "emit\_parent" emits the original parent record unchanged.
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### ExponentialBackoffStrategy `object`[](#/definitions/ExponentialBackoffStrategy "Direct link to /definitions/ExponentialBackoffStrategy")
Backoff strategy with an exponential backoff interval. The interval is defined as factor \* 2^attempt\_count.
Properties:
* #### factor
Multiplicative constant applied on each retry.
Type:
* `number`
* `string`
Available variables:
* [config](#/variables/config)
Examples:
```
5
```
```
5.5
```
```
10
```
* #### jitter\_range\_in\_seconds `number`
Optional additive jitter range in seconds. When set, the backoff time is uniformly distributed between computed\_backoff and computed\_backoff + (jitter\_range\_in\_seconds \* 2), so jitter only increases the computed backoff.
Example:
```
2
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### GroupByKeyMergeStrategy[](#/definitions/GroupByKeyMergeStrategy "Direct link to /definitions/GroupByKeyMergeStrategy")
Record merge strategy that combines records according to fields on the record.
Properties:
* #### key
The name of the field on the record whose value will be used to group properties that were retrieved through multiple API requests.
Type:
* `string`
* `array`
Examples:
```
id
```
```
[
"parent_id",
"end_date"
]
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### SessionTokenAuthenticator `object`[](#/definitions/SessionTokenAuthenticator "Direct link to /definitions/SessionTokenAuthenticator")
Authenticator for requests using the session token as an API key that's injected into the request.
Properties:
* #### login\_requester `#/definitions/HttpRequester`
Description of the request to perform to obtain a session token to perform data requests. The response body is expected to be a JSON object with a session token property.
Example:
```
{
"type": "HttpRequester",
"url_base": "https://my_api.com",
"path": "/login",
"authenticator": {
"type": "BasicHttpAuthenticator",
"username": "{{ config.username }}",
"password": "{{ config.password }}"
}
}
```
* #### session\_token\_path `array`
The path in the response body returned from the login requester to the session token.
Examples:
```
[
"access_token"
]
```
```
[
"result",
"token"
]
```
* #### expiration\_duration `string`
The duration in ISO 8601 duration notation after which the session token expires, starting from the time it was obtained. Omitting it will result in the session token being refreshed for every request.
* **PT1H**: 1 hour
* **P1D**: 1 day
* **P1W**: 1 week
* **P1M**: 1 month
* **P1Y**: 1 year
Examples:
```
PT1H
```
```
P1D
```
* #### request\_authentication
Authentication method to use for requests sent to the API, specifying how to inject the session token.
Type:
* [`#/definitions/SessionTokenRequestApiKeyAuthenticator`](#/definitions/SessionTokenRequestApiKeyAuthenticator)
* [`#/definitions/SessionTokenRequestBearerAuthenticator`](#/definitions/SessionTokenRequestBearerAuthenticator)
* #### decoder
Component used to decode the response.
Type:
* [`#/definitions/JsonDecoder`](#/definitions/JsonDecoder)
* [`#/definitions/XmlDecoder`](#/definitions/XmlDecoder)
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### SessionTokenRequestApiKeyAuthenticator `object`[](#/definitions/SessionTokenRequestApiKeyAuthenticator "Direct link to /definitions/SessionTokenRequestApiKeyAuthenticator")
Authenticator for requests using the session token as an API key that's injected into the request.
Properties:
* #### inject\_into `#/definitions/RequestOption`
Configure how the API Key will be sent in requests to the source API.
Examples:
```
{
"inject_into": "header",
"field_name": "Authorization"
}
```
```
{
"inject_into": "request_parameter",
"field_name": "authKey"
}
```
* #### api\_token `string`
A template for the token value to inject. Use {{ session\_token }} to reference the session token. For example, use "Token {{ session\_token }}" for APIs that expect "Authorization: Token \".
Available variables:
* [config](#/variables/config)
* [session\_token](#/variables/session_token)
Examples:
```
{{ session_token }}
```
```
Token {{ session_token }}
```
```
Bearer {{ session_token }}
```
### SessionTokenRequestBearerAuthenticator[](#/definitions/SessionTokenRequestBearerAuthenticator "Direct link to /definitions/SessionTokenRequestBearerAuthenticator")
Authenticator for requests using the session token as a standard bearer token.
Properties:
### HttpRequester `object`[](#/definitions/HttpRequester "Direct link to /definitions/HttpRequester")
Requester submitting HTTP requests and extracting records from the response.
Properties:
* #### url\_base `string`
Deprecated, use the `url` instead. Base URL of the API source. Do not put sensitive information (e.g. API tokens) into this field - Use the Authenticator component for this.
Available variables:
* [config](#/variables/config)
* [next\_page\_token](#/variables/next_page_token)
* [stream\_interval](#/variables/stream_interval)
* [stream\_partition](#/variables/stream_partition)
* [stream\_slice](#/variables/stream_slice)
* [creation\_response](#/variables/creation_response)
* [polling\_response](#/variables/polling_response)
* [download\_target](#/variables/download_target)
Examples:
```
https://connect.squareup.com/v2
```
```
{{ config['base_url'] or 'https://app.posthog.com'}}/api
```
```
https://connect.squareup.com/v2/quotes/{{ stream_partition['id'] }}/quote_line_groups
```
```
https://example.com/api/v1/resource/{{ next_page_token['id'] }}
```
* #### url `string`
The URL of the source API endpoint. Do not put sensitive information (e.g. API tokens) into this field - Use the Authenticator component for this.
Available variables:
* [config](#/variables/config)
* [next\_page\_token](#/variables/next_page_token)
* [stream\_interval](#/variables/stream_interval)
* [stream\_partition](#/variables/stream_partition)
* [stream\_slice](#/variables/stream_slice)
* [creation\_response](#/variables/creation_response)
* [polling\_response](#/variables/polling_response)
* [download\_target](#/variables/download_target)
Examples:
```
https://connect.squareup.com/v2
```
```
{{ config['url'] or 'https://app.posthog.com'}}/api
```
```
https://connect.squareup.com/v2/quotes/{{ stream_partition['id'] }}/quote_line_groups
```
```
https://example.com/api/v1/resource/{{ next_page_token['id'] }}
```
* #### path `string`
Deprecated, use the `url` instead. Path the specific API endpoint that this stream represents. Do not put sensitive information (e.g. API tokens) into this field - Use the Authenticator component for this.
Available variables:
* [config](#/variables/config)
* [next\_page\_token](#/variables/next_page_token)
* [stream\_interval](#/variables/stream_interval)
* [stream\_partition](#/variables/stream_partition)
* [stream\_slice](#/variables/stream_slice)
* [creation\_response](#/variables/creation_response)
* [polling\_response](#/variables/polling_response)
* [download\_target](#/variables/download_target)
Examples:
```
/products
```
```
/quotes/{{ stream_partition['id'] }}/quote_line_groups
```
```
/trades/{{ config['symbol_id'] }}/history
```
* #### http\_method `string`
The HTTP method used to fetch data from the source (can be GET or POST).
Examples:
```
GET
```
```
POST
```
* #### authenticator
Authentication method to use for requests sent to the API.
Type:
* [`#/definitions/ApiKeyAuthenticator`](#/definitions/ApiKeyAuthenticator)
* [`#/definitions/BasicHttpAuthenticator`](#/definitions/BasicHttpAuthenticator)
* [`#/definitions/BearerAuthenticator`](#/definitions/BearerAuthenticator)
* [`#/definitions/OAuthAuthenticator`](#/definitions/OAuthAuthenticator)
* [`#/definitions/JwtAuthenticator`](#/definitions/JwtAuthenticator)
* [`#/definitions/SessionTokenAuthenticator`](#/definitions/SessionTokenAuthenticator)
* [`#/definitions/SelectiveAuthenticator`](#/definitions/SelectiveAuthenticator)
* [`#/definitions/CustomAuthenticator`](#/definitions/CustomAuthenticator)
* [`#/definitions/NoAuth`](#/definitions/NoAuth)
* [`#/definitions/LegacySessionTokenAuthenticator`](#/definitions/LegacySessionTokenAuthenticator)
* [`#/definitions/RateLimitedMultipleTokenAuthenticator`](#/definitions/RateLimitedMultipleTokenAuthenticator)
* #### fetch\_properties\_from\_endpoint `#/definitions/PropertiesFromEndpoint`
Allows for retrieving a dynamic set of properties from an API endpoint which can be injected into outbound request using the stream\_partition.extra\_fields.
* #### query\_properties `#/definitions/QueryProperties`
For APIs that require explicit specification of the properties to query for, this component will take a static or dynamic set of properties (which can be optionally split into chunks) and allow them to be injected into an outbound request by accessing stream\_partition.extra\_fields.
* #### request\_parameters
Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
Type:
* `object`
* `string`
Available variables:
* [next\_page\_token](#/variables/next_page_token)
* [stream\_interval](#/variables/stream_interval)
* [stream\_partition](#/variables/stream_partition)
* [stream\_slice](#/variables/stream_slice)
Examples:
```
{
"unit": "day"
}
```
```
{
"query": "last_event_time BETWEEN TIMESTAMP \"{{ stream_interval.start_time }}\" AND TIMESTAMP \"{{ stream_interval.end_time }}\""
}
```
```
{
"searchIn": "{{ ','.join(config.get('search_in', [])) }}"
}
```
```
{
"sort_by[asc]": "updated_at"
}
```
* #### request\_headers
Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
Type:
* `object`
* `string`
Available variables:
* [next\_page\_token](#/variables/next_page_token)
* [stream\_interval](#/variables/stream_interval)
* [stream\_partition](#/variables/stream_partition)
* [stream\_slice](#/variables/stream_slice)
Examples:
```
{
"Output-Format": "JSON"
}
```
```
{
"Version": "{{ config['version'] }}"
}
```
* #### request\_body\_data
Specifies how to populate the body of the request with a non-JSON payload. Plain text will be sent as is, whereas objects will be converted to a urlencoded form.
Type:
* `object`
* `string`
Available variables:
* [next\_page\_token](#/variables/next_page_token)
* [stream\_interval](#/variables/stream_interval)
* [stream\_partition](#/variables/stream_partition)
* [stream\_slice](#/variables/stream_slice)
Example:
```
[{"clause": {"type": "timestamp", "operator": 10, "parameters":
[{"value": {{ stream_interval['start_time'] | int * 1000 }} }]
}, "orderBy": 1, "columnName": "Timestamp"}]/
```
* #### request\_body\_json
Specifies how to populate the body of the request with a JSON payload. Can contain nested objects.
Type:
* `object`
* `string`
Available variables:
* [next\_page\_token](#/variables/next_page_token)
* [stream\_interval](#/variables/stream_interval)
* [stream\_partition](#/variables/stream_partition)
* [stream\_slice](#/variables/stream_slice)
Examples:
```
{
"sort_order": "ASC",
"sort_field": "CREATED_AT"
}
```
```
{
"key": "{{ config['value'] }}"
}
```
```
{
"sort": {
"field": "updated_at",
"order": "ascending"
}
}
```
* #### request\_body
Specifies how to populate the body of the request with a payload. Can contain nested objects.
Type:
* [`#/definitions/RequestBodyPlainText`](#/definitions/RequestBodyPlainText)
* [`#/definitions/RequestBodyUrlEncodedForm`](#/definitions/RequestBodyUrlEncodedForm)
* [`#/definitions/RequestBodyJsonObject`](#/definitions/RequestBodyJsonObject)
* [`#/definitions/RequestBodyGraphQL`](#/definitions/RequestBodyGraphQL)
Available variables:
* [next\_page\_token](#/variables/next_page_token)
* [stream\_interval](#/variables/stream_interval)
* [stream\_partition](#/variables/stream_partition)
* [stream\_slice](#/variables/stream_slice)
* #### error\_handler
Error handler component that defines how to handle errors.
Type:
* [`#/definitions/DefaultErrorHandler`](#/definitions/DefaultErrorHandler)
* [`#/definitions/CompositeErrorHandler`](#/definitions/CompositeErrorHandler)
* [`#/definitions/CustomErrorHandler`](#/definitions/CustomErrorHandler)
* #### use\_cache `boolean`
Enables stream requests caching. When set to true, repeated requests to the same URL will return cached responses. Parent streams automatically have caching enabled. Only set this to false if you are certain that caching should be disabled, as it may negatively impact performance when the same data is needed multiple times (e.g., for scroll-based pagination APIs where caching causes duplicate records).
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### HttpResponseFilter `object`[](#/definitions/HttpResponseFilter "Direct link to /definitions/HttpResponseFilter")
A filter that is used to select on properties of the HTTP response received. When used with additional filters, a response will be selected if it matches any of the filter's criteria.
Properties:
* #### action `string`
Action to execute if a response matches the filter.
Examples:
```
SUCCESS
```
```
FAIL
```
```
RETRY
```
```
IGNORE
```
```
RESET_PAGINATION
```
```
RATE_LIMITED
```
```
REFRESH_TOKEN_THEN_RETRY
```
* #### failure\_type `string`
Failure type of traced exception if a response matches the filter.
Examples:
```
system_error
```
```
config_error
```
```
transient_error
```
* #### error\_message `string`
Error Message to display if the response matches the filter.
Available variables:
* [config](#/variables/config)
* [response](#/variables/response)
* [headers](#/variables/headers)
* #### error\_message\_contains `string`
Match the response if its error message contains the substring.
* #### http\_codes `array`
Match the response if its HTTP code is included in this list.
Examples:
```
[
420,
429
]
```
```
[
500
]
```
* #### predicate `string`
Match the response if the predicate evaluates to true.
Available variables:
* [response](#/variables/response)
* [headers](#/variables/headers)
Examples:
```
{{ 'Too much requests' in response }}
```
```
{{ 'error_code' in response and response['error_code'] == 'ComplexityException' }}
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### ComplexFieldType `object`[](#/definitions/ComplexFieldType "Direct link to /definitions/ComplexFieldType")
(This component is experimental. Use at your own risk.) Represents a complex field type.
Properties:
* #### field\_type `string`
* #### items
Type:
* `string`
* [`#/definitions/ComplexFieldType`](#/definitions/ComplexFieldType)
### TypesMap `object`[](#/definitions/TypesMap "Direct link to /definitions/TypesMap")
(This component is experimental. Use at your own risk.) Represents a mapping between a current type and its corresponding target type.
Properties:
* #### target\_type
Type:
* `string`
* `array`
* [`#/definitions/ComplexFieldType`](#/definitions/ComplexFieldType)
* #### current\_type
Type:
* `string`
* `array`
* #### condition `string`
Available variables:
* [raw\_schema](#/variables/raw_schema)
### SchemaTypeIdentifier `object`[](#/definitions/SchemaTypeIdentifier "Direct link to /definitions/SchemaTypeIdentifier")
(This component is experimental. Use at your own risk.) Identifies schema details for dynamic schema extraction and processing.
Properties:
* #### schema\_pointer `array`
List of nested fields defining the schema field path to extract. Defaults to \[].
Available variables:
* [config](#/variables/config)
* #### key\_pointer `array`
List of potentially nested fields describing the full path of the field key to extract.
Available variables:
* [config](#/variables/config)
* #### type\_pointer `array`
List of potentially nested fields describing the full path of the field type to extract.
Available variables:
* [config](#/variables/config)
* #### types\_mapping `array` `#/definitions/TypesMap`
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### DynamicSchemaLoader `object`[](#/definitions/DynamicSchemaLoader "Direct link to /definitions/DynamicSchemaLoader")
(This component is experimental. Use at your own risk.) Loads a schema by extracting data from retrieved records.
Properties:
* #### retriever
Component used to coordinate how records are extracted across stream slices and request pages.
Type:
* [`#/definitions/SimpleRetriever`](#/definitions/SimpleRetriever)
* [`#/definitions/AsyncRetriever`](#/definitions/AsyncRetriever)
* [`#/definitions/CustomRetriever`](#/definitions/CustomRetriever)
* #### schema\_filter
Responsible for filtering fields to be added to json schema.
Type:
* [`#/definitions/RecordFilter`](#/definitions/RecordFilter)
* [`#/definitions/CustomRecordFilter`](#/definitions/CustomRecordFilter)
* #### schema\_transformations `array`
A list of transformations to be applied to the schema.
* #### schema\_type\_identifier `#/definitions/SchemaTypeIdentifier`
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### InlineSchemaLoader `object`[](#/definitions/InlineSchemaLoader "Direct link to /definitions/InlineSchemaLoader")
Loads a schema that is defined directly in the manifest file.
Properties:
* #### schema `object`
Describes a streams' schema. Refer to the \Data Types documentation\ for more details on which types are valid.
### JsonFileSchemaLoader `object`[](#/definitions/JsonFileSchemaLoader "Direct link to /definitions/JsonFileSchemaLoader")
Loads the schema from a json file.
Properties:
* #### file\_path `string`
Path to the JSON file defining the schema. The path is relative to the connector module's root.
Available variables:
* [config](#/variables/config)
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### JsonDecoder `object`[](#/definitions/JsonDecoder "Direct link to /definitions/JsonDecoder")
Select 'JSON' if the response is formatted as a JSON object.
Properties:
### JsonItemsDecoder `object`[](#/definitions/JsonItemsDecoder "Direct link to /definitions/JsonItemsDecoder")
Select 'JSON Items (Streaming)' to stream-decode a single JSON document by yielding each element of a nested array, one at a time. Use this for very large single-document JSON responses (e.g. a wrapping object containing a multi-GB array) where buffering the whole document into memory would cause out-of-memory errors. Powered by the `ijson` streaming parser.
Properties:
* #### items\_path `string`
Dot-separated path to the JSON array whose elements should be yielded as records. Uses `ijson` path syntax (e.g. `data.users`), not JSONPath syntax — do not include leading `$.` or trailing `[*]`.
Examples:
```
dataByDepartmentAndSearchTerm
```
```
dataByAsin
```
```
data.users
```
* #### encoding `string`
Text encoding used to decode the streamed bytes before JSON parsing.
### JsonlDecoder `object`[](#/definitions/JsonlDecoder "Direct link to /definitions/JsonlDecoder")
Select 'JSON Lines' if the response consists of JSON objects separated by new lines ('\n') in JSONL format.
Properties:
### JsonSchemaPropertySelector `object`[](#/definitions/JsonSchemaPropertySelector "Direct link to /definitions/JsonSchemaPropertySelector")
When configured, the JSON schema supplied in the catalog containing which columns are selected for the current stream will be used to reduce which query properties will be included in the outbound API request. This can improve the performance of API requests, especially for those requiring multiple requests to get a complete record.
Properties:
* #### transformations `array`
A list of transformations to be applied on the customer configured schema that will be used to filter out unselected fields when specifying query properties for API requests.
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### KeysToLower `object`[](#/definitions/KeysToLower "Direct link to /definitions/KeysToLower")
A transformation that renames all keys to lower case.
Properties:
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### KeysToSnakeCase `object`[](#/definitions/KeysToSnakeCase "Direct link to /definitions/KeysToSnakeCase")
A transformation that renames all keys to snake case.
Properties:
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### FlattenFields `object`[](#/definitions/FlattenFields "Direct link to /definitions/FlattenFields")
A transformation that flatten record to single level format.
Properties:
* #### flatten\_lists `boolean`
Whether to flatten lists or leave it as is. Default is True.
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### KeyTransformation `object`[](#/definitions/KeyTransformation "Direct link to /definitions/KeyTransformation")
Properties:
* #### prefix `string`
Prefix to add for object keys. If not provided original keys remain unchanged.
Example:
```
flattened_
```
* #### suffix `string`
Suffix to add for object keys. If not provided original keys remain unchanged.
Example:
```
_flattened
```
### DpathFlattenFields `object`[](#/definitions/DpathFlattenFields "Direct link to /definitions/DpathFlattenFields")
A transformation that flatten field values to the to top of the record.
Properties:
* #### field\_path `array`
A path to field that needs to be flattened.
Examples:
```
[
"data"
]
```
```
[
"data",
"*",
"field"
]
```
* #### delete\_origin\_value `boolean`
Whether to delete the origin value or keep it. Default is False.
* #### replace\_record `boolean`
Whether to replace the origin record or not. Default is False.
* #### key\_transformation `object` `#/definitions/KeyTransformation`
Transformation for object keys. If not provided, original key will be used.
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### KeysReplace `object`[](#/definitions/KeysReplace "Direct link to /definitions/KeysReplace")
A transformation that replaces symbols in keys.
Properties:
* #### old `string`
Old value to replace.
Available variables:
* [config](#/variables/config)
* [record](#/variables/record)
* [stream\_slice](#/variables/stream_slice)
Examples:
```
```
```
{{ record.id }}
```
```
{{ config['id'] }}
```
```
{{ stream_slice['id'] }}
```
* #### new `string`
New value to set.
Available variables:
* [config](#/variables/config)
* [record](#/variables/record)
* [stream\_slice](#/variables/stream_slice)
Examples:
```
_
```
```
{{ record.id }}
```
```
{{ config['id'] }}
```
```
{{ stream_slice['id'] }}
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### IterableDecoder `object`[](#/definitions/IterableDecoder "Direct link to /definitions/IterableDecoder")
Select 'Iterable' if the response consists of strings separated by new lines (`\n`). The string will then be wrapped into a JSON object with the `record` key.
Properties:
### XmlDecoder `object`[](#/definitions/XmlDecoder "Direct link to /definitions/XmlDecoder")
Select 'XML' if the response consists of XML-formatted data.
Properties:
### CustomDecoder `object`[](#/definitions/CustomDecoder "Direct link to /definitions/CustomDecoder")
Use this to implement custom decoder logic.
Properties:
* #### class\_name `string`
Fully-qualified name of the class that will be implementing the custom decoding. Has to be a sub class of Decoder. The format is `source_..`.
Example:
```
source_amazon_ads.components.GzipJsonlDecoder
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### ZipfileDecoder `object`[](#/definitions/ZipfileDecoder "Direct link to /definitions/ZipfileDecoder")
Select 'ZIP file' for response data that is returned as a zipfile. Requires specifying an inner data type/decoder to parse the unzipped data.
Properties:
* #### decoder
Parser to parse the decompressed data from the zipfile(s).
Type:
* [`#/definitions/CsvDecoder`](#/definitions/CsvDecoder)
* [`#/definitions/GzipDecoder`](#/definitions/GzipDecoder)
* [`#/definitions/JsonDecoder`](#/definitions/JsonDecoder)
* [`#/definitions/JsonItemsDecoder`](#/definitions/JsonItemsDecoder)
* [`#/definitions/JsonlDecoder`](#/definitions/JsonlDecoder)
### ListPartitionRouter `object`[](#/definitions/ListPartitionRouter "Direct link to /definitions/ListPartitionRouter")
A Partition router that specifies a list of attributes where each attribute describes a portion of the complete data set for a stream. During a sync, each value is iterated over and can be used as input to outbound API requests.
Properties:
* #### cursor\_field `string`
While iterating over list values, the name of field used to reference a list value. The partition value can be accessed with string interpolation. e.g. "{{ stream\_partition\['my\_key'] }}" where "my\_key" is the value of the cursor\_field.
Available variables:
* [config](#/variables/config)
Examples:
```
section
```
```
{{ config['section_key'] }}
```
* #### values
The list of attributes being iterated over and used as input for the requests made to the source API.
Type:
* `string`
* `array`
Available variables:
* [config](#/variables/config)
Examples:
```
[
"section_a",
"section_b",
"section_c"
]
```
```
{{ config['sections'] }}
```
* #### request\_option `#/definitions/RequestOption`
A request option describing where the list value should be injected into and under what field name if applicable.
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### MinMaxDatetime `object`[](#/definitions/MinMaxDatetime "Direct link to /definitions/MinMaxDatetime")
Compares the provided date against optional minimum or maximum times. The max\_datetime serves as the ceiling and will be returned when datetime exceeds it. The min\_datetime serves as the floor.
Properties:
* #### datetime `string`
Datetime value.
Available variables:
* [config](#/variables/config)
Examples:
```
2021-01-01
```
```
2021-01-01T00:00:00Z
```
```
{{ config['start_time'] }}
```
```
{{ now_utc().strftime('%Y-%m-%dT%H:%M:%SZ') }}
```
* #### datetime\_format `string`
Format of the datetime value. Defaults to "%Y-%m-%dT%H:%M:%S.%f%z" if left empty. Use placeholders starting with "%" to describe the format the API is using. The following placeholders are available:
* **%s**: Epoch unix timestamp - `1686218963`
* **%s\_as\_float**: Epoch unix timestamp in seconds as float with microsecond precision - `1686218963.123456`
* **%ms**: Epoch unix timestamp - `1686218963123`
* **%a**: Weekday (abbreviated) - `Sun`
* **%A**: Weekday (full) - `Sunday`
* **%w**: Weekday (decimal) - `0` (Sunday), `6` (Saturday)
* **%d**: Day of the month (zero-padded) - `01`, `02`, ..., `31`
* **%b**: Month (abbreviated) - `Jan`
* **%B**: Month (full) - `January`
* **%m**: Month (zero-padded) - `01`, `02`, ..., `12`
* **%y**: Year (without century, zero-padded) - `00`, `01`, ..., `99`
* **%Y**: Year (with century) - `0001`, `0002`, ..., `9999`
* **%H**: Hour (24-hour, zero-padded) - `00`, `01`, ..., `23`
* **%I**: Hour (12-hour, zero-padded) - `01`, `02`, ..., `12`
* **%p**: AM/PM indicator
* **%M**: Minute (zero-padded) - `00`, `01`, ..., `59`
* **%S**: Second (zero-padded) - `00`, `01`, ..., `59`
* **%f**: Microsecond (zero-padded to 6 digits) - `000000`, `000001`, ..., `999999`
* **%\_ms**: Millisecond (zero-padded to 3 digits) - `000`, `001`, ..., `999`
* **%z**: UTC offset - `(empty)`, `+0000`, `-04:00`
* **%Z**: Time zone name - `(empty)`, `UTC`, `GMT`
* **%j**: Day of the year (zero-padded) - `001`, `002`, ..., `366`
* **%U**: Week number of the year (Sunday as first day) - `00`, `01`, ..., `53`
* **%W**: Week number of the year (Monday as first day) - `00`, `01`, ..., `53`
* **%c**: Date and time representation - `Tue Aug 16 21:30:00 1988`
* **%x**: Date representation - `08/16/1988`
* **%X**: Time representation - `21:30:00`
* **%%**: Literal '%' character
Some placeholders depend on the locale of the underlying system - in most cases this locale is configured as en/US. For more information see the [Python documentation](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes).
Examples:
```
%Y-%m-%dT%H:%M:%S.%f%z
```
```
%Y-%m-%d
```
```
%s
```
* #### max\_datetime `string`
Ceiling applied on the datetime value. Must be formatted with the datetime\_format field.
Available variables:
* [config](#/variables/config)
Examples:
```
2021-01-01T00:00:00Z
```
```
2021-01-01
```
* #### min\_datetime `string`
Floor applied on the datetime value. Must be formatted with the datetime\_format field.
Available variables:
* [config](#/variables/config)
Examples:
```
2010-01-01T00:00:00Z
```
```
2010-01-01
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### NoAuth `object`[](#/definitions/NoAuth "Direct link to /definitions/NoAuth")
Authenticator for requests requiring no authentication.
Properties:
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### NoPagination `object`[](#/definitions/NoPagination "Direct link to /definitions/NoPagination")
Pagination implementation that never returns a next page.
Properties:
### OAuthConfigSpecification `object`[](#/definitions/OAuthConfigSpecification "Direct link to /definitions/OAuthConfigSpecification")
Specification describing how an 'advanced' Auth flow would need to function.
Properties:
* #### oauth\_user\_input\_from\_connector\_config\_specification `object`
OAuth specific blob. This is a Json Schema used to validate Json configurations used as input to OAuth. Must be a valid non-nested JSON that refers to properties from ConnectorSpecification.connectionSpecification using special annotation 'path\_in\_connector\_config'. These are input values the user is entering through the UI to authenticate to the connector, that might also shared as inputs for syncing data via the connector. Examples: if no connector values is shared during oauth flow, oauth\_user\_input\_from\_connector\_config\_specification=\[] if connector values such as 'app\_id' inside the top level are used to generate the API url for the oauth flow, oauth\_user\_input\_from\_connector\_config\_specification={ app\_id: { type: string path\_in\_connector\_config: \['app\_id'] } } if connector values such as 'info.app\_id' nested inside another object are used to generate the API url for the oauth flow, oauth\_user\_input\_from\_connector\_config\_specification={ app\_id: { type: string path\_in\_connector\_config: \['info', 'app\_id'] } }
Examples:
```
{
"app_id": {
"type": "string",
"path_in_connector_config": [
"app_id"
]
}
}
```
```
{
"app_id": {
"type": "string",
"path_in_connector_config": [
"info",
"app_id"
]
}
}
```
* #### oauth\_connector\_input\_specification `object`
The DeclarativeOAuth specific blob. Pertains to the fields defined by the connector relating to the OAuth flow.
Interpolation capabilities:
* The variables placeholders are declared as `{{my_var}}`.
* The nested resolution variables like `{{ {{my_nested_var}} }}` is allowed as well.
* The allowed interpolation context is:
* base64Encoder - encode to `base64`, {{ {{my\_var\_a}}:{{my\_var\_b}} | base64Encoder }}
* base64Decorer - decode from `base64` encoded string, {{ {{my\_string\_variable\_or\_string\_value}} | base64Decoder }}
* urlEncoder - encode the input string to URL-like format, {{ https\://test.host.com/endpoint | urlEncoder}}
* urlDecorer - decode the input url-encoded string into text format, {{ urlDecoder:https%3A%2F%2Fairbyte.io | urlDecoder}}
* codeChallengeS256 - get the `codeChallenge` encoded value to provide additional data-provider specific authorisation values, {{ {{state\_value}} | codeChallengeS256 }}
Examples:
* The TikTok Marketing DeclarativeOAuth spec: { "oauth\_connector\_input\_specification": { "type": "object", "additionalProperties": false, "properties": { "consent\_url": "https\://ads.tiktok.com/marketing\_api/auth?{{client\_id\_key}}={{client\_id\_value}}&{{redirect\_uri\_key}}={{ {{redirect\_uri\_value}} | urlEncoder}}&{{state\_key}}={{state\_value}}", "access\_token\_url": "https\://business-api.tiktok.com/open\_api/v1.3/oauth2/access\_token/", "access\_token\_params": { "{{ auth\_code\_key }}": "{{ auth\_code\_value }}", "{{ client\_id\_key }}": "{{ client\_id\_value }}", "{{ client\_secret\_key }}": "{{ client\_secret\_value }}" }, "access\_token\_headers": { "Content-Type": "application/json", "Accept": "application/json" }, "extract\_output": \["data.access\_token"], "client\_id\_key": "app\_id", "client\_secret\_key": "secret", "auth\_code\_key": "auth\_code" } } }
* #### complete\_oauth\_output\_specification `object`
OAuth specific blob. This is a Json Schema used to validate Json configurations produced by the OAuth flows as they are returned by the distant OAuth APIs. Must be a valid JSON describing the fields to merge back to `ConnectorSpecification.connectionSpecification`. For each field, a special annotation `path_in_connector_config` can be specified to determine where to merge it, Examples: complete\_oauth\_output\_specification={ refresh\_token: { type: string, path\_in\_connector\_config: \['credentials', 'refresh\_token'] } }
Example:
```
{
"refresh_token": {
"type": "string,",
"path_in_connector_config": [
"credentials",
"refresh_token"
]
}
}
```
* #### complete\_oauth\_server\_input\_specification `object`
OAuth specific blob. This is a Json Schema used to validate Json configurations persisted as Airbyte Server configurations. Must be a valid non-nested JSON describing additional fields configured by the Airbyte Instance or Workspace Admins to be used by the server when completing an OAuth flow (typically exchanging an auth code for refresh token). Examples: complete\_oauth\_server\_input\_specification={ client\_id: { type: string }, client\_secret: { type: string } }
Example:
```
{
"client_id": {
"type": "string"
},
"client_secret": {
"type": "string"
}
}
```
* #### complete\_oauth\_server\_output\_specification `object`
OAuth specific blob. This is a Json Schema used to validate Json configurations persisted as Airbyte Server configurations that also need to be merged back into the connector configuration at runtime. This is a subset configuration of `complete_oauth_server_input_specification` that filters fields out to retain only the ones that are necessary for the connector to function with OAuth. (some fields could be used during oauth flows but not needed afterwards, therefore they would be listed in the `complete_oauth_server_input_specification` but not `complete_oauth_server_output_specification`) Must be a valid non-nested JSON describing additional fields configured by the Airbyte Instance or Workspace Admins to be used by the connector when using OAuth flow APIs. These fields are to be merged back to `ConnectorSpecification.connectionSpecification`. For each field, a special annotation `path_in_connector_config` can be specified to determine where to merge it, Examples: complete\_oauth\_server\_output\_specification={ client\_id: { type: string, path\_in\_connector\_config: \['credentials', 'client\_id'] }, client\_secret: { type: string, path\_in\_connector\_config: \['credentials', 'client\_secret'] } }
Example:
```
{
"client_id": {
"type": "string,",
"path_in_connector_config": [
"credentials",
"client_id"
]
},
"client_secret": {
"type": "string,",
"path_in_connector_config": [
"credentials",
"client_secret"
]
}
}
```
### OffsetIncrement `object`[](#/definitions/OffsetIncrement "Direct link to /definitions/OffsetIncrement")
Pagination strategy that returns the number of records reads so far and returns it as the next page token.
Properties:
* #### page\_size
The number of records to include in each pages.
Type:
* `integer`
* `string`
Available variables:
* [config](#/variables/config)
* [response](#/variables/response)
Examples:
```
100
```
```
{{ config['page_size'] }}
```
* #### inject\_on\_first\_request `boolean`
Using the `offset` with value `0` during the first request
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### PageIncrement `object`[](#/definitions/PageIncrement "Direct link to /definitions/PageIncrement")
Pagination strategy that returns the number of pages reads so far and returns it as the next page token.
Properties:
* #### page\_size
The number of records to include in each pages.
Type:
* `integer`
* `string`
Available variables:
* [config](#/variables/config)
Examples:
```
100
```
```
100
```
```
{{ config['page_size'] }}
```
* #### start\_from\_page `integer`
Index of the first page to request.
Examples:
```
0
```
```
1
```
* #### inject\_on\_first\_request `boolean`
Using the `page number` with value defined by `start_from_page` during the first request
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### ParentStreamConfig `object`[](#/definitions/ParentStreamConfig "Direct link to /definitions/ParentStreamConfig")
Describes how to construct partitions from the records retrieved from the parent stream..
Properties:
* #### stream
Reference to the parent stream.
Type:
* [`#/definitions/DeclarativeStream`](#/definitions/DeclarativeStream)
* [`#/definitions/StateDelegatingStream`](#/definitions/StateDelegatingStream)
* #### parent\_key `string`
The primary key of records from the parent stream that will be used during the retrieval of records for the current substream. This parent identifier field is typically a characteristic of the child records being extracted from the source API.
Examples:
```
id
```
```
{{ config['parent_record_id'] }}
```
* #### partition\_field `string`
While iterating over parent records during a sync, the parent\_key value can be referenced by using this field.
Examples:
```
parent_id
```
```
{{ config['parent_partition_field'] }}
```
* #### request\_option `#/definitions/RequestOption`
A request option describing where the parent key value should be injected into and under what field name if applicable.
* #### incremental\_dependency `boolean`
Indicates whether the parent stream should be read incrementally based on updates in the child stream.
* #### lazy\_read\_pointer `array`
If set, this will enable lazy reading, using the initial read of parent records to extract child records.
Available variables:
* [config](#/variables/config)
* #### extra\_fields `array`
Array of field paths to include as additional fields in the stream slice. Each path is an array of strings representing keys to access fields in the respective parent record. Accessible via `stream_slice.extra_fields`. Missing fields are set to `None`.
Available variables:
* [config](#/variables/config)
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### PrimaryKey[](#/definitions/PrimaryKey "Direct link to /definitions/PrimaryKey")
The stream field to be used to distinguish unique records. Can either be a single field, an array of fields representing a composite key, or an array of arrays representing a composite key where the fields are nested fields.
Examples:
```
id
```
```
[
"code",
"type"
]
```
### PropertiesFromEndpoint `object`[](#/definitions/PropertiesFromEndpoint "Direct link to /definitions/PropertiesFromEndpoint")
Defines the behavior for fetching the list of properties from an API that will be loaded into the requests to extract records. Note that stream\_slices can't be interpolated from this retriever.
Properties:
* #### property\_field\_path `array`
Describes the path to the field that should be extracted
Available variables:
* [config](#/variables/config)
* [parameters](#/variables/parameters)
Example:
```
[
"name"
]
```
* #### retriever
Requester component that describes how to fetch the properties to query from a remote API endpoint.
Type:
* [`#/definitions/SimpleRetriever`](#/definitions/SimpleRetriever)
* [`#/definitions/CustomRetriever`](#/definitions/CustomRetriever)
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### PropertyChunking `object`[](#/definitions/PropertyChunking "Direct link to /definitions/PropertyChunking")
For APIs with restrictions on the amount of properties that can be requester per request, property chunking can be applied to make multiple requests with a subset of the properties.
Properties:
* #### property\_limit\_type
The type used to determine the maximum number of properties per chunk
* #### property\_limit `integer`
The maximum amount of properties that can be retrieved per request according to the limit type.
* #### record\_merge\_strategy `#/definitions/GroupByKeyMergeStrategy`
Dictates how to records that require multiple requests to get all properties should be emitted to the destination
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### QueryProperties `object`[](#/definitions/QueryProperties "Direct link to /definitions/QueryProperties")
For APIs that require explicit specification of the properties to query for, this component specifies which property fields and how they are supplied to outbound requests.
Properties:
* #### property\_list
The set of properties that will be queried for in the outbound request. This can either be statically defined or dynamic based on an API endpoint
Type:
* `array`
* [`#/definitions/PropertiesFromEndpoint`](#/definitions/PropertiesFromEndpoint)
* #### always\_include\_properties `array`
The list of properties that should be included in every set of properties when multiple chunks of properties are being requested.
* #### property\_chunking `#/definitions/PropertyChunking`
Defines how query properties will be grouped into smaller sets for APIs with limitations on the number of properties fetched per API request.
* #### property\_selector `#/definitions/JsonSchemaPropertySelector`
Defines where to look for and which query properties that should be sent in outbound API requests. For example, you can specify that only the selected columns of a stream should be in the request.
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### RecordFilter `object`[](#/definitions/RecordFilter "Direct link to /definitions/RecordFilter")
Filter applied on a list of records.
Properties:
* #### condition `string`
The predicate to filter a record. Records will be removed if evaluated to False.
Available variables:
* [config](#/variables/config)
* [next\_page\_token](#/variables/next_page_token)
* [record](#/variables/record)
* [stream\_interval](#/variables/stream_interval)
* [stream\_partition](#/variables/stream_partition)
* [stream\_slice](#/variables/stream_slice)
Examples:
```
{{ record['created_at'] >= stream_interval['start_time'] }}
```
```
{{ record.status in ['active', 'expired'] }}
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### RecordSelector `object`[](#/definitions/RecordSelector "Direct link to /definitions/RecordSelector")
Responsible for translating an HTTP response into a list of records by extracting records from the response and optionally filtering records based on a heuristic.
Properties:
* #### extractor
Type:
* [`#/definitions/DpathExtractor`](#/definitions/DpathExtractor)
* [`#/definitions/CustomRecordExtractor`](#/definitions/CustomRecordExtractor)
* #### record\_filter
Responsible for filtering records to be emitted by the Source.
Type:
* [`#/definitions/RecordFilter`](#/definitions/RecordFilter)
* [`#/definitions/CustomRecordFilter`](#/definitions/CustomRecordFilter)
* #### schema\_normalization
Responsible for normalization according to the schema.
Type:
* [`#/definitions/SchemaNormalization`](#/definitions/SchemaNormalization)
* [`#/definitions/CustomSchemaNormalization`](#/definitions/CustomSchemaNormalization)
* #### transform\_before\_filtering `boolean`
If true, transformation will be applied before record filtering.
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### SchemaNormalization `string`[](#/definitions/SchemaNormalization "Direct link to /definitions/SchemaNormalization")
Responsible for normalization according to the schema.
Examples:
```
Default
```
```
None
```
### RemoveFields `object`[](#/definitions/RemoveFields "Direct link to /definitions/RemoveFields")
A transformation which removes fields from a record. The fields removed are designated using FieldPointers. During transformation, if a field or any of its parents does not exist in the record, no error is thrown.
Properties:
* #### condition `string`
The predicate to filter a property by a property value. Property will be removed if it is empty OR expression is evaluated to True.,
Available variables:
* [config](#/variables/config)
* [property](#/variables/property)
* [parameters](#/variables/parameters)
Examples:
```
{{ property|string == '' }}
```
```
{{ property is integer }}
```
```
{{ property|length > 5 }}
```
```
{{ property == 'some_string_to_match' }}
```
* #### field\_pointers `array`
Array of paths defining the field to remove. Each item is an array whose field describe the path of a field to remove.
Examples:
```
[
"tags"
]
```
```
[
[
"content",
"html"
],
[
"content",
"plain_text"
]
]
```
### RequestPath `object`[](#/definitions/RequestPath "Direct link to /definitions/RequestPath")
The URL path to be used for the HTTP request.
Properties:
### RequestOption `object`[](#/definitions/RequestOption "Direct link to /definitions/RequestOption")
Specifies the key field or path and where in the request a component's value should be injected.
Properties:
* #### inject\_into
Configures where the descriptor should be set on the HTTP requests. Note that request parameters that are already encoded in the URL path will not be duplicated.
Examples:
```
request_parameter
```
```
header
```
```
body_data
```
```
body_json
```
* #### field\_name `string`
Configures which key should be used in the location that the descriptor is being injected into. We hope to eventually deprecate this field in favor of `field_path` for all request\_options, but must currently maintain it for backwards compatibility in the Builder.
Available variables:
* [config](#/variables/config)
* [parameters](#/variables/parameters)
Example:
```
segment_id
```
* #### field\_path `array`
Configures a path to be used for nested structures in JSON body requests (e.g. GraphQL queries)
Available variables:
* [config](#/variables/config)
* [parameters](#/variables/parameters)
Example:
```
[
"data",
"viewer",
"id"
]
```
### Schemas `object`[](#/definitions/Schemas "Direct link to /definitions/Schemas")
The stream schemas representing the shape of the data emitted by the stream.
### LegacySessionTokenAuthenticator `object`[](#/definitions/LegacySessionTokenAuthenticator "Direct link to /definitions/LegacySessionTokenAuthenticator")
Deprecated - use SessionTokenAuthenticator instead. Authenticator for requests authenticated using session tokens. A session token is a random value generated by a server to identify a specific user for the duration of one interaction session.
Properties:
* #### header `string`
The name of the session token header that will be injected in the request
Example:
```
X-Session
```
* #### login\_url `string`
Path of the login URL (do not include the base URL)
Example:
```
session
```
* #### session\_token `string`
Session token to use if using a pre-defined token. Not needed if authenticating with username + password pair
* #### session\_token\_response\_key `string`
Name of the key of the session token to be extracted from the response
Example:
```
id
```
* #### username `string`
Username used to authenticate and obtain a session token
Example:
```
{{ config['username'] }}
```
* #### password `string`
Password used to authenticate and obtain a session token
Examples:
```
{{ config['password'] }}
```
```
```
* #### validate\_session\_url `string`
Path of the URL to use to validate that the session token is valid (do not include the base URL)
Example:
```
user/current
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### StateDelegatingStream `object`[](#/definitions/StateDelegatingStream "Direct link to /definitions/StateDelegatingStream")
(This component is experimental. Use at your own risk.) Orchestrate the retriever's usage based on the state value.
Properties:
* #### name `string`
The stream name.
* #### full\_refresh\_stream `#/definitions/DeclarativeStream`
Component used to coordinate how records are extracted across stream slices and request pages when the state is empty or not provided.
* #### incremental\_stream `#/definitions/DeclarativeStream`
Component used to coordinate how records are extracted across stream slices and request pages when the state provided.
* #### api\_retention\_period `string`
The data retention period of the incremental API (ISO8601 duration). If the cursor value is older than this retention period, the connector will automatically fall back to a full refresh to avoid data loss. This is useful for APIs like Stripe Events API which only retain data for 30 days.
* **PT1H**: 1 hour
* **P1D**: 1 day
* **P1W**: 1 week
* **P1M**: 1 month
* **P1Y**: 1 year
* **P30D**: 30 days
Examples:
```
P30D
```
```
P90D
```
```
P1Y
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### SimpleRetriever `object`[](#/definitions/SimpleRetriever "Direct link to /definitions/SimpleRetriever")
Retrieves records by synchronously sending requests to fetch records. The retriever acts as an orchestrator between the requester, the record selector, the paginator, and the partition router.
Properties:
* #### requester
Requester component that describes how to prepare HTTP requests to send to the source API.
Type:
* [`#/definitions/HttpRequester`](#/definitions/HttpRequester)
* [`#/definitions/CustomRequester`](#/definitions/CustomRequester)
* #### decoder
Component decoding the response so records can be extracted.
Type:
* [`#/definitions/JsonDecoder`](#/definitions/JsonDecoder)
* [`#/definitions/JsonItemsDecoder`](#/definitions/JsonItemsDecoder)
* [`#/definitions/XmlDecoder`](#/definitions/XmlDecoder)
* [`#/definitions/CsvDecoder`](#/definitions/CsvDecoder)
* [`#/definitions/JsonlDecoder`](#/definitions/JsonlDecoder)
* [`#/definitions/GzipDecoder`](#/definitions/GzipDecoder)
* [`#/definitions/IterableDecoder`](#/definitions/IterableDecoder)
* [`#/definitions/ZipfileDecoder`](#/definitions/ZipfileDecoder)
* [`#/definitions/CustomDecoder`](#/definitions/CustomDecoder)
* #### record\_selector `#/definitions/RecordSelector`
Component that describes how to extract records from a HTTP response.
* #### paginator
Paginator component that describes how to navigate through the API's pages.
Type:
* [`#/definitions/DefaultPaginator`](#/definitions/DefaultPaginator)
* [`#/definitions/NoPagination`](#/definitions/NoPagination)
* #### pagination\_reset `#/definitions/PaginationReset`
Describes what triggers pagination reset and how to handle it.
* #### ignore\_stream\_slicer\_parameters\_on\_paginated\_requests `boolean`
If true, the partition router and incremental request options will be ignored when paginating requests. Request options set directly on the requester will not be ignored.
* #### partition\_router
Used to iteratively execute requests over a set of values, such as a parent stream's records or a list of constant values.
Type:
* [`#/definitions/SubstreamPartitionRouter`](#/definitions/SubstreamPartitionRouter)
* [`#/definitions/ListPartitionRouter`](#/definitions/ListPartitionRouter)
* [`#/definitions/GroupingPartitionRouter`](#/definitions/GroupingPartitionRouter)
* [`#/definitions/UnionPartitionRouter`](#/definitions/UnionPartitionRouter)
* [`#/definitions/CustomPartitionRouter`](#/definitions/CustomPartitionRouter)
* `array`
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### PaginationReset `object`[](#/definitions/PaginationReset "Direct link to /definitions/PaginationReset")
Describes what triggers pagination reset and how to handle it. If SPLIT\_USING\_CURSOR, the connector developer is accountable for ensuring that the records are returned in ascending order.
Properties:
* #### action `string`
* #### limits `#/definitions/PaginationResetLimits`
### PaginationResetLimits `object`[](#/definitions/PaginationResetLimits "Direct link to /definitions/PaginationResetLimits")
Describes the limits that trigger pagination reset
Properties:
* #### number\_of\_records `integer`
### GzipDecoder `object`[](#/definitions/GzipDecoder "Direct link to /definitions/GzipDecoder")
Select 'gzip' for response data that is compressed with gzip. Requires specifying an inner data type/decoder to parse the decompressed data.
Properties:
* #### decoder
Type:
* [`#/definitions/CsvDecoder`](#/definitions/CsvDecoder)
* [`#/definitions/GzipDecoder`](#/definitions/GzipDecoder)
* [`#/definitions/JsonDecoder`](#/definitions/JsonDecoder)
* [`#/definitions/JsonItemsDecoder`](#/definitions/JsonItemsDecoder)
* [`#/definitions/JsonlDecoder`](#/definitions/JsonlDecoder)
### CsvDecoder `object`[](#/definitions/CsvDecoder "Direct link to /definitions/CsvDecoder")
Select 'CSV' for response data that is formatted as CSV (comma-separated values). Can specify an encoding (default: 'utf-8') and a delimiter (default: ',').
Properties:
* #### encoding `string`
* #### delimiter `string`
* #### set\_values\_to\_none `array`
### AsyncJobStatusMap `object`[](#/definitions/AsyncJobStatusMap "Direct link to /definitions/AsyncJobStatusMap")
Matches the api job status to Async Job Status.
Properties:
* #### running `array`
* #### completed `array`
* #### failed `array`
* #### timeout `array`
* #### skipped `array`
Statuses that indicate the job was skipped because there is no data to return. Jobs with these statuses will not be retried and no records will be fetched.
### AsyncRetriever `object`[](#/definitions/AsyncRetriever "Direct link to /definitions/AsyncRetriever")
Retrieves records by Asynchronously sending requests to fetch records. The retriever acts as an orchestrator between the requester, the record selector, the paginator, and the partition router.
Properties:
* #### record\_selector `#/definitions/RecordSelector`
Component that describes how to extract records from a HTTP response.
* #### status\_mapping
Async Job Status to Airbyte CDK Async Job Status mapping.
Type:
* [`#/definitions/AsyncJobStatusMap`](#/definitions/AsyncJobStatusMap)
* #### status\_extractor
Responsible for fetching the actual status of the async job.
Type:
* [`#/definitions/DpathExtractor`](#/definitions/DpathExtractor)
* [`#/definitions/CustomRecordExtractor`](#/definitions/CustomRecordExtractor)
* #### download\_target\_extractor
Responsible for fetching the final result `urls` provided by the completed / finished / ready async job.
Type:
* [`#/definitions/DpathExtractor`](#/definitions/DpathExtractor)
* [`#/definitions/CustomRecordExtractor`](#/definitions/CustomRecordExtractor)
* #### download\_extractor
Responsible for fetching the records from provided urls.
Type:
* [`#/definitions/DpathExtractor`](#/definitions/DpathExtractor)
* [`#/definitions/CustomRecordExtractor`](#/definitions/CustomRecordExtractor)
* [`#/definitions/ResponseToFileExtractor`](#/definitions/ResponseToFileExtractor)
* #### creation\_requester
Requester component that describes how to prepare HTTP requests to send to the source API to create the async server-side job.
Type:
* [`#/definitions/HttpRequester`](#/definitions/HttpRequester)
* [`#/definitions/CustomRequester`](#/definitions/CustomRequester)
* #### polling\_requester
Requester component that describes how to prepare HTTP requests to send to the source API to fetch the status of the running async job.
Type:
* [`#/definitions/HttpRequester`](#/definitions/HttpRequester)
* [`#/definitions/CustomRequester`](#/definitions/CustomRequester)
* #### polling\_job\_timeout
The time in minutes after which the single Async Job should be considered as Timed Out.
Type:
* `integer`
* `string`
Available variables:
* [config](#/variables/config)
* #### failed\_retry\_wait\_time\_in\_seconds
Time in seconds to wait before retrying a failed async job. Only applies to jobs that ran on the API side and reported a FAILED status (e.g. report generation failed due to a cooldown). Creation failures (HTTP errors when starting a job, such as 429s) and TIMED\_OUT jobs are retried immediately and are not affected by this setting. When set, the orchestrator defers retry of real failed jobs until the wait time has elapsed, without blocking other jobs.
Type:
* `integer`
* `string`
Available variables:
* [config](#/variables/config)
* #### download\_target\_requester
Requester component that describes how to prepare HTTP requests to send to the source API to extract the url from polling response by the completed async job.
Type:
* [`#/definitions/HttpRequester`](#/definitions/HttpRequester)
* [`#/definitions/CustomRequester`](#/definitions/CustomRequester)
* #### download\_requester
Requester component that describes how to prepare HTTP requests to send to the source API to download the data provided by the completed async job.
Type:
* [`#/definitions/HttpRequester`](#/definitions/HttpRequester)
* [`#/definitions/CustomRequester`](#/definitions/CustomRequester)
* #### download\_paginator
Paginator component that describes how to navigate through the API's pages during download.
Type:
* [`#/definitions/DefaultPaginator`](#/definitions/DefaultPaginator)
* [`#/definitions/NoPagination`](#/definitions/NoPagination)
* #### abort\_requester
Requester component that describes how to prepare HTTP requests to send to the source API to abort a job once it is timed out from the source's perspective.
Type:
* [`#/definitions/HttpRequester`](#/definitions/HttpRequester)
* [`#/definitions/CustomRequester`](#/definitions/CustomRequester)
* #### delete\_requester
Requester component that describes how to prepare HTTP requests to send to the source API to delete a job once the records are extracted.
Type:
* [`#/definitions/HttpRequester`](#/definitions/HttpRequester)
* [`#/definitions/CustomRequester`](#/definitions/CustomRequester)
* #### partition\_router
PartitionRouter component that describes how to partition the stream, enabling incremental syncs and checkpointing.
Type:
* [`#/definitions/ListPartitionRouter`](#/definitions/ListPartitionRouter)
* [`#/definitions/SubstreamPartitionRouter`](#/definitions/SubstreamPartitionRouter)
* [`#/definitions/GroupingPartitionRouter`](#/definitions/GroupingPartitionRouter)
* [`#/definitions/UnionPartitionRouter`](#/definitions/UnionPartitionRouter)
* [`#/definitions/CustomPartitionRouter`](#/definitions/CustomPartitionRouter)
* `array`
* #### decoder
Component decoding the response so records can be extracted.
Type:
* [`#/definitions/CsvDecoder`](#/definitions/CsvDecoder)
* [`#/definitions/GzipDecoder`](#/definitions/GzipDecoder)
* [`#/definitions/JsonDecoder`](#/definitions/JsonDecoder)
* [`#/definitions/JsonItemsDecoder`](#/definitions/JsonItemsDecoder)
* [`#/definitions/JsonlDecoder`](#/definitions/JsonlDecoder)
* [`#/definitions/IterableDecoder`](#/definitions/IterableDecoder)
* [`#/definitions/XmlDecoder`](#/definitions/XmlDecoder)
* [`#/definitions/ZipfileDecoder`](#/definitions/ZipfileDecoder)
* [`#/definitions/CustomDecoder`](#/definitions/CustomDecoder)
* #### download\_decoder
Component decoding the download response so records can be extracted.
Type:
* [`#/definitions/CsvDecoder`](#/definitions/CsvDecoder)
* [`#/definitions/GzipDecoder`](#/definitions/GzipDecoder)
* [`#/definitions/JsonDecoder`](#/definitions/JsonDecoder)
* [`#/definitions/JsonItemsDecoder`](#/definitions/JsonItemsDecoder)
* [`#/definitions/JsonlDecoder`](#/definitions/JsonlDecoder)
* [`#/definitions/IterableDecoder`](#/definitions/IterableDecoder)
* [`#/definitions/XmlDecoder`](#/definitions/XmlDecoder)
* [`#/definitions/ZipfileDecoder`](#/definitions/ZipfileDecoder)
* [`#/definitions/CustomDecoder`](#/definitions/CustomDecoder)
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### Spec `object`[](#/definitions/Spec "Direct link to /definitions/Spec")
A source specification made up of connector metadata and how it can be configured.
Properties:
* #### connection\_specification `object`
A connection specification describing how a the connector can be configured.
* #### documentation\_url `string`
URL of the connector's documentation page.
Example:
```
https://docs.airbyte.com/integrations/sources/dremio
```
* #### advanced\_auth `#/definitions/AuthFlow`
Advanced specification for configuring the authentication flow.
* #### config\_normalization\_rules `object`
### ConfigMigration `object`[](#/definitions/ConfigMigration "Direct link to /definitions/ConfigMigration")
A config migration that will be applied on the incoming config at the start of a sync.
Properties:
* #### description `string`
The description/purpose of the config migration.
* #### transformations `array`
The list of transformations that will attempt to be applied on an incoming unmigrated config. The transformations will be applied in the order they are defined.
### StreamGroup `object`[](#/definitions/StreamGroup "Direct link to /definitions/StreamGroup")
A group of streams that share a common resource and should not be read simultaneously. Streams in the same group will be blocked from concurrent reads based on the specified action.
Properties:
* #### streams `array`
List of references to streams that belong to this group.
* #### action `#/definitions/BlockSimultaneousSyncsAction`
The action to apply to streams in this group.
### BlockSimultaneousSyncsAction `object`[](#/definitions/BlockSimultaneousSyncsAction "Direct link to /definitions/BlockSimultaneousSyncsAction")
Action that prevents streams in the same group from being read concurrently. When applied to a stream group, streams with this action will be deferred if another stream in the same group is currently active. This is useful for APIs that don't allow concurrent access to the same endpoint or session. Only applies to ConcurrentDeclarativeSource.
Properties:
### SubstreamPartitionRouter `object`[](#/definitions/SubstreamPartitionRouter "Direct link to /definitions/SubstreamPartitionRouter")
Partition router that is used to retrieve records that have been partitioned according to records from the specified parent streams. An example of a parent stream is automobile brands and the substream would be the various car models associated with each branch.
Properties:
* #### parent\_stream\_configs `array` `#/definitions/ParentStreamConfig`
Specifies which parent streams are being iterated over and how parent records should be used to partition the child stream data set.
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### ValueType `string`[](#/definitions/ValueType "Direct link to /definitions/ValueType")
A schema type.
### WaitTimeFromHeader `object`[](#/definitions/WaitTimeFromHeader "Direct link to /definitions/WaitTimeFromHeader")
Extract wait time from a HTTP header in the response.
Properties:
* #### header `string`
The name of the response header defining how long to wait before retrying.
Available variables:
* [config](#/variables/config)
Example:
```
Retry-After
```
* #### regex `string`
Optional regex to apply on the header to extract its value. The regex should define a capture group defining the wait time.
Example:
```
([-+]?\d+)
```
* #### max\_waiting\_time\_in\_seconds
Stop the stream instead of waiting, when the value extracted from the header is greater than or equal to this value. Can be a hardcoded number, or a string interpolated from the connector config so that the bound can be changed without a connector release. A value of 0 means never wait. Not evaluated when a rate-limited retry can rotate to another credential with quota; any other retryable error still consults this strategy.
Type:
* `number`
* `string`
Available variables:
* [config](#/variables/config)
Examples:
```
3600
```
```
{{ config['max_waiting_time'] * 60 }}
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### GroupingPartitionRouter `object`[](#/definitions/GroupingPartitionRouter "Direct link to /definitions/GroupingPartitionRouter")
A decorator on top of a partition router that groups partitions into batches of a specified size. This is useful for APIs that support filtering by multiple partition keys in a single request. Note that per-partition incremental syncs may not work as expected because the grouping of partitions might change between syncs, potentially leading to inconsistent state tracking.
Properties:
* #### group\_size `integer`
The number of partitions to include in each group. This determines how many partition values are batched together in a single slice.
Examples:
```
10
```
```
50
```
* #### underlying\_partition\_router
The partition router whose output will be grouped. This can be any valid partition router component.
Type:
* [`#/definitions/ListPartitionRouter`](#/definitions/ListPartitionRouter)
* [`#/definitions/SubstreamPartitionRouter`](#/definitions/SubstreamPartitionRouter)
* [`#/definitions/UnionPartitionRouter`](#/definitions/UnionPartitionRouter)
* [`#/definitions/CustomPartitionRouter`](#/definitions/CustomPartitionRouter)
* #### deduplicate `boolean`
If true, ensures that partitions are unique within each group by removing duplicates based on the partition key.
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### UnionPartitionRouter `object`[](#/definitions/UnionPartitionRouter "Direct link to /definitions/UnionPartitionRouter")
A partition router that yields the deduplicated union of the partitions produced by its child partition routers. Every emitted partition is normalized to a single key defined by `partition_field`; any other partition keys coming from a child router (such as a SubstreamPartitionRouter's `parent_slice`) are moved into the slice's extra fields. The first occurrence of a partition value wins and later duplicates are skipped. Partition values must be hashable; deduplication holds every distinct value in memory for the duration of the sync, and the resulting partition count is the deduplicated sum of the child routers' partition counts, which affects per-partition state size. Because child routers must emit scalar partition values, GroupingPartitionRouter (which emits list-valued partitions) cannot be used as a child.
Properties:
* #### partition\_field `string`
The single partition key that all child partition routers' slices are normalized to. Each child router must emit this key in its partitions. Interpolation is evaluated once when the connector is built, using the connector config and $parameters.
Available variables:
* [config](#/variables/config)
* [parameters](#/variables/parameters)
Examples:
```
repository
```
```
{{ config['partition_field'] }}
```
* #### partition\_routers `array`
The child partition routers whose partitions are unioned. Request options are not supported on child partition routers; partition values should be consumed via interpolation (e.g. `stream_partition`).
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### WaitUntilTimeFromHeader `object`[](#/definitions/WaitUntilTimeFromHeader "Direct link to /definitions/WaitUntilTimeFromHeader")
Extract time at which we can retry the request from response header and wait for the difference between now and that time.
Properties:
* #### header `string`
The name of the response header defining how long to wait before retrying.
Available variables:
* [config](#/variables/config)
Example:
```
wait_time
```
* #### min\_wait
Minimum time to wait before retrying.
Type:
* `number`
* `string`
Available variables:
* [config](#/variables/config)
Examples:
```
10
```
```
60
```
* #### regex `string`
Optional regex to apply on the header to extract its value. The regex should define a capture group defining the wait time.
Available variables:
* [config](#/variables/config)
Example:
```
([-+]?\d+)
```
* #### max\_waiting\_time\_in\_seconds
Stop the stream instead of waiting, when the wait this strategy computes is greater than or equal to this value. The comparison is against the computed wait rather than the raw header, since the header holds an absolute timestamp, and it is applied after `min_wait`, so a cap below the floor still wins -- including for the fallback where the header is absent and `min_wait` supplies the wait on its own. Can be a hardcoded number, or a string interpolated from the connector config so that the bound can be changed without a connector release. A value of 0 means never wait. Not evaluated when a rate-limited retry can rotate to another credential with quota; any other retryable error still consults this strategy.
Type:
* `number`
* `string`
Available variables:
* [config](#/variables/config)
Examples:
```
3600
```
```
{{ config['max_waiting_time'] * 60 }}
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### ComponentMappingDefinition `object`[](#/definitions/ComponentMappingDefinition "Direct link to /definitions/ComponentMappingDefinition")
(This component is experimental. Use at your own risk.) Specifies a mapping definition to update or add fields in a record or configuration. This allows dynamic mapping of data by interpolating values into the template based on provided contexts.
Properties:
* #### field\_path `array`
A list of potentially nested fields indicating the full path where value will be added or updated.
Available variables:
* [config](#/variables/config)
* [components\_values](#/variables/components_values)
* [stream\_slice](#/variables/stream_slice)
* [stream\_template\_config](#/variables/stream_template_config)
Examples:
```
[
"name"
]
```
```
[
"retriever",
"requester",
"url"
]
```
```
[
"retriever",
"requester",
"{{ components_values.field }}"
]
```
```
[
"*",
"**",
"name"
]
```
* #### value `string`
The dynamic or static value to assign to the key. Interpolated values can be used to dynamically determine the value during runtime.
Available variables:
* [config](#/variables/config)
* [stream\_template\_config](#/variables/stream_template_config)
* [components\_values](#/variables/components_values)
* [stream\_slice](#/variables/stream_slice)
Examples:
```
{{ components_values['updates'] }}
```
```
{{ components_values['MetaData']['LastUpdatedTime'] }}
```
```
{{ config['segment_id'] }}
```
```
{{ stream_slice['parent_id'] }}
```
```
{{ stream_slice['extra_fields']['name'] }}
```
* #### value\_type `#/definitions/ValueType`
The expected data type of the value. If omitted, the type will be inferred from the value provided.
* #### create\_or\_update `boolean`
Determines whether to create a new path if it doesn't exist (true) or only update existing paths (false). When set to true, the resolver will create new paths in the stream template if they don't exist. When false (default), it will only update existing paths.
* #### condition `string`
A condition that must be met for the mapping to be applied. This property is only supported for `ConfigComponentsResolver`.
Available variables:
* [config](#/variables/config)
* [stream\_template\_config](#/variables/stream_template_config)
* [components\_values](#/variables/components_values)
* [stream\_slice](#/variables/stream_slice)
Examples:
```
{{ components_values.get('cursor_field', None) }}
```
```
{{ '_incremental' in components_values.get('stream_name', '') }}
```
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### HttpComponentsResolver `object`[](#/definitions/HttpComponentsResolver "Direct link to /definitions/HttpComponentsResolver")
(This component is experimental. Use at your own risk.) Component resolve and populates stream templates with components fetched via an HTTP retriever.
Properties:
* #### retriever
Component used to coordinate how records are extracted across stream slices and request pages.
Type:
* [`#/definitions/SimpleRetriever`](#/definitions/SimpleRetriever)
* [`#/definitions/AsyncRetriever`](#/definitions/AsyncRetriever)
* [`#/definitions/CustomRetriever`](#/definitions/CustomRetriever)
* #### components\_mapping `array` `#/definitions/ComponentMappingDefinition`
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### StreamConfig `object`[](#/definitions/StreamConfig "Direct link to /definitions/StreamConfig")
(This component is experimental. Use at your own risk.) Describes how to get streams config from the source config.
Properties:
* #### configs\_pointer `array`
A list of potentially nested fields indicating the full path in source config file where streams configs located.
Available variables:
* [parameters](#/variables/parameters)
Examples:
```
[
"data"
]
```
```
[
"data",
"streams"
]
```
```
[
"data",
"{{ parameters.name }}"
]
```
* #### default\_values `array`
A list of default values, each matching the structure expected from the parsed component value.
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### ConfigComponentsResolver `object`[](#/definitions/ConfigComponentsResolver "Direct link to /definitions/ConfigComponentsResolver")
(This component is experimental. Use at your own risk.) Resolves and populates stream templates with components fetched from the source config.
Properties:
* #### stream\_config
Type:
* `array`
* [`#/definitions/StreamConfig`](#/definitions/StreamConfig)
* #### components\_mapping `array` `#/definitions/ComponentMappingDefinition`
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### StreamParametersDefinition `object`[](#/definitions/StreamParametersDefinition "Direct link to /definitions/StreamParametersDefinition")
(This component is experimental. Use at your own risk.) Represents a stream parameters definition to set up dynamic streams from defined values in manifest.
Properties:
* #### list\_of\_parameters\_for\_stream `array`
A list of object of parameters for stream, each object in the list represents params for one stream.
Example:
```
[
{
"name": "test stream",
"$parameters": {
"entity": "test entity"
},
"primary_key": "test key"
}
]
```
### ParametrizedComponentsResolver `object`[](#/definitions/ParametrizedComponentsResolver "Direct link to /definitions/ParametrizedComponentsResolver")
(This component is experimental. Use at your own risk.) Resolves and populates dynamic streams from defined parametrized values in manifest.
Properties:
* #### stream\_parameters `#/definitions/StreamParametersDefinition`
* #### components\_mapping `array` `#/definitions/ComponentMappingDefinition`
* #### $parameters `object`
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
### DynamicDeclarativeStream `object`[](#/definitions/DynamicDeclarativeStream "Direct link to /definitions/DynamicDeclarativeStream")
(This component is experimental. Use at your own risk.) A component that described how will be created declarative streams based on stream template.
Properties:
* #### name `string`
The dynamic stream name.
* #### stream\_template
Reference to the stream template.
Type:
* [`#/definitions/DeclarativeStream`](#/definitions/DeclarativeStream)
* [`#/definitions/StateDelegatingStream`](#/definitions/StateDelegatingStream)
* #### components\_resolver
Component resolve and populates stream templates with components values.
Type:
* [`#/definitions/HttpComponentsResolver`](#/definitions/HttpComponentsResolver)
* [`#/definitions/ConfigComponentsResolver`](#/definitions/ConfigComponentsResolver)
* [`#/definitions/ParametrizedComponentsResolver`](#/definitions/ParametrizedComponentsResolver)
* #### use\_parent\_parameters `boolean`
Whether or not to prioritize parent parameters over component parameters when constructing dynamic streams. Defaults to true for backward compatibility.
### RequestBodyPlainText `object`[](#/definitions/RequestBodyPlainText "Direct link to /definitions/RequestBodyPlainText")
Request body value is sent as plain text
Properties:
* #### value `string`
### RequestBodyUrlEncodedForm `object`[](#/definitions/RequestBodyUrlEncodedForm "Direct link to /definitions/RequestBodyUrlEncodedForm")
Request body value is converted into a url-encoded form
Properties:
* #### value `object`
### RequestBodyJsonObject `object`[](#/definitions/RequestBodyJsonObject "Direct link to /definitions/RequestBodyJsonObject")
Request body value converted into a JSON object
Properties:
* #### value `object`
### RequestBodyGraphQL `object`[](#/definitions/RequestBodyGraphQL "Direct link to /definitions/RequestBodyGraphQL")
Request body value converted into a GraphQL query object
Properties:
* #### value `#/definitions/RequestBodyGraphQlQuery`
### RequestBodyGraphQlQuery `object`[](#/definitions/RequestBodyGraphQlQuery "Direct link to /definitions/RequestBodyGraphQlQuery")
Request body GraphQL query object
Properties:
* #### query `string`
The GraphQL query to be executed
### DpathValidator `object`[](#/definitions/DpathValidator "Direct link to /definitions/DpathValidator")
Validator that extracts the value located at a given field path.
Properties:
* #### field\_path `array`
List of potentially nested fields describing the full path of the field to validate. Use "\*" to validate all values from an array.
Available variables:
* [config](#/variables/config)
Examples:
```
[
"data"
]
```
```
[
"data",
"records"
]
```
```
[
"data",
"{{ parameters.name }}"
]
```
```
[
"data",
"*",
"record"
]
```
* #### validation\_strategy
The condition that the specified config value will be evaluated against
Type:
* [`#/definitions/ValidateAdheresToSchema`](#/definitions/ValidateAdheresToSchema)
* [`#/definitions/CustomValidationStrategy`](#/definitions/CustomValidationStrategy)
### PredicateValidator `object`[](#/definitions/PredicateValidator "Direct link to /definitions/PredicateValidator")
Validator that applies a validation strategy to a specified value.
Properties:
* #### value `stringnumberobjectarraybooleannull`
The value to be validated. Can be a literal value or interpolated from configuration.
Available variables:
* [config](#/variables/config)
Examples:
```
test-value
```
```
{{ config['api_version'] }}
```
```
{{ config['tenant_id'] }}
```
```
123
```
* #### validation\_strategy
The validation strategy to apply to the value.
Type:
* [`#/definitions/ValidateAdheresToSchema`](#/definitions/ValidateAdheresToSchema)
* [`#/definitions/CustomValidationStrategy`](#/definitions/CustomValidationStrategy)
### ValidateAdheresToSchema `object`[](#/definitions/ValidateAdheresToSchema "Direct link to /definitions/ValidateAdheresToSchema")
Validates that a user-provided schema adheres to a specified JSON schema.
Properties:
* #### base\_schema `stringobject`
The base JSON schema against which the user-provided schema will be validated.
Available variables:
* [config](#/variables/config)
Examples:
```
{{ config['report_validation_schema'] }}
```
```
'{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Person",
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "The person's name"
},
"age": {
"type": "integer",
"minimum": 0,
"description": "The person's age"
}
},
"required": ["name", "age"]
}'
```
```
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Person",
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "The person's name"
},
"age": {
"type": "integer",
"minimum": 0,
"description": "The person's age"
}
},
"required": [
"name",
"age"
]
}
```
### CustomValidationStrategy `object`[](#/definitions/CustomValidationStrategy "Direct link to /definitions/CustomValidationStrategy")
Custom validation strategy that allows for custom validation logic.
Properties:
* #### class\_name `string`
Fully-qualified name of the class that will be implementing the custom validation strategy. Has to be a sub class of ValidationStrategy. The format is `source_..`.
Example:
```
source_declarative_manifest.components.MyCustomValidationStrategy
```
### ConfigRemapField `object`[](#/definitions/ConfigRemapField "Direct link to /definitions/ConfigRemapField")
Transformation that remaps a field's value to another value based on a static map.
Properties:
* #### map `objectstring`
A mapping of original values to new values. When a field value matches a key in this map, it will be replaced with the corresponding value.
Available variables:
* [config](#/variables/config)
Examples:
```
{
"pending": "in_progress",
"done": "completed",
"cancelled": "terminated"
}
```
```
{{ config['status_mapping'] }}
```
* #### field\_path `array`
The path to the field whose value should be remapped. Specified as a list of path components to navigate through nested objects.
Available variables:
* [config](#/variables/config)
Examples:
```
[
"status"
]
```
```
[
"data",
"status"
]
```
```
[
"data",
"{{ config.name }}",
"status"
]
```
```
[
"data",
"*",
"status"
]
```
### ConfigAddFields `object`[](#/definitions/ConfigAddFields "Direct link to /definitions/ConfigAddFields")
Transformation that adds fields to a config. The path of the added field can be nested.
Properties:
* #### fields `array` `#/definitions/AddedFieldDefinition`
A list of transformations (path and corresponding value) that will be added to the config.
* #### condition `string`
Fields will be added if expression is evaluated to True.
Available variables:
* [config](#/variables/config)
* [property](#/variables/property)
Examples:
```
{{ config['environemnt'] == 'sandbox' }}
```
```
{{ property is integer }}
```
```
{{ property|length > 5 }}
```
```
{{ property == 'some_string_to_match' }}
```
### ConfigRemoveFields `object`[](#/definitions/ConfigRemoveFields "Direct link to /definitions/ConfigRemoveFields")
Transformation that removes a field from the config.
Properties:
* #### field\_pointers `array`
A list of field pointers to be removed from the config.
Examples:
```
[
"tags"
]
```
```
[
[
"content",
"html"
],
[
"content",
"plain_text"
]
]
```
* #### condition `string`
Fields will be removed if expression is evaluated to True.
Available variables:
* [config](#/variables/config)
* [property](#/variables/property)
Examples:
```
{{ config['environemnt'] == 'sandbox' }}
```
```
{{ property is integer }}
```
```
{{ property|length > 5 }}
```
```
{{ property == 'some_string_to_match' }}
```
### CustomConfigTransformation `object`[](#/definitions/CustomConfigTransformation "Direct link to /definitions/CustomConfigTransformation")
A custom config transformation that can be used to transform the connector configuration.
Properties:
* #### class\_name `string`
Fully-qualified name of the class that will be implementing the custom config transformation. The format is `source_..`.
Example:
```
source_declarative_manifest.components.MyCustomConfigTransformation
```
* #### $parameters `object`
Additional parameters to be passed to the custom config transformation.
Set parameters that are inherited to all children. See the [section in the advanced topics](/connector-development/config-based/advanced-topics/parameters) for more details.
## Interpolation variables[](#variables "Direct link to Interpolation variables")
All string properties that list out available variables allow [jinja expressions](https://jinja.palletsprojects.com/en/3.0.x/templates/#expressions). They can be used by placing them in double curly braces: `{{ config.property }}`. The following variables are available
### config `object`[](#/variables/config "Direct link to /variables/config")
The connector configuration. The object's keys are the same as the the keys defined in the connection specification.
Example:
```
{
"start_date": "2010-01-01",
"api_key": "*****"
}
```
### parameters `object`[](#/variables/parameters "Direct link to /variables/parameters")
Additional runtime parameters, to be used for string interpolation. Parameters can be passed down from a parent component to its subcomponents using the $parameters key. This can be used to avoid repetitions.
Example:
```
{
"path": "automations",
"data_export_path": "automations",
"cursor_field": "updated_at"
}
```
### headers `object`[](#/variables/headers "Direct link to /variables/headers")
The HTTP headers from the last response received from the API. The object's keys are the header names from the response.
Example:
```
{
"Server": "nginx",
"Date": "Mon, 24 Apr 2023 20:17:21 GMT",
"Content-Type": "application/json",
"Content-Length": "420",
"Connection": "keep-alive",
"referrer-policy": "strict-origin-when-cross-origin",
"x-content-type-options": "nosniff",
"x-ratelimit-limit": "600",
"x-ratelimit-remaining": "598",
"x-ratelimit-reset": "39"
}
```
### last\_record `object`[](#/variables/last_record "Direct link to /variables/last_record")
Last record extracted from the response received from the API.
Example:
```
{
"name": "Test List: 19",
"id": "0236d6d2",
"contact_count": 20,
"_metadata": {
"self": "https://api.sendgrid.com/v3/marketing/lists/0236d6d2"
}
}
```
### last\_page\_size `object`[](#/variables/last_page_size "Direct link to /variables/last_page_size")
Number of records extracted from the last response received from the API.
Example:
```
2
```
### next\_page\_token `object`[](#/variables/next_page_token "Direct link to /variables/next_page_token")
Object describing the token to fetch the next page of records. The object has a single key "next\_page\_token".
Examples:
```
{
"next_page_token": 3
}
```
```
{
"next_page_token": "https://api.sendgrid.com/v3/marketing/lists/0236d6d2-75d2-42c5-962d-603e0deaf8d1"
}
```
### record `object`[](#/variables/record "Direct link to /variables/record")
The record being processed. The object's keys are the same keys as the records produced by the RecordSelector.
### response `object`[](#/variables/response "Direct link to /variables/response")
The body of the last response received from the API. The object's keys are the same keys as the response body's.
Example:
```
{
"result": [
{
"name": "Test List: 19",
"id": "0236d6d2-75d2-42c5-962d-603e0deaf8d1",
"contact_count": 20,
"_metadata": {
"self": "https://api.sendgrid.com/v3/marketing/lists/0236d6d2"
}
}
],
"_metadata": {
"self": "https://api.sendgrid.com/v3/marketing/lists?page_size=1&page_token=",
"next": "https://api.sendgrid.com/v3/marketing/lists?page_size=1&page_token=0236d6d2",
"count": 82
}
}
```
### creation\_response `object`[](#/variables/creation_response "Direct link to /variables/creation_response")
The response received from the creation\_requester in the AsyncRetriever component.
Example:
```
{
"id": "1234"
}
```
### polling\_response `object`[](#/variables/polling_response "Direct link to /variables/polling_response")
The response received from the polling\_requester in the AsyncRetriever component.
Example:
```
{
"id": "1234"
}
```
### download\_target `string`[](#/variables/download_target "Direct link to /variables/download_target")
The `URL` received from the polling\_requester in the AsyncRetriever with jobStatus as `COMPLETED`.
Example:
```
https://api.sendgrid.com/v3/marketing/lists?page_size=1&page_token=0236d6d2&filename=xxx_yyy_zzz.csv
```
### stream\_interval `object`[](#/variables/stream_interval "Direct link to /variables/stream_interval")
The current stream interval being processed. The keys are defined by the incremental sync component. Default keys are `start_time` and `end_time`.
Example:
```
{
"start_time": "2020-01-01 00:00:00.000+00:00",
"end_time": "2020-01-02 00:00:00.000+00:00"
}
```
### stream\_partition `object`[](#/variables/stream_partition "Direct link to /variables/stream_partition")
The current stream partition being processed. The keys are defined by the partition router component.
Examples:
```
{
"survey_id": 1234
}
```
```
{
"strategy": "DESKTOP"
}
```
```
{
"survey_id": 1234,
"strategy": "MOBILE"
}
```
### stream\_slice `object`[](#/variables/stream_slice "Direct link to /variables/stream_slice")
This variable is deprecated. Use stream\_interval or stream\_partition instead.
### components\_values `object`[](#/variables/components_values "Direct link to /variables/components_values")
The record object produced by the components resolver for which a stream will be generated.
Example:
```
{
"name": "accounts",
"id": 1234
}
```
## Interpolation macros[](#macros "Direct link to Interpolation macros")
Besides referencing variables, the following macros can be called as part of [jinja expressions](https://jinja.palletsprojects.com/en/3.0.x/templates/#expressions), for example like this: `{{ now_utc() }}`.
### now\_utc[](#/macros/now_utc "Direct link to now_utc")
Returns the current date and time in the UTC timezone.
Examples:
```
'{{ now_utc() }}' -> '2021-09-01 00:00:00+00:00'
```
```
'{{ now_utc().strftime('%Y-%m-%d') }}' -> '2021-09-01'
```
### today\_utc[](#/macros/today_utc "Direct link to today_utc")
Returns the current date in UTC timezone. The output is a date object.
Examples:
```
'{{ today_utc() }}' -> '2021-09-01'
```
```
'{{ today_utc().strftime('%Y/%m/%d')}}' -> '2021/09/01'
```
### timestamp[](#/macros/timestamp "Direct link to timestamp")
Converts a number or a string representing a datetime (formatted as ISO8601) to a timestamp. If the input is a number, it is converted to an int. If no timezone is specified, the string is interpreted as UTC.
Arguments:
* `datetime`:
A string formatted as ISO8601 or an integer representing a unix timestamp
Examples:
```
'{{ timestamp(1646006400) }}' -> 1646006400
```
```
'{{ timestamp('2022-02-28') }}' -> 1646006400
```
```
'{{ timestamp('2022-02-28T00:00:00Z') }}' -> 1646006400
```
```
'{{ timestamp('2022-02-28 00:00:00Z') }}' -> 1646006400
```
```
'{{ timestamp('2022-02-28T00:00:00-08:00') }}' -> 1646035200
```
### max[](#/macros/max "Direct link to max")
Returns the largest object of a iterable, or or two or more arguments.
Arguments:
* `args`:
iterable or a sequence of two or more arguments
Examples:
```
'{{ max(2, 3) }}' -> 3
```
```
'{{ max([2, 3]) }}' -> 3
```
### day\_delta[](#/macros/day_delta "Direct link to day_delta")
Returns the datetime of now() + num\_days.
Arguments:
* `num_days`:
The number of days to add to now
* `format`:
How to format the output string
Examples:
```
'{{ day_delta(1) }}' -> '2021-09-02T00:00:00.000000+0000'
```
```
'{{ day_delta(-1) }}' -> '2021-08-31:00:00.000000+0000'
```
```
'{{ day_delta(25, format='%Y-%m-%d') }}' -> '2021-09-02'
```
### duration[](#/macros/duration "Direct link to duration")
Converts an ISO8601 duration to datetime timedelta.
Arguments:
* `duration_string`:
A string representing an ISO8601 duration. See https\://www\.digi.com/resources/documentation/digidocs//90001488-13/reference/r\_iso\_8601\_duration\_format.htm for more details.
Examples:
```
'{{ duration('P1D') }}' -> '1 day, 0:00:00'
```
```
'{{ duration('P6DT23H') }}' -> '6 days, 23:00:00'
```
```
'{{ (now_utc() - duration('P1D')).strftime('%Y-%m-%dT%H:%M:%SZ') }}' -> '2021-08-31T00:00:00Z'
```
### format\_datetime[](#/macros/format_datetime "Direct link to format_datetime")
Converts a datetime or a datetime-string to the specified format.
Arguments:
* `datetime`:
The datetime object or a string to convert. If datetime is a string, it must be formatted as ISO8601.
* `format`:
The datetime format.
* `input_format`:
(optional) The datetime format in the case it is an string.
Examples:
```
{{ format_datetime(config['start_time'], '%Y-%m-%d') }}
```
```
{{ format_datetime(config['start_date'], '%Y-%m-%dT%H:%M:%S.%fZ') }}
```
```
{{ format_datetime(config['start_date'], '%Y-%m-%dT%H:%M:%S.%fZ', '%a, %d %b %Y %H:%M:%S %z') }}
```
### str\_to\_datetime[](#/macros/str_to_datetime "Direct link to str_to_datetime")
Converts a string to a datetime object with UTC timezone.
Arguments:
* `s`:
The string to convert.
Examples:
```
{{ str_to_datetime('2022-01-14') }}
```
```
{{ str_to_datetime('2022-01-01 13:45:30') }}
```
```
{{ str_to_datetime('2022-01-01T13:45:30+00:00') }}
```
```
{{ str_to_datetime('2022-01-01T13:45:30.123456Z') }}
```
## Interpolation filters[](#filters "Direct link to Interpolation filters")
The following filters can be called as part of [jinja expressions](https://jinja.palletsprojects.com/en/3.0.x/templates/#expressions), for example like this: `{{ 1 | string }}`.
### hash[](#/filters/hash "Direct link to hash")
Convert the specified value to a hashed string.
Arguments:
* `hash_type`:
Valid hash type for converts ('md5' as default value).
* `salt`:
An additional value to further protect sensitive data.
Examples:
```
{{ 'Test client_secret' | hash() }} -> '3032d57a12f76b61a820e47b9a5a0cbb'
```
```
{{ 'Test client_secret' | hash('md5') }} -> '3032d57a12f76b61a820e47b9a5a0cbb'
```
```
{{ 'Test client_secret' | hash('md5', salt='salt') }} -> '5011a0168579c2d94cbbe1c6ad14327c'
```
### base64encode[](#/filters/base64encode "Direct link to base64encode")
Convert the specified value to a string in the base64 format.
Example:
```
{{ 'Test client_secret' | base64encode }} -> 'VGVzdCBjbGllbnRfc2VjcmV0'
```
### base64decode[](#/filters/base64decode "Direct link to base64decode")
Decodes the specified base64 format value into a common string.
Example:
```
{{ 'ZmFrZSByZWZyZXNoX3Rva2VuIHZhbHVl' | base64decode }} -> 'fake refresh_token value'
```
### string[](#/filters/string "Direct link to string")
Converts the specified value to a string.
Examples:
```
{{ 1 | string }} -> "1"
```
```
{{ ["hello", "world" | string }} -> "["hello", "world"]"
```
### regex\_search[](#/filters/regex_search "Direct link to regex_search")
Match the input string against a regular expression and return the first match.
Arguments:
* `regex`:
The regular expression to search for. It must include a capture group.
Example:
```
{{ "goodbye, cruel world" | regex_search("goodbye,\s(.*)$") }} -> "cruel world"
```
### regex\_replace[](#/filters/regex_replace "Direct link to regex_replace")
Replace all occurrences in the string that match the provided regex pattern with the specified replacement string.
Arguments:
* `regex`:
The regular expression pattern to match against.
* `replacement`:
The string to replace matched occurrences with.
Examples:
```
{{ "hello world" | regex_replace("world", "universe") }} -> "hello universe"
```
```
{{ "hello-world_foo" | regex_replace("[_-]", " ") }} -> "hello world foo"
```