YAML reference

This reference lists the components and interpolation helpers available when defining a low code YAML file.

See the raw schema on GitHub for the technical schema used to validate manifests.

Variables · Macros · Filters

Component	Description
DeclarativeSource	An API source that extracts data according to its declarative components.
AddFields	Transformation which adds field to an output record.
AddedFieldDefinition	Defines the field to add on a record.
ApiKeyAuthenticator	Authenticator for requests authenticated with an API token injected as an HTTP request header.
AsyncJobStatusMap	Matches the api job status to Async Job Status.
AsyncRetriever	Retrieves records by Asynchronously sending requests to fetch records.
AuthFlow	Additional and optional specification object to describe what an 'advanced' Auth flow would need to function.
BasicHttpAuthenticator	Authenticator for requests authenticated with the Basic HTTP authentication scheme, which encodes a username and an optional password in the Authorization request header.
BearerAuthenticator	Authenticator for requests authenticated with a bearer token injected as a request header of the form Authorization: Bearer <token>.
BlockSimultaneousSyncsAction	Action that prevents streams in the same group from being read concurrently.
CheckDynamicStream	(This component is experimental.
CheckStream	Defines the streams to try reading when running a check operation.
CombineMode	How the records of the sub-extractors of a CombinedExtractor are combined.
CombinedExtractor	Record extractor that combines the output of several sub-extractors.
ComplexFieldType	(This component is experimental.
ComponentMappingDefinition	(This component is experimental.
CompositeErrorHandler	Error handler that sequentially iterates over a list of error handlers.
ConcurrencyLevel	Defines the amount of parallelization for the streams that are being synced.
ConditionalStreams	Streams that are only available while performing a connector operation when the condition is met.
ConfigAddFields	Transformation that adds fields to a config.
ConfigComponentsResolver	(This component is experimental.
ConfigMigration	A config migration that will be applied on the incoming config at the start of a sync.
ConfigRemapField	Transformation that remaps a field's value to another value based on a static map.
ConfigRemoveFields	Transformation that removes a field from the config.
ConstantBackoffStrategy	Backoff strategy with a constant backoff interval.
CsvDecoder	Select 'CSV' for response data that is formatted as CSV (comma-separated values).
CursorPagination	Pagination strategy that evaluates an interpolated string to define the next page to fetch.
CustomAuthenticator	Authenticator component whose behavior is derived from a custom code implementation of the connector.
CustomBackoffStrategy	Backoff strategy component whose behavior is derived from a custom code implementation of the connector.
CustomConfigTransformation	A custom config transformation that can be used to transform the connector configuration.
CustomDecoder	Use this to implement custom decoder logic.
CustomErrorHandler	Error handler component whose behavior is derived from a custom code implementation of the connector.
CustomPaginationStrategy	Pagination strategy component whose behavior is derived from a custom code implementation of the connector.
CustomPartitionRouter	Partition router component whose behavior is derived from a custom code implementation of the connector.
CustomRecordExtractor	Record extractor component whose behavior is derived from a custom code implementation of the connector.
CustomRecordFilter	Record filter component whose behavior is derived from a custom code implementation of the connector.
CustomRequester	Requester component whose behavior is derived from a custom code implementation of the connector.
CustomRetriever	Retriever component whose behavior is derived from a custom code implementation of the connector.
CustomSchemaLoader	Schema Loader component whose behavior is derived from a custom code implementation of the connector.
CustomSchemaNormalization	Schema normalization component whose behavior is derived from a custom code implementation of the connector.
CustomStateMigration	Apply a custom transformation on the input state.
CustomTransformation	Transformation component whose behavior is derived from a custom code implementation of the connector.
CustomValidationStrategy	Custom validation strategy that allows for custom validation logic.
DatetimeBasedCursor	Cursor to provide incremental capabilities over datetime.
DeclarativeStream	A stream whose behavior is described by a set of declarative low code components.
DefaultErrorHandler	Component defining how to handle errors.
DefaultPaginator	Default pagination implementation to request pages of results with a fixed size until the pagination strategy no longer returns a next_page_token.
DpathExtractor	Record extractor that searches a decoded response over a path defined as an array of fields.
DpathFlattenFields	A transformation that flatten field values to the to top of the record.
DpathValidator	Validator that extracts the value located at a given field path.
DynamicDeclarativeStream	(This component is experimental.
DynamicSchemaLoader	(This component is experimental.
DynamicStreamCheckConfig
ExponentialBackoffStrategy	Backoff strategy with an exponential backoff interval.
FixedWindowCallRatePolicy	A policy that allows a fixed number of calls within a specific time window.
FlattenFields	A transformation that flatten record to single level format.
GroupByKeyMergeStrategy	Record merge strategy that combines records according to fields on the record.
GroupingPartitionRouter	A decorator on top of a partition router that groups partitions into batches of a specified size.
GzipDecoder	Select 'gzip' for response data that is compressed with gzip.
HTTPAPIBudget	Defines how many requests can be made to the API in a given time frame.
HttpComponentsResolver	(This component is experimental.
HttpRequestRegexMatcher	Matches HTTP requests based on method, base URL, URL path pattern, query parameters, and headers.
HttpRequester	Requester submitting HTTP requests and extracting records from the response.
HttpResponseFilter	A filter that is used to select on properties of the HTTP response received.
IncrementingCountCursor	Cursor that allows for incremental sync according to a continuously increasing integer.
InlineSchemaLoader	Loads a schema that is defined directly in the manifest file.
IterableDecoder	Select 'Iterable' if the response consists of strings separated by new lines (\n).
JsonDecoder	Select 'JSON' if the response is formatted as a JSON object.
JsonFileSchemaLoader	Loads the schema from a json file.
JsonItemsDecoder	Select 'JSON Items (Streaming)' to stream-decode a single JSON document by yielding each element of a nested array, one at a time.
JsonSchemaPropertySelector	When configured, the JSON schema supplied in the catalog containing which columns are selected for the current stream will be used to reduce which query properties will be included in the outbound API request.
JsonlDecoder	Select 'JSON Lines' if the response consists of JSON objects separated by new lines ('\n') in JSONL format.
JwtAuthenticator	Authenticator for requests using JWT authentication flow.
KeyTransformation
KeysReplace	A transformation that replaces symbols in keys.
KeysToLower	A transformation that renames all keys to lower case.
KeysToSnakeCase	A transformation that renames all keys to snake case.
LegacySessionTokenAuthenticator	Deprecated - use SessionTokenAuthenticator instead.
LegacyToPerPartitionStateMigration	Transforms the input state for per-partitioned streams from the legacy format to the low-code format.
ListPartitionRouter	A Partition router that specifies a list of attributes where each attribute describes a portion of the complete data set for a stream.
MinMaxDatetime	Compares the provided date against optional minimum or maximum times.
MovingWindowCallRatePolicy	A policy that allows a fixed number of calls within a moving time window.
NoAuth	Authenticator for requests requiring no authentication.
NoPagination	Pagination implementation that never returns a next page.
OAuthAuthenticator	Authenticator for requests using OAuth 2.0 authorization flow.
OAuthConfigSpecification	Specification describing how an 'advanced' Auth flow would need to function.
OffsetIncrement	Pagination strategy that returns the number of records reads so far and returns it as the next page token.
PageIncrement	Pagination strategy that returns the number of pages reads so far and returns it as the next page token.
PageSizeReduction	Describes how the page size is reduced when an error handler resolves to the REDUCE_PAGE_SIZE action.
PaginationReset	Describes what triggers pagination reset and how to handle it.
PaginationResetLimits	Describes the limits that trigger pagination reset
ParametrizedComponentsResolver	(This component is experimental.
ParentFieldPath	One value to copy from the record being expanded onto each expanded item.
ParentStreamConfig	Describes how to construct partitions from the records retrieved from the parent stream..
PredicateValidator	Validator that applies a validation strategy to a specified value.
PrimaryKey	The stream field to be used to distinguish unique records.
PropertiesFromEndpoint	Defines the behavior for fetching the list of properties from an API that will be loaded into the requests to extract records.
PropertyChunking	For APIs with restrictions on the amount of properties that can be requester per request, property chunking can be applied to make multiple requests with a subset of the properties.
QueryProperties	For APIs that require explicit specification of the properties to query for, this component specifies which property fields and how they are supplied to outbound requests.
QuotaStatusSource	Describes the endpoint from which a token's current rate limit quota status can be fetched.
Rate	Defines a rate limit with a specific number of calls allowed within a time interval.
RateLimitedMultipleTokenAuthenticator	Authenticator that rotates between multiple interchangeable tokens, tracking each token's remaining call quota.
RecordExpander	Expands records by extracting items from a nested array field.
RecordFilter	Filter applied on a list of records.
RecordSelector	Responsible for translating an HTTP response into a list of records by extracting records from the response and optionally filtering records based on a heuristic.
RemoveFields	A transformation which removes fields from a record.
RequestBodyGraphQL	Request body value converted into a GraphQL query object
RequestBodyGraphQlQuery	Request body GraphQL query object
RequestBodyJsonObject	Request body value converted into a JSON object
RequestBodyPlainText	Request body value is sent as plain text
RequestBodyUrlEncodedForm	Request body value is converted into a url-encoded form
RequestOption	Specifies the key field or path and where in the request a component's value should be injected.
RequestPath	The URL path to be used for the HTTP request.
RequestWindowSplitting	Describes how the request window is split when an error handler resolves to the SPLIT_REQUEST_WINDOW action.
ResponseToFileExtractor	A record extractor designed for handling large responses that may exceed memory limits (to prevent OOM issues).
SchemaNormalization	Responsible for normalization according to the schema.
SchemaTypeIdentifier	(This component is experimental.
Schemas	The stream schemas representing the shape of the data emitted by the stream.
SelectiveAuthenticator	Authenticator that selects concrete authenticator based on config property.
SessionTokenAuthenticator	Authenticator for requests using the session token as an API key that's injected into the request.
SessionTokenRequestApiKeyAuthenticator	Authenticator for requests using the session token as an API key that's injected into the request.
SessionTokenRequestBearerAuthenticator	Authenticator for requests using the session token as a standard bearer token.
SimpleRetriever	Retrieves records by synchronously sending requests to fetch records.
Spec	A source specification made up of connector metadata and how it can be configured.
StateDelegatingStream	(This component is experimental.
StreamConfig	(This component is experimental.
StreamGroup	A group of streams that share a common resource and should not be read simultaneously.
StreamParametersDefinition	(This component is experimental.
SubstreamPartitionRouter	Partition router that is used to retrieve records that have been partitioned according to records from the specified parent streams.
SyncTokenCursor	Incremental sync driven by a sync token that the API returns at the end of a pagination pass.
TokenQuota	A named per-token quota pool with matchers that classify outgoing requests into the pool.
TypesMap	(This component is experimental.
UnionPartitionRouter	A partition router that yields the deduplicated union of the partitions produced by its child partition routers.
UnlimitedCallRatePolicy	A policy that allows unlimited calls for specific requests.
ValidateAdheresToSchema	Validates that a user-provided schema adheres to a specified JSON schema.
ValueType	A schema type.
WaitTimeFromHeader	Extract wait time from a HTTP header in the response.
WaitTimeFromResponse	Compute the wait time, in relative seconds, from the JSON response body or the response headers.
WaitUntilTimeFromHeader	Extract time at which we can retry the request from response header and wait for the difference between now and that time.
XmlDecoder	Select 'XML' if the response consists of XML-formatted data.
ZipfileDecoder	Select 'ZIP file' for response data that is returned as a zipfile.