DeclarativeStream

A stream whose behavior is described by a set of declarative low code components.

• type: object
• required: ["type","retriever"]

type

• Required: Yes
• type: string
• enum: ["DeclarativeStream"]

name

The stream name.

• Required: No
• type: string
• default: ``

retriever

Component used to coordinate how records are extracted across stream slices and request pages.

• Required: Yes

• anyOf:

  • Allowed type: SimpleRetriever

• anyOf:

  • Allowed type: AsyncRetriever

• anyOf:

  • Allowed type: CustomRetriever

incremental_sync

Component used to fetch data incrementally based on a time field in the data.

• Required: No

• anyOf:

  • Allowed type: DatetimeBasedCursor

• anyOf:

  • Allowed type: IncrementingCountCursor

• anyOf:

  • Allowed type: SyncTokenCursor

primary_key

• Required: No
• default: ``
• Allowed type: PrimaryKey

schema_loader

One or many schema loaders can be used to retrieve the schema for the current stream. When multiple schema loaders are defined, schema properties will be merged together. Schema loaders defined first taking precedence in the event of a conflict.

• Required: No

• anyOf:

  • Allowed type: InlineSchemaLoader

• anyOf:

  • Allowed type: DynamicSchemaLoader

• anyOf:

  • Allowed type: JsonFileSchemaLoader

• anyOf:

  • type: array

  • items:

    • anyOf:

      • Allowed type: InlineSchemaLoader

    • anyOf:

      • Allowed type: DynamicSchemaLoader

    • anyOf:

      • Allowed type: JsonFileSchemaLoader

    • anyOf:

      • Allowed type: CustomSchemaLoader

• anyOf:

  • Allowed type: CustomSchemaLoader

transformations

A list of transformations to be applied to each output record.

• Required: No

• type: array

• items:

  • anyOf:

    • Allowed type: AddFields

  • anyOf:

    • Allowed type: RemoveFields

  • anyOf:

    • Allowed type: KeysToLower

  • anyOf:

    • Allowed type: KeysToSnakeCase

  • anyOf:

    • Allowed type: FlattenFields

  • anyOf:

    • Allowed type: DpathFlattenFields

  • anyOf:

    • Allowed type: KeysReplace

  • anyOf:

    • Allowed type: CustomTransformation

state_migrations

Array of state migrations to be applied on the input state

• Required: No

• type: array

• default: []

• items:

  • anyOf:

    • Allowed type: LegacyToPerPartitionStateMigration

  • anyOf:

    • Allowed type: CustomStateMigration

file_uploader

(experimental) Describes how to fetch a file

• Required: No

• type: object

• required: ["type","requester","download_target_extractor"]

• type

  • Required: Yes
  • type: string
  • enum: ["FileUploader"]

• requester

  Requester component that describes how to prepare HTTP requests to send to the source API.

  • Required: Yes

  • anyOf:

    • Allowed type: HttpRequester

  • anyOf:

    • Allowed type: CustomRequester

• download_target_extractor

  Responsible for fetching the url where the file is located. This is applied on each records and not on the HTTP response

  • Required: Yes

  • anyOf:

    • Allowed type: DpathExtractor

  • anyOf:

    • Allowed type: CustomRecordExtractor

  • anyOf:

    • Allowed type: CombinedExtractor

• file_extractor

  Responsible for fetching the content of the file. If not defined, the assumption is that the whole response body is the file content

  • Required: No

  • anyOf:

    • Allowed type: DpathExtractor

  • anyOf:

    • Allowed type: CustomRecordExtractor

• filename_extractor

  Defines the name to store the file. Stream name is automatically added to the file path. File unique ID can be used to avoid overwriting files. Random UUID will be used if the extractor is not provided.

  • Required: No
  • type: string
  • Available variables: config, record

  Examples:

  {{ record.id }}/{{ record.file_name }}/

  {{ record.id }}_{{ record.file_name }}/

• $parameters

  • Required: No
  • type: object

$parameters

• Required: No
• type: object