Skip to main content

AsyncRetriever

Retrieves records by Asynchronously sending requests to fetch records. The retriever acts as an orchestrator between the requester, the record selector, the paginator, and the partition router.

  • type: object
  • required: ["type","record_selector","status_mapping","creation_requester","polling_requester","download_requester","status_extractor"]

type​

  • Required: Yes
  • type: string
  • enum: ["AsyncRetriever"]

record_selector​

Component that describes how to extract records from a HTTP response.

status_mapping​

Async Job Status to Airbyte CDK Async Job Status mapping.

status_extractor​

Responsible for fetching the actual status of the async job.

download_target_extractor​

Responsible for fetching the final result urls provided by the completed / finished / ready async job.

download_extractor​

Responsible for fetching the records from provided urls.

creation_requester​

Requester component that describes how to prepare HTTP requests to send to the source API to create the async server-side job.

polling_requester​

Requester component that describes how to prepare HTTP requests to send to the source API to fetch the status of the running async job.

polling_job_timeout​

The time in minutes after which the single Async Job should be considered as Timed Out.

  • Required: No

  • Available variables: config

  • anyOf:

    • type: integer

  • anyOf:

    • type: string

failed_retry_wait_time_in_seconds​

Time in seconds to wait before retrying a failed async job. Only applies to jobs that ran on the API side and reported a FAILED status (e.g. report generation failed due to a cooldown). Creation failures (HTTP errors when starting a job, such as 429s) and TIMED_OUT jobs are retried immediately and are not affected by this setting. When set, the orchestrator defers retry of real failed jobs until the wait time has elapsed, without blocking other jobs.

  • Required: No

  • Available variables: config

  • anyOf:

    • type: integer
    • minimum: 1

  • anyOf:

    • type: string

download_target_requester​

Requester component that describes how to prepare HTTP requests to send to the source API to extract the url from polling response by the completed async job.

download_requester​

Requester component that describes how to prepare HTTP requests to send to the source API to download the data provided by the completed async job.

download_paginator​

Paginator component that describes how to navigate through the API's pages during download.

abort_requester​

Requester component that describes how to prepare HTTP requests to send to the source API to abort a job once it is timed out from the source's perspective.

delete_requester​

Requester component that describes how to prepare HTTP requests to send to the source API to delete a job once the records are extracted.

partition_router​

PartitionRouter component that describes how to partition the stream, enabling incremental syncs and checkpointing.

decoder​

Component decoding the response so records can be extracted.

download_decoder​

Component decoding the download response so records can be extracted.

$parameters​

  • Required: No
  • type: object
Was this page helpful?