Skip to main content

SimpleRetriever

Retrieves records by synchronously sending requests to fetch records. The retriever acts as an orchestrator between the requester, the record selector, the paginator, and the partition router.

  • type: object
  • required: ["type","record_selector","requester"]

type​

  • Required: Yes
  • type: string
  • enum: ["SimpleRetriever"]

requester​

Requester component that describes how to prepare HTTP requests to send to the source API.

decoder​

Component decoding the response so records can be extracted.

record_selector​

Component that describes how to extract records from a HTTP response.

paginator​

Paginator component that describes how to navigate through the API's pages.

pagination_reset​

Describes what triggers pagination reset and how to handle it.

page_size_reduction​

Describes how the page size is reduced when an error handler resolves to the REDUCE_PAGE_SIZE action. Requires a DefaultPaginator that defines both page_size_option and a pagination strategy with a page_size. Cannot be combined with query properties, a file uploader, or a parent stream read lazily through lazy_read_pointer, because in those cases records of the failing page have already been emitted and re-issuing the page would emit them twice. A page_token_option of type RequestPath is rejected as well, because the next page is then a URL built by the API which already carries the page size.

request_window_splitting​

Describes how the request window is split when an error handler resolves to the SPLIT_REQUEST_WINDOW action. Requires an incremental_sync cursor that defines cursor_granularity. Distinct from page_size_reduction: this splits the cursor's window rather than a paginator's page size, and applies even when the API rejects the request before returning a single page.

ignore_stream_slicer_parameters_on_paginated_requests​

If true, the partition router and incremental request options will be ignored when paginating requests. Request options set directly on the requester will not be ignored.

  • Required: No
  • type: boolean
  • default: false

partition_router​

Used to iteratively execute requests over a set of values, such as a parent stream's records or a list of constant values.

$parameters​

  • Required: No
  • type: object
Was this page helpful?