PageSizeReduction

Describes how the page size is reduced when an error handler resolves to the REDUCE_PAGE_SIZE action. On such a response, the connector re-issues the same page with a smaller page size instead of retrying the identical request. Only supported on a DefaultPaginator that has a page_size_option and whose pagination_strategy defines a page_size and is CursorPagination, OffsetIncrement, or a CustomPaginationStrategy whose next_page_token accepts a page_size_override keyword argument. PageIncrement is not supported because a smaller page size moves every following page boundary and would skip records. It is also rejected when the stream uses query properties, a file uploader, or reads its parent stream lazily through lazy_read_pointer, since re-issuing a page whose records were already emitted would duplicate them. A page_token_option of type RequestPath is rejected too: the next page is then a URL built by the API which already carries the page size, so the reduced page size would be sent next to the original one. Each reduction waits a short, growing amount of time before re-issuing the page so that an endpoint failing at every page size is not hit in a burst. That wait is the CDK's own: the error handler's backoff_strategies and any Retry-After header are not consulted on this path, the same way they are not for RESET_PAGINATION, and backoff_seconds is what sets its length.

• type: object
• required: ["type"]

type

• Required: Yes
• type: string
• enum: ["PageSizeReduction"]

reduction_factor

Divisor applied to the page size on each reduction. The new page size is floor(current page size / reduction factor).

• Required: No
• type: number
• default: 2
• exclusiveMinimum: 1

Examples:

2

4

minimum_page_size

Page size below which the connector stops reducing and fails the sync. It must be smaller than the page size configured on the pagination strategy, otherwise no reduction could ever be applied. It is one of two bounds on the reduction and whichever is tighter wins: an unbroken run of failing pages divides the page size by reduction_factor at most max_attempts times, so reaching this floor in a single run needs max_attempts of at least log(page_size / minimum_page_size) / log(reduction_factor) - with the defaults, a page size of 1000 bottoms out at 31 records per page and a floor of 10 is never reached. Pages that succeed in between restart the max_attempts budget, so the floor is still reachable over a partition.

• Required: No
• type: integer
• default: 1
• minimum: 1

Examples:

1

10

max_attempts

Maximum number of page size reductions made in a row without a single page succeeding, before the sync fails with a transient error. Every reduction follows a request that failed, so at most max_attempts + 1 failing requests are issued before giving up. The budget restarts after every page that succeeds, under either reset_policy, so it bounds the reductions needed to get a single page through and not the number of pages a partition may have: a stream that needs a reduction every now and then reads to the end however long it is. Under NEVER the page size also strictly decreases, so minimum_page_size bounds the reductions of the whole partition on its own. The wait between reduction attempts is the CDK's own - it grows with each attempt - and does not consult the error handler's backoff_strategies or a Retry-After header.

• Required: No
• type: integer
• default: 5
• minimum: 1

Examples:

5

10

backoff_seconds

Base number of seconds to wait before the page is re-issued, multiplied by the number of attempts made in a row, so the second attempt waits twice as long as the first. A REDUCE_PAGE_SIZE response never reaches the error handler's retry budget, backoff_strategies or a Retry-After header, so this is the only thing spacing those requests out. Raise it on an API whose error also means "we are briefly unwell" rather than only "your page is too big", since the default spaces the whole run of attempts over a few seconds.

• Required: No
• type: number
• default: 0.5
• minimum: 0

Examples:

0.5

5

retries_at_minimum_page_size

Number of times the same page is re-issued unchanged, each after the backoff wait, once the page size cannot be shrunk any further, before the sync fails with a transient error. The default of 0 fails on the first response received at minimum_page_size. Raise it when the API returns the same error for a page that is too big and for a server-side hiccup: at the floor, reducing is no longer an option but waiting still is, and without this budget those responses end the stream on the first one. This budget is separate from max_attempts, which only counts reductions, and it restarts on every page that succeeds. It applies however the page size arrived at the floor, whether by reduction or because page_size was already there; a page size that minimum_page_size blocks from ever being reduced is still reported as a configuration error, but only once this budget is spent.

• Required: No
• type: integer
• default: 0
• minimum: 0

Examples:

0

3

failure_message

Sentence appended to the error message shown to the user when the connector runs out of reductions, either because max_attempts was reached or because the page size is already at minimum_page_size. Use it to tell the user what they can do about it in terms of this specific API, for instance which filter narrows the query down. Without it the message only states that the API kept rejecting every page size the connector asked for.

• Required: No
• type: string

Examples:

Narrow the sync down by selecting fewer fields on this stream.

Set a more recent start date so that each page covers less data.

reset_policy

When to restore the page size configured on the pagination strategy. NEVER keeps the reduced page size for the rest of the partition. AFTER_SUCCESSFUL_PAGE restores it as soon as one page succeeds, which means hitting the same error again on every page - use it only when the reduction is worth one extra request per page, for instance because the configured page size usually works and only some pages are too heavy. It only controls the page size: the max_attempts budget restarts on every page that succeeds under both policies, so there is no limit on how many reductions a partition may make in total. What is bounded is the reductions that get no page through.

• Required: No
• type: string
• default: NEVER
• enum: ["NEVER","AFTER_SUCCESSFUL_PAGE"]