Skip to main content

DatetimeBasedCursor

Cursor to provide incremental capabilities over datetime.

  • type: object
  • required: ["type","cursor_field","datetime_format","start_datetime"]

type​

  • Required: Yes
  • type: string
  • enum: ["DatetimeBasedCursor"]

clamping​

This option is used to adjust the upper and lower boundaries of each datetime window to beginning and end of the provided target period (day, week, month)

  • Required: No

  • type: object

  • required: ["target"]

  • target

    The period of time that datetime windows will be clamped by

    • Required: Yes
    • type: string
    • Available variables: config

    Examples:

    DAY
    WEEK
    MONTH
    {{ config['target'] }}

  • target_details

    • Required: No
    • type: object

cursor_field​

The location of the value on a record that will be used as a bookmark during sync. To ensure no data loss, the API must return records in ascending order based on the cursor field. Nested fields are not supported, so the field must be at the top level of the record. You can use a combination of Add Field and Remove Field transformations to move the nested field to the top.

  • Required: Yes
  • type: string
  • Available variables: config

Examples:

created_at
{{ config['record_cursor'] }}

allow_catalog_defined_cursor_field​

Whether the cursor allows users to override the default cursor_field when configuring their connection. The user defined cursor field will be specified from within the configured catalog.

  • Required: No
  • type: boolean

cursor_datetime_formats​

The possible formats for the cursor field, in order of preference. The first format that matches the cursor field value will be used to parse it. If not provided, the Outgoing Datetime Format will be used. Use placeholders starting with "%" to describe the format the API is using. The following placeholders are available:

  • %s: Epoch unix timestamp - 1686218963
  • %s_as_float: Epoch unix timestamp in seconds as float with microsecond precision - 1686218963.123456
  • %ms: Epoch unix timestamp - 1686218963123
  • %a: Weekday (abbreviated) - Sun
  • %A: Weekday (full) - Sunday
  • %w: Weekday (decimal) - 0 (Sunday), 6 (Saturday)
  • %d: Day of the month (zero-padded) - 01, 02, ..., 31
  • %b: Month (abbreviated) - Jan
  • %B: Month (full) - January
  • %m: Month (zero-padded) - 01, 02, ..., 12
  • %y: Year (without century, zero-padded) - 00, 01, ..., 99
  • %Y: Year (with century) - 0001, 0002, ..., 9999
  • %H: Hour (24-hour, zero-padded) - 00, 01, ..., 23
  • %I: Hour (12-hour, zero-padded) - 01, 02, ..., 12
  • %p: AM/PM indicator
  • %M: Minute (zero-padded) - 00, 01, ..., 59
  • %S: Second (zero-padded) - 00, 01, ..., 59
  • %f: Microsecond (zero-padded to 6 digits) - 000000, 000001, ..., 999999
  • %_ms: Millisecond (zero-padded to 3 digits) - 000, 001, ..., 999
  • %z: UTC offset - (empty), +0000, -04:00
  • %Z: Time zone name - (empty), UTC, GMT
  • %j: Day of the year (zero-padded) - 001, 002, ..., 366
  • %U: Week number of the year (Sunday as first day) - 00, 01, ..., 53
  • %W: Week number of the year (Monday as first day) - 00, 01, ..., 53
  • %c: Date and time representation - Tue Aug 16 21:30:00 1988
  • %x: Date representation - 08/16/1988
  • %X: Time representation - 21:30:00
  • %%: Literal '%' character

Some placeholders depend on the locale of the underlying system - in most cases this locale is configured as en/US. For more information see the Python documentation.

  • Required: No
  • type: array

Examples:

%Y-%m-%d
%Y-%m-%d %H:%M:%S
%Y-%m-%dT%H:%M:%S
%Y-%m-%dT%H:%M:%SZ
%Y-%m-%dT%H:%M:%S%z
%Y-%m-%dT%H:%M:%S.%fZ
%Y-%m-%dT%H:%M:%S.%f%z
%Y-%m-%d %H:%M:%S.%f+00:00
%s
%ms

  • items:

    • type: string

start_datetime​

The datetime that determines the earliest record that should be synced.

  • Required: Yes
  • Available variables: config

Examples:

2020-01-1T00:00:00Z
{{ config['start_time'] }}

start_time_option​

Optionally configures how the start datetime will be sent in requests to the source API.

end_datetime​

The datetime that determines the last record that should be synced. If not provided, {{ now_utc() }} will be used.

  • Required: No
  • Available variables: config

Examples:

2021-01-1T00:00:00Z
{{ now_utc() }}
{{ day_delta(-1) }}

end_time_option​

Optionally configures how the end datetime will be sent in requests to the source API.

datetime_format​

The datetime format used to format the datetime values that are sent in outgoing requests to the API. Use placeholders starting with "%" to describe the format the API is using. The following placeholders are available:

  • %s: Epoch unix timestamp - 1686218963
  • %s_as_float: Epoch unix timestamp in seconds as float with microsecond precision - 1686218963.123456
  • %ms: Epoch unix timestamp (milliseconds) - 1686218963123
  • %a: Weekday (abbreviated) - Sun
  • %A: Weekday (full) - Sunday
  • %w: Weekday (decimal) - 0 (Sunday), 6 (Saturday)
  • %d: Day of the month (zero-padded) - 01, 02, ..., 31
  • %b: Month (abbreviated) - Jan
  • %B: Month (full) - January
  • %m: Month (zero-padded) - 01, 02, ..., 12
  • %y: Year (without century, zero-padded) - 00, 01, ..., 99
  • %Y: Year (with century) - 0001, 0002, ..., 9999
  • %H: Hour (24-hour, zero-padded) - 00, 01, ..., 23
  • %I: Hour (12-hour, zero-padded) - 01, 02, ..., 12
  • %p: AM/PM indicator
  • %M: Minute (zero-padded) - 00, 01, ..., 59
  • %S: Second (zero-padded) - 00, 01, ..., 59
  • %f: Microsecond (zero-padded to 6 digits) - 000000
  • %_ms: Millisecond (zero-padded to 3 digits) - 000
  • %z: UTC offset - (empty), +0000, -04:00
  • %Z: Time zone name - (empty), UTC, GMT
  • %j: Day of the year (zero-padded) - 001, 002, ..., 366
  • %U: Week number of the year (starting Sunday) - 00, ..., 53
  • %W: Week number of the year (starting Monday) - 00, ..., 53
  • %c: Date and time - Tue Aug 16 21:30:00 1988
  • %x: Date standard format - 08/16/1988
  • %X: Time standard format - 21:30:00
  • %%: Literal '%' character

Some placeholders depend on the locale of the underlying system - in most cases this locale is configured as en/US. For more information see the Python documentation.

  • Required: Yes
  • type: string

Examples:

%Y-%m-%dT%H:%M:%S.%f%z
%Y-%m-%d
%s
%ms
%s_as_float

cursor_granularity​

Smallest increment the datetime_format has (ISO 8601 duration) that is used to ensure the start of a slice does not overlap with the end of the previous one, e.g. for %Y-%m-%d the granularity should be P1D, for %Y-%m-%dT%H:%M:%SZ the granularity should be PT1S. Given this field is provided, step needs to be provided as well.

  • PT0.000001S: 1 microsecond
  • PT0.001S: 1 millisecond
  • PT1S: 1 second
  • PT1M: 1 minute
  • PT1H: 1 hour
  • P1D: 1 day
  • Required: No
  • type: string

Examples:

PT1S

is_data_feed​

A data feed API is an API that does not allow filtering and paginates the content from the most recent to the least recent. Given this, the CDK needs to know when to stop paginating and this field will generate a stop condition for pagination. The last page fetched still holds records that fall outside the cursor window, and those are filtered out as well, so Client-side Incremental Filtering does not need to be enabled alongside this field. Records are kept when their cursor value is within the window that starts at the previous sync's cursor value minus the lookback window, if one is set (never before the start date), and ends at the end date, defaulting to the current time, so records dated in the future are filtered out too.

  • Required: No
  • type: boolean

is_client_side_incremental​

Set to True if the target API endpoint does not take cursor values to filter records and returns all records anyway. This will cause the connector to filter out records locally, keeping only the ones whose cursor value falls within the window that starts at the previous sync's cursor value minus the lookback window, if one is set (never before the start date), and ends at the end date, defaulting to the current time. This means that all records would be read from the API, but only the records within that window will be emitted to the destination. This is not needed when Data Feed API is enabled, as a data feed already filters on the same window.

  • Required: No
  • type: boolean

is_compare_strictly​

Set to True if the target API does not accept queries where the start time equal the end time. This will cause those requests to be skipped.

  • Required: No
  • type: boolean
  • default: false

global_substream_cursor​

Setting to True causes the connector to store the cursor as one value, instead of per-partition. This setting optimizes performance when the parent stream has thousands of partitions. Notably, the substream state is updated only at the end of the sync, which helps prevent data loss in case of a sync failure. See more info in the docs.

  • Required: No
  • type: boolean
  • default: false

lookback_window​

Time interval (ISO8601 duration) before the start_datetime to read data for, e.g. P1M for looking back one month.

  • PT1H: 1 hour
  • P1D: 1 day
  • P1W: 1 week
  • P1M: 1 month
  • P1Y: 1 year
  • Required: No
  • type: string
  • Available variables: config

Examples:

P1D
P{{ config['lookback_days'] }}D

partition_field_end​

Name of the partition start time field.

  • Required: No
  • type: string

Examples:

ending_time

partition_field_start​

Name of the partition end time field.

  • Required: No
  • type: string

Examples:

starting_time

step​

The size of the time window (ISO8601 duration). Given this field is provided, cursor_granularity needs to be provided as well.

  • PT1H: 1 hour
  • P1D: 1 day
  • P1W: 1 week
  • P1M: 1 month
  • P1Y: 1 year
  • Required: No
  • type: string

Examples:

P1W
{{ config['step_increment'] }}

$parameters​

  • Required: No
  • type: object
Was this page helpful?