Skip to main content

HttpRequester

Requester submitting HTTP requests and extracting records from the response.

  • type: object
  • required: ["type"]

type​

  • Required: Yes
  • type: string
  • enum: ["HttpRequester"]

url_base​

Deprecated, use the url instead. Base URL of the API source. Do not put sensitive information (e.g. API tokens) into this field - Use the Authenticator component for this.

Examples:

https://connect.squareup.com/v2
{{ config['base_url'] or 'https://app.posthog.com'}}/api
https://connect.squareup.com/v2/quotes/{{ stream_partition['id'] }}/quote_line_groups
https://example.com/api/v1/resource/{{ next_page_token['id'] }}

url​

The URL of the source API endpoint. Do not put sensitive information (e.g. API tokens) into this field - Use the Authenticator component for this.

Examples:

https://connect.squareup.com/v2
{{ config['url'] or 'https://app.posthog.com'}}/api
https://connect.squareup.com/v2/quotes/{{ stream_partition['id'] }}/quote_line_groups
https://example.com/api/v1/resource/{{ next_page_token['id'] }}

path​

Deprecated, use the url instead. Path the specific API endpoint that this stream represents. Do not put sensitive information (e.g. API tokens) into this field - Use the Authenticator component for this.

Examples:

/products
/quotes/{{ stream_partition['id'] }}/quote_line_groups
/trades/{{ config['symbol_id'] }}/history

http_method​

The HTTP method used to fetch data from the source (can be GET or POST).

  • Required: No
  • type: string
  • default: GET
  • enum: ["GET","POST"]

Examples:

GET
POST

authenticator​

Authentication method to use for requests sent to the API.

fetch_properties_from_endpoint​

Allows for retrieving a dynamic set of properties from an API endpoint which can be injected into outbound request using the stream_partition.extra_fields.

query_properties​

For APIs that require explicit specification of the properties to query for, this component will take a static or dynamic set of properties (which can be optionally split into chunks) and allow them to be injected into an outbound request by accessing stream_partition.extra_fields.

request_parameters​

Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.

Examples:

unit: day
query: last_event_time BETWEEN TIMESTAMP "{{ stream_interval.start_time }}" AND TIMESTAMP "{{ stream_interval.end_time }}"
searchIn: '{{ '',''.join(config.get(''search_in'', [])) }}'
sort_by[asc]: updated_at

  • anyOf:

    • type: object

    • additionalProperties:

  • anyOf:

    • type: string

request_headers​

Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.

Examples:

Output-Format: JSON
Version: '{{ config[''version''] }}'

  • anyOf:

    • type: object

    • additionalProperties:

      • type: string

  • anyOf:

    • type: string

request_body_data​

Specifies how to populate the body of the request with a non-JSON payload. Plain text will be sent as is, whereas objects will be converted to a urlencoded form.

Examples:

[{"clause": {"type": "timestamp", "operator": 10, "parameters":
    [{"value": {{ stream_interval['start_time'] | int * 1000 }} }]
  }, "orderBy": 1, "columnName": "Timestamp"}]/

  • anyOf:

    • type: object

    • additionalProperties:

      • type: string

  • anyOf:

    • type: string

request_body_json​

Specifies how to populate the body of the request with a JSON payload. Can contain nested objects.

Examples:

sort_order: ASC
sort_field: CREATED_AT
key: '{{ config[''value''] }}'
sort:
  field: updated_at
  order: ascending

  • anyOf:

    • type: object

  • anyOf:

    • type: string

request_body​

Specifies how to populate the body of the request with a payload. Can contain nested objects.

error_handler​

Error handler component that defines how to handle errors.

use_cache​

Enables stream requests caching. When set to true, repeated requests to the same URL will return cached responses. Parent streams automatically have caching enabled. Only set this to false if you are certain that caching should be disabled, as it may negatively impact performance when the same data is needed multiple times (e.g., for scroll-based pagination APIs where caching causes duplicate records).

  • Required: No
  • type: boolean
  • default: false

connect_timeout_in_seconds​

Maximum time to establish the connection (TCP + TLS handshake). When unset, no connect timeout is applied.

  • Required: No
  • type: number
  • exclusiveMinimum: 0

Examples:

30

read_timeout_in_seconds​

The maximum time to wait for the server to send any data (the first byte of the response, or the next chunk while the body is read). It is NOT a limit on the total request duration. When unset, no read timeout is applied. Only set it when every endpoint served by this requester responds well within the value; synchronous report or export endpoints can stay silent for minutes. Each timed-out attempt counts against the error handler's retry window (600 s for manifest connectors), so a value near 300 allows only a few retries.

  • Required: No
  • type: number
  • exclusiveMinimum: 0

Examples:

300

use_tcp_keepalive​

Enables TCP keepalive (60 s idle, 10 s probe interval, 6 probes) on this requester's connections, so dead or NAT-dropped connections are detected and fail instead of hanging forever. Does not fail slow but alive servers; use read_timeout_in_seconds to bound those.

  • Required: No
  • type: boolean
  • default: false

Examples:

true

$parameters​

  • Required: No
  • type: object
Was this page helpful?