How the Framework Works
- Given the connection config and an optional stream state, the
PartitionRouter
computes the partitions that should be routed to read data. - Iterate over all the partitions defined by the stream's partition router.
- For each partition,
- Submit a request to the partner API as defined by the requester
- Select the records from the response
- Repeat for as long as the paginator points to a next page