UnionPartitionRouter
A partition router that yields the deduplicated union of the partitions produced by its child partition routers. Every emitted partition is normalized to a single key defined by partition_field; any other partition keys coming from a child router (such as a SubstreamPartitionRouter's parent_slice) are moved into the slice's extra fields. The first occurrence of a partition value wins and later duplicates are skipped. Partition values must be hashable; deduplication holds every distinct value in memory for the duration of the sync, and the resulting partition count is the deduplicated sum of the child routers' partition counts, which affects per-partition state size. Because child routers must emit scalar partition values, GroupingPartitionRouter (which emits list-valued partitions) cannot be used as a child.
- type:
object - required:
["type","partition_field","partition_routers"]
type
- Required: Yes
- type:
string - enum:
["UnionPartitionRouter"]
partition_field
The single partition key that all child partition routers' slices are normalized to. Each child router must emit this key in its partitions. Interpolation is evaluated once when the connector is built, using the connector config and $parameters.
- Required: Yes
- type:
string - Available variables:
config,parameters
Examples:
repository
{{ config['partition_field'] }}
partition_routers
The child partition routers whose partitions are unioned. Request options are not supported on child partition routers; partition values should be consumed via interpolation (e.g. stream_partition).
-
Required: Yes
-
type:
array -
minItems:
2 -
items:
-
anyOf:
- Allowed type:
ListPartitionRouter
- Allowed type:
-
anyOf:
- Allowed type:
SubstreamPartitionRouter
- Allowed type:
-
anyOf:
- Allowed type:
UnionPartitionRouter
- Allowed type:
-
anyOf:
- Allowed type:
CustomPartitionRouter
- Allowed type:
-
$parameters
- Required: No
- type:
object