Skip to main content

CombinedExtractor

Record extractor that combines the output of several sub-extractors. Use it when a single response holds records under more than one path, when the records can be found under one of several alternative paths, or when a record has to be assembled from several parallel lists.

  • type: object
  • required: ["type","extractors"]

Examples:

type: CombinedExtractor
mode: union
extractors:
  - type: DpathExtractor
    field_path:
      - data
      - repository
      - issues
      - nodes
  - type: DpathExtractor
    field_path:
      - data
      - repository
      - pullRequests
      - nodes
type: CombinedExtractor
mode: first_match
skip_empty_records: true
extractors:
  - type: DpathExtractor
    field_path:
      - data
      - boards
      - '*'
      - items_page
      - items
      - '*'
  - type: DpathExtractor
    field_path:
      - data
      - next_items_page
      - items

type​

  • Required: Yes
  • type: string
  • enum: ["CombinedExtractor"]

extractors​

The record extractors to combine. At least one is required. Each sub-extractor is given the same HTTP response and decodes it independently, so the response is parsed once per sub-extractor. Streaming decoders (CsvDecoder, JsonlDecoder, JsonItemsDecoder, GzipDecoder, IterableDecoder) can only read the response body once and are rejected.

mode​

How the records of the sub-extractors are combined. "union" (default) yields every record of every sub-extractor, in the order the extractors are declared. "first_match" yields the records of the first sub-extractor that produces at least one record and skips the remaining ones; nothing is yielded if none of them produces a record. Note that a paginator which counts the records of a page counts the combined records: under "union" that is the sum over all sub-extractors, which overshoots the API page size, so "union" is rejected with an OffsetIncrement paginator because the offset would skip records. Under "first_match" the count is the count of the winning sub-extractor, which is usually the number of records the API returned for the page.

  • Required: No
  • default: union
  • Allowed type: CombineMode

skip_empty_records​

Whether to drop empty records - null, {}, [], "" - yielded by a sub-extractor before the records are combined. Off by default, so the output of a sub-extractor is passed through as it is. Turn it on when the API can return nulls in the middle of a record list, which a GraphQL API does when it answers a partial response and reports the failure in a sibling error field. Under "first_match" this also changes which sub-extractor wins: a sub-extractor whose records are all empty no longer counts as a match, so the next one is tried. The dropped records still count toward the page size of an OffsetIncrement or PageIncrement paginator, so a full page holding a null does not end pagination early.

  • Required: No
  • type: boolean
  • default: false

$parameters​

  • Required: No
  • type: object
Was this page helpful?