Skip to main content

End-to-End Testing Source

Overview

This is a mock source for testing the Airbyte pipeline. It can generate arbitrary data streams.

Mode

Continuous Feed

This is the only mode available starting from 2.0.0.

This mode allows users to specify a single-stream or multi-stream catalog with arbitrary schema. The schema should be compliant with Json schema draft-07.

The single-stream catalog config exists just for convenient, since in many testing cases, one stream is enough. If only one stream is specified in the multi-stream catalog config, it is equivalent to a single-stream catalog config.

Here is its configuration:

Mock Catalog TypeParametersTypeRequiredDefaultNotes
Single-streamstream namestringyesName of the stream in the catalog.
stream schemajsonyesJson schema of the stream in the catalog. It must be a valid Json schema.
stream duplicationintegerno1Duplicate the stream N times to quickly create a multi-stream catalog.
Multi-streamstreams and schemasjsonyesA Json object specifying multiple data streams and their schemas. Each key in this object is one stream name. Each value is the schema for that stream.
Bothmax recordsintegeryes100The number of record messages to emit from this connector. Min 1. Max 100 billion.
random seedintegernocurrent time millisThe seed is used in random Json object generation. Min 0. Max 1 million.
message intervalintegerno0The time interval between messages in millisecond. Min 0 ms. Max 60000 ms (1 minute).

Example Stream Schemas

If you need a stream for testing performance simulating a wide table, we have an example 500 column stream or use the form below to generate your own with an arbitrary width, then copy+paste the resulting schema into your configuration.

Generated Schema:

{"type":"object","properties":{"id":{"type":"integer"},"updated_at":{"type":"string","format":"date-time","airbyte_type":"timestamp_with_timezone"},"string_2":{"type":"string"},"boolean_3":{"type":"boolean"},"date_4":{"type":["null","string"],"format":"date"},"timestamp_wo_tz_5":{"type":["null","string"],"format":"date-time","airbyte_type":"timestamp_without_timezone"},"timestamp_w_tz_6":{"type":["null","string"],"format":"date-time","airbyte_type":"timestamp_with_timezone"},"time_wo_tz_7":{"type":["null","string"],"format":"time","airbyte_type":"time_without_timezone"},"time_w_tz_8":{"type":["null","string"],"format":"time","airbyte_type":"time_with_timezone"},"integer_9":{"type":"integer"}}}

Legacy Infinite Feed

This is a legacy mode used in Airbyte integration tests. It has been removed since 2.0.0. It has a simple catalog with one data stream that has the following schema:

{
"type": "object",
"properties": {
"column1": { "type": "string" }
}
}

The value of column1 will be an increasing number starting from 1.

This mode can generate infinite number of records, which can be dangerous. That's why it is excluded from the Cloud variant of this connector. Usually this mode should not be used.

There are two configurable parameters:

ParametersTypeRequiredDefaultNotes
max recordsintegernonullNumber of message records to emit. When it is left empty, the connector will generate infinite number of messages.
message intervalintegernonullTime interval between messages in millisecond.

Exception after N

This is a legacy mode used in Airbyte integration tests. It has been removed since 2.0.0. It throws an IllegalStateException after certain number of messages. The number of messages to emit before exception is the only parameter for this mode.

This mode is also excluded from the Cloud variant of this connector.

Changelog

Expand to review

The OSS and Cloud variants have the same version number. The Cloud variant was initially released at version 1.0.0.

VersionDatePull requestSubject
2.2.22024-04-2537581bump jsonschemafriend to 0.12.4
2.2.12024-02-1335231Adopt JDK 0.20.4.
2.2.02023-12-1833485Remove LEGACY state
2.1.52023-10-0431092Bump jsonschemafriend dependency version to fix bug
2.1.42023-03-0123656Add speed benchmark mode to e2e test
2.1.32022-08-2515591Declare supported sync modes in catalogs
2.1.12022-06-1713864Updated stacktrace format for any trace message errors
2.1.02021-02-12#10298Support stream duplication to quickly create a multi-stream catalog.
2.0.02021-02-01#9954Remove legacy modes. Use more efficient Json generator.
1.0.12021-01-29#9745Integrate with Sentry.
1.0.02021-01-23#9720Add new continuous feed mode that supports arbitrary catalog specification. Initial release to cloud.
0.1.22022-10-18#18100Set supported sync mode on streams
0.1.12021-12-16#8217Fix sleep time in infinite feed mode.
0.1.02021-07-23#3290 #4939Initial release.