Skip to main content

Connector UX handbook

Use this handbook when you make decisions that affect how people configure or use connectors. Each form rule has a stable ID (for example, UX-01) so reviews and automated checks can link to it. Automated checks report violations as warnings, which don't block merge.

Personas​

PersonaWhat you can expect
Data analystUses spreadsheets, SQL, and dashboards. May not read API documentation, but can follow clear steps to generate an API key.
Analytics engineerUses SQL, dbt, Git, and a scripting language. May know containers, but less about infrastructure than a data engineer.
Data engineerUses SQL, dbt, multiple programming languages, cloud services, and orchestration tools. Builds or consumes APIs and manages infrastructure.

The mix of personas varies by connector. A technical connector such as Kafka may have few data analysts.

Guiding principles​

Airbyte is infrastructure in a user's data pipeline. Users rely on it to move data securely, accurately, and reliably; they should not need to debug the infrastructure.

The hierarchy of user needs is security, data integrity, reliability, speed, ease of use, and feature set. Protect credentials and data, preserve records and their meaning, and make connectors reliable. Then optimize speed and usability while covering useful cases.

Security means protecting confidential data and credentials. Do not expose credentials in logs or plain text.

Data integrity means replicating complete and correct data. For example, preserve leading zeroes in postal codes and account for database time zones.

Reliability means handling large, unusual, and incorrectly formatted inputs without requiring users to debug the connector.

Speed reduces the time users wait for value and lets them iterate on configurations.

Ease of use means sensible defaults, predictable output, and little unnecessary friction.

Feature set means supporting as many useful cases as practical without making configuration harder to understand.

How the connector form renders your spec​

  • required: Required fields appear in the main form unless they have a default. Required fields with a default collapse like optional fields. A default of false, "", null, or an empty array doesn't count, and neither does a default on an airbyte_secret field. Root-level collapsed fields without order go into the collapsed Advanced section at the end of the form. Collapsed fields with order, and nested collapsed fields, collapse in place.
  • default: When the web app creates a new configuration, it fills initial form values from schema defaults. API, Terraform, and PyAirbyte callers do not receive defaults from the web app.
  • order: Orders fields within their required or optional sections; required fields still precede optional fields.
  • group: Places root-level fields in a named group. Group titles come from the form's group metadata.
  • always_show: Keeps a field expanded with the main form fields, including a required field with a default.
  • airbyte_hidden: Hides a field from the form.
  • oneOf: Renders a selector for the alternatives. Each option's title is its selector label. oneOf blocks and groups other than resource allocation are always expanded.
  • enum: Renders a dropdown for a scalar field.
  • examples: Displays example values next to the field's description when users open the field's info icon.
  • type: string with format: date or format: date-time: Renders a date picker; date-time includes a time control. pattern alone does not select the date picker.
  • pattern and pattern_descriptor: pattern validates the value. pattern_descriptor displays a readable format hint beside the field.
  • airbyte_secret: Uses a secret input that masks the value.

Form rules​

IDRule
UX-01Every field has a title and description
UX-02Titles use sentence case
UX-03Description says more than the title
UX-04Required fields have a default when a safe value exists
UX-05pattern has a readable pattern_descriptor
UX-06Secrets marked airbyte_secret
UX-07Fields with a default are required (recommended, not checked)
UX-08Explain meaning and impact
UX-09Say where to get each value
UX-10Doc links point to anchors
UX-11Auth is not entirely optional
UX-12Date pattern also sets format: date/date-time

UX-01: Every field has a title and description​

Rule: Give every user-facing field a clear title and description. The description should help the user understand what to enter.

Why: Without a title or description, users see a raw property name and have no explanation of the value.

Exceptions: A oneOf selector and its discriminator const fields do not need their own title and description.

Poor
spec:
  connection_specification:
    type: object
    properties:
      data_format:
        type: string
        enum: [csv, json]
Good
spec:
  connection_specification:
    type: object
    properties:
      aws_region_name:
        type: string
        title: AWS region
        description: AWS region in which CloudTrail stores trail events. Choose the region used by your trail in the AWS console.

UX-02: Titles use sentence case​

Rule: Write titles in sentence case: capitalize the first word and proper nouns, not every word. Keep product names and acronyms correctly capitalized.

Why: Sentence case makes field labels easier to scan and keeps the form consistent.

Exceptions: Preserve capitalization in product names and acronyms, such as “API key” and “Azure Blob Storage.”

Poor
spec:
  connection_specification:
    type: object
    properties:
      account_name:
        type: string
        title: Account Name
        description: Name to assign to the account created by this connector.
Good
spec:
  connection_specification:
    type: object
    properties:
      account_name:
        type: string
        title: Account name
        description: Name to assign to the account created by this connector.

UX-03: Description says more than the title​

Rule: Explain the field's purpose in its description instead of repeating its title. Add context that helps the user choose a value.

Why: A description that repeats the label does not help users decide what to enter.

Poor
spec:
  connection_specification:
    type: object
    properties:
      lookback_window:
        type: integer
        title: Lookback window
        description: Lookback window in days.
Good
spec:
  connection_specification:
    type: object
    properties:
      lookback_window:
        type: integer
        title: Lookback window
        description: Also known as the attribution or conversion window, this sets how many days of past records the connector revisits. A larger window can capture late-arriving updates but increases the records processed.

UX-04: Required fields have a default when a safe value exists​

Rule: Give required fields a default when a safe value exists. A common region or is_sandbox: false can be a safe default; never default credentials or IDs.

Why: A safe default pre-fills the form and saves users from choosing a value that works for most configurations. The field also collapses out of the main form, unless it sets always_show: true.

Poor
spec:
  connection_specification:
    type: object
    required: [aws_region_name]
    properties:
      aws_region_name:
        type: string
        title: AWS region
        description: AWS region in which CloudTrail stores trail events. Choose the region used by your trail in the AWS console.
Good
spec:
  connection_specification:
    type: object
    required: [aws_region_name]
    properties:
      aws_region_name:
        type: string
        title: AWS region
        description: AWS region in which CloudTrail stores trail events. Choose the region used by your trail in the AWS console.
        default: us-east-1

UX-05: pattern has a readable pattern_descriptor​

Rule: When you validate a value with pattern, provide a readable pattern_descriptor. Describe the expected format rather than relying on users to interpret a regular expression.

Why: The form shows a readable hint beside the field when pattern_descriptor is set or the web application recognizes the pattern. An explicit descriptor keeps other patterns understandable.

Poor
spec:
  connection_specification:
    type: object
    properties:
      account_identifier:
        type: string
        title: Account identifier
        description: Account identifier used to select the account to replicate. Find it in the account settings page.
        pattern: "^[A-Z]{3}-[0-9]{4}$"
Good
spec:
  connection_specification:
    type: object
    properties:
      account_identifier:
        type: string
        title: Account identifier
        description: Account identifier used to select the account to replicate. Find it in the account settings page.
        pattern: "^[A-Z]{3}-[0-9]{4}$"
        pattern_descriptor: Enter three uppercase letters, a hyphen, and four digits.

UX-06: Secrets marked airbyte_secret​

Rule: Mark every secret field with airbyte_secret: true. Do not mark public values, IDs, or other non-secret fields as secrets.

Why: Unmarked credentials appear in a regular text control instead of a masked secret control.

Poor
spec:
  connection_specification:
    type: object
    required: [aws_secret_key]
    properties:
      aws_secret_key:
        type: string
        title: AWS secret access key
        description: Used to sign CloudTrail API requests. Create an access key for the connector's IAM user in AWS IAM.
Good
spec:
  connection_specification:
    type: object
    required: [aws_secret_key]
    properties:
      aws_secret_key:
        type: string
        title: AWS secret access key
        description: Used to sign CloudTrail API requests. Create an access key for the connector's IAM user in AWS IAM.
        airbyte_secret: true

UX-07: Fields with a default are required​

Rule: If a field has a default, add it to the parent's list of required properties. This ensures every saved configuration contains the value that determines the connector's behavior.

Why: The web app fills defaults only when someone creates a configuration in the UI. API, Terraform, and PyAirbyte callers do not get those defaults, so an optional field with a default may be missing from saved configurations. Required fields with a default collapse out of the main form, so this rule doesn't add clutter. Set always_show: true when users should see the field right away.

warning

UX-07 is a recommendation. Automated checks don't report it. Applying it to an existing connector means making an optional field required, which is a breaking change: API and Terraform updates of saved configurations that lack the field fail validation. A breaking change is rarely worth it for this rule, so apply it to new fields and be judicious about existing ones.

Poor
spec:
  connection_specification:
    type: object
    properties:
      num_workers:
        type: integer
        title: Worker count
        description: Number of workers used to process the replication job; larger counts can process data faster but use more resources.
        default: 2
Good
spec:
  connection_specification:
    type: object
    required: [aws_region_name]
    properties:
      aws_region_name:
        type: string
        title: AWS region
        description: AWS region in which CloudTrail stores trail events. Choose the region used by your trail in the AWS console.
        default: us-east-1

UX-08: Explain meaning and impact​

Rule: Explain what a field controls and how changing it affects the data, sync, or output. Give users enough context to choose a value.

Why: A vague label or short description leaves users guessing about the effect of a setting.

Poor
spec:
  connection_specification:
    type: object
    properties:
      lookback_window:
        type: integer
        title: Lookback window
        description: Number of days before the cursor to check for records.
Good
spec:
  connection_specification:
    type: object
    properties:
      lookback_window:
        type: integer
        title: Lookback window
        description: Number of days before the cursor to revisit during incremental sync. A larger window can capture late-arriving updates but increases the records and API requests processed.

UX-09: Say where to get each value​

Rule: Tell users where to find each value they must provide. Link to setup instructions when they can explain the steps.

Why: Users should not have to search a vendor's site or guess which credential or identifier you need.

Exceptions: Do not apply this rule to fields OAuth fills in through advanced_auth.complete_oauth_output_specification.

Poor
spec:
  connection_specification:
    type: object
    required: [access_key]
    properties:
      access_key:
        type: string
        title: Gong access key
        description: Key used to authenticate requests to the Gong API.
        airbyte_secret: true
Good
spec:
  connection_specification:
    type: object
    required: [api_key]
    properties:
      api_key:
        type: string
        title: API key
        description: API key for your Amplitude project. Create it in your organization's settings using the <a href="https://docs.airbyte.com/integrations/sources/amplitude#setup-guide">setup guide</a>.
        airbyte_secret: true

UX-10: Doc links point to anchors​

Rule: Link to the relevant heading or setup section when a page has one. Use a stable, current URL rather than sending users to a page's top-level overview.

Why: An anchored link takes users directly to the instructions they need.

Poor
spec:
  connection_specification:
    type: object
    required: [access_token]
    properties:
      access_token:
        type: string
        title: Private app access token
        description: Create a private app in HubSpot and copy its access token. See the <a href="https://docs.airbyte.com/integrations/sources/hubspot">HubSpot setup guide</a>.
        airbyte_secret: true
Good
spec:
  connection_specification:
    type: object
    required: [api_key]
    properties:
      api_key:
        type: string
        title: API key
        description: API key for your Amplitude project. Create it in your organization's settings using the <a href="https://docs.airbyte.com/integrations/sources/amplitude#setup-guide">setup guide</a>.
        airbyte_secret: true

UX-11: Auth is not entirely optional​

Rule: Put the method selector (credentials) in the top-level required list and define one oneOf option per authentication method. If anonymous access is supported, represent No auth as an explicit option rather than making the selector optional; GitHub, HubSpot, and Jira manifests demonstrate required selectors with method options.

Why: If the selector is optional, users can save a configuration without choosing or providing an authentication method.

Exceptions: Deliberately optional credentials, such as a public API key or a database password that can be omitted, are still reported. You can ignore the finding when the connector intentionally supports that configuration.

Poor
spec:
  connection_specification:
    type: object
    properties:
      credentials:
        type: object
        title: Authentication
        description: Choose API key authentication or no authentication.
        oneOf:
          - title: API key
            type: object
            required: [auth_type, api_key]
            properties:
              auth_type:
                type: string
                const: API_KEY
              api_key:
                type: string
                title: API key
                description: Create an API key in the provider's developer console.
                airbyte_secret: true
          - title: No auth
            type: object
            required: [auth_type]
            properties:
              auth_type:
                type: string
                const: NO_AUTH
Good
spec:
  connection_specification:
    type: object
    required: [credentials]
    properties:
      credentials:
        type: object
        title: Authentication
        description: Choose API key authentication or no authentication.
        oneOf:
          - title: API key
            type: object
            required: [auth_type, api_key]
            properties:
              auth_type:
                type: string
                const: API_KEY
              api_key:
                type: string
                title: API key
                description: Create an API key in the provider's developer console.
                airbyte_secret: true
          - title: No auth
            type: object
            required: [auth_type]
            properties:
              auth_type:
                type: string
                const: NO_AUTH

UX-12: Date pattern also sets format: date/date-time​

Rule: For a date or date-time field, set type: string, the matching format, and a pattern with a readable pattern_descriptor. Use date for a date and date-time when the value includes a time.

Why: The date picker requires the string type and date format. A pattern without a format renders a plain text field.

Poor
spec:
  connection_specification:
    type: object
    properties:
      start_date:
        type: string
        title: Start date
        description: Date from which to start syncing records; earlier records are not replicated.
        pattern: "^[0-9]{4}-[0-9]{2}-[0-9]{2}$"
        pattern_descriptor: YYYY-MM-DD
Good
spec:
  connection_specification:
    type: object
    properties:
      start_date:
        type: string
        title: Start date
        description: UTC date from which to start syncing Google Ads records; earlier records are not replicated.
        format: date
        pattern: "^[0-9]{4}-[0-9]{2}-[0-9]{2}$"
        pattern_descriptor: YYYY-MM-DD

Other form guidance​

Configuration adds work and creates opportunities for confusion. Avoid asking users to configure a behavior when the connector can choose a safe value or handle it automatically. Avoid exposing implementation details such as retry counts unless users need them to reach a goal.

Use order and group to arrange related fields. Keep technical or niche options out of the main flow with collapsed groups, and use always_show only when users need to see a field that would otherwise collapse. Give every oneOf option a title, add useful examples, and use enum instead of free text when the accepted values are fixed.

Fail fast and provide an actionable error. Use pattern to reject malformed values and pattern_descriptor to explain the expected format. See UX-05 and UX-12.

Output data and schemas​

Strongly favor ELT over ETL​

Extract-load-transform (ELT) means a connector loads data into the destination in the shape the source provides it, and users transform it afterward. Extract-transform-load (ETL) means the connector reshapes data before loading it.

A connector's features shouldn't stand between users and their goals. When a connector honors the most literal representation of the source API and leaves transformation to the user's warehouse, every user gets a faithful copy of their data and full control over its final shape. ELT is the most democratic model Airbyte can offer, for the following reasons.

Removes Airbyte as a bottleneck​

If a connector shapes its output for one way of using the data, users who need a different shape must ask for connector changes and wait for them. When the connector loads data the way the source provides it, each user can transform it in their destination, on their own schedule and with their own tools.

Will always be backwards compatible​

APIs already follow strong conventions to maintain backwards compatibility. By transforming data, we break this guarantee, which means we may break downstream flows for our users.

Future proof​

We may have a vision of what a user needs today. But if our persona evolves next year, then we’ll probably also need to adapt our transformation logic, which would require significant development and data migration efforts.

More flexible​

Current users have different needs from data. By being opinionated on how they should consume data, we are effectively favoring one user persona over the other. While there might be some cases where this is warranted, it should be done with extreme intentionality.

More efficient​

With ETL, if the “T” ever needs to change, then we need to re-extract all data for all users. This is computationally and financially expensive and will place a lot of pressure on the source systems as we re-extract all data.

Describe output schemas as completely and reliably as possible​

Our most popular destinations use strongly typed systems such as Postgres and BigQuery, and file formats such as Parquet.

Being strongly typed enables optimizations and syntactic sugar to make it very easy and efficient for the user to query data.

To provide the best UX when moving data to these destinations, Airbyte source connectors should describe their schema in as much detail as correctness allows.

In some cases, describing schemas is impossible to do reliably. For example, MongoDB doesn’t have any schemas. To infer the schema, one needs to read all the records in a particular table. And even then, once new records are added, they also must all be read in order to update the inferred schema. At the time of writing, this is infeasible to do efficiently in Airbyte since we do not have an intermediate staging area to do this. In this case, we should do the “best we can” to describe the schema, keeping in mind that reliability of the described schema is more important than expressiveness.

That is, we would rather not describe a schema at all than describe it incorrectly, as incorrect descriptions will lead to failures downstream.

To keep schema descriptions reliable, automate schema generation whenever possible.

Be very cautious about breaking changes to output schemas​

Assuming we follow ELT over ETL, and automate generation of output schemas, this should come up very rarely. However, it is still important enough to warrant mention.

If for any reason we need to change the output schema declared by a connector in a backwards breaking way, consider it a necessary evil that should be avoided if possible. Basically, the only reasons for a backwards breaking change should be:

  • A connector previously had an incorrect schema.
  • A connector was not following ELT principles and is now being changed to follow them.

Other breaking changes should probably be escalated for approval.

Prerequisite configurations and assumptions​

Document all assumptions​

If a connector makes assumptions about the underlying data source, then these assumptions must be documented. For example: for some features of the Facebook Pages connector to work, a user must have an Instagram Business account linked to an Instagram page linked to their Facebook Page. In this case, the externally facing documentation page for the connector must be very clear about this.

Provide how-tos for prerequisite configuration​

If a user needs to set up their data source in a particular way to pull data, then we must provide documentation for how they should do it.

For example, to set up CDC for databases, a user must create logical replication slots and do a few other things. These steps should be documented with examples or screenshots wherever possible (for example, the SQL statements to run, the required permissions, and the relevant screen).

External documentation​

Write external-facing connector documentation on docs.airbyte.com, such as the Amazon Seller Partner page.

Documentation should communicate persona-impacting behaviors​

When writing documentation ask: who is the intended target persona for a piece of documentation, and what information do they need to understand how this connector impacts their workflows?

For example, data analysts and analytics engineers probably do not care which implementation we use for database replication. To them, the important thing is that we provide Change Data Capture (CDC). The implementation is a detail. Therefore, when communicating information about database replication, we should emphasize the end behaviors rather than implementation details.

Example: Don’t say: “The CDC implementation cannot process UTF-16 characters.”

Instead, say: “When using CDC, UTF-16 characters are not currently supported.”

A user who does not know which implementation we use might be confused by the first phrasing, so we should use the second phrasing.

Was this page helpful?