Skip to main content

GitHub

This page contains the setup guide and reference information for the GitHub source connector.

Prerequisites

  • Start date
  • GitHub Repositories
  • Branch (Optional)
  • Page size for large streams (Optional)

For Airbyte Cloud:

  • Access Token

For Airbyte OSS:

  • Personal Access Token

Setup guide

Step 1: Set up GitHub

Create a GitHub Account.

Airbyte Open Source additional setup steps

Log into GitHub and then generate a personal access token. To load balance your API quota consumption across multiple API tokens, input multiple tokens separated with ,.

Step 2: Set up the GitHub connector in Airbyte

For Airbyte Cloud:

  1. Log into your Airbyte Cloud account.
  2. In the left navigation bar, click Sources. In the top-right corner, click + new source.
  3. On the source setup page, select GitHub from the Source type dropdown and enter a name for this connector.
  4. Click Authenticate your GitHub account.
  5. Log in and Authorize to the GitHub account.
  6. Start date - The date from which you'd like to replicate data for streams: comments, commit_comment_reactions, commit_comments, commits, deployments, events, issue_comment_reactions, issue_events, issue_milestones, issue_reactions, issues, project_cards, project_columns, projects, pull_request_comment_reactions, pull_requests, pull_requeststats, releases, review_comments, reviews, stargazers, workflow_runs, workflows.
  7. GitHub Repositories - Space-delimited list of GitHub organizations/repositories, e.g. airbytehq/airbyte for single repository, airbytehq/airbyte airbytehq/another-repo for multiple repositories. If you want to specify the organization to receive data from all its repositories, then you should specify it according to the following example: airbytehq/*.
  8. Branch (Optional) - Space-delimited list of GitHub repository branches to pull commits for, e.g. airbytehq/airbyte/master. If no branches are specified for a repository, the default branch will be pulled. (e.g. airbytehq/airbyte/master airbytehq/airbyte/my-branch).
  9. Page size for large streams (Optional) - The GitHub connector contains several streams with a large load. The page size of such streams depends on the size of your repository. Recommended to specify values between 10 and 30.

For Airbyte OSS:

  1. Authenticate with Personal Access Token.

Supported sync modes

The GitHub source connector supports the following sync modes:

FeatureSupported?
Full Refresh SyncYes
Incremental - Append SyncYes
Replicate Incremental DeletesComing soon
SSL connectionYes
NamespacesNo

Supported Streams

This connector outputs the following full refresh streams:

This connector outputs the following incremental streams:

Notes

  1. Only 4 streams (comments, commits, issues and review comments) from the above 24 incremental streams are pure incremental meaning that they:

    • read only new records;
    • output only new records.
  2. Stream workflow_runs is almost pure incremental:

    • read new records and some portion of old records (in past 30 days) docs;
    • output only new records.
  3. Other 19 incremental streams are also incremental but with one difference, they:

    • read all records;
    • output only new records. Please, consider this behaviour when using those 19 incremental streams because it may affect you API call limits.
  4. We are passing few parameters (since, sort and direction) to GitHub in order to filter records and sometimes for large streams specifying very distant start_date in the past may result in keep on getting error from GitHub instead of records (respective WARN log message will be outputted). In this case Specifying more recent start_date may help. The "Start date" configuration option does not apply to the streams below, because the GitHub API does not include dates which can be used for filtering:

  • assignees
  • branches
  • collaborators
  • issue_labels
  • organizations
  • pull_request_commits
  • pull_request_stats
  • repositories
  • tags
  • teams
  • users

Permissions and scopes

If you use OAuth authentication method, the oauth2.0 application requests the next list of scopes: repo, read:org, read:repo_hook, read:user, read:discussion, workflow. For personal access token it need to manually select needed scopes.

Your token should have at least the repo scope. Depending on which streams you want to sync, the user generating the token needs more permissions:

  • For syncing Collaborators, the user which generates the personal access token must be a collaborator. To become a collaborator, they must be invited by an owner. If there are no collaborators, no records will be synced. Read more about access permissions here.
  • Syncing Teams is only available to authenticated members of a team's organization. Personal user accounts and repositories belonging to them don't have access to Teams features. In this case no records will be synced.
  • To sync the Projects stream, the repository must have the Projects feature enabled.

Performance considerations

The GitHub connector should not run into GitHub API limitations under normal usage. Please create an issue if you see any rate limit issues that are not automatically retried successfully.

Changelog

VersionDatePull RequestSubject
0.2.412022-07-0314376Add Retry for GraphQL API Resource limitations
0.2.402022-07-0114338Revert: "Rename field mergeable to is_mergeable"
0.2.392022-06-3014274Rename field mergeable to is_mergeable
0.2.382022-06-2713989Use GraphQL for reviews stream
0.2.372022-06-2113955Fix "secondary rate limit" not retrying
0.2.362022-06-2013926Break point added for workflows_runs stream
0.2.352022-06-1613763Use GraphQL for pull_request_stats stream
0.2.342022-06-1413707Fix API sorting, fix get_starting_point caching
0.2.332022-06-0813558Enable caching only for parent streams
0.2.322022-06-0713531Fix different result from get_starting_point when reading by pages
0.2.312022-05-2413115Add incremental support for streams WorkflowRuns
0.2.302022-05-0912294Add incremental support for streams CommitCommentReactions, IssueCommentReactions, IssueReactions, PullRequestCommentReactions, Repositories, Workflows
0.2.292022-05-04#12482Update input configuration copy
0.2.282022-04-2111893Add new streams TeamMembers, TeamMemberships
0.2.272022-04-0211678Fix "PAT Credentials" in spec
0.2.262022-03-3111623Re-factored incremental sync for Reviews stream
0.2.252022-03-3111567Improve code for better error handling
0.2.242022-03-309251Add Streams Workflow and WorkflowRuns
0.2.232022-03-1711212Improve documentation and spec for Beta
0.2.222022-03-1010878Fix error handling for unavailable streams with 404 status code
0.2.212022-03-0410749Add new stream ProjectCards
0.2.202022-02-1610385Add new stream Deployments, ProjectColumns, PullRequestCommits
0.2.192022-02-0710211Add human-readable error in case of incorrect organization or repo name
0.2.182021-02-0910193Add handling secondary rate limits
0.2.172021-02-029999Remove BAD_GATEWAY code from backoff_time
0.2.162021-02-029868Add log message for streams that are restricted for OAuth. Update oauth scopes.
0.2.152021-01-269802Add missing fields for auto_merge in pull request stream
0.2.142021-01-219664Add custom pagination size for large streams
0.2.132021-01-209619Fix logging for function should_retry
0.2.112021-01-179492Remove optional parameter Accept for reaction`s streams to fix error with 502 HTTP status code in response
0.2.102021-01-037250Use CDK caching and convert PR-related streams to incremental
0.2.92021-12-299179Use default retry delays on server error responses
0.2.82021-12-078524Update connector fields title/description
0.2.72021-12-068518Add connection retry with GitHub
0.2.62021-11-248030Support start date property for PullRequestStats and Reviews streams
0.2.52021-11-218170Fix slow check connection for organizations with a lot of repos
0.2.42021-11-117856Resolve $ref fields in some stream schemas
0.2.32021-10-066833Fix config backward compatability
0.2.22021-10-056761Add oauth worflow specification
0.2.12021-09-226223Add option to pull commits from user-specified branches
0.2.02021-09-195898 and 6227Don't minimize any output fields & add better error handling
0.1.112021-09-155949Add caching for all streams
0.1.102021-09-095860Add reaction streams
0.1.92021-09-025788Handling empty repository, check method using RepositoryStats stream
0.1.82021-09-015757Add more streams
0.1.72021-08-275696Handle negative backoff values
0.1.62021-08-185456Add MultipleTokenAuthenticator
0.1.52021-08-185456Fix set up validation
0.1.42021-08-135136Support syncing multiple repositories/organizations
0.1.32021-08-035156Extended existing schemas with users property for certain streams
0.1.22021-07-134708Fix bug with IssueEvents stream and add handling for rate limiting
0.1.12021-07-074590Fix schema in the pull_request stream
0.1.02021-07-064174New Source: GitHub