Skip to main content

GitLab

Availability
Available CoreAvailable StandardAvailable PlusAvailable ProAvailable Enterprise FlexAvailable PyAirbyte
Connector type (beta)
Available Data replicationAvailable Agent(learn more)
Support Level
Airbyte
Connector Version
4.5.1 (last updated 6 days ago)
Sync Success Rate
High
Usage Rate
High
Data workers used
0.2 per sync (Learn more)
Definition ID
5e6175e5-68e1-4c17-bff9-56103bbb0d80

This page contains the setup guide and reference information for the GitLab source connector.

Prerequisites​

  • A GitLab account on GitLab.com or a self-hosted GitLab instance (v4 API)

For Airbyte Cloud:

For Airbyte Open Source:

Setup guide​

Step 1: Set up GitLab​

Create a GitLab account or set up a self-hosted GitLab instance.

If you authenticate with a personal access token, generate one in Settings > Access Tokens with the read_api scope. The read_api scope grants read-only access to the API, including all groups, projects, the container registry, and the package registry.

Step 2: Set up the GitLab connector in Airbyte​

For Airbyte Cloud:

  1. Log into your Airbyte Cloud account.
  2. In the left navigation bar, click Sources. In the top-right corner, click + new source.
  3. On the source setup page, select GitLab from the Source type dropdown and enter a name for this connector.
  4. Click Authenticate your GitLab account by selecting OAuth or Personal Access Token for authentication.
  5. Log in and authorize the GitLab account.
  6. API URL (Optional) - The host of your self-hosted GitLab instance, or gitlab.com (default).
  7. Start date (Optional) - The date from which you'd like to replicate data for streams, in the format YYYY-MM-DDT00:00:00Z.
  8. Groups (Optional) - List of GitLab group paths, e.g. airbytehq for a single group.
  9. Projects (Optional) - List of GitLab project paths, e.g. airbytehq/airbyte.
  10. Number of Concurrent Threads (Optional) - The number of concurrent threads used for syncing. Higher values can speed up syncs but may hit rate limits. Defaults to 4, with an allowed range of 2 to 25. Adjust based on your GitLab instance's rate limits.
  11. Click Set up source.

For Airbyte Open Source:

  1. Navigate to the Airbyte UI and click Sources > + New source.
  2. Select GitLab and enter a name for the connector.
  3. Authenticate with a Personal Access Token.
  4. API URL (Optional) - The host of your self-hosted GitLab instance, or gitlab.com (default).
  5. Start date (Optional) - The date from which you'd like to replicate data for streams, in the format YYYY-MM-DDT00:00:00Z.
  6. Groups (Optional) - List of GitLab group paths, e.g. airbytehq for a single group.
  7. Projects (Optional) - List of GitLab project paths, e.g. airbytehq/airbyte.
  8. Number of Concurrent Threads (Optional) - The number of concurrent threads used for syncing. Higher values can speed up syncs but may hit rate limits. Defaults to 4, with an allowed range of 2 to 25. Adjust based on your GitLab instance's rate limits.
  9. Click Set up source.

Choose which groups and projects to sync​

Almost every stream in this connector is scoped to a group or a project, so Groups and Projects determine how much data the sync covers.

  • If you leave both blank, the connector discovers every group your token can see and syncs the projects in those groups.
  • If you set Groups, the connector syncs those groups, their descendant groups, and the projects in them.
  • If you set both, Projects acts as a filter on the projects discovered from your groups. If that filter matches nothing, the connector falls back to syncing the project paths in Projects directly, which is how you sync a project that isn't in one of the listed groups.

Use full paths, not display names or numeric IDs: airbytehq for a group and airbytehq/airbyte for a project.

API URL format​

The API URL takes a host, not a full API endpoint. The connector appends /api/v4/ itself, so gitlab.company.org and https://gitlab.company.org both work, but a value containing a path, such as gitlab.company.org/api/v4, fails validation. Only http and https schemes are accepted, and Airbyte Cloud rejects http.

Supported sync modes​

The GitLab source connector supports the following sync modes:

Supported streams​

This connector outputs the following streams:

Limitations​

API version​

This connector uses GitLab API v4. It works with both GitLab.com and self-hosted GitLab instances.

Incremental sync window​

Incremental streams request data in 180-day windows, so a first sync of a long-lived project issues many requests. Most incremental streams filter on updated_at, but commits filters on created_at using GitLab's since and until parameters. If you leave Start date blank, incremental streams start from 2014-01-01, which is effectively all history for most projects. Set a start date to cut the initial sync short.

Incremental pipeline child streams​

jobs, pipelines_extended, and pipeline_trigger_jobs make one request per pipeline. In Incremental mode, a sync only makes these requests for the pipelines returned by the incremental pipelines query, which are the pipelines whose updated_at is at or after the previous sync's cursor. All three streams use pipeline_updated_at, the pipeline's updated_at as returned by pipelines, as their cursor. Keep the following in mind:

  • Changes that don't update the parent pipeline's updated_at, such as erasing a job or job artifacts expiring, aren't picked up. Use Full Refresh for these streams if you need those changes.
  • When a job is retried, GitLab returns only the new attempt, under a new job id. The row for the earlier attempt keeps the status it had when it was last synced, which can be running or pending. To get one row per job, keep the highest id per pipeline_id and name. Deleted pipelines and their jobs are not removed from the destination.
  • In pipeline_trigger_jobs, the downstream_pipeline fields, such as status, reflect the downstream pipeline when the upstream pipeline last changed. A trigger job without strategy: depend finishes as soon as the downstream pipeline is created, so later status changes in the downstream pipeline don't update the upstream pipeline, and the row isn't read again. To get the downstream pipeline's current status, join downstream_pipeline_id to the pipelines stream, or use Full Refresh for pipeline_trigger_jobs.
  • The cursor has one-second precision and GitLab's updated_after filter includes the boundary. On every sync, the most recently updated regular pipeline and the most recently updated child pipeline in each project are requested again, so up to two pipelines per project. Use Incremental | Append + Deduped to avoid duplicate rows.
  • A project that first appears after the first incremental sync starts from the latest pipelines cursor minus the duration of the previous sync, not from Start date. This applies to projects added to the connector configuration and to projects created in or moved into a configured group. Older pipelines in that project aren't read. Refresh these streams to backfill them.
  • Changing a stream's sync mode clears its saved state. The first Incremental sync after switching from Full Refresh reads every pipeline from Start date. With Incremental | Append, rows written by earlier Full Refresh syncs stay in the destination; use Append + Deduped or clear the stream when you switch.
  • In Full Refresh mode, a failed attempt can resume instead of reading every pipeline again when the destination supports refreshes and the schema has been refreshed after upgrading to 4.5.0. This also applies to existing Full Refresh connections that never switch to Incremental. An attempt resumes from the last completed 180-day pipelines window in each project, so some pipelines are read twice; use Full Refresh | Overwrite + Deduped to avoid duplicate rows. Each new Full Refresh sync still starts from Start date.
  • Pipelines are paginated by page number. If a pipeline is updated while a sync is paging through results, another pipeline can shift to an earlier page and not be returned. That pipeline's child records are skipped until it's updated again.

Child pipelines on very large instances​

Since version 4.4.41, pipelines reads each project twice: once for regular pipelines and once with source=parent_pipeline for child pipelines. Existing connections keep their per-project cursor for regular pipelines and backfill child pipelines from Start date automatically.

The backfill is not guaranteed when the stream tracks more than 10,000 partitions (more than roughly 5,000 projects, since each project now has two partitions). Above that limit the connector stores a single shared cursor for the whole stream instead of one per project, and child pipelines older than that cursor are not read. If your connection is that large, reset the pipelines, pipelines_extended, jobs, and pipeline_trigger_jobs streams once after upgrading to load historical child pipelines.

Rate limits​

The connector respects per-endpoint rate limits based on GitLab.com's documented defaults for the Groups API, Members API, Projects API, and general authenticated traffic:

EndpointRate limit
GET /groups200 requests/min
GET /groups/:id400 requests/min
GET /groups/:id/descendant_groups200 requests/min
GET /projects/:id400 requests/min
GET /groups/:id/members/all200 requests/min
GET /projects/:id/members/all200 requests/min
All other authenticated requests2,000 requests/min

Starting October 19, 2026, GitLab.com also applies rate limits by subscription tier: Free namespaces get 5,000 requests per hour and 100 requests per minute per user, and Premium and Ultimate follow in January 2027. See rate limits on GitLab.com. Under these limits, syncs can pause for long periods while the connector waits for the limit to reset.

Self-hosted GitLab instances may have different rate limits configured by the administrator. The connector also reads the RateLimit-Remaining and RateLimit-Reset response headers. When GitLab returns HTTP 429, the connector waits for the time in the Retry-After header, up to about one hour, and then retries. If the header is missing, it retries with exponential backoff. If you encounter persistent rate limit errors, create an issue.

You can adjust the Number of Concurrent Threads setting to control how many parallel requests the connector makes. Lower this value if you share your API quota with other integrations or if you experience rate limiting.

How the connector handles API errors​

ResponseBehavior
401 UnauthorizedThe sync fails with Unable to refresh the access_token, please re-authenticate in Sources > Settings. Re-authenticate the source, or replace an expired or revoked personal access token.
403 ForbiddenThe connector skips the group, project, or resource and continues. Nothing fails, so partial data is the only symptom.
404 Not FoundThe sync fails with Groups and/or projects that you provide are invalid or you don't have permission to view it. Check the paths in Groups and Projects for typos and confirm your token can see them.
500 Internal Server ErrorRetried.

Because 403 responses are skipped silently, missing data usually means a permissions gap rather than a bug. Verify that your token has access to every group and project you expect to sync.

Connection checks read the groups endpoint. A check can succeed while individual project streams still return no data, if the token can list groups but can't read the projects inside them.

IP allow list​

If you use Airbyte Cloud and your organization restricts access to specific IPs, add the Airbyte Cloud IP addresses to your allow list.

Reference​

Config fields reference

Field
Type
Property name
object
credentials
string
api_url
string
groups
array<string>
groups_list
integer
num_workers
string
projects
array<string>
projects_list
string
start_date

Changelog​

Expand to review
VersionDatePull RequestSubject
4.5.22026-10-1388467Honor GitLab Retry-After headers for HTTP 429 rate limits and raise heartbeat timeout to 90 minutes
4.5.12026-10-1388629Update dependencies
4.5.02026-10-0787655Add incremental sync support to the pipelines_extended, jobs, and pipeline_trigger_jobs streams. Failed Full Refresh attempts of these streams can now resume, which can write duplicate rows under Full Refresh | Overwrite
4.4.432026-10-0687865Update dependencies
4.4.422026-09-2987138Update dependencies
4.4.412026-09-2486933Include child pipelines in the pipelines stream (and therefore pipelines_extended and jobs) and add the pipeline_trigger_jobs stream
4.4.402026-09-2286599Update dependencies
4.4.392026-09-1586063Update dependencies
4.4.382026-09-0885485Update dependencies
4.4.372026-08-1884568Update dependencies
4.4.362026-08-1183941Update dependencies
4.4.352026-07-2882903Update dependencies
4.4.342026-07-2182429Update dependencies
4.4.332026-07-1481830Update dependencies
4.4.322026-06-3081041Update dependencies
4.4.312026-06-2380476Update dependencies
4.4.302026-06-1777516Lower default concurrency from 8 to 4; set HTTPAPIBudget to documented GitLab.com rate limits
4.4.292026-06-1679874Update dependencies
4.4.282026-06-0979295Update dependencies
4.4.272026-06-0278724Update dependencies
4.4.262026-04-2877265Update dependencies
4.4.252026-04-2176608Update dependencies
4.4.242026-04-1376276Rename "concurrent workers" to "concurrent threads" in connector spec
4.4.232026-04-1076210Promote 4.4.23-rc.5 to GA: HTTPAPIBudget with concurrency 8
4.4.23-rc.52026-04-0175994Revert concurrency to 8 with HTTPAPIBudget for progressive rollout tuning
4.4.23-rc.42026-03-3175915Re-enable HTTPAPIBudget with concurrency 10 for rate limit protection during progressive rollout
4.4.23-rc.32026-03-2775537Increase default concurrency from 8 to 10 for progressive rollout tuning
4.4.23-rc.22026-03-2575480Comment out HTTPAPIBudget to isolate concurrency tuning during progressive rollout
4.4.23-rc.12026-03-0970858Add HTTPAPIBudget and concurrency_level for improved sync performance
4.4.222026-02-2473774Update dependencies
4.4.212026-02-1773401Update dependencies
4.4.202026-02-1071398Update dependencies
4.4.192026-02-0572262Update CDK version from 7.6.1 to 7.6.5
4.4.182025-12-1870698Update dependencies
4.4.172025-12-1170839Change check stream from projects to groups to fix timeout for large GitLab organizations
4.4.162025-11-2569875Update dependencies
4.4.152025-11-1869411Update dependencies
4.4.142025-10-2969030Update dependencies
4.4.132025-10-2168297Update dependencies
4.4.122025-10-1468000Update dependencies
4.4.112025-10-0767266Update dependencies
4.4.102025-09-3066076Update dependencies
4.4.92025-09-1065997Update dependencies
4.4.82025-08-2665556Update dependencies
4.4.72025-08-2365362Update dependencies
4.4.62025-08-0964579Update dependencies
4.4.52025-08-0264222Update dependencies
4.4.42025-07-2663913Update dependencies
4.4.32025-07-1963489Update dependencies
4.4.22025-07-1863309Adds type property to config_normalization_rules in manifest
4.4.12025-07-1261471Update dependencies
4.4.02025-07-0762831Promoting release candidate 4.4.0-rc.1 to a main version.
4.4.0-rc.12025-07-0262480Migrate to manifest-only
4.3.72025-06-0257264Update dependencies
4.3.62025-03-2954945Update dependencies
4.3.52025-02-2351695Update dependencies
4.3.42025-01-1544671Starting with this version, the Docker image is now rootless. Please note that this and future versions will not be compatible with Airbyte versions earlier than 0.64
4.3.32024-08-1844207Update dependencies
4.3.22024-08-1243856Update dependencies
4.3.12024-08-0443058Update dependencies
4.3.02024-07-3142920Migrate to CDK v4.1.0
4.2.22024-07-2842601Update dependencies
4.2.12024-07-2142295Update dependencies
4.2.02024-07-1742085Migrate to CDK v2.4.0
4.1.02024-07-1742021Migrate to CDK v1.8.0
4.0.82024-07-1441835Update dependencies
4.0.72024-07-1141470Update dependencies
4.0.62024-07-1041100Update dependencies
4.0.52024-07-0740894Update dependencies
4.0.42024-06-2640417Update dependencies
4.0.32024-06-2340102Update dependencies
4.0.22024-05-0736637Schema descriptions and CDK 0.80.0
4.0.12024-04-2337505Set error code 500 as retryable
4.0.02024-04-0135989Migrate to low-code
3.0.02024-02-1534548Fix merge_request_commits stream to return commits for each merge request
2.1.22024-02-1335167Manage dependencies with Poetry.
2.1.12024-01-1634203prepare for airbyte-lib
2.1.02024-01-0933676Add fields to Commits (extended_trailers), Groups (emails_enabled, service_access_tokens_expiration_enforced) and Projects (code_suggestions, model_registry_access_level) streams
2.0.02023-10-2631700Add correct date-time format for Deployments, Projects and Groups Members streams
1.8.42023-10-1931599Base image migration: remove Dockerfile and use the python-connector-base image
1.8.32023-10-1831547Add validation for invalid groups_list and/or projects_list
1.8.22023-10-1731492Expand list of possible error status codes when handling expired access_token
1.8.12023-10-1231375Mark start_date as optional, migrate groups and projects to array
1.8.02023-10-1231339Add undeclared fields to streams schemas, validate date/date-time format in stream schemas
1.7.12023-10-1031210Added expired access_token handling, while checking the connection
1.7.02023-08-0829203Add Deployments stream
1.6.02023-06-3027869Add shared_runners_setting field to groups
1.5.12023-06-2427679Fix formatting
1.5.02023-06-1527392Make API URL an optional parameter in spec.
1.4.22023-06-1527346Partially revert changes made in version 1.0.4, disallow http calls in cloud.
1.4.12023-06-1327351Fix OAuth token expiry date.
1.4.02023-06-1227234Skip stream slices on 403/404 errors, do not fail syncs.
1.3.12023-06-0827147Improve connectivity check for connections with no projects/groups
1.3.02023-06-0827150Update stream schemas
1.2.12023-06-0226947New field name added to Pipelines and PipelinesExtended stream schema
1.2.02023-05-1722293Preserve data in records with flattened keys
1.1.12023-05-2326422Fix error 404 Repository Not Found when syncing project with Repository feature disabled
1.1.02023-05-1025948Introduce two new fields in the Projects stream schema
1.0.42023-04-2021373Accept api_url with or without scheme
1.0.32023-02-1422992Specified date formatting in specification
1.0.22023-01-2722001Set AvailabilityStrategy for streams explicitly to None
1.0.12023-01-2321713Fix missing data issue
1.0.02022-12-057506Add OAuth2.0 authentication option
0.1.122022-12-1520542Revert HttpAvailability changes, run on cdk 0.15.0
0.1.112022-12-1420479Use HttpAvailabilityStrategy + add unit tests
0.1.102022-12-1220384Fetch groups along with their subgroups
0.1.92022-12-1120348Fix 403 error when syncing EpicIssues stream
0.1.82022-12-0220023Fix duplicated records issue for Projects stream
0.1.72022-12-0119986Fix GroupMilestones stream schema
0.1.62022-06-2313252Add GroupIssueBoards stream
0.1.52022-05-0211907Fix null projects param and container_expiration_policy
0.1.42022-03-2311140Ingest All Accessible Groups if not Specified in Config
0.1.32021-12-218991Update connector fields title/description
0.1.22021-10-187108Allow all domains to be used as api_url
0.1.12021-10-126932Fix pattern field in spec file, remove unused fields from config files, use cache from CDK
0.1.02021-07-064174Initial Release
Was this page helpful?