Skip to main content

Shopify

Availability:
Available Airbyte CloudAvailable Airbyte OSSAvailable PyAirbyte
Support Level:
Airbyte Connector
Connector Version:
2.5.12(Last updated 8 days ago)
CDK Version:
5.17.0
Sync Success Rate:
Usage Rate:

This page contains the setup guide and reference information for the Shopify source connector.

Prerequisites​

Setup guide​

This connector supports OAuth2.0 and API Password (for private applications) authentication methods.

Set up Shopify​

note

For existing Airbyte Cloud customers, if you are currently using the API Password authentication method, please switch to OAuth2.0, as the API Password will be deprecated shortly. This change will not affect Airbyte Open Source connections.

For Airbyte Cloud:​

  1. Log into your Airbyte Cloud account.
  2. Click Sources and then click + New source.
  3. On the Set up the source page, select Shopify from the Source type dropdown.
  4. Enter a name for the Shopify connector.

Connect using OAuth2.0​

  1. Choose a Source name.
  2. Click Authenticate your Shopify account.
  3. Click Install to install the Airbyte application. Log in to your account, if you are not already logged in. Select the store you want to sync and review the consent form. Click Install app to finish the installation.
  1. The Shopify Store field will be automatically filled after you authenticate your Shopify account based on the store you selected. Once populated, confirm the value is accurate.
  1. (Optional) You may set a Replication Start Date as the starting point for your data replication. Any data created before this date will not be synced. Defaults to January 1st, 2020.
  1. Click Set up source and wait for the connection test to complete.

For Airbyte Open Source:​

  1. Navigate to the Airbyte Open Source dashboard.
  2. Click Sources and then click + New source.
  3. On the Set up the source page, select Shopify from the Source type dropdown.
  4. Enter a name for the Shopify connector.

Create a custom app​

Authentication to the Shopify API requires a custom application. Follow these instructions to create a custom app and find your Admin API Access Token.

  1. Log in to your Shopify account.
  2. In the dashboard, navigate to Settings > App and sales channels > Develop apps > Create an app.
  3. Select a name for your new app.
  4. Select Configure Admin API scopes.
  5. Grant access to the following list of scopes. Only select scopes prefixed with read_, not write_ (e.g. read_locations,read_price_rules, etc ).
  6. Click Install app to give this app access to your data.
  7. Once installed, go to API Credentials to copy the Admin API Access Token. You are now ready to set up the source in Airbyte!

Connect using API Password​

  1. Enter a Source name.
  2. Enter your Shopify Store name. You can find this in your URL when logged in to Shopify or within the Store details section of your Settings.
  3. For API Password, enter your custom application's Admin API access token.
  4. (Optional) You may set a Replication Start Date as the starting point for your data replication. Any data created before this date will not be synced. Please note that this defaults to January 1st, 2020.
  5. Click Set up source and wait for the connection test to complete.

Custom app scopes​

Add the following scopes to your custom app to ensure Airbyte can sync all available data. For more information on access scopes, see the Shopify docs.

  • read_analytics
  • read_assigned_fulfillment_orders
  • read_content
  • read_customers
  • read_discounts
  • read_draft_orders
  • read_fulfillments
  • read_gdpr_data_request
  • read_gift_cards
  • read_inventory
  • read_legal_policies
  • read_locations
  • read_locales
  • read_marketing_events
  • read_merchant_managed_fulfillment_orders
  • read_online_store_pages
  • read_order_edits
  • read_orders
  • read_price_rules
  • read_product_listings
  • read_products
  • read_publications
  • read_reports
  • read_resource_feedbacks
  • read_script_tags
  • read_shipping
  • read_shopify_payments_accounts
  • read_shopify_payments_bank_accounts
  • read_shopify_payments_disputes
  • read_shopify_payments_payouts
  • read_themes
  • read_third_party_fulfillment_orders
  • read_translations

Supported sync modes​

The Shopify source connector supports the following sync modes:

FeatureSupported?
Full Refresh SyncYes
Incremental SyncYes

The Shopify source supports both Full Refresh and Incremental syncs. You can choose if this connector will copy only the new or updated data, or all rows in the tables and columns you set up for replication, every time a sync is run.

This source can sync data for the Shopify REST API and the Shopify GraphQL API and the Shopify GraphQL BULK API

Supported Streams​

Entity-Relationship Diagram (ERD)​

Capturing deleted records​

The connector captures deletions for records in the Articles, Blogs, CustomCollections, Orders, Pages, PriceRules and Products streams.

When a record is deleted, the connector outputs a record with the ID of that record and the deleted_at, deleted_message, and deleted_description fields filled out. No other fields are filled out for the deleted records.

Check the following Shopify documentation for more information about retrieving deleted records.

Marketing Attribution data​

Data related to marketing attribution can be found across a few different streams. Sync these streams to understand marketing performance:

  • Customer Journey Summary (firstVisit.source, firstVisit.sourcetype)
  • Orders (referring_site, source_name)
  • Abandoned Checkouts (referring_site, source_name)

Features​

FeatureSupported?(Yes/No)
Full Refresh SyncYes
Incremental - Append SyncYes
NamespacesNo

Data type map​

Integration TypeAirbyte Type
stringstring
numbernumber
arrayarray
objectobject
booleanboolean

Limitations & Troubleshooting​

Expand to see details about Shopify connector limitations and troubleshooting

Connector limitations​

Rate limiting​

Shopify has some rate limit restrictions. Typically, there should not be issues with throttling or exceeding the rate limits but, in some edge cases, you may encounter the following warning message:

"Caught retryable error '<some_error> or null' after <some_number> tries.
Waiting <some_number> seconds then retrying..."

This is expected when the connector hits a 429 - Rate Limit Exceeded HTTP Error. The sync operation will continue successfully after a short backoff period.

For all Shopify GraphQL BULK api requests these limitations are applied: https://shopify.dev/docs/api/usage/bulk-operations/queries#operation-restrictions

Troubleshooting​

  • If you encounter access errors while using OAuth2.0 authentication, please make sure you've followed this Shopify Article to request the access to the client's store first. Once the access is granted, you should be able to proceed with OAuth2.0 authentication.
  • Check out common troubleshooting issues for the Shopify source connector on our Airbyte Forum here.

Reference​

Config fields reference

Field
Type
Property name
string
shop
object
credentials
string
start_date
integer
bulk_window_in_days
boolean
fetch_transactions_user_id
boolean
job_product_variants_include_pres_prices
integer
job_termination_threshold
integer
job_checkpoint_interval

Changelog​

Expand to review
VersionDatePull RequestSubject
2.5.122024-11-2548661Starting with this version, the Docker image is now rootless. Please note that this and future versions will not be compatible with Airbyte versions earlier than 0.64
2.5.112024-11-0748402Add image_src, image_url, including options to Product Variants stream
2.5.102024-11-0548322Update dependencies
2.5.92024-10-2947814Update dependencies
2.5.82024-10-2847044Update dependencies
2.5.72024-10-1446552Add parent state tracking for BULK sub-streams
2.5.62024-10-1246844Update dependencies
2.5.52024-10-0546578Raise exception on missing stream
2.5.42024-10-0545759Update dependencies
2.5.32024-09-2746095Fixed duplicates for Product Images, Metafield Product Images and Metafield Products streams for Incremental syncs
2.5.22024-09-1745633Adds read_inventory as a required scope for product_variants stream
2.5.12024-09-1445255Update dependencies
2.5.02024-09-0645190Migrate to CDK v5
2.4.242024-09-0345116Have message and description be nullable for custom_collections deleted events
2.4.232024-08-3144971Update dependencies
2.4.222024-08-2444723Update dependencies
2.4.212024-08-1744318Update dependencies
2.4.202024-08-1243834Update dependencies
2.4.192024-08-1043194Update dependencies
2.4.182024-08-0643326Added missing type type for customer_journey_summary field for Customer Journey Summary stream schema
2.4.172024-08-0242973Fixed FAILED Job handling for no-checkpointing BULK Streams, fixed STATE collision for REST Streams with Deleted Events
2.4.162024-07-2142095Added the Checkpointing for the BULK streams, fixed the store redirection
2.4.152024-07-2742806Update dependencies
2.4.142024-07-2042150Update dependencies
2.4.132024-07-1341809Update dependencies
2.4.122024-07-1041103Update dependencies
2.4.112024-07-0941068Added options field to Product Variants stream
2.4.102024-07-0941042Use latest CDK: 3.0.0
2.4.92024-07-0640768Update dependencies
2.4.82024-07-0340707Fixed the bug when product_images stream emitted records with no primary_key
2.4.72024-06-2740593Use latest CDK version possible
2.4.62024-06-2640526Made BULK Job termination threshold limit adjustable from input configuration, increased the default value to 1 hour.
2.4.52024-06-2540484Update dependencies
2.4.42024-06-1939594Extended the Discount Codes, Fulfillment Orders, Inventory Items, Inventory Levels, Products, Product Variants and Transactions stream schemas
2.4.32024-06-0638084add resiliency on some transient errors using the HttpClient
2.4.12024-06-2039651Update dependencies
2.4.02024-06-1739527Added new stream Order Agreements
2.3.02024-06-1439487Added new stream Customer Journey Summary
2.2.32024-06-0638084add resiliency on some transient errors using the HttpClient
2.2.22024-06-0439019[autopull] Upgrade base image to v1.2.1
2.2.12024-05-3038769Have products stream return all the tags comma separated
2.2.02024-05-2938746Updated countries schema
2.1.42024-05-2438610Updated the source API Version to 2024-04
2.1.32024-05-2338464Added missing fields to Products stream
2.1.22024-05-2338352Migrated Order Risks stream to GraphQL BULK
2.1.12024-05-2038251Replace AirbyteLogger with logging.Logger
2.1.02024-05-0237767Migrated Products, Product Images and Product Variants to GraphQL BULK
2.0.82024-05-0237589Added retry for known HTTP Errors for BULK streams
2.0.72024-04-2436660Schema descriptions
2.0.62024-04-2237468Fixed one time retry for Internal Server Error for BULK streams
2.0.52024-04-0336788Added ability to dynamically adjust the size of the slice
2.0.42024-03-2236355Update CDK version to ensure Per-Stream Error Messaging and Record Counts In State (features were already there so just upping the version)
2.0.32024-03-1536170Fixed the STATE messages emittion frequency for the nested sub-streams
2.0.22024-03-1236000Fix and issue where invalid shop name causes index out of bounds error
2.0.12024-03-1135952Fixed the issue when start date is missing but the stream required it
2.0.02024-02-1232345Fixed the issue with state causing the substreams to skip the records, made metafield_*: collections, customers, draft_orders, locations, orders, product_images, product_variants, products, and fulfillment_orders, collections, discount_codes, inventory_levels, inventory_items, transactions_graphql, customer_address streams to use BULK Operations instead of REST
1.1.82024-02-1235166Manage dependencies with Poetry.
1.1.72024-01-1933804Updated documentation with list of all supported streams
1.1.62024-01-0433414Prepare for airbyte-lib
1.1.52023-12-2833827Fix GraphQL query
1.1.42023-10-1931599Base image migration: remove Dockerfile and use the python-connector-base image
1.1.32023-10-1731500Fixed the issue caused by the missing access token while setup the new source and not yet authenticated
1.1.22023-10-1331381Fixed the issue caused by the state presence while fetching the deleted events with pagination
1.1.12023-09-1830560Performance testing - include socat binary in docker image
1.1.02023-09-0730246Added ability to fetch destroyed records for Articles, Blogs, CustomCollections, Orders, Pages, PriceRules, Products
1.0.02023-08-1129361Migrate to the 2023-07 Shopify API Version
0.6.22023-08-0929302Handle the Internal Server Error when entity could be fetched
0.6.12023-08-0828291Allow shop field to accept *.myshopify.com shop names, updated OAuth Spec
0.6.02023-08-0228770Added Disputes stream
0.5.12023-07-1328700Improved error messages with more user-friendly description, refactored code
0.5.02023-06-1327732License Update: Elv2
0.4.02023-06-1327083Added CustomerSavedSearch, CustomerAddress and Countries streams
0.3.42023-05-1025961Added validation for shop in input configuration (accepts non-url-like inputs)
0.3.32023-04-1225110Fixed issue when cursor_field is "None", added missing properties to stream schemas, fixed access_scopes validation error
0.3.22023-02-2723473Fixed OOM / Memory leak issue for Airbyte Cloud
0.3.12023-01-1621461Added discount_applications to orders stream
0.3.02022-11-1619492Added support for graphql and add a graphql products stream
0.2.02022-10-2118298Updated API version to the 2022-10, make stream schemas backward cpmpatible
0.1.392022-10-1317962Added metafield streams; support for nested list streams
0.1.382022-10-1017777Fixed 404 for configured streams, fix missing cursor error for old records
0.1.372022-04-3012500Improve input configuration copy
0.1.362022-03-229850Added BalanceTransactions stream
0.1.352022-03-0710915Fixed a bug which caused full-refresh syncs of child REST entities configured for incremental
0.1.342022-03-0210794Minor specification re-order, fixed links in documentation
0.1.332022-02-1710419Fixed wrong field type for tax_exemptions for Abandoned_checkouts stream
0.1.322022-02-1810449Added tender_transactions stream
0.1.312022-02-0810175Fixed compatibility issues for legacy user config
0.1.302022-01-249648Added permission validation before sync
0.1.292022-01-209049Added shop_url to the record for all streams
0.1.282022-01-199591Implemented OAuth2.0 authentication method for Airbyte Cloud
0.1.272021-12-229049Updated connector fields title/description
0.1.262021-12-148597Fixed mismatched number of tables for base-normalization, increased performance of order_refunds stream
0.1.252021-12-028297Added Shop stream
0.1.242021-11-307783Reviewed and corrected schemas for all streams
0.1.232021-11-157973Added InventoryItems
0.1.222021-10-187101Added FulfillmentOrders, Fulfillments streams
0.1.212021-10-147382Fixed InventoryLevels primary key
0.1.202021-10-147063Added Location and InventoryLevels as streams
0.1.192021-10-116951Added support of OAuth 2.0 authorisation option
0.1.182021-09-216056Added pre_tax_price to the orders/line_items schema
0.1.172021-09-175244Created data type enforcer for converting prices into numbers
0.1.162021-09-095965Fixed the connector's performance for Incremental refresh
0.1.152021-09-025853Fixed amount type in order_refund schema
0.1.142021-09-025801Fixed line_items/discount allocations & duties parts of orders schema
0.1.132021-08-175470Fixed rate limits throttling
0.1.122021-08-095276Added status property to product schema
0.1.112021-07-234943Fixed products schema up to API 2021-07
0.1.102021-07-194830Fixed for streams json schemas, upgrade to API version 2021-07
0.1.92021-07-044472Incremental sync is now using updated_at instead of since_id by default
0.1.82021-06-294121Added draft orders stream
0.1.72021-06-264290Fixed the bug when limiting output records to 1 caused infinity loop
0.1.62021-06-244009Added pages, price rules and discount codes streams
0.1.52021-06-103973Added AIRBYTE_ENTRYPOINT for Kubernetes support
0.1.42021-06-093926New attributes to Orders schema
0.1.32021-06-083787Added Native Shopify Source Connector
Was this page helpful?