Skip to main content

Hugging Face - Datasets

Availability
Available CoreAvailable StandardAvailable PlusAvailable ProAvailable Enterprise FlexAvailable PyAirbyte
Connector type (beta)
Available Data replicationNot available Agent(learn more)
Support Level
Marketplace
Connector Version
0.0.67 (last updated 4 days ago)
Sync Success Rate
Low
Usage Rate
Low
Data workers used
0.2 per sync (Learn more)
Definition ID
38438040-03d9-406d-b10b-af83beadd3ef

Imports datasets from Hugging Face (https://huggingface.co/datasets)

Only datasets with Parquet exports can be imported with this connector.

The connector reads data through the Hugging Face dataset viewer API (https://datasets-server.huggingface.co). Anonymous requests are rate limited per IP address, so syncs of larger datasets may fail with 429 Too Many Requests unless you provide a Hugging Face user token. Private and gated datasets always require a token with access to the dataset.

Configuration​

InputTypeDescriptionDefault Value
dataset_namestringDataset Name.
dataset_subsetsarrayDataset Subsets. Dataset Subsets to import. Will import all of them if nothing is provided (see https://huggingface.co/docs/dataset-viewer/en/configs_and_splits for more details)
dataset_splitsarrayDataset Splits. Splits to import. Will import all of them if nothing is provided (see https://huggingface.co/docs/dataset-viewer/en/configs_and_splits for more details)
api_tokenstringHugging Face API Token. Optional for public datasets, but required for private or gated datasets and strongly recommended to avoid anonymous per-IP rate limits (see https://huggingface.co/docs/hub/en/rate-limits). Create one at https://huggingface.co/settings/tokens.

Streams​

Stream NamePrimary KeyPaginationSupports Full SyncSupports Incremental
rowsDefaultPaginator✅❌
splitsNo pagination✅❌

IP allow list​

If you use Airbyte Cloud and your organization restricts access to specific IPs, add the Airbyte Cloud IP addresses to your allow list.

Reference​

Config fields reference

Field
Type
Property name
Dataset Name
required
string
dataset_name
string
api_token
array
dataset_splits
array
dataset_subsets

Changelog​

Expand to review
VersionDatePull RequestSubject
0.0.672026-10-1388676Update dependencies
0.0.662026-10-07PRAdd optional Hugging Face API token, honor rate-limit headers with longer 429 backoff, and surface clear 401/404 errors
0.0.652026-10-0687904Update dependencies
0.0.642026-09-2987224Update dependencies
0.0.632026-09-2286680Update dependencies
0.0.622026-09-1586089Update dependencies
0.0.612026-09-0885544Update dependencies
0.0.602026-08-1884611Update dependencies
0.0.592026-08-1183976Update dependencies
0.0.582026-08-0483490Update dependencies
0.0.572026-07-2882964Update dependencies
0.0.562026-07-2182457Update dependencies
0.0.552026-07-1481860Update dependencies
0.0.542026-06-3081111Update dependencies
0.0.532026-06-2380530Update dependencies
0.0.522026-06-1679903Update dependencies
0.0.512026-06-0979343Update dependencies
0.0.502026-06-0278783Update dependencies
0.0.492026-04-2877280Update dependencies
0.0.482026-04-2176633Update dependencies
0.0.472026-03-3175656Update dependencies
0.0.462026-03-1774523Update dependencies
0.0.452026-03-0373521Update dependencies
0.0.442026-02-1073049Update dependencies
0.0.432026-01-2071918Update dependencies
0.0.422026-01-1471724Update dependencies
0.0.412025-12-1870468Update dependencies
0.0.402025-11-2570018Update dependencies
0.0.392025-11-1869378Update dependencies
0.0.382025-10-2968799Update dependencies
0.0.372025-10-2168218Update dependencies
0.0.362025-10-1467867Update dependencies
0.0.352025-10-0767407Update dependencies
0.0.342025-09-3066397Update dependencies
0.0.332025-09-0966054Update dependencies
0.0.322025-08-2365368Update dependencies
0.0.312025-08-0964600Update dependencies
0.0.302025-08-0264206Update dependencies
0.0.292025-07-2663917Update dependencies
0.0.282025-07-1963503Update dependencies
0.0.272025-07-1263096Update dependencies
0.0.262025-07-0562652Update dependencies
0.0.252025-06-2161825Update dependencies
0.0.242025-06-1461131Update dependencies
0.0.232025-05-2460604Update dependencies
0.0.222025-05-1059877Update dependencies
0.0.212025-05-0359255Update dependencies
0.0.202025-04-2658760Update dependencies
0.0.192025-04-1957716Update dependencies
0.0.182025-04-0557037Update dependencies
0.0.172025-03-2956694Update dependencies
0.0.162025-03-2256059Update dependencies
0.0.152025-03-0855433Update dependencies
0.0.142025-03-0154762Update dependencies
0.0.132025-02-2254324Update dependencies
0.0.122025-02-1553812Update dependencies
0.0.112025-02-0853292Update dependencies
0.0.102025-02-0152789Update dependencies
0.0.92025-01-2552244Update dependencies
0.0.82025-01-1851820Update dependencies
0.0.72025-01-1151202Update dependencies
0.0.62024-12-2850621Update dependencies
0.0.52024-12-2150079Update dependencies
0.0.42024-12-1449609Update dependencies
0.0.32024-12-1249233Update dependencies
0.0.22024-12-1148911Starting with this version, the Docker image is now rootless. Please note that this and future versions will not be compatible with Airbyte versions earlier than 0.64
0.0.12024-11-28Initial release by @michel-tricot via Connector Builder
Was this page helpful?