Dataset

Warning

The pyarrow.dataset module is experimental (specifically the classes), and a stable API is not yet guaranteed.

Factory functions

dataset(source[, schema, format, …])	Open a dataset.
parquet_dataset(metadata_path[, schema, …])	Create a FileSystemDataset from a _metadata file created via pyarrrow.parquet.write_metadata.
partitioning([schema, field_names, flavor, …])	Specify a partitioning scheme.
field(name)	Reference a named column of the dataset.
scalar(value)	Expression representing a scalar value.

Classes

FileFormat
ParquetFileFormat
Partitioning
PartitioningFactory
DirectoryPartitioning	A Partitioning based on a specified Schema.
HivePartitioning	A Partitioning for “/$key=$value/” nested directories as found in Apache Hive.
Dataset	Collection of data fragments and potentially child datasets.
FileSystemDataset	A Dataset of file fragments.
FileSystemFactoryOptions	Influences the discovery of filesystem paths.
FileSystemDatasetFactory	Create a DatasetFactory from a list of paths with schema inspection.
UnionDataset	A Dataset wrapping child datasets.
Scanner	A materialized scan operation with context and options bound.
Expression