Dataset#

Vortex files implement the Arrow Dataset interface permitting efficient use of a Vortex file within query engines like DuckDB and Polars. In particular, Vortex will read data proportional to the number of rows passing a filter condition and the number of columns in a selection. For most Vortex encodings, this property holds true even when the filter condition specifies a single row.

VortexDataset

Read Vortex files with row filter and column selection pushdown.

VortexScanner

A PyArrow Dataset Scanner that reads from a Vortex Array.
class vortex.dataset.VortexDataset(dataset)#

Read Vortex files with row filter and column selection pushdown.

This class implements the pyarrow.dataset.Dataset interface which enables its use with Polars, DuckDB, Pandas and others.

count_rows(filter: Expression | None = None, batch_size: int | None = None, batch_readahead: int | None = None, fragment_readahead: int | None = None, fragment_scan_options: FragmentScanOptions | None = None, use_threads: bool | None = None, memory_pool: MemoryPool | None = None) → int#

Not implemented.

filter(expression: Expression) → VortexDataset#

Not implemented.

get_fragments(filter: Expression | None = None) → Iterator[Fragment]#

Not implemented.

head(num_rows: int, columns: list[str] | None = None, filter: Expression | None = None, batch_size: int | None = None, batch_readahead: int | None = None, fragment_readahead: int | None = None, fragment_scan_options: FragmentScanOptions | None = None, use_threads: bool | None = None, memory_pool: MemoryPool | None = None) → Table#

Load the first num_rows of the dataset.

Parameters:

  • num_rows (int) – The number of rows to load.

  • columns (list of str) – The columns to keep, identified by name.

  • filter (pyarrow.dataset.Expression) – Keep only rows for which this expression evalutes to True. Any rows for which this expression evaluates to Null is removed.

  • batch_size (int) – The maximum number of rows per batch.

  • batch_readahead (int) – Not implemented.

  • fragment_readahead (int) – Not implemented.

  • fragment_scan_options (pyarrow.dataset.FragmentScanOptions) – Not implemented.

  • use_threads (bool) – Not implemented.

  • memory_pool (pyarrow.MemoryPool) – Not implemented.

Returns:

table

Return type:

pyarrow.Table

join(right_dataset, keys, right_keys=None, join_type=None, left_suffix=None, right_suffix=None, coalesce_keys=True, use_threads: bool | None = None) → InMemoryDataset#

Not implemented.

join_asof(right_dataset, on, by, tolerance, right_on=None, right_by=None) → InMemoryDataset#

Not implemented.

replace_schema(schema: Schema)#

Not implemented.

scanner(columns: list[str] | None = None, filter: Expression | None = None, batch_size: int | None = None, batch_readahead: int | None = None, fragment_readahead: int | None = None, fragment_scan_options: FragmentScanOptions | None = None, use_threads: bool | None = None, memory_pool: MemoryPool | None = None) → Scanner#

Construct a pyarrow.dataset.Scanner.

Parameters:

  • columns (list of str) – The columns to keep, identified by name.

  • filter (pyarrow.dataset.Expression) – Keep only rows for which this expression evalutes to True. Any rows for which this expression evaluates to Null is removed.

  • batch_size (int) – The maximum number of rows per batch.

  • batch_readahead (int) – Not implemented.

  • fragment_readahead (int) – Not implemented.

  • fragment_scan_options (pyarrow.dataset.FragmentScanOptions) – Not implemented.

  • use_threads (bool) – Not implemented.

  • memory_pool (pyarrow.MemoryPool) – Not implemented.

Returns:

table

Return type:

pyarrow.Table

property schema: Schema#

The common schema of the full Dataset

sort_by(sorting, **kwargs) → InMemoryDataset#

Not implemented.

take(indices: Array | Any, columns: list[str] | None = None, filter: Expression | None = None, batch_size: int | None = None, batch_readahead: int | None = None, fragment_readahead: int | None = None, fragment_scan_options: FragmentScanOptions | None = None, use_threads: bool | None = None, memory_pool: MemoryPool | None = None) → Table#

Load a subset of rows identified by their absolute indices.

Parameters:

  • indices (pyarrow.Array) – A numeric array of absolute indices into self indicating which rows to keep.

  • columns (list of str) – The columns to keep, identified by name.

  • filter (pyarrow.dataset.Expression) – Keep only rows for which this expression evalutes to True. Any rows for which this expression evaluates to Null is removed.

  • batch_size (int) – The maximum number of rows per batch.

  • batch_readahead (int) – Not implemented.

  • fragment_readahead (int) – Not implemented.

  • fragment_scan_options (pyarrow.dataset.FragmentScanOptions) – Not implemented.

  • use_threads (bool) – Not implemented.

  • memory_pool (pyarrow.MemoryPool) – Not implemented.

Returns:

table

Return type:

pyarrow.Table

to_batches(columns: list[str] | None = None, filter: Expression | None = None, batch_size: int | None = None, batch_readahead: int | None = None, fragment_readahead: int | None = None, fragment_scan_options: FragmentScanOptions | None = None, use_threads: bool | None = None, memory_pool: MemoryPool | None = None) → Iterator[RecordBatch]#

Construct an iterator of pyarrow.RecordBatch.

Parameters:

  • columns (list of str) – The columns to keep, identified by name.

  • filter (pyarrow.dataset.Expression) – Keep only rows for which this expression evalutes to True. Any rows for which this expression evaluates to Null is removed.

  • batch_size (int) – The maximum number of rows per batch.

  • batch_readahead (int) – Not implemented.

  • fragment_readahead (int) – Not implemented.

  • fragment_scan_options (pyarrow.dataset.FragmentScanOptions) – Not implemented.

  • use_threads (bool) – Not implemented.

  • memory_pool (pyarrow.MemoryPool) – Not implemented.

Returns:

table

Return type:

pyarrow.Table

to_record_batch_reader(columns: list[str] | None = None, filter: Expression | None = None, batch_size: int | None = None, batch_readahead: int | None = None, fragment_readahead: int | None = None, fragment_scan_options: FragmentScanOptions | None = None, use_threads: bool | None = None, memory_pool: MemoryPool | None = None) → RecordBatchReader#

Construct a pyarrow.RecordBatchReader.

Parameters:

  • columns (list of str) – The columns to keep, identified by name.

  • filter (pyarrow.dataset.Expression) – Keep only rows for which this expression evalutes to True. Any rows for which this expression evaluates to Null is removed.

  • batch_size (int) – The maximum number of rows per batch.

  • batch_readahead (int) – Not implemented.

  • fragment_readahead (int) – Not implemented.

  • fragment_scan_options (pyarrow.dataset.FragmentScanOptions) – Not implemented.

  • use_threads (bool) – Not implemented.

  • memory_pool (pyarrow.MemoryPool) – Not implemented.

Returns:

table

Return type:

pyarrow.Table

to_table(columns: list[str] | None = None, filter: Expression | None = None, batch_size: int | None = None, batch_readahead: int | None = None, fragment_readahead: int | None = None, fragment_scan_options: FragmentScanOptions | None = None, use_threads: bool | None = None, memory_pool: MemoryPool | None = None) → Table#

Construct an Arrow pyarrow.Table.

Parameters:

  • columns (list of str) – The columns to keep, identified by name.

  • filter (pyarrow.dataset.Expression) – Keep only rows for which this expression evalutes to True. Any rows for which this expression evaluates to Null is removed.

  • batch_size (int) – The maximum number of rows per batch.

  • batch_readahead (int) – Not implemented.

  • fragment_readahead (int) – Not implemented.

  • fragment_scan_options (pyarrow.dataset.FragmentScanOptions) – Not implemented.

  • use_threads (bool) – Not implemented.

  • memory_pool (pyarrow.MemoryPool) – Not implemented.

Returns:

table

Return type:

pyarrow.Table

class vortex.dataset.VortexScanner(dataset: VortexDataset, columns: list[str] | None = None, filter: Expression | None = None, batch_size: int | None = None, batch_readahead: int | None = None, fragment_readahead: int | None = None, fragment_scan_options: FragmentScanOptions | None = None, use_threads: bool | None = None, memory_pool: MemoryPool | None = None)#

A PyArrow Dataset Scanner that reads from a Vortex Array.

Parameters:

  • dataset (VortexDataset) – The dataset to scan.

  • columns (list of str) – The columns to keep, identified by name.

  • filter (pyarrow.dataset.Expression) – Keep only rows for which this expression evalutes to True. Any rows for which this expression evaluates to Null is removed.

  • batch_size (int) – The maximum number of rows per batch.

  • batch_readahead (int) – Not implemented.

  • fragment_readahead (int) – Not implemented.

  • fragment_scan_options (pyarrow.dataset.FragmentScanOptions) – Not implemented.

  • use_threads (bool) – Not implemented.

  • memory_pool (pyarrow.MemoryPool) – Not implemented.

Returns:

table

Return type:

pyarrow.Table

count_rows(self)#

Count rows matching the scanner filter.

Returns:

count

Return type:

int

head(num_rows: int) → Table#

Load the first num_rows of the dataset.

Parameters:

num_rows (int) – The number of rows to read.

Returns:

table

Return type:

pyarrow.Table

scan_batches() → Iterator[TaggedRecordBatch]#

Not implemented.

to_batches() → Iterator[RecordBatch]#

Construct an iterator of pyarrow.RecordBatch.

Returns:

table

Return type:

pyarrow.Table

to_reader() → RecordBatchReader#

Construct a pyarrow.RecordBatchReader.

Returns:

table

Return type:

pyarrow.Table

to_table() → Table#

Construct an Arrow pyarrow.Table.

Returns:

table

Return type:

pyarrow.Table