Trait vortex_file::LayoutReader

pub trait LayoutReader: Debug + Send {
    // Required methods
    fn add_splits(
        &self,
        row_offset: usize,
        splits: &mut BTreeSet<usize>,
    ) -> VortexResult<()>;
    fn poll_read(
        &self,
        selector: &RowMask,
    ) -> VortexResult<Option<PollRead<ArrayData>>>;

    // Provided methods
    fn poll_metadata(
        &self,
    ) -> VortexResult<Option<PollRead<Vec<Option<ArrayData>>>>> { ... }
    fn poll_prune(
        &self,
        _begin: usize,
        _end: usize,
    ) -> VortexResult<PollRead<Prune>> { ... }
}

A reader for a layout, a serialized sequence of Vortex arrays.

Some layouts are horizontally divisible: they can read a sub-sequence of rows independently of other sub-sequences. A layout advertises its subdivisions in its add_splits method. Any layout which is or contains a chunked layout is horizontally divisible.

The poll_read method accepts and applies a RowMask, reading only the subdivisions which contain the selected rows.

State management

Layout readers are synchronous and stateful. A request to read a given row range may trigger a request for more messages, which will be handled by the caller, placing the messages back into the message cache for this layout as a result.

Required Methods

fn add_splits( &self, row_offset: usize, splits: &mut BTreeSet<usize>, ) -> VortexResult<()>

Register all horizontal row boundaries of this layout.

Layout should register all indivisible absolute row boundaries of the data stored in itself and its children. row_offset gives the relative row position of this layout to the beginning of the file.

fn poll_read( &self, selector: &RowMask, ) -> VortexResult<Option<PollRead<ArrayData>>>

Reads the data from the underlying layout within given selection

Layout is required to return all data for given selection in one batch. Layout can either return a batch of data (i.e., an Array) or ask for more layout messages to be read. When requesting messages to be read the caller should populate the message cache used when creating the invoked instance of this trait and then call back into this function.

The layout is finished producing data for selection when it returns None

Provided Methods

fn poll_metadata( &self, ) -> VortexResult<Option<PollRead<Vec<Option<ArrayData>>>>>

Reads the metadata of the layout, if it exists.

LayoutReaders can override the default behavior, which is to return no metadata.

fn poll_prune( &self, _begin: usize, _end: usize, ) -> VortexResult<PollRead<Prune>>

Introspect to determine if we can prune the given [begin, end) row range.

LayoutReaders can opt out of the default implementation, which is to not prune.

Implementors