Struct vortex_file::builder::VortexReadBuilder

pub struct VortexReadBuilder<R> { /* private fields */ }

Builder for reading Vortex files.

Succinctly, the file format specification is as follows:

1. Data is written first, in a form that is describable by a Layout (typically Array IPC Messages). a. To allow for more efficient IO & pruning, our writer implementation first writes the “data” arrays, and then writes the “metadata” arrays (i.e., per-column statistics)
2. We write what is collectively referred to as the “Footer”, which contains: a. An optional Schema, which if present is a valid flatbuffer representing a message::Schema b. The Layout, which is a valid footer::Layout flatbuffer, and describes the physical byte ranges & relationships amongst the those byte ranges that we wrote in part 1. c. The Postscript, which is a valid footer::Postscript flatbuffer, containing the absolute start offsets of the Schema & Layout flatbuffers within the file. d. The End-of-File marker, which is 8 bytes, and contains the u16 version, u16 postscript length, and 4 magic bytes.

Reified File Format

┌────────────────────────────┐
│                            │
│            Data            │
│    (Array IPC Messages)    │
│                            │
├────────────────────────────┤
│                            │
│   Per-Column Statistics    │
│                            │
├────────────────────────────┤
│                            │
│     Schema Flatbuffer      │
│                            │
├────────────────────────────┤
│                            │
│     Layout Flatbuffer      │
│                            │
├────────────────────────────┤
│                            │
│    Postscript Flatbuffer   │
│  (Schema & Layout Offsets) │
│                            │
├────────────────────────────┤
│     8-byte End of File     │
│(Version, Postscript Length,│
│       Magic Bytes)         │
└────────────────────────────┘

Implementations

impl<R: VortexReadAt + Unpin> VortexReadBuilder<R>

pub fn new(read_at: R, layout_serde: LayoutDeserializer) -> Self

pub fn with_file_size(self, size: u64) -> Self

pub fn with_projection(self, projection: Projection) -> Self

pub fn with_indices(self, array: ArrayData) -> Self

pub fn with_row_filter(self, row_filter: RowFilter) -> Self

pub fn with_io_dispatcher(self, dispatcher: Arc<IoDispatcher>) -> Self

pub fn with_initial_read(self, initial_read: InitialRead) -> Self

pub async fn build(self) -> VortexResult<VortexFileArrayStream<R>>