Module vortex_flatbuffers::footer

A file format footer containining a serialized vortex-serde Layout.

footer.fbs:

/// A `Buffer` is a simple container for the `begin` and `end` byte offsets within the file.
/// These offsets are absolute (i.e., relative to the start of the file).
struct Buffer {
    begin: uint64;
    end: uint64;
}

/// A `Layout` is a recursive data structure that describes the physical layout of the data in a Vortex file.
/// As a starting, concrete example, the first three Layout encodings are defined as:
///
/// 1. encoding == 1, `Flat` -> one buffer, zero child Layouts
/// 2. encoding == 2, `Chunked` -> zero buffers, one or more child Layouts (used for chunks of rows)
/// 3. encoding == 3, `Columnar` -> zero buffers, one or more child Layouts (used for columns of structs)
///
/// The `row_count` represents the number of rows represented by this Layout. This is very useful for
/// pruning the Layout tree based on row filters.
///
/// The `metadata` field is fully opaque at this layer, and allows the Layout implementation corresponding to
/// `encoding` to embed additional information that may be useful for the reader. For example, the `ChunkedLayout`
/// uses the first byte of the `metadata` array as a boolean to indicate whether the first child Layout represents
/// the statistics table for the other chunks. 
table Layout {
    encoding: uint16;
    buffers: [Buffer];
    children: [Layout];
    row_count: uint64;
    metadata: [ubyte];
}

/// The `Postscript` is guaranteed by the file format to never exceed 65528 bytes (i.e., u16::MAX - 8 bytes)
/// in length, and is immediately followed by an 8-byte `EndOfFile` struct.
///
/// The `EndOfFile` struct cannot change size without breaking backwards compatibility. It is not written/read
/// using flatbuffers, but the equivalent flatbuffer definition would be:
///
/// struct EndOfFile {
///     version: uint16;
///     footer_length: uint16;
///     magic: [uint8; 4]; // "VTXF"
/// }
///
table Postscript {
    schema_offset: uint64;
    layout_offset: uint64;
}

root_type Layout;
root_type Postscript;

Structs

• Buffer
  A Buffer is a simple container for the begin and end byte offsets within the file. These offsets are absolute (i.e., relative to the start of the file).
• Layout
  A Layout is a recursive data structure that describes the physical layout of the data in a Vortex file. As a starting, concrete example, the first three Layout encodings are defined as:
• LayoutArgs
• LayoutBuilder
• Postscript
  The Postscript is guaranteed by the file format to never exceed 65528 bytes (i.e., u16::MAX - 8 bytes) in length, and is immediately followed by an 8-byte EndOfFile struct.
• PostscriptArgs
• PostscriptBuilder

Enums

• LayoutOffset
• PostscriptOffset

Functions

• finish_postscript_buffer
• finish_size_prefixed_postscript_buffer
• root_as_postscript
  Verifies that a buffer of bytes contains a Postscript and returns it. Note that verification is still experimental and may not catch every error, or be maximally performant. For the previous, unchecked, behavior use root_as_postscript_unchecked.
• root_as_postscript_unchecked^⚠
  Assumes, without verification, that a buffer of bytes contains a Postscript and returns it.
• root_as_postscript_with_opts
  Verifies, with the given options, that a buffer of bytes contains a Postscript and returns it. Note that verification is still experimental and may not catch every error, or be maximally performant. For the previous, unchecked, behavior use root_as_postscript_unchecked.
• size_prefixed_root_as_postscript
  Verifies that a buffer of bytes contains a size prefixed Postscript and returns it. Note that verification is still experimental and may not catch every error, or be maximally performant. For the previous, unchecked, behavior use size_prefixed_root_as_postscript_unchecked.
• size_prefixed_root_as_postscript_unchecked^⚠
  Assumes, without verification, that a buffer of bytes contains a size prefixed Postscript and returns it.
• size_prefixed_root_as_postscript_with_opts
  Verifies, with the given verifier options, that a buffer of bytes contains a size prefixed Postscript and returns it. Note that verification is still experimental and may not catch every error, or be maximally performant. For the previous, unchecked, behavior use root_as_postscript_unchecked.