Crate vortex_array

Vortex crate containing core logic for encoding and memory representation of arrays.

At the heart of Vortex are arrays and encodings. Arrays are typed views of memory buffers that hold scalars. These buffers can be held in a number of physical encodings to perform lightweight compression that exploits the particular data distribution of the array’s values.

Every data type recognized by Vortex also has a canonical physical encoding format, which arrays can be canonicalized into for ease of access in compute functions.

Re-exports

• pub use paste;

Modules

• accessor
• aliases
• array
  All the built-in encoding schemes and arrays.
• arrow
  Utilities to work with Arrow data and types.
• builders
• compress
• compute
  Compute kernels on top of Vortex Arrays.
• encoding
  Traits and types to define shared unique encoding identifiers.
• flatbuffers
  Re-exported autogenerated code from the core Vortex flatbuffer definitions.
• iter
• nbytes
• patches
• stats
  Traits and utilities to compute and access array statistics.
• stream
• tree
• validity
  Array validity and nullability behavior, used by arrays and compute functions.
• variants
  This module defines array traits for each Vortex DType.
• visitor

Macros

• impl_encoding
  Macro to generate all the necessary code for a new type of array encoding. Including:

Structs

• ArrayChildrenIterator
  A depth-first pre-order iterator over a ArrayData.
• ArrayData
  A central type for all Vortex arrays, which are known length sequences of typed and possibly compressed data.
• ChildrenCollector
• Context
  A mapping between an encoding’s ID to an EncodingRef, used to have a shared view of all available encoding schemes.
• NamedChildrenCollector

Enums

• Canonical
  The set of canonical array encodings, also the set of encodings that can be transferred to Arrow with zero-copy.

Traits

• ArrayDType
• ArrayLen
• ArrayMetadata
  Dynamic trait used to represent opaque owned Array metadata
• ArrayTrait
  Collects together the behavior of an array.
• GetArrayMetadata
• IntoArrayData
  Consume self and turn it into an ArrayData infallibly.
• IntoArrayVariant
  Trait for types that can be converted from an owned type into an owned array variant.
• IntoCanonical
  Support trait for transmuting an array into the canonical encoding for its [vortex_dtype::DType].
• IntoCanonicalVTable
  Encoding VTable for canonicalizing an array.
• MetadataVTable
• ToArrayData
• TryDeserializeArrayMetadata
• TrySerializeArrayMetadata