Enum vortex_array::Canonical

pub enum Canonical {
    Null(NullArray),
    Bool(BoolArray),
    Primitive(PrimitiveArray),
    Struct(StructArray),
    List(ListArray),
    VarBinView(VarBinViewArray),
    Extension(ExtensionArray),
}

The set of canonical array encodings, also the set of encodings that can be transferred to Arrow with zero-copy.

Note that a canonical form is not recursive, i.e. a StructArray may contain non-canonical child arrays, which may themselves need to be canonicalized.

Logical vs. Physical encodings

Vortex separates logical and physical types, however this creates ambiguity with Arrow, there is no separation. Thus, if you receive an Arrow array, compress it using Vortex, and then decompress it later to pass to a compute kernel, there are multiple suitable Arrow array variants to hold the data.

To disambiguate, we choose a canonical physical encoding for every Vortex [DType], which will correspond to an arrow-rs [arrow_schema::DataType].

Views support

Binary and String views, also known as “German strings” are a better encoding format for nearly all use-cases. Variable-length binary views are part of the Apache Arrow spec, and are fully supported by the Datafusion query engine. We use them as our canonical string encoding for all Utf8 and Binary typed arrays in Vortex.

Variants

Null(NullArray)

Bool(BoolArray)

Primitive(PrimitiveArray)

Struct(StructArray)

List(ListArray)

VarBinView(VarBinViewArray)

Extension(ExtensionArray)

Implementations

impl Canonical

pub fn into_arrow(self) -> VortexResult<ArrayRef>

Convert a canonical array into its equivalent [ArrayRef](Arrow array).

Scalar arrays such as Bool and Primitive canonical arrays should convert with zero copies, while more complex variants such as Struct may require allocations if its child arrays require decompression.

impl Canonical

pub fn empty(dtype: &DType) -> VortexResult<Canonical>

impl Canonical

pub fn into_null(self) -> VortexResult<NullArray>

pub fn into_bool(self) -> VortexResult<BoolArray>

pub fn into_primitive(self) -> VortexResult<PrimitiveArray>

pub fn into_struct(self) -> VortexResult<StructArray>

pub fn into_list(self) -> VortexResult<ListArray>

pub fn into_varbinview(self) -> VortexResult<VarBinViewArray>

pub fn into_extension(self) -> VortexResult<ExtensionArray>

Trait Implementations

impl AsRef<ArrayData> for Canonical

fn as_ref(&self) -> &ArrayData

Converts this type into a shared reference of the (usually inferred) input type.

impl Clone for Canonical

fn clone(&self) -> Canonical

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Canonical

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<Canonical> for ArrayData

This conversion is always “free” and should not touch underlying data. All it does is create an owned pointer to the underlying concrete array type.

This combined with the above IntoCanonical impl for ArrayData allows simple two-way conversions between arbitrary Vortex encodings and canonical Arrow-compatible encodings.

fn from(value: Canonical) -> Self

Converts to this type from the input type.

impl IntoArrayData for Canonical

fn into_array(self) -> ArrayData