Vortex is a general purpose toolkit for working with columnar data built around the latest research from the database community.
Vortex in-memory arrays support:
Vortex ships with an extensible file format supporting:
Vortex defines a work-in-progress IPC format for sending possibly compressed arrays over the wire.
Vortex is designed to be incredibly extensible. Almost all reader and writer logic is extensible at compile-time by providing various implementations of Rust traits, and encodings and layouts are extensible at runtime with dynamically loaded libraries or WebAssembly kernels.
Please reach out to us if you’d like to extend Vortex with your own encodings, layouts, or other functionality.
Vortex is more like an ecosystem of building blocks than it is a specific format or specification. Almost everything in Vortex is extensible, enabling it to be used for both general-purpose columnar data processing, and niche embedded use-cases where specific encodings and performance characteristics are required.
This section of the documentation covers the core concepts of Vortex and how they piece together.
---
maxdepth: 3
includehidden:
caption: Concepts
---
Arrays <concepts/arrays>
Layouts <concepts/layouts>
Data Types <concepts/dtypes>
Compute <concepts/compute>
Vortex is currently available for both Python and Rust. The user guides for each language provide a comprehensive overview of the Vortex API and how to use it.
---
maxdepth: 3
includehidden:
caption: User Guide
---
Python <python/index>
Rust <rust/index>
Vortex currently defines two serialization formats: a file format and an IPC format. The file format is designed for random access to Vortex Layouts on disk, while the IPC format is designed to efficiently send possibly compressed Vortex Arrays over the wire.
---
maxdepth: 3
includehidden:
caption: Specifications
---
specs/file-format
specs/ipc-format
specs/dtype-format
---
hidden:
caption: Project Links
---
references
Spiral <https://spiraldb.com>
GitHub <https://github.com/spiraldb/vortex>
PyPI <https://pypi.org/project/vortex-array>
Crates <https://crates.io/crates/vortex>