tectonicdb is a very fast, highly compressed standalone datastore and streaming protocol for order book ticks. Achieves ~50x compression on disk usage, ~100x compression on bandwidth usage compared to PostgreSQL. Easily scale to billions of records since file read is cheap and fast.
This software is motivated by reducing expenditure. 1TB stored on Google Cloud PostgreSQL was too expensive and too slow. Since financial data is usually read and stored in bulk, it is possible to convert into a more efficient format.
-
Uses a simple binary file format: Dense Tick Format(DTF) ...
-
Stores order book tick data tuple of shape:
(timestamp, seq, is\_trade, is\_bid, price, size). -
Sorted by seq/timestamp
-
12 bytes per row
-
11.5MB per 1 million row
There are several ways to install tectonicdb.
- Binaries
Binaries are available for download. Make sure to put the path to the binary into your PATH. Currently only released for Linux x86_64.
- Crates.io
Requires Rust. Once you have Rust installed, simply run:
cargo install tectonicdb
This will download and compile tectonic-server and tectonic-cli.
- GitHub
To contribute you will need the copy of the source code on your local machine.
git clone https://github.com/rickyhan/tectonic
cd tectonic
cargo build --lib
cargo build --bin tectonic-server
cargo build --bin tectonic-cli
The binaries can be found under target/release/debug folder.
It is possible to use the dtf protocol/format as a separate package. Works nicely with any buffer implementing the Write trait.
TectonicDB is a standalone service.
Operating system:
-
Ubuntu
-
macOS
Language bindings:
-
Rust
-
Python
-
JavaScript
-
Usage statistics like Cloud SQL
-
Query by timestamp