tickpack

A small lossless compressor for exchange tick data in CSV form. Written in C++ as a take-home exercise in 2016-2017; published as-is in 2026.

Each input line is one trade or quote:

symbol,exchange,side,condition,sendtime,recvtime,price,size
6AH8,F,B,0,60306043,60306043,90.51,10
GEM8-GEU8,F,A,0,60303042,60303042,-0.14,115

sendtime and recvtime are integer timestamps (milliseconds since midnight in the original data); price is kept as its decimal string so it round-trips exactly. Lines are CRLF-terminated.

Encoding scheme

The file is read fully, then written as a small ASCII header followed by one fixed-width binary record per row.

1. Dictionaries for low-cardinality fields. symbol, price and size are each collected into a sorted, de-duplicated set and copied to a vector; the record stores the vector index instead of the value. exchange, side and condition are concatenated into one three-character string (ESC) and dictionary-coded the same way. Prices are dictionary-coded as strings, not doubles, so 145.124 and 145.124535 remain distinct and lower_bound lookups match exactly.
2. Delta timestamps. sendtime is stored as the delta from the previous row's sendtime (the first absolute value goes in the header); recvtime is stored as the delta from the same row's sendtime. The set of distinct deltas is then dictionary-coded like the other fields, so a stream that mostly advances by a handful of common intervals costs only a few bits per row.
3. Computed bit widths. Each field's width is floor(log2(dictionary size)) + 1 bits. Widths and single-bit masks are derived at compression time and rederived from the header at decompression time; nothing is fixed in the format.
4. 128-bit record packing. The six indices are shifted and OR-ed into one __int128 (a union with a char[16]), and only ceil(total_bits / 8) bytes of it are written per row. The header is plain ASCII (widths, dictionary sizes, dictionary contents; symbols are @-delimited because some contain spaces).

On the CME futures file the exercise was set against (500,000 rows, 19,113,524 bytes, 834 symbols, 5,779 distinct prices), the widths came out 10+2+12+1+13+12 = 50 bits, packed into 7 bytes per row: 3,500,000 bytes of records plus a 70,597-byte header, a 5.35x compression ratio including the dictionaries (5.46x on the record payload alone). Decompression reproduced the input byte for byte. That data set is licensed exchange data and is not included here. The figures above were re-measured in 2026 by running this build against the same file; the transcript in the comment at the top of src/compression.cpp is that run.

The header comment in src/compression.cpp also lists the improvements the author considered at the time — closing the 50-bits-in-56 packing gap (worth ~10%), price deltas per symbol, entropy-coding the packed stream, and packing by symbol or time slice rather than one record per row.

Build

Requires g++ with __int128 support (any x86-64 or AArch64 GCC/Clang) and GNU make.

make                    # debug build (-O0 -ggdb3), binary at install/compress
make config=release     # -O3

Usage

install/compress -c input.csv  output.tp    # compress
install/compress -d output.tp  restored.csv # decompress

Compression prints the dictionary sizes, per-field widths, bytes per record, and the compression ratio with and without the header.

Test

No real market data ships with the repository. tools/gen_ticks.py emits a deterministic synthetic tick stream in the same format (futures-style symbols including calendar spreads, per-symbol random-walk prices on a fixed tick, mostly-repeating timestamp deltas), and make test runs the full round trip on it:

make test               # generate 5,000 rows -> compress -> decompress -> cmp
make test TEST_ROWS=50000

On the default 5,000-row synthetic file the round trip is byte-identical and the ratio is 6.35x with dictionaries (8.18x on the payload; 5 bytes per record). The synthetic stream is deliberately simpler than real data (43 symbols, six distinct send-time deltas), so it compresses better than the CME file did; it exercises correctness, not the headline ratio.

Limitations

• The whole input is held in memory (a vector of tuples) before anything is written; this was written for a single-session file, not a multi-GB stream.
• Input is parsed with std::regex; a line that does not match the expected eight fields is silently skipped.
• Records are limited to 128 bits total. (The 2017 code shifted each field's index as a 64-bit size_t before inserting it into the 128-bit word, which would have dropped any field placed above bit 63; fixed on import by widening before the shift. Real tick data sits around 50 bits, so the original never hit it.)
• The format has no magic number, version, or checksum.

Provenance

Written by Wade Stone in 2016-2017 as an interview take-home exercise and kept in a private archive. Imported into this repository in 2026 as a single commit. Changes made at import time: -std=c++11 -> -std=c++17, three missing standard headers added (<cmath>, <numeric>, <sstream>) so it builds on current GCC, one range-for copy turned into a reference, the Makefile made to create its output directories and given the test target, the synthetic generator added, and the data-file name in the header comment replaced with a description. No algorithmic changes.

License

MIT, copyright (c) 2016-2026 Wade Stone. See LICENSE.