Event-sourced Bitcoin market data platform that turns raw Binance archives into checksum-verified trade tables, rebuildable bar projections, and replayable research datasets.
Origo ingests Binance BTCUSDT market data into ClickHouse under Dagster orchestration, records every ingested source file in a checksum ledger, and rebuilds every daily-partitioned projection deterministically from that raw record; the rolling _latest tables and live depth snapshots capture the span after the last daily load and sit outside that rebuild guarantee. The Dagster deployment in this repository is Origo's control plane — the term names that orchestration component, not the project. The project evolves from tdw-control-plane, this repository's original identity.
Origo is not:
- a trade execution system
- a signal research or backtesting engine
- a multi-exchange market data vendor
In the wider Vaquum architecture, Origo sits upstream as the data layer. Limen consumes its outputs downstream as the research engine, and Nexus, Praxis, and Veritas sit further downstream for decisioning, execution, and oversight.
- Revisioned spot source: checksum-verified Binance BTCUSDT daily trade archives (from 2017-08-17) built per day into raw trades, 1-minute klines, dollar, volume, tick and dollar-imbalance bars and the aligned 1-minute rows, with provisional minute intervals covering the open day, activation history, reconciliation and rollback
- Legacy spot table names (
binance_daily_spot_trades,binance_spot_klines, the bar tables and the_latestnames) served as views over the source's current state; the spot aligned rows arebinance_spot_trades_aligned_current - Revisioned perp source: checksum-verified Binance USDT-M perpetual BTCUSDT daily trade archives (from 2019-09-08) built per day into raw trades, 1-minute klines, dollar, volume, tick and dollar-imbalance bars and the aligned 1-minute rows, with provisional minute intervals covering the open day, activation history, reconciliation and rollback
- Revisioned spot aggregate source (CANARY): checksum-verified Binance BTCUSDT daily aggregate-trade archives (from 2017-08-17) built per day into raw aggregates, 1-minute klines, dollar, volume, tick and dollar-imbalance bars and the aligned 1-minute rows, with provisional minute intervals covering the open day; tick bars count aggregate events and dollar bars accumulate aggregate notional
- Revisioned perp aggregate source (CANARY): checksum-verified Binance USDT-M perpetual BTCUSDT daily aggregate-trade archives (from 2019-12-31) built per day into raw aggregates, 1-minute klines, dollar, volume, tick and dollar-imbalance bars and the aligned 1-minute rows, with provisional minute intervals covering the open day; tick bars count aggregate events and dollar bars accumulate aggregate notional
- Binance spot order-book depth snapshots (20- and 200-level) with 1-minute projections and per-minute reconciliation
- BTCUSDT spot market state cube from 2021-01-01: pinned sparse Arrow queries with quote/base volume, trade counts, path, dwell and OHLC measures; exact First hit, Controlled advance and Swing rally memberships with local filter/grid/replay reuse
- Hugging Face publishing of twelve kline datasets (six time intervals and six dollar-bar sizes), the local monthly Parquet mirror of the same series and the versioned mmap-ready Arrow bar store, all published by the revisioned spot source from one pinned state
- Ratcheted CI gates on every PR: strict pyright typing, fail-loud (no silent fallbacks), Conventional Commits, and version plus CHANGELOG trails
- Automatic production deploy of merged
mainthrough GitHub Actions
The first runnable path is the local Docker Compose stack: prepare the spot source, backfill one day, and read it back through the query module.
- Clone the repository and set the one required secret:
git clone https://github.com/Vaquum/Origo.git
cd Origo
export CLICKHOUSE_PASSWORD=<choose-a-password>Supported runtime: the containers run Python 3.11 (python:3.11.12) and the package requires Python >=3.11; the host needs Docker with Compose. CLICKHOUSE_PASSWORD is the only variable without a default — CLICKHOUSE_HOST, CLICKHOUSE_PORT, CLICKHOUSE_USER, and CLICKHOUSE_DATABASE default to the in-network values (clickhouse, 9000 native / 8123 HTTP, default, origo). HF_TOKEN and HUGGINGFACE_DATASET_REPO_ID are needed only for Hugging Face publishing, and the BINANCE_SPOT_DEPTH20_* / BINANCE_SPOT_DEPTH200_* pairs only for the live depth collectors; docker-compose.deploy.yml requires them all, the local docker-compose.yml does not. The local compose file passes CLICKHOUSE_PASSWORD only to the ClickHouse server, so the commands below inject it into the Dagster container with -e.
- Start the stack:
docker compose up -d --build- Prepare the spot source (the stack's bootstrap does this on start; the job repeats it idempotently):
docker compose exec -e CLICKHOUSE_PASSWORD dagster \
dagster job launch -m origo.definitions -j prepare_revisioned_sources_job- Backfill one day of the spot source — raw trades, every bar projection and the aligned rows, then the public files:
docker compose exec -e CLICKHOUSE_PASSWORD dagster \
dagster job backfill -m origo.definitions \
-j backfill_binance_spot_trades_source_job \
--partitions 2024-01-02 --noprompt- Read the day back through the query module:
docker compose exec -e CLICKHOUSE_PASSWORD dagster python -c "
from origo.query.binance_spot_kline_rollups import dollar_month, time_month
print(time_month(interval_minutes=60, year=2024, month=1))
print(dollar_month(ratio=15, year=2024, month=1))
"time_month rolls the 1-minute projection up to any minute interval, and dollar_month rolls the 1M-dollar bar base up by an integer ratio. Both read the legacy table names, which are views over the spot source: binance_spot_klines and binance_spot_dollar_klines hold the canonical days, and binance_spot_klines_latest and binance_spot_trades_latest the provisional minutes of the open day. Beyond the quickstart, every job, schedule, and sensor is defined in origo/definitions.py and can be launched from the Dagster UI at http://localhost:4000.
Origo is research software. Data outputs are not investment advice, trading advice, execution simulation, regulatory approval, or a promise of future performance. Past performance is not predictive, and trading digital assets can result in total loss of capital.
- Start with the full documentation hub
- See What is Origo, Product Boundaries, and System Architecture for scope and shape
- Use Get Started Locally for the local stack
- Run your first native query and your first aligned query against the warehouse
- Rebuild derived tables with Rebuild Projections and check coverage with Understand Historical Coverage
- Export datasets with Export Data
- Export Binance spot rallies with their trades and order book using the rally export guide and reference
- Query the BTCUSDT spot market state cube and exact rally memberships through the cube query reference; see projection, rollout and acceptance
- Explore cube measures, profiles, Lines and Inspect in the separate Cube Explorer; selectable rally marks and membership dimming remain Explorer #50
- See the mechanical PR gates in AGENTS.md and their implementations under tools
- Follow the change record in CHANGELOG.md
- Contribute through Contributing and the Developer docs
Contribution starts through AGENTS.md, the Contributing guide, or open issues. Before contributing, start with the Developer docs.
Use open issues for support requests and scope questions.
Report vulnerabilities privately to the repository owner, @mikkokotila. Do not report vulnerabilities through public issues.
Published work should cite:
Vaquum Origo [Computer software]. (2026). Retrieved from GitHub.