GoldSentinel

GoldSentinel is a research pipeline and paper-trading platform that studies whether public attention and sentiment around gold help explain or predict future gold-market behavior. It is English-only in this version and ships with sample fixture data so it runs fully offline, with no API keys.

The active MVP pipeline uses three sources: Google Trends, English- language news (GDELT + RSS), and gold/macro market data. Reddit is implemented but currently disabled — see Reddit (currently disabled) below.

Reddit (currently disabled)

Reddit API access is not available for this project, so Reddit is not part of the active MVP source list (sources.disabled in configs/default.yaml) and no CLI command ingests it by default.

The collector itself (collectors/reddit.py) is kept fully implemented, isolated, and tested — including a working fixture mode — so it can be re-enabled later purely by moving reddit from sources.disabled to sources.enabled and (for live collection) supplying Reddit OAuth credentials. It was designed around Reddit's official, read-only OAuth API from the start:

• Read-only access to publicly available posts from selected subreddits (Gold, PreciousMetals, investing, stocks, economics, Wallstreetsilver).
• No posting, voting, messaging, following, or any other interaction with Reddit users or discussions.
• No scraping — only the official OAuth API.
• No collection of Reddit usernames.

Reddit content, when enabled, normalizes into the same ContentItem model as every other source and is used only as statistical research input, not redistributed as a standalone Reddit-content product.

This is a research and paper-trading project only. It does not execute trades, connect to a broker, hold credentials for any exchange, or manage real capital. Nothing it produces is financial advice. See Disclaimer below.

Research hypothesis

• Extremely high attention around gold combined with extremely positive sentiment may indicate crowding or excessive optimism (a contrarian sell signal).
• Extremely high attention combined with extremely negative sentiment may indicate panic or excessive pessimism (a contrarian buy signal).
• Low attention alone must not be treated as an automatic buy signal.
• These hypotheses are tested objectively through historical data and walk-forward backtesting — the system reports results, it does not assume the hypothesis is true.

Non-goals

• No live trading, broker integration, order execution, or leverage.
• No real-money portfolio management.
• No personalized investment advice.
• No claim that any backtest result predicts future profitability.
• No X/Twitter collection in this version (see Adding X/Twitter later) and no web scraping of any kind — only RSS/documented APIs and fixture data.
• No non-English content translation — non-English content is excluded, never translated.

Architecture

collectors/   -> normalize each source into ContentItem / TrendObservation / MarketBar
processing/   -> language filter, gold-relevance classifier, dedup, engagement, sentiment
features/     -> daily aggregation, rolling z-scores, attention/sentiment indices, contrarian score
strategy/     -> rule-based signal engine, position sizing, benchmark strategies
backtest/     -> daily backtest engine, performance metrics, walk-forward splitting
db/           -> DuckDB schema + repository (all SQL lives here)
pipelines/    -> thin orchestration wiring the above together
cli.py        -> Typer CLI exposing each pipeline stage as a command

Downstream code (processing/, features/, strategy/, backtest/) only ever depends on the normalized domain models in models/ — never on a collector's source-specific payload shape. That boundary is what lets a new data source be added later (see below) without touching anything past the collector.

Each active source has a fixture provider (always available, no credentials) plus an isolated, optional live provider:

Source	Fixture (default)	Live provider
Google Trends	data/fixtures/trends_sample.csv	pytrends (optional dep, no API key exists)
News	data/fixtures/news_sample.json	GDELT DOC 2.0 API (free, keyless) + RSS feeds
Market	data/fixtures/market_sample.csv	yfinance (optional dep) + FRED fredgraph.csv (keyless)
Reddit	data/fixtures/reddit_sample.json	Reddit OAuth API — disabled, see above

Data flows through DuckDB tables in this order:

raw_content ──┐
trend_observations ──┼─► processed_content ─► daily_features ─┐
market_bars ──┘                                                ├─► signals ─► backtest_runs / backtest_trades
                                    daily_market_features ──────┘

Installation

Requires Python 3.11+.

python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

Environment setup

Copy .env.example to .env and adjust as needed. The system runs fully in fixture mode (GOLDSENTINEL_FIXTURE_MODE=true, the default) with no API keys at all. Research parameters (keywords, subreddits, RSS feeds, source weights, rolling windows, thresholds, cost assumptions) live in configs/default.yaml, which is safe to commit — secrets never go there.

cp .env.example .env

Live-mode credentials are only needed if you flip GOLDSENTINEL_FIXTURE_MODE=false for a given source — see Running the real-data collectors. Reddit's OAuth credentials specifically are unused unless you also re-enable reddit in configs/default.yaml's sources.enabled (see Reddit (currently disabled)). Never commit a real .env file.

Fixture-mode quickstart

Everything below runs with no network access and no API keys, using the synthetic sample data in data/fixtures/ (see Data limitations — this is not real market or social data).

goldsentinel init-db
goldsentinel collect-fixtures
goldsentinel process-content
goldsentinel build-features
goldsentinel generate-signals
goldsentinel run-backtest

Or run the whole thing in one command:

goldsentinel run-pipeline

Check what each collector can see without ingesting anything (also shows which sources are active vs. disabled):

goldsentinel health-check

Individual source commands are also available:

goldsentinel collect-trends
goldsentinel collect-news
goldsentinel collect-market

Every command supports --help, e.g. goldsentinel run-backtest --help.

Running the real-data collectors

Fixture mode is the default and is what CI/tests use. To collect real data for a given source, flip fixture mode off and (for Google Trends / market data) install the optional provider extras:

pip install -e ".[providers]"   # installs pytrends + yfinance

# .env
GOLDSENTINEL_FIXTURE_MODE=false

Then, per source:

• Google Trends — no API key exists for Trends at all; live collection uses pytrends (installed above). Unofficial and rate-limited by Google (HTTP 429), so expect occasional throttling — a 429 is handled specifically: Retry-After is honored when present, otherwise exponential backoff with jitter is used (base_backoff_seconds / max_backoff_seconds / jitter_seconds in configs/default.yaml), and a per-keyword-group group_cooldown_seconds pause keeps steady request pressure low even when nothing fails. A keyword group that still fails after max_retries is recorded as incomplete — its keywords are simply absent from the result (never faked as zero attention) — and reported via goldsentinel collect-trends's warnings output and failed_keywords metadata, not silently swallowed.
• News — GDELT's DOC 2.0 API needs no key and works immediately with fixture mode off. RSS feeds (collectors.news.rss_feeds in configs/default.yaml) also need no key.
• Market data — requires both GOLDSENTINEL_FIXTURE_MODE=false and collectors.market.provider: yfinance in configs/default.yaml; fixture_mode is authoritative, so setting provider: yfinance alone (fixture mode still on) makes no network request and keeps collecting from the fixture CSV — see "Fixture vs. live data isolation" below. Every logical symbol (XAUUSD/USDTRY/DXY/US10Y/VIX) maps to an independently verified Yahoo ticker via YFINANCE_TICKER_MAP in collectors/market.py — none of them are assumed to work as literal Yahoo tickers (XAUUSD=X, for instance, does not resolve at all). These are documented approximations, not guaranteed equivalents of spot/interbank rates (e.g. XAUUSD -> GC=F COMEX gold futures, not spot). The US 10-year yield prefers FRED's keyless fredgraph.csv endpoint (DGS10, the official constant-maturity series) whenever collectors.market.fred.enabled is true; otherwise yfinance's ^TNX is used as a live-market fallback proxy. A ticker that returns no data is logged and reported via warnings()/goldsentinel collect-market's output — never silently replaced with fixture data or a fabricated value.
• Reddit — disabled; see Reddit (currently disabled).

goldsentinel collect-trends
goldsentinel collect-news
goldsentinel collect-market

Fixture vs. live data isolation

Every row collected into DuckDB — raw_content, processed_content, trend_observations, market_bars, daily_features, daily_market_features, signals, and backtest_runs — is tagged with a collection_mode: fixture, live, or unknown (rows written before this field existed; see the migration note below). collection_mode is part of the primary key on every table that stores per-item data, so a live collection run can never silently overwrite (or be shadowed by) a previously-collected fixture row for the same logical key — both persist, distinguishable by collection_mode.

Ingestion tags each item with the mode of the specific provider that produced it (e.g. FixtureNewsProvider vs. RssNewsProvider). Every stage after ingestion — processing, feature-building, signal generation, backtesting — reads and writes a single collection_mode per run, derived from GOLDSENTINEL_FIXTURE_MODE (fixture when true, live when false). GOLDSENTINEL_FIXTURE_MODE is authoritative for every collector, including market data: MarketCollector uses the fixture CSV whenever fixture_mode is true, full stop, regardless of collectors.market.provider in configs/default.yaml — that field only chooses which live provider to use once fixture mode is off ("yfinance" to go live, or "fixture" to deliberately stay on fixture data even while every other collector is live). This is deliberate, not incidental: market data used to be the one collector driven purely by provider, independent of fixture_mode, which meant a config with provider: yfinance made real Yahoo Finance requests during ordinary "fixture-mode" use — including the test suite. collection_mode_for_market_provider (in models/common.py) computes the same effective mode MarketCollector uses, so feature-building reads back exactly the rows a given run actually wrote. This means fixture and live data can safely coexist in the same data/goldsentinel.duckdb — e.g. after testing with fixtures and then switching a source to live — without a pipeline run ever mixing the two: Repository.fetch_* methods all accept an optional collection_mode filter, and every pipeline stage passes it.

Migrating an existing database. A database created before collection_mode existed is upgraded automatically and non-destructively the next time any command runs init_schema (i.e. any goldsentinel command, or explicitly goldsentinel init-db): existing rows are copied into the new schema with collection_mode = 'unknown', nothing is dropped, and the migration is idempotent (safe to run repeatedly, a no-op once already migrated). No manual steps are required.

Starting from a clean database instead, e.g. if you'd rather not carry forward old unknown-tagged rows: stop any running goldsentinel process, then delete the DuckDB file (default data/goldsentinel.duckdb, or whatever GOLDSENTINEL_DB_PATH / database.path in configs/default.yaml points at) and re-run goldsentinel init-db. This is a destructive, irreversible action — back up the file first (cp data/goldsentinel.duckdb data/goldsentinel.duckdb.bak) if you might want the existing data later.

Running tests

pytest

The test suite runs fully offline against fixture data — no network access or credentials required.

Running linting

ruff check .
ruff format .

Adding a new data provider

1. Add a ContentSource (or reuse an existing one) in models/content.py if needed.
2. Implement a class in collectors/ that subclasses collectors.base.Collector[T] (T = ContentItem, TrendObservation, or MarketBar) with source_name(), collect(start_date, end_date), and health_check(). Return normalized domain objects — never raw source payloads — and never let one failed request crash the whole collection run (catch/log and continue; see collectors/base.py's request_with_retries helper).
3. Support a fixture mode that needs no credentials, mirroring the existing collectors, so tests and the default local workflow keep working offline.
4. Add configuration under collectors: in configs/default.yaml (a Pydantic model in config.py) and wire the collector into pipelines/ingest.py and the CLI.
5. Nothing in processing/, features/, strategy/, or backtest/ should need to change — they only depend on the normalized models.

Adding X/Twitter later

X/Twitter is intentionally not implemented in this version — see collectors/x_placeholder.py for the full rationale and the exact steps to add a real integration. In short: implement a Collector[ContentItem] that returns items with source=ContentSource.X using an official or licensed API (no scraping), wire it into pipelines/ingest.py and the CLI the same way other collectors are wired in, and nothing downstream of the collector boundary needs to change.

Data limitations

• All fixture data in data/fixtures/ and data/goldsentinel.duckdb (once generated) is synthetic, deterministically generated for development and testing. It is not real news, not real Google Trends data, not real market prices, and (for the disabled, isolated Reddit fixture) not real Reddit content. Do not draw market conclusions from it.
• Google Trends values are normalized relative-interest scores (0–100), not absolute search volume, and are normalized independently within each request (pytrends compares at most 5 keywords per request, so a 10-keyword configuration means at least 2 independently-normalized groups). Values from different groups are not directly comparable without adjustment. This collector uses an anchor-keyword strategy to make them approximately usable together: anchor_keyword ("gold" by default) is forced into every group, and every non-reference group is rescaled so its anchor value matches the reference group's anchor value on each date, then that same scale factor is applied to the rest of the group. This is a documented approximation, not a guarantee of exact comparability — see the full methodology and caveats in collectors/google_trends.py's module docstring. A day where the anchor is missing from either side of a rebase is left un-rebased and recorded as a warning rather than silently guessed.
• A Google Trends day with incomplete keyword coverage (e.g. a keyword group that failed to collect) is not treated as zero attention: it is aggregated from whatever keywords did report and its data_quality_score is scaled down to the coverage ratio (see features.daily_aggregation.aggregate_daily_trends_features and collectors.google_trends.min_keyword_coverage), which the signal engine's data-quality gate then uses to withhold a signal on badly incomplete days rather than trade on partial data.
• GDELT's DOC API article-list mode returns only title, URL, publisher, and timestamp — no article body/summary. Relevance and sentiment scoring for GDELT-sourced items therefore run on the title alone, which the pipeline already handles safely but is inherently less accurate than scoring full text.
• The yfinance market-data tickers are documented approximations (e.g. COMEX gold futures GC=F as a proxy for XAUUSD spot), not guaranteed equivalents of spot/interbank pricing. FRED-sourced series (e.g. US10Y) provide only a daily close level — no open/high/low/volume.
• The English-language filter (processing/language.py) is a lightweight, dependency-free heuristic tuned for this project's short, finance- flavored text. It is not a general-purpose language identifier.
• The sentiment scorer (processing/sentiment.py) is a deterministic, hand-curated lexicon. It has no negation handling, no sarcasm detection, and is not production-quality NLP. Both are designed to be swapped for real models later behind the same interface.
• The gold-relevance classifier (processing/relevance.py) uses transparent term matching, not a trained model, so every decision is auditable via matched_terms — but it will miss relevant content that doesn't use the configured vocabulary and can still misfire on unusual phrasing.
• Deduplication (processing/deduplication.py) treats the same story syndicated across multiple publishers as reduced-weight, not zero — see that module's docstring for the exact matching rules and the bounded time window used to avoid conflating unrelated recurring headlines.

Look-ahead bias precautions

• Every rolling statistic (features/attention.py's rolling_zscore and everything built on it) is trailing: the window ending at day t uses only observations at or before t.
• The backtest engine (backtest/engine.py) defaults to next-period execution: a position decided from day t's close is applied starting with the return realized from t to t+1, never t-1 to t. execution="same_period" exists only to demonstrate what look-ahead bias does to results and must never be used to report performance.
• The walk-forward interface (backtest/walk_forward.py) keeps train/calibration and out-of-sample test windows explicitly separate; backtests are only ever run on the test window.

Legal and terms-of-service considerations

• Reddit is currently disabled (no API access available for this project); the collector is designed around Reddit's official, read-only OAuth API and does no scraping, but it is not part of the active pipeline — see Reddit (currently disabled).
• News collection uses GDELT's documented, free DOC API and RSS feeds only — no scraping of article bodies.
• Google Trends has no official API; live collection uses pytrends, an unofficial wrapper, which is why it is isolated behind _collect_from_provider and kept optional. Review Google's terms before relying on it beyond casual research use.
• yfinance is likewise an unofficial wrapper around Yahoo Finance data; review Yahoo's terms before relying on it beyond development/research use. FRED's fredgraph.csv endpoint is an official, documented, keyless data source.
• X/Twitter is excluded entirely in this version specifically because an unofficial/scraping implementation would violate the platform's terms of service. Only an official or licensed API should ever be used.
• You are responsible for reviewing and complying with the terms of service of any data source you configure, especially before switching any collector out of fixture mode.

Disclaimer

GoldSentinel is a research and educational tool. It does not execute trades, does not connect to any broker or exchange, does not manage real capital, and does not provide personalized investment advice. Backtest results describe historical, largely synthetic fixture data and must not be interpreted as a prediction or guarantee of future performance. Nothing in this repository is financial advice.