Paper Library

A curated, offline-ready library of investment and finance academic papers with an on-device AI assistant. PDFs are converted to markdown, their sources/URLs are backfilled (DOI → Crossref → arXiv → SSRN → Google Scholar), and a journal-style card catalog makes them browsable and searchable. An on-device Gemma 4 RAG chatbot (running entirely in the browser via WebGPU) answers questions about the papers — nothing leaves the user's device.

This project mirrors the architecture of the sibling reading_library (books) and adapts it for academic papers.

---

What's in here

• PDFs → Markdown: each PDF in the source folder is converted to a single papers/<slug>/index.md with <!-- rl-section ... --> section markers (load-bearing for the SPA router and the RAG chunker).
• Source backfill: every paper gets a meta.json with the best-known DOI, source URL, venue, authors, and year — resolved from the PDF content and free APIs (Crossref, arXiv, SSRN). Low-confidence matches are flagged.
• Journal-style catalog: the front page is a journal table-of-contents card grid (title, authors, venue/year, source badge, abstract preview, tags) — not a bookshelf.
• On-device chatbot: Gemma 4 E2B (~3.1 GB, cached after first load) + embeddinggemma-300m, via transformers.js v4 + WebGPU. Cross-paper RAG with a prebuilt embedding cache.

The original PDFs are not committed — they stay in the user's OneDrive as the source of truth. The repo stores only the converted markdown, metadata, thumbnails, and RAG artifacts.

---

Adding a paper

Single CLI entry point (scripts/add_paper.py):

# from a local PDF file
uv run python scripts/add_paper.py --pdf "C:\path\to\paper.pdf"

# from a URL (PDF is downloaded first)
uv run python scripts/add_paper.py --url "https://arxiv.org/pdf/2401.12345"
uv run python scripts/add_paper.py --url "https://papers.ssrn.com/sol3/papers.cfm?abstract_id=12345"
uv run python scripts/add_paper.py --url "https://doi.org/10.1111/jofi.12345"

This runs convert → backfill → catalog sync, then prompts to regenerate the RAG index. Pass --no-rebuild to skip the prompt.

Batch convert from the OneDrive source folder

uv run python scripts/convert_pdf.py --batch-dir "C:\Users\hevan\OneDrive\Stocks" --no-recurse

---

Pipeline (the full regeneration sequence)

After converting or backfilling, regenerate the derived RAG artifacts in this strict order:

uv run python scripts/backfill_sources.py            # resolve DOI/source/venue
uv run python scripts/sync_papers_catalog.py         # rebuild papers.json + FALLBACK_PAPERS
uv run python scripts/build_chatbot_index.py         # chunks → assets/chatbot_chunks.json
cd scripts && node build_chatbot_embeddings.mjs      # vectors → assets/chatbot_embeddings.bin (auto GPU)
cp web_assets/spa.css web_assets/spa.js web_assets/chatbot.css web_assets/chatbot.js assets/
cp web_assets/vendor/marked.esm.js assets/vendor/
uv run python scripts/wire_chatbot.py                # wire SPA + chatbot into index.html

---

On-device chatbot

• Runs entirely in the browser via WebGPU — no question ever leaves the user's device.
• Gemma 4 E2B (~3.1 GB, downloaded once and cached) with embeddinggemma-300m for retrieval.
• Scope toggle: This section / This paper / All papers controls which slice of the corpus the assistant searches.
• Cites other papers by title (e.g. [Paper: Title · Section]) and links to them through the SPA.

Terminal chatbot (CLI)

Same local Gemma 4 models and RAG index as the browser assistant — no external LLM APIs.

cd scripts && npm install          # once — transformers.js + onnxruntime-node
uv sync --extra cli                # numpy for RAG retrieval

uv run --extra cli python scripts/chatbot_cli.py -q "What does the VIX white paper say about mean reversion?"
uv run --extra cli python scripts/chatbot_cli.py --model e4b --device webgpu
uv run --extra cli python scripts/chatbot_cli.py --paper predicting-absolute-stock-returns-option-iv --scope paper

---

PDF conversion notes

• Default path: pymupdf4llm — fast, pure-Python, good general quality, handles tables and basic layout.
• Math-heavy fallback: papers detected as equation-dense are routed through Nougat (Meta's academic-paper model) for better equation handling. Nougat is a heavy optional install: uv sync --extra nougat.
• Some PDFs have opaque filenames (e.g. work606.pdf, wp807.pdf); title and authors are extracted from PDF metadata or the largest text on page 1, and flagged low_confidence if extraction is uncertain.

---

Repository layout

paper_library/
├── index.html              # SPA shell: journal card catalog + reader view (GitHub Pages index)
├── papers.json             # Catalog DB (source of truth for cards)
├── pyproject.toml          # Python deps (pymupdf4llm, requests, bs4, lxml)
├── papers/                 # Per-paper content (fetched by the SPA at runtime)
│   ├── catalog.json        # Public catalog/status metadata
│   └── <slug>/
│       ├── index.md        # Converted paper (front matter + rl-section markers)
│       ├── meta.json       # Extracted + backfilled metadata (DOI, source_url, ...)
│       └── assets/         # First-page thumbnail + extracted figures
├── assets/                 # Served SPA + chatbot assets (copied from web_assets/)
├── web_assets/             # Source of truth for spa/chatbot css/js
└── scripts/                # Converters, backfill, catalog sync, RAG build, CLI chatbot

See AGENTS.md for the full architecture and conventions.