Knowledge Compiler

Knowledge Compiler is a local-first tool for turning source material into structured, durable Markdown knowledge artifacts.

It keeps raw inputs and compiled topics in a local vault, uses OpenAI structured generation to classify and synthesize material, and exposes Bun-powered CLI commands for seeding, compiling, and asking questions over the compiled knowledge base.

Current Status

The project has an end-to-end local wiki workflow:

1. Seed a raw note from a question.
2. Compile raw Markdown sources into topic pages.
3. Repair and validate generated topic links and frontmatter.
4. Regenerate the wiki index and append compile/ask/memory events to the log.
5. Ask questions using selected compiled topic pages as context.
6. Optionally save durable answers back into raw sources and recompile the topic.

This is still early software. The workflow is intentionally small, local, and file-based while the project boundaries settle.

Requirements

• Bun
• An OpenAI API key

Install dependencies:

bun install

Create a .env file:

OPENAI_API_KEY=your_api_key
KNOWLEDGE_VAULT_PATH=/absolute/path/to/your/knowledge-vault

KNOWLEDGE_VAULT_PATH is optional. If it is not set, the app uses ~/knowledge-vault.

Optional:

OPENAI_BASE_URL=https://your-compatible-openai-endpoint.example

Commands

Seed a raw source note from a question:

bun run seed "How does consistent hashing work?"

Compile all raw sources into topic pages:

bun run compile

Compile a specific domain/topic:

bun run compile --domain "System Design" --topic "Consistent Hashing"

Ask a question over compiled topics:

bun run ask "When should I use consistent hashing?"

Check vault quality:

bun run check

Repair legacy vault issues without writing changes:

bun run doctor --dry-run

Apply safe deterministic vault repairs:

bun run doctor --apply

Run the yargs-based command shell:

bun run cli repl

Run the test suite:

bun test

Vault Layout

Knowledge Compiler reads and writes Markdown files inside the configured vault.

knowledge-vault/
  00-raw/
    consistent-hashing.md
    answers/
      when-to-use-consistent-hashing.md
  04-topics/
    index.md
    log.md
    System Design/
      consistent-hashing.md

Raw source files must include frontmatter that matches the ingest schema:

---
title: Consistent Hashing
domain: System Design
topic: Consistent Hashing
tags:
  - distributed-systems
source_type: note
source_url: https://example.com/optional-source
---

Your raw source material goes here.

Compiled topic pages include structured sections such as summary, key concepts, deep dives, related topics, open questions, and sources.

Vault Quality

Generated topics are repaired and validated before they are written. Validation errors block new compiled output when required sections, frontmatter, source counts, or Obsidian links are structurally wrong.

bun run check scans existing compiled topics and reports:

• ERROR: structural problems that can break traceability or retrieval, such as bad source links, bad related links, missing source links, and source count mismatches.
• WARNING: quality signals that are useful but not blocking, such as orphan topics, weak summaries, broken legacy source paths, or duplicate raw source titles.

bun run doctor --apply fixes safe legacy issues deterministically, without asking the LLM to rewrite note content.

Project Interaction Map

The main flow is intentionally layered:

1. CLI entry points in src/cli/ and local scripts in scripts/ receive user commands.
2. Core use cases in src/core/ coordinate classification, compiling, repair, validation, answering, retrieval, and answer memory.
3. Ingest code in src/ingest/ reads raw Markdown sources from the vault.
4. LLM code in src/llm/ owns OpenAI client access, structured output, metadata-rich topic selection, contextual answering, and answer-memory decisions.
5. Render code in src/render/ converts structured knowledge into Markdown plus frontmatter and Obsidian links.
6. Storage code in src/storage/ reads compiled topics, validates topic quality, and writes topic pages, answer memories, the wiki index, and the wiki log.
7. Schema code in src/schema/ defines the structured knowledge shape expected from the LLM.

CLI/scripts
  -> core
    -> ingest
    -> llm
    -> render
    -> storage
      -> local knowledge vault

Project Structure

knowledge-compiler/
  src/
    cli/        command-line entry points and argument handling
    config/     environment and runtime configuration
    core/       compiler use cases and domain orchestration
    ingest/     source adapters and input normalization
    llm/        AI provider boundary and structured generation
    pipeline/   placeholder for composable workflow stages
    render/     Markdown and metadata rendering
    schema/     structured knowledge schemas
    storage/    filesystem and vault persistence
  tests/
    unit/        isolated module tests
  scripts/       local development and maintenance scripts
  docs/          decisions, plans, and reference notes

Function Documentation

Each implementation folder has its own README with function-level details:

• src/core/README.md documents orchestration functions such as compileTopic, askQuestion, and askQuestionWithAutoCompile.
• src/llm/README.md documents LLM boundaries such as callStructured, answerWithContext, selectTopicsFromIndex, and decideAnswerMemory.
• src/storage/README.md documents filesystem and quality functions such as resolveTopicDomainPath, readCompiledTopics, writeAnswerMemory, regenerateWikiIndex, appendWikiLogEntry, and topic validation helpers.
• src/ingest/README.md, src/render/README.md, src/config/README.md, and src/schema/README.md document schemas, helpers, and rendering/config behavior.
• scripts/README.md documents raw seed creation helpers and vault maintenance scripts.
• tests/README.md documents test helpers and module coverage.

Development Approach

Implementation happens incrementally:

1. Keep the core workflow small and testable.
2. Add tests around behavior before expanding capabilities.
3. Preserve clear module boundaries between CLI, core orchestration, storage, rendering, ingest, and LLM access.
4. Document design decisions as the structure evolves.

Notes

• LLM calls currently use gpt-5-mini.
• Raw sources are scanned from 00-raw.
• Compiled topics are written under 04-topics.
• Answer memories are written under 00-raw/answers.
• Compile automatically repairs rendered links and validates the topic before writing.
• Retrieval selection uses the wiki index plus compiled topic metadata such as summary, tags, related topics, and source paths.
• The project uses path aliases from tsconfig.json, with @/* mapped to src/*.