Caveman Compression Benchmark

A benchmark dataset for evaluating RAG context compression methods. This dataset accompanies the paper "Caveman Compression: MLM-Guided Token Removal for LLM Cost Reduction".

Dataset

The benchmark contains 79 topics with 948 question-answer pairs designed to evaluate whether compressed text retains sufficient information for accurate question answering.

Structure

data/rag-chunks-qa/
├── topic-name/
│   ├── content.txt      # Source document (~300-600 words)
│   ├── qa.json          # 12 questions with answers and difficulty
│   └── metadata.json    # Topic metadata (content_type, domain)

Question Format

Each qa.json contains questions stratified by difficulty:

• Easy (4 questions): Surface-level facts directly stated
• Medium (4 questions): Require connecting multiple sentences
• Hard (4 questions): Require inference or synthesis

{
  "questions": [
    {
      "question": "What is the primary mechanism?",
      "answer": "The resonance coupling between...",
      "difficulty": "hard",
      "type": "mechanism"
    }
  ]
}

Content Types

• fiction (16 topics): Original synthetic narratives to avoid training data contamination
• factual (63 topics): Technical, scientific, and general knowledge content

Benchmark Scripts

MLM Compression Benchmark

Evaluates compression using masked language model predictability scores:

python src/benchmark_rag_compression.py \
    --data-dir data/rag-chunks-qa \
    --thresholds 0.000001 0.00001 0.0001 0.001 \
    --content-type fiction

Baseline: Random Removal

python src/benchmark_random_removal.py \
    --data-dir data/rag-chunks-qa \
    --removal-rates 0.15 0.30

Baseline: Stopword Removal

python src/benchmark_stopword_removal.py \
    --data-dir data/rag-chunks-qa

Requirements

torch
transformers
spacy
anthropic
openai

Install spaCy model:

python -m spacy download en_core_web_sm

Evaluation Protocol

1. Compress document using the method under test
2. Query LLM (GPT-5) with compressed context + question
3. Judge answer correctness using Claude 4 Sonnet
4. Report raw accuracy (correct/total)

Citation

@article{caveman2025,
  title={Caveman Compression: MLM-Guided Token Removal for LLM Cost Reduction},
  author={Peltomaeki, William},
  year={2025}
}

License

MIT License