Correct isolated Level 0 into a realistic GPT-2-BPE nanoGPT baseline

level_0_baseline/README.md

@@ -1,76 +1,121 @@
-# Level 0 Baseline
+# Isolated Level 0 nanoGPT Baseline
 
-A deliberately self-contained nanoGPT baseline. It does not import the repository's existing experiment framework or WW-PGD code.
+This subtree is an independent, auditable nanoGPT baseline. It does not import the repository's WW-PGD experiment framework or apply WW-PGD. Its purpose is to establish a credible AdamW language-model baseline and measure WeightWatcher layer spectra before adding an optimizer extension.
 
-## Scope
+## Corrected baseline
 
-- one transformer block, one ordinary Q/K/V attention head, width 64, context 256
-- byte-level next-token language modeling on a fixed FineWeb-Edu subset
-- AdamW or Muon with a global warmup/cosine schedule
-- Muon applies only to hidden 2-D matrices; AdamW handles embeddings, tied LM head, LayerNorm parameters, and other non-matrix parameters
-- deterministic seeds for initialization and sampled training windows
-- immutable train, validation, and test splits
-- CSV logging of loss, next-token accuracy, perplexity, validation/test generalization gaps, gradient norm, weight norm, tokens, and elapsed time
-- optional checkpoint-time WeightWatcher layer analysis
-- single-seed and multi-seed notebooks; multi-seed plots use mean ± one standard deviation shaded bands
+The original isolated baseline was an 82K-parameter, one-layer byte model. This version uses a materially more realistic MacBook-scale configuration:
 
-## Install
+- pinned FineWeb-Edu `sample-10BT` source;
+- GPT-2 BPE tokenization (`tiktoken`, vocabulary 50,257);
+- 16M training, 1M validation, and 1M test tokens stored as `uint16`;
+- 4 transformer blocks, 4 attention heads, width 128, context 256;
+- 7,253,248 parameters with tied token embedding/output weights;
+- AdamW with decoupled weight decay, gradient clipping, 100-step warmup, and cosine decay;
+- microbatch 4 with 8 accumulation steps: 8,192 tokens per optimizer update and 16.384M tokens over 2,000 updates;
+- fixed, independent train/validation/test probes that do not advance the training RNG;
+- test evaluation only at the final and validation-selected checkpoints;
+- non-randomized WeightWatcher analysis of the 24 transformer block matrices only. The large embedding/output matrix is excluded from periodic spectral analysis.
 
-```bash
-cd level_0_baseline
-python -m venv .venv
-source .venv/bin/activate
-pip install -e '.[data,analysis,test]'
-```
-
-## Paths
+The default layer learning-rate multiplier is flat (`layer_lr_decay: 1.0`). Layerwise decay is available as an explicit ablation, not silently enabled in the baseline.
 
-Defaults are under `/tmp/nanogpt-level0`. Override them without editing code:
+## Install in the existing Conda environment
 
 ```bash
-export NANOGPT_LEVEL0_DATA_ROOT=/tmp/my-level0/data
-export NANOGPT_LEVEL0_RESULTS_ROOT=/tmp/my-level0/results
-export NANOGPT_LEVEL0_CACHE_ROOT=/tmp/my-level0/cache
+conda activate ww_prod310
+cd ~/Desktop/work/nanoGPT/nanogpt-experiments/level_0_baseline
+python -m pip install -e '.[data,analysis,test]'
 ```
 
-## Prepare the real corpus
+## Paths
 
-This prepares fixed 50 MB training, 2 MB validation, and 2 MB test byte-token splits from streamed FineWeb-Edu:
+The corrected format uses a new root so the earlier raw-byte files cannot be mistaken for GPT-2-tokenized data:
 
 ```bash
-level0-prepare-data --dataset fineweb-edu
+export NANOGPT_LEVEL0_ROOT=/tmp/nanogpt-level0-gpt2
+export NANOGPT_LEVEL0_DATA_ROOT=$NANOGPT_LEVEL0_ROOT/data
+export NANOGPT_LEVEL0_RESULTS_ROOT=$NANOGPT_LEVEL0_ROOT/results
 ```
 
-To monitor the streamed download and preparation, enable heartbeat logging:
+## Prepare the pinned FineWeb-Edu corpus
 
 ```bash
 level0-prepare-data \
+  --config configs/level0.yaml \
   --dataset fineweb-edu \
   --verbose \
   --log-interval-seconds 10
 ```
 
-Verbose output reports documents processed, bytes collected, completion percentage, elapsed time, average throughput, estimated time remaining, and how long the stream has produced no new bytes. The heartbeat continues while the streaming iterator is blocked, making a network or dataset stall visible.
+The preparer reports documents, GPT-2 tokens, elapsed time, throughput, ETA, and time since the stream last produced tokens. It validates compatible existing data and reuses it; pass `--force` to rebuild it.
+
+A successful preparation produces:
 
-## Run one seed
+```text
+/tmp/nanogpt-level0-gpt2/data/train.bin
+/tmp/nanogpt-level0-gpt2/data/val.bin
+/tmp/nanogpt-level0-gpt2/data/test.bin
+/tmp/nanogpt-level0-gpt2/data/meta.json
+```
+
+## Run one AdamW seed
 
 ```bash
-./scripts/run_one.sh adamw 1337
-./scripts/run_one.sh muon 1337
+./scripts/run_one.sh adamw 1337 \
+  2>&1 | tee /tmp/level0-gpt2-adamw-seed1337.log
 ```
 
-## Run multiple seeds
+The command refuses to overwrite an existing nonempty run. To intentionally replace it:
 
 ```bash
-NANOGPT_LEVEL0_SEEDS=1337,2027,4099 ./scripts/run_multiseed.sh
+NANOGPT_LEVEL0_OVERWRITE=1 ./scripts/run_one.sh adamw 1337
 ```
 
-The notebooks read `NANOGPT_LEVEL0_RESULTS_ROOT`. Select the single-seed run with `NANOGPT_LEVEL0_NOTEBOOK_OPTIMIZER` and `NANOGPT_LEVEL0_NOTEBOOK_SEED`.
+The run writes periodic metrics and checkpoints, validation-selected and final test metrics, WeightWatcher CSV files, and `run_complete.json` under:
+
+```text
+/tmp/nanogpt-level0-gpt2/results/adamw_seed_1337
+```
 
-For a bounded infrastructure smoke test, override the run length and batch size:
+## Run several AdamW seeds
 
 ```bash
-NANOGPT_LEVEL0_MAX_STEPS=2 NANOGPT_LEVEL0_BATCH_SIZE=2 NANOGPT_LEVEL0_EVAL_INTERVAL=1 ./scripts/run_one.sh adamw 1337
+NANOGPT_LEVEL0_SEEDS=1337,2027,4099 \
+NANOGPT_LEVEL0_OPTIMIZERS=adamw \
+  ./scripts/run_multiseed.sh
 ```
 
-Next-token error is `1 - next-token accuracy`; the notebooks derive and plot it explicitly.
+Muon remains available for a later baseline comparison:
+
+```bash
+NANOGPT_LEVEL0_OPTIMIZERS=adamw,muon ./scripts/run_multiseed.sh
+```
+
+## Analyze one seed
+
+```bash
+export NANOGPT_LEVEL0_RESULTS_ROOT=/tmp/nanogpt-level0-gpt2/results
+export NANOGPT_LEVEL0_NOTEBOOK_OPTIMIZER=adamw
+export NANOGPT_LEVEL0_NOTEBOOK_SEED=1337
+jupyter lab notebooks/01_single_seed.ipynb
+```
+
+The notebook plots train/validation loss, perplexity, exact next-GPT-2-token accuracy, generalization gap, learning rate, final versus validation-selected test metrics, and WeightWatcher alpha by transformer matrix.
+
+## Bounded smoke run
+
+The smoke run checks the software path only; it is not a scientific result:
+
+```bash
+NANOGPT_LEVEL0_MAX_STEPS=2 \
+NANOGPT_LEVEL0_BATCH_SIZE=1 \
+NANOGPT_LEVEL0_GRAD_ACCUM_STEPS=1 \
+NANOGPT_LEVEL0_EVAL_INTERVAL=1 \
+NANOGPT_LEVEL0_WEIGHTWATCHER=0 \
+NANOGPT_LEVEL0_OVERWRITE=1 \
+  ./scripts/run_one.sh adamw 1337
+```
+
+## Primary outcomes
+
+Use held-out cross-entropy and perplexity as the principal language-model outcomes. Exact token accuracy is reported as a secondary diagnostic and should not be compared numerically with the old 256-byte-vocabulary accuracy.

level_0_baseline/configs/level0.yaml

@@ -1,17 +1,30 @@
+data:
+  dataset_name: HuggingFaceFW/fineweb-edu
+  dataset_config: sample-10BT
+  dataset_split: train
+  dataset_revision: 593b3a867298afb8ce42625a270ef20ddcad28f9
+  tokenizer: gpt2
+  dtype: uint16
+  train_tokens: 16000000
+  val_tokens: 1000000
+  test_tokens: 1000000
 model:
-  vocab_size: 256
+  vocab_size: 50257
   block_size: 256
-  n_layer: 1
-  n_head: 1
-  n_embd: 64
+  n_layer: 4
+  n_head: 4
+  n_embd: 128
   dropout: 0.0
   bias: false
+  tie_weights: true
 training:
-  batch_size: 16
-  grad_accum_steps: 1
+  batch_size: 4
+  grad_accum_steps: 8
   max_steps: 2000
   eval_interval: 50
   eval_batches: 20
+  eval_batch_size: 4
+  test_eval_batches: 40
   checkpoint_interval: 250
   learning_rate: 0.0006
   muon_learning_rate: 0.02
@@ -21,7 +34,9 @@ training:
   weight_decay: 0.1
   beta1: 0.9
   beta2: 0.95
+  epsilon: 1.0e-8
   grad_clip: 1.0
+  layer_lr_decay: 1.0
   optimizer: adamw
   muon_momentum: 0.95
   muon_nesterov: true
@@ -30,4 +45,4 @@ training:
 analysis:
   weightwatcher: true
   weightwatcher_interval: 250
-  randomize: true
+  randomize: false