A decoder-only transformer language model trained from scratch on a ~1.1 MB corpus of Shakespeare, generating text one character at a time. Built for a Machine Learning course at Taylor University.
Attribution: the model architecture follows Andrej Karpathy's Let's build GPT: from scratch, in code, spelled out and his nanoGPT. I worked through that implementation to understand the mechanics, then ran my own hyperparameter experiments on top of it — documented below.
| Component | Value |
|---|---|
| Layers | 6 transformer blocks |
| Attention heads | 6 (64-dim each) |
| Embedding dim | 384 |
| Context window | 1024 characters |
| Dropout | 0.2 |
| Optimizer | AdamW, lr 3e-4 |
| Training iterations | 20,000 |
| Vocabulary | 65 characters (character-level, no subword tokenizer) |
Implemented in raw PyTorch — scaled dot-product attention with a causal mask, multi-head
attention, position embeddings, pre-norm residual blocks, and a feed-forward network. No
nn.Transformer, no HuggingFace.
The baseline walkthrough uses a 256-character context and 5,000 training iterations. I raised both and compared the output:
- Context window 256 → 1024. More context per prediction, so the model can carry syntactic structure across a longer span rather than losing the thread mid-sentence.
- Iterations 5,000 → 20,000. More gradient steps to converge on Shakespearean syntax, which the model has to learn from scratch at the character level — it starts with no notion of what an English word even is.
Both changes improved output quality, and together they moved the samples from word-shaped noise to noticeably more coherent paragraphs with consistent speaker-name formatting and line structure.
pip install -r requirements.txt
python train.pyPrints train/val loss every 300 steps, then generates 500 characters of sample output.
Uses CUDA when available and falls back to CPU — on CPU, expect the full 20,000-iteration
run to take a long time; lower max_iters for a quick check.
train.py— model, training loop, and sampling in a single fileinput.txt— the tinyshakespeare corpus (public domain)