Exploring Transformer Architecture

Uses presidential speech data for classification (3-way speaker prediction) and language modeling tasks.

Directory Structure

src/                      # Source code
│   ├── main.py           # Main entry point: classify, generate, sparse
│   ├── transformer.py    # Encoder, Decoder, Classifier; full and causal multi-head attention
│   ├── sparse_attention.py # SparseEncoder with local window attention
│   ├── tokenizer.py      # Word-based tokenizer (NLTK)
│   ├── dataset.py       # SpeechesClassificationDataset, LanguageModelingDataset
│   └── utilities.py     # Attention visualization and sanity checks
speechesdataset/      # Data directory
│   ├── train_CLS.tsv     # Classification training (label, text)
│   ├── test_CLS.tsv      # Classification test
│   ├── train_LM.txt      # Language modeling training
│   ├── test_LM_obama.txt
│   ├── test_LM_wbush.txt
│   └── test_LM_hbush.txt
results/              # Output directory (created when running)
    ├── encoder/          # Classification attention plots
    ├── sparse_encoder/   # Sparse classification attention plots
    └── decoder/         # Language modeling attention plots

Dependencies

• Python 3
• PyTorch
• NLTK (for word_tokenize)
• matplotlib

Install NLTK data if needed:

python -c "import nltk; nltk.download('punkt')"

Usage

Run from the src/ directory. Data is in speechesdataset/. Results are written to results/.

Classification (classify)

Train an encoder + classifier to predict speaker (3 classes) from speech text.

cd src
python3 main.py --task classify

Produces attention sanity-check plots in results/encoder/.

Language Modeling (generate)

Train a decoder for language modeling. Reports perplexity on Obama, W. Bush, and H. Bush test sets.

cd src
python3 main.py --task generate

Produces attention sanity-check plots in results/decoder/.

Sparse Attention (sparse)

Train a classification model with SparseEncoder (local window attention instead of full attention).

cd src
python3 main.py --task sparse

Produces attention plots in results/sparse_encoder/.

Module Overview

File	Purpose
src/transformer.py	Encoder, Decoder, Classifier; full and causal multi-head attention
src/sparse_attention.py	SparseEncoder, SparseMultiHeadAttention with local window attention
src/tokenizer.py	SimpleTokenizer (word-level, NLTK-based)
src/dataset.py	SpeechesClassificationDataset, LanguageModelingDataset
src/utilities.py	Utilities for attention visualization and sanity checks

Hyperparameters (from src/main.py)

Parameter	Value	Description
batch_size	16	Batch size
block_size	32	Max context length
learning_rate	1e-3	Adam learning rate
n_embd	64	Embedding dimension
n_heads	2	Number of attention heads
n_layers	4	Number of transformer layers
epochs	15	Epochs for classification
max_epochs	500	Max LM training iterations
eval_epochs	100	Evaluate perplexity every N iterations