Validate DPO on Qwen3.5-9B-Base LoRA

[micahtyong]

DPO works on Spindle today through the Tinker SDK's forward_backward_custom, using the existing qwen35-9b-lora-16k deployment. No new deployment config or wrapper is needed: the stock tinker-cookbook DPO recipe runs unchanged with base_url pointed at a Spindle server.

Validation. The cookbook DPO recipe covers reference-sampler publication, compute_logprobs, forward_backward_custom, and optim_step. With the cookbook settings (HHH, Qwen3.5-9B-Base, rank 32, β 0.1, batch 256) at lr 1e-4 for 10 steps:

Metric	Step 0	Step 9
dpo_loss	0.6939	0.6795
accuracy	0.480	0.557
margin	−0.0011	0.0502

The command and results are in docs/lora_validation.md, and the run is on W&B.

Fixes found while deploying from scratch:

• spindle deploy failed on macOS because socket.TCP_KEEPIDLE doesn't exist there. It now falls back to TCP_KEEPALIVE, the macOS name for the same option.
• The README generated API keys without the tml- prefix, which the Tinker SDK rejects. It now generates tml- keys.

Testing:

• pytest tests/inference passes on macOS.
• The cookbook DPO recipe ran end to end against a fresh qwen35-9b-lora-16k deployment with tinker 0.24.1.

🤖 Generated with Claude Code

[micahtyong]

Versions used for the DPO validation run in docs/lora_validation.md:

• Client: tinker==0.24.1 and tinker-cookbook at c8ed9c7, Python 3.12. Cookbook main requires tinker>=0.30.1, which Spindle doesn't support yet, so the run pinned an older cookbook commit.
• Server: Spindle at this branch, qwen35-9b-lora-16k config.

The docs show the plain cookbook command, which works once Spindle supports tinker 0.30. Until then, run it with:

uv run --python 3.12 --with "tinker>=0.24,<0.25" \
  --with "tinker-cookbook @ git+https://github.com/thinking-machines-lab/tinker-cookbook.git@c8ed9c764b59161391156f980102d82f05014765" \
  python -m tinker_cookbook.recipes.preference.dpo.train \
  base_url=$TINKER_BASE_URL model_name=Qwen/Qwen3.5-9B-Base dataset=hhh \
  renderer_name=role_colon learning_rate=1e-4 dpo_beta=0.1 max_steps=10 \
  log_path=/tmp/dpo-hhh-experiment