[nathanhubens]

RegularizeCallback computed a per-group L1 and then summed it over the groups. The groups partition the weights, so the value was the layer's elementwise L1 whatever the granularity: that parameter never changed the penalty.

What changes

• RegularizeCallback(..., *, norm=1).
  • norm=1 (default) is unchanged, byte for byte: a test compares the loss and every gradient to the previous formula.
  • norm=2 sums the L2 norm (not squared) of each group of the chosen granularity, i.e. a group lasso within a layer, via torch.linalg.vector_norm.
  • Any other value is refused.
• Docs, API page and tutorial:
  • They now say what each norm does.
  • They carry the BatchNorm caveat. In a network where each convolution is followed by BatchNorm, the BatchNorm rescales each filter's output, so penalizing the convolution weights does not select filters. To prepare structured pruning there, penalize the BatchNorm scale or use a group penalty that includes it.
  • The tutorial example becomes RegularizeCallback(large_final, 'filter', 1e-3, norm=2, ...).
  • The tutorial previously claimed that whole groups were pushed towards zero with the default L1; that sentence is gone.

Evidence

• Full suite nbdev-test passes in a clean clone, with CI's nbdev 3.3.21. Clean and export leave the tree unchanged.
• The tutorial was re-executed: every stored output comes from the visible code.
• Tests in nbs/tests/test_regularize_callback.ipynb:
  • norm=1 byte identity;
  • granularity invariance under norm=1, which pins the old behaviour, and a difference under norm=2;
  • hand-computed norm=2 values;
  • a finite, zero gradient on an all-zero group;
  • invalid norm refused;
  • a #| slow Learner fit for both norms.