feat(trainer): support configurable entropy regularization in PPO and GRPO actor

[hsusul]

Description

In large-scale reinforcement learning for reasoning models (e.g. DeepSeek-R1, QwQ, and mathematical reasoning / tool use), policy optimization often suffers from premature entropy collapse, where the policy rapidly concentrates on repetitive degenerate sequences or narrow sub-optimal generation paths. In classical and modern RL (TRL, CleanRL, OpenRLHF, verl), an entropy regularization bonus (-\beta \mathcal{H}(\pi)) is standard practice to encourage token exploration.

Previously in AReaL:

1. PPOActorConfig lacked an entropy_coeff option.
2. In grpo_loss_fn (areal/trainer/ppo/actor.py), entropy was unconditionally detached (entropy = entropy.detach()) purely for metric logging, making it impossible to propagate gradients back to model weights even when engines (such as FSDP or Megatron with entropy_requires_grad=True) compute differentiable entropy.

This PR adds configurable entropy regularization:

• Configuration: Adds entropy_coeff (default 0.0) to PPOActorConfig with non-negative validation in __post_init__.
• Differentiable Entropy Loss: In grpo_loss_fn, when entropy_coeff > 0.0, entropy is kept attached to the autograd graph, and a masked token-level entropy loss term (\mathcal{L}{\text{entropy}} = -\frac{1}{|\mathcal{M}|} \sum{i \in \mathcal{M}} \mathcal{H}(\pi_\theta(s_i))) is added to the objective scaled by entropy_coeff.
• Zero-Overhead Detached Default: When entropy_coeff == 0.0, entropy is immediately detached, preserving the zero-autograd-overhead default.
• Observability: Records entropy_loss in stats_tracker for logging to WandB/SwanLab/TensorBoard.
• CLI Docs: Regenerated documentation in docs/en/cli_reference.md and docs/zh/cli_reference.md.

Verification Plan

• Unit test suite in tests/test_ppo_entropy.py:
  • test_ppo_actor_config_entropy_coeff_validation: verifies default, positive acceptance, and rejection of negative values.
  • test_grpo_loss_fn_entropy_bonus_gradient: verifies that when entropy_coeff > 0.0, gradients flow through entropy to model logits.
  • test_grpo_loss_fn_entropy_zero_is_detached: verifies that when entropy_coeff == 0.0, entropy receives no gradients.
  • test_grpo_loss_fn_entropy_bonus_modifies_loss: verifies mathematical correctness of the entropy bonus in the overall loss.
• Verified existing PPO tests in tests/test_ppo_stats.py (15 passed).
• Passed all 16 pre-commit hooks (ruff check, ruff format, generate-cli-docs, etc.).

[Le8r0nJames]

This looks useful to support as an opt-in feature. Have you run any end-to-end training experiments with a positive entropy_coeff? It would be helpful to share the model/backend, coefficient, and how entropy and validation performance compare with the zero-coefficient baseline.

[Le8r0nJames]

Could we validate the backend requirements when entropy_coeff > 0? With Megatron's enable_chunked_logits=True and the default entropy_requires_grad=False, entropy is non-differentiable upstream, so this term changes the loss value without contributing gradients.

Please reject incompatible configurations or enable a supported differentiable path, and add a test using engine-produced entropy.