feat(reward): support composable reasoning rewards with format verification and length regulation

[hsusul]

Description

In reasoning RL post-training and RLVR (Reinforcement Learning with Verifiable Rewards, e.g. DeepSeek-R1 style training on GSM8K, MATH, and Olympiad tasks), models trained exclusively on binary task accuracy suffer from two critical failure modes:

1. Format Degradation: Models frequently drop closing reasoning tags (</think>), generate multiple unclosed <think> blocks, output empty thinking traces, or fail to structure their final answer into standard delimiters (\boxed{...}, <answer>...</answer>, ####).
2. Length Hacking / Verbosity Exploitation: Under policy gradient algorithms (PPO, GRPO), models often discover that generating repetitive, rambling tokens artificially delays penalty or exploits length biases.

As highlighted in community discussion (#162), AReaL previously lacked modular support for format rewards, length penalties, and composable reward functions.

This PR introduces a clean, composable reasoning reward module in areal.reward:

1. FormatReward & FormatRewardConfig:
   • Validates <think>...</think> tag structure (detects unclosed, missing, duplicate, or reversed tags).
   • Validates and extracts answers from LaTeX boxed expressions (\boxed{...} with robust balanced brace parsing), XML tags (<answer>...</answer>), and GSM8K delimiters (####).
   • Computes configurable structure rewards, answer format rewards, and malformed tag penalties.
   • Provides extract_reasoning_and_answer utility to isolate the reasoning chain from the scoreable final answer.
2. LengthPenalty & LengthPenaltyConfig:
   • Implements bounded anti-length-hacking penalties supporting threshold ($-\alpha \max(0, L - L_{\text{target}})$), linear, and soft_tanh modes.
   • Operates on token IDs (completion_ids) or decoded text.
3. CompositeReward:
   • Computes $R = w_{\text{acc}} R_{\text{acc}} + w_{\text{format}} R_{\text{format}} + w_{\text{len}} R_{\text{len}}$.
   • Automatically reports sub-reward breakdown (reward_accuracy, reward_format, reward_length_penalty, reward_composite) to stats_tracker for real-time training observability.
   • Works seamlessly with RLVRWorkflow and MultiTurnWorkflow.
4. get_deepseek_r1_math_reward:
   • Out-of-the-box factory helper matching the canonical DeepSeek-R1 math reasoning reward specification.

Related Issue

Relates to #162

Type of Change

• 🐛 Bug fix
• ✨ New feature
• 💥 Breaking change
• 📝 Documentation update
• ♻️ Refactoring
• ⚡ Performance improvement
• ✅ Test coverage improvement

Checklist

• I have read the Contributing Guide
• Pre-commit hooks pass (pre-commit run --all-files)
• Relevant tests pass; new tests added for new functionality
• Documentation updated (if applicable; built with ./docs/build_all.sh)
• Branch is up to date with main
• Self-reviewed via /review-pr command
• This PR was created by a coding agent via /create-pr
• This PR is a breaking change

Additional Context

• 13/13 new unit tests in tests/test_reasoning_reward.py passing cleanly.
• Full reward test suite (98/98 tests) passing cleanly.
• All 16 pre-commit hooks passing across the entire repository.

[Le8r0nJames]

The claimed MultiTurnWorkflow compatibility needs a regression test. Its retry loop continues only while reward == 0, but this composite returns a positive reward for an incorrect, well-formatted answer, ending retries prematurely. Could we separate task success from the shaped training reward, or explicitly document this incompatibility?

[Le8r0nJames]

Could we validate a non-empty answer outside the thinking block? answer or text falls back to the full completion when the answer is missing, so <think>...\boxed{42}</think> receives full format reward. An empty \boxed{} is also accepted. Please add negative tests for both cases.

[Le8r0nJames]

Could we propagate accuracy scorer exceptions by default? Returning zero accuracy while still awarding format rewards can turn a broken scorer into format-only training without stopping training. If this fallback is intentional, please make it opt-in and distinguish scorer failures from incorrect answers.