TextualRL: Textual Reinforcement Learning for Context Management

TextualRL improves an agent's shared textual instructions while keeping the target model's weights fixed. It collects multiple rollouts for each training task, critiques outcome groups, reviews proposed edits against pooled trajectory evidence, and selects candidate contexts using held-out validation rewards.

Framework

Groupwise Policy Critique keeps same-task rollout groups together. Mixed outcomes support within-task contrast, all-success groups support cross-task preservation, and all-failure groups support cross-task repair. Critics return edit proposals and evidence cards separately, including evidence with no proposed edit.

Cross-Group Policy Update consolidates proposals, reviews their scope against current-step evidence, and applies a bounded patch. The ordinary step accepts a candidate only when its configured held-out validation score improves. Within-epoch feedback informs subsequent critique calls.

Install

Use Python 3.10 or newer from the repository directory:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e .

Install the extra needed by your benchmark, for example:

python -m pip install -e '.[searchqa]'
python -m pip install -e '.[spreadsheetbench]'
python -m pip install -e '.[alfworld]'

See installation for all six benchmarks. Prepare datasets using the data guide.

Inspect, train, and evaluate

First inspect the resolved configuration. Dry-run does not make model calls or require API credentials or benchmark data:

python -m textualrl train \
  --config configs/searchqa.yaml \
  --data-dir ./data \
  --output-dir ./outputs/searchqa/run1 \
  --dry-run

Copy .env.example to .env, replace its placeholders with your two endpoints and credentials, and export them:

cp .env.example .env
# Edit .env before sourcing it.
set -a
source .env
set +a

The default target is Qwen3.8-27B with thinking explicitly disabled. The default optimizer is gpt-5.5 with medium reasoning. The endpoint must expose the API and model name configured for its role; model names may need to match your provider's deployment names.

python -m textualrl train \
  --config configs/searchqa.yaml \
  --data-dir ./data \
  --output-dir ./outputs/searchqa/run1

python -m textualrl eval \
  --config configs/searchqa.yaml \
  --data-dir ./data \
  --output-dir ./outputs/searchqa/eval-best \
  --skill ./outputs/searchqa/run1/best_skill.md \
  --split test

Evaluation uses the target model; training also requires the optimizer. To resume an interrupted run, supply the same configuration, data, and run directory:

python -m textualrl train \
  --config configs/searchqa.yaml \
  --data-dir ./data \
  --resume-from ./outputs/searchqa/run1

See running experiments for configuration controls, checkpoint selection, and artifacts. A normal run can make many model calls: four rollouts for each of 40 tasks already produce 160 training trajectories per full batch, before critique and validation.

Codex and Claude Code harnesses

The target can also run through the native Codex CLI or Claude Code CLI while the optimizer continues to use an API endpoint. Ten harness presets cover SearchQA, SpreadsheetBench, OfficeQA, DocVQA, and LiveMathematicianBench, with gpt-5.5 configured for both roles:

python -m textualrl train \
  --config configs/harness/codex_searchqa.yaml \
  --data-dir ./data \
  --output-dir ./outputs/harness/codex/searchqa/run1 \
  --dry-run

Use claude_code_searchqa.yaml for the Claude Code harness. Harness and model are separate choices: GPT-5.5 inside Claude Code requires a user-configured Anthropic Messages-compatible gateway that routes to GPT-5.5; a stock Anthropic account or a model-name change is insufficient. Native targets use their CLI's own authentication, while training also needs optimizer API credentials. See the harness guide for setup, all presets, selected-skill evaluation, resume, and execution controls.

Documentation

• Installation and optional dependencies
• Datasets and split layout
• Configuration and endpoint controls
• Codex and Claude Code harnesses
• Training, evaluation, resume, and saved artifacts
• Method-to-code map

Offline checks

python -m pip install -e '.[dev]'
python -m pytest -q
python -m build

The tests cover outcome routing, evidence review, edit checks, context overflow splitting, generation settings, checkpoint behavior, and split preparation. An end-to-end smoke test runs the trainer, completed-run resume, and evaluator against a local HTTP fixture using synthetic tasks. These checks use no paid model service. They do not measure benchmark accuracy.

Citation

If you use TextualRL in your research, please cite:

@article{huai2026textualrl,
  title={TextualRL: Textual Reinforcement Learning for Context Management},
  author={Huai, Tianyu and Zhou, Jie and Mao, Bo and Chen, Qin and He, Liang and Qiu, Xipeng},
  journal={arXiv preprint arXiv:2610.18216},
  year={2026}
}

License and acknowledgements

We thank the authors of SkillOpt, which we consulted when developing our implementation.

The MIT license and Microsoft copyright notice are preserved.