From f9c63ad1b9489546b97912d0e8ba1dbfa5fc903d Mon Sep 17 00:00:00 2001 From: LeeLeno <1846301824@qq.com> Date: Thu, 5 Nov 2026 23:28:20 +0800 Subject: [PATCH 1/8] docs: versioned site scaffold + P0 user reference pages (#706) Versioning (mjlab-style, main only for now): - build docs with sphinx-multiversion; branch whitelist defaults to main, tag whitelist empty (SMV_* env vars enable future releases) - Furo sidebar version switcher template + styles - deploy job: fetch-depth 0, root redirect to main/, keep_files so existing root-level pages and future version dirs survive deploys - html_baseurl points at /UniLab-doc/main/ for sitemap correctness P0 reference content (bilingual en/zh_CN): - 5-reference/6-manager_terms: reward/observation/termination term tables (framework-level unilab.envs.mdp + task-layer terms) - 2-user_guide/5-domain_randomization/2-field_reference: per-field DR configuration reference for the 14 event terms - 2-user_guide/2-algorithms/*: per-algo Hydra configuration reference (PPO/APPO/SAC/FlashSAC; efficiency / policy / training-runtime / reward-weight groups) --- .github/workflows/docs.yml | 25 +- docs/README.md | 3 +- docs/sphinx/AGENTS.md | 3 +- docs/sphinx/README.md | 37 +- docs/sphinx/requirements.txt | 2 + docs/sphinx/source/_static/css/custom.css | 29 ++ .../source/_templates/sidebar/versioning.html | 17 + docs/sphinx/source/conf.py | 20 +- .../2-user_guide/1-training/2-hydra_config.md | 11 + .../en/2-user_guide/2-algorithms/1-ppo.md | 78 ++++ .../en/2-user_guide/2-algorithms/2-appo.md | 84 ++++ .../en/2-user_guide/2-algorithms/3-sac.md | 94 ++++ .../2-user_guide/2-algorithms/4-flash_sac.md | 112 +++++ .../5-domain_randomization/0-index.md | 1 + .../2-field_reference.md | 350 ++++++++++++++ docs/sphinx/source/en/5-reference/0-index.md | 7 + .../en/5-reference/6-manager_terms/0-index.md | 128 +++++ .../5-reference/6-manager_terms/1-rewards.md | 439 ++++++++++++++++++ .../6-manager_terms/2-observations.md | 308 ++++++++++++ .../6-manager_terms/3-terminations.md | 138 ++++++ .../2-user_guide/1-training/2-hydra_config.md | 10 + .../zh_CN/2-user_guide/2-algorithms/1-ppo.md | 77 +++ .../zh_CN/2-user_guide/2-algorithms/2-appo.md | 83 ++++ .../zh_CN/2-user_guide/2-algorithms/3-sac.md | 93 ++++ .../2-user_guide/2-algorithms/4-flash_sac.md | 111 +++++ .../5-domain_randomization/0-index.md | 1 + .../2-field_reference.md | 327 +++++++++++++ .../source/zh_CN/5-reference/0-index.md | 7 + .../5-reference/6-manager_terms/0-index.md | 122 +++++ .../5-reference/6-manager_terms/1-rewards.md | 424 +++++++++++++++++ .../6-manager_terms/2-observations.md | 290 ++++++++++++ .../6-manager_terms/3-terminations.md | 123 +++++ 32 files changed, 3541 insertions(+), 13 deletions(-) create mode 100644 docs/sphinx/source/_templates/sidebar/versioning.html create mode 100644 docs/sphinx/source/en/2-user_guide/5-domain_randomization/2-field_reference.md create mode 100644 docs/sphinx/source/en/5-reference/6-manager_terms/0-index.md create mode 100644 docs/sphinx/source/en/5-reference/6-manager_terms/1-rewards.md create mode 100644 docs/sphinx/source/en/5-reference/6-manager_terms/2-observations.md create mode 100644 docs/sphinx/source/en/5-reference/6-manager_terms/3-terminations.md create mode 100644 docs/sphinx/source/zh_CN/2-user_guide/5-domain_randomization/2-field_reference.md create mode 100644 docs/sphinx/source/zh_CN/5-reference/6-manager_terms/0-index.md create mode 100644 docs/sphinx/source/zh_CN/5-reference/6-manager_terms/1-rewards.md create mode 100644 docs/sphinx/source/zh_CN/5-reference/6-manager_terms/2-observations.md create mode 100644 docs/sphinx/source/zh_CN/5-reference/6-manager_terms/3-terminations.md diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 9c3cb8774..77bd0bcee 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -77,6 +77,10 @@ jobs: steps: - name: Checkout uses: actions/checkout@v4 + with: + # sphinx-multiversion needs full git refs to build every + # whitelisted branch/tag. + fetch-depth: 0 - name: Install uv uses: astral-sh/setup-uv@v8.0.0 @@ -105,9 +109,22 @@ jobs: echo "::warning::UniLab install failed — deploying prose-only docs" echo "UNILAB_DOCS_SKIP_AUTODOC=1" >> "$GITHUB_ENV" - - name: Build full HTML + - name: Build full HTML (all whitelisted versions) + working-directory: docs/sphinx + # sphinx-multiversion builds one subdirectory per whitelisted ref + # (default: main only; see smv_* in source/conf.py). The prose-only + # fallback still applies — UNILAB_DOCS_SKIP_AUTODOC is honoured by + # conf.py inside each per-version build. + run: uv run --no-sync sphinx-multiversion source build/html + + - name: Add root redirect + working-directory: docs/sphinx + # Versioned builds land in build/html//; send / to main. + run: echo '' > build/html/index.html + + - name: Remove Sphinx doctree caches working-directory: docs/sphinx - run: uv run --no-sync sphinx-build -j auto -b html -n source build/html + run: find build/html -type d -name .doctrees -exec rm -rf {} + - name: Push to UniLab-doc gh-pages uses: peaceiris/actions-gh-pages@v4 @@ -118,6 +135,10 @@ jobs: external_repository: unilabsim/UniLab-doc publish_branch: gh-pages publish_dir: docs/sphinx/build/html + # Keep files not touched by this deploy: older version directories + # and the pre-existing root-level pages stay reachable, so old + # external links keep working. + keep_files: true user_name: "github-actions[bot]" user_email: "github-actions[bot]@users.noreply.github.com" commit_message: "docs: deploy from UniLab@${{ github.sha }}" diff --git a/docs/README.md b/docs/README.md index f94ed9a2b..8503e940c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,7 +1,8 @@ # UniLab Documentation UniLab 的全部文档源码统一在 [`docs/sphinx/`](sphinx/) 下,通过 Sphinx 构建,部署到 -。 +。站点按版本子路径发布(当前仅 `main/`, +如 `/UniLab-doc/main/en/0-index.html`),根路径自动重定向到 `main/`。 发布入口: diff --git a/docs/sphinx/AGENTS.md b/docs/sphinx/AGENTS.md index 29b5932e4..aa6224c6a 100644 --- a/docs/sphinx/AGENTS.md +++ b/docs/sphinx/AGENTS.md @@ -28,7 +28,8 @@ source/ ├── _static/ └── _templates/ ├── sidebar/ - │ └── lang_switcher.html # language dropdown in sidebar + │ ├── lang_switcher.html # language dropdown in sidebar + │ └── versioning.html # version dropdown (rendered only under sphinx-multiversion) └── autosummary/ ``` diff --git a/docs/sphinx/README.md b/docs/sphinx/README.md index 080d2d48b..7d52e1121 100644 --- a/docs/sphinx/README.md +++ b/docs/sphinx/README.md @@ -3,9 +3,10 @@ UniLab 的文档源码全部在这里,**双语平行结构**——`source/en/` 是英文版,`source/zh_CN/` 是中文版,`adr/`、`api_reference/`、`glossary.md`、`changelog.md`、`_static/` 共享。 -CI 在 push to `main` 时构建 Sphinx,通过 deploy key 推到独立仓 +CI 在 push to `main` 时用 sphinx-multiversion 构建 Sphinx,通过 deploy key 推到独立仓 [`unilabsim/UniLab-doc`](https://github.com/unilabsim/UniLab-doc) 的 `gh-pages` 分支, -由 GitHub Pages 发布到 。 +由 GitHub Pages 发布到 。站点按版本目录发布 +(当前仅 `main/`),根路径重定向到 `main/`。 ## 目录结构 @@ -41,7 +42,8 @@ docs/sphinx/ └── agents/ ← Agent 速查(中文,01-agent-quick-reference.md) ``` -URL 模式:`/`(语言 picker)、`/en/...`、`/zh_CN/...`。两种语言路径 1:1 镜像。 +URL 模式:`/`(重定向到 `main/en/0-index.html`)、`/main/en/...`、`/main/zh_CN/...`。站点按 +版本目录发布(当前仅 `main/`),版本目录内两种语言路径 1:1 镜像。 ## 本地构建 @@ -73,7 +75,8 @@ UNILAB_DOCS_SKIP_AUTODOC=1 uv run --no-project --with-requirements requirements. CI 工作流在 `.github/workflows/docs.yml`: -- **PR**: prose-only HTML build,跳过 API reference,failed 阻塞 PR +- **PR**: prose-only HTML build(plain `sphinx-build`),跳过 API reference, + failed 阻塞 PR。plain build 下 `versions` 上下文不存在,侧栏版本切换器不渲染 - **push to main**: prose-only build 通过后,`deploy` job 执行完整构建(安装 UniLab + autodoc)并通过 deploy key(`UNILAB_DOC_DEPLOY_KEY` secret)推送到 [`unilabsim/UniLab-doc`](https://github.com/unilabsim/UniLab-doc) 的 @@ -82,6 +85,19 @@ CI 工作流在 `.github/workflows/docs.yml`: - **手动**: `workflow_dispatch` 在 `main` 上触发时走同一套构建 + 部署; 在其他分支触发时只做 prose-only build +deploy job 用 `sphinx-multiversion source build/html` 构建,按版本目录发布: +每个白名单 ref 构建到 `build/html//`(当前白名单只有 `main`,见 +`source/conf.py` 的 `smv_*` 配置),随后在 `build/html/index.html` 写入根 +redirect(`meta refresh` → `main/en/0-index.html`),再清理各版本目录下的 +`.doctrees` 缓存后发布。发布步骤带 `keep_files: true`,旧版本目录和存量根目录 +页面不会被覆盖删除,旧外链保持可用。 + +多版本放行方式:默认 `smv_branch_whitelist = ^(main)$`、`smv_tag_whitelist = ^$` +(不匹配任何 tag)。未来发版时通过环境变量放行,例如在部署环境设置 +`SMV_TAG_WHITELIST='^v[1-9]\d*\.\d+\.\d+$'`(分支侧对应 +`SMV_BRANCH_WHITELIST`)。注意 `html_baseurl` 当前固定带 `main/` 后缀,启用多版本 +前需将其按版本动态化(见 `conf.py` 注释)。 + PR 与 push 的检查性构建明确设置 `UNILAB_DOCS_SKIP_AUTODOC=1`,不安装 UniLab 运行时依赖,也不跑 linkcheck。下面的本地发布流程保留为 CI 部署不可用时的 fallback。 @@ -97,14 +113,18 @@ fallback。 ### 本地发布到 UniLab-doc 完整 API 文档在本地构建完成后,把 `docs/sphinx/build/html/` 的内容同步到 -`unilabsim/UniLab-doc` 的 `gh-pages` 分支根目录: +`unilabsim/UniLab-doc` 的 `gh-pages` 分支根目录。构建方式与 CI 一致,用 +sphinx-multiversion(需要本地有对应 git ref;`main` 分支存在即可,shallow clone +也能构建),构建后补写根 redirect 并清理 `.doctrees`: ```bash # 在 UniLab 仓库根目录 uv sync uv pip install -r docs/sphinx/requirements.txt cd docs/sphinx -uv run --no-sync sphinx-build -j auto -b html -n source build/html +uv run --no-sync sphinx-multiversion source build/html +echo '' > build/html/index.html +find build/html -type d -name .doctrees -exec rm -rf {} + cd ../.. ``` @@ -121,10 +141,11 @@ git -C ../UniLab-doc checkout gh-pages git -C ../UniLab-doc pull --ff-only ``` -同步构建产物并推送: +同步构建产物并推送。**不要加 `--delete`**:CI 发布用 `keep_files: true` 保留旧版本 +目录和存量根目录页面,本地手动发布也要保留它们: ```bash -rsync -a --delete --exclude .git docs/sphinx/build/html/ ../UniLab-doc/ +rsync -a --exclude .git docs/sphinx/build/html/ ../UniLab-doc/ touch ../UniLab-doc/.nojekyll SOURCE_SHA=$(git rev-parse --short HEAD) diff --git a/docs/sphinx/requirements.txt b/docs/sphinx/requirements.txt index ea64032ef..c2555b050 100644 --- a/docs/sphinx/requirements.txt +++ b/docs/sphinx/requirements.txt @@ -10,3 +10,5 @@ sphinxcontrib-mermaid>=0.9 sphinx-togglebutton>=0.3 linkify-it-py>=2.0 sphinx-sitemap>=2.6 +sphinx-multiversion>=0.2 +beautifulsoup4>=4.12 diff --git a/docs/sphinx/source/_static/css/custom.css b/docs/sphinx/source/_static/css/custom.css index 65f4d2e94..7c2983dde 100644 --- a/docs/sphinx/source/_static/css/custom.css +++ b/docs/sphinx/source/_static/css/custom.css @@ -231,6 +231,35 @@ kbd { border-color: var(--color-brand-primary); } +/* ---------- Sidebar version switcher (sphinx-multiversion) ---------- */ +.sidebar-version-switcher { + display: flex; + align-items: center; + gap: 0.5rem; + padding: 0.4rem 0.8rem; + margin: 0 0 0.2rem 0; +} +.sidebar-version-label { + color: var(--color-foreground-secondary); + font-size: 0.85rem; + font-weight: 600; +} +.sidebar-version-select { + flex: 1; + min-width: 0; + padding: 0.35rem 0.6rem; + font-size: 0.85rem; + border: 1px solid var(--color-foreground-border); + border-radius: 6px; + background: var(--color-background-secondary); + color: var(--color-foreground-primary); + cursor: pointer; + appearance: auto; +} +.sidebar-version-select:hover { + border-color: var(--color-brand-primary); +} + /* Hide old body-injected language switcher */ .language-switcher { display: none !important; } .landing-language-picker { display: none !important; } diff --git a/docs/sphinx/source/_templates/sidebar/versioning.html b/docs/sphinx/source/_templates/sidebar/versioning.html new file mode 100644 index 000000000..c90837cbf --- /dev/null +++ b/docs/sphinx/source/_templates/sidebar/versioning.html @@ -0,0 +1,17 @@ +{# Furo sidebar component: version switcher (sphinx-multiversion). + Renders only under `sphinx-multiversion` builds, which inject `versions` + and `current_version` into the HTML context; plain `sphinx-build` runs + leave `versions` undefined and the guard hides the switcher. #} +{% if versions %} +
+ + +
+{% endif %} diff --git a/docs/sphinx/source/conf.py b/docs/sphinx/source/conf.py index 85a9aebd4..9faa8e017 100644 --- a/docs/sphinx/source/conf.py +++ b/docs/sphinx/source/conf.py @@ -70,6 +70,7 @@ "sphinxcontrib.video", "sphinxcontrib.mermaid", "sphinx_sitemap", + "sphinx_multiversion", ] # Only enable autodoc / autosummary when UniLab is importable. Otherwise # autosummary's recursive import probe blows up the whole build. @@ -174,6 +175,18 @@ ".rst": "restructuredtext", } templates_path = ["_templates"] + +# --------------------------------------------------------------------------- +# sphinx-multiversion — build one output subdirectory per whitelisted ref. +# Defaults build only `main` and match no tags; historical releases are out +# of scope for now. When releases are cut, allow them via the environment, +# e.g. SMV_TAG_WHITELIST='^v[1-9]\d*\.\d+\.\d+$'. +# Plain `sphinx-build` ignores these settings. +# --------------------------------------------------------------------------- +smv_remote_whitelist = r"^.*$" +smv_branch_whitelist = os.getenv("SMV_BRANCH_WHITELIST", r"^(main)$") +smv_tag_whitelist = os.getenv("SMV_TAG_WHITELIST", r"^$") + exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"] # When autodoc is disabled, skip the entire API reference tree — its pages # only contain autosummary directives that would otherwise fail. @@ -193,13 +206,18 @@ html_css_files = ["css/custom.css"] html_favicon = None # add _static/favicon.ico when ready # html_logo = "_static/images/logo.png" -html_baseurl = "https://unilabsim.github.io/UniLab-doc/" +# The site is published one subdirectory per version by sphinx-multiversion +# (currently only `main`), so canonical URLs live under /UniLab-doc/main/. +# If more versions are ever enabled, this must become version-aware (e.g. +# derived from the smv env vars) so each version's sitemap is correct. +html_baseurl = "https://unilabsim.github.io/UniLab-doc/main/" sitemap_url_scheme = "{link}" html_sidebars = { "**": [ "sidebar/brand.html", "sidebar/lang_switcher.html", + "sidebar/versioning.html", "sidebar/search.html", "sidebar/scroll-start.html", "sidebar/navigation.html", diff --git a/docs/sphinx/source/en/2-user_guide/1-training/2-hydra_config.md b/docs/sphinx/source/en/2-user_guide/1-training/2-hydra_config.md index fba914c3a..67f6a1751 100644 --- a/docs/sphinx/source/en/2-user_guide/1-training/2-hydra_config.md +++ b/docs/sphinx/source/en/2-user_guide/1-training/2-hydra_config.md @@ -70,3 +70,14 @@ is not an independent backend switch. Use the unified CLI `--sim` flag to select the backend. See the developer contract in {doc}`../../4-developer_guide/2-contracts/3-task_owner`. + +## Per-Algorithm Reference + +Field-by-field tables — training efficiency, policy performance (GAE lambda, entropy +coefficient, learning rate, network shape), training runtime, and reward term weights — +live on each algorithm page: + +- {doc}`../2-algorithms/1-ppo` +- {doc}`../2-algorithms/2-appo` +- {doc}`../2-algorithms/3-sac` +- {doc}`../2-algorithms/4-flash_sac` diff --git a/docs/sphinx/source/en/2-user_guide/2-algorithms/1-ppo.md b/docs/sphinx/source/en/2-user_guide/2-algorithms/1-ppo.md index 6f7d8ed5f..a66e87e33 100644 --- a/docs/sphinx/source/en/2-user_guide/2-algorithms/1-ppo.md +++ b/docs/sphinx/source/en/2-user_guide/2-algorithms/1-ppo.md @@ -90,3 +90,81 @@ smoke coverage for `go2_joystick_flat` and `g1_motion_tracking`; this does not claim multi-node or Motrix multi-GPU support. On the currently validated RTX 6000D host, the launcher inherits the production NCCL compatibility defaults `NCCL_P2P_DISABLE=1` and `NCCL_SHM_DISABLE=1`; explicit environment values win. + +## Configuration reference + +Defaults come from the typed configs in `src/unilab/structured_configs.py` +(`PPOConfig`); `src/unilab/conf/ppo/config.yaml` carries the same values unless noted. +Every field can be overridden on the command line: + +```bash +uv run train --algo ppo --task go2_joystick_flat --sim mujoco \ + algo.algorithm.lam=0.92 \ + algo.algorithm.entropy_coef=0.005 +``` + +### Training efficiency + +| Override key | Type | Default | Practical effect | +| --- | --- | --- | --- | +| `algo.num_envs` | int | 4096 | Parallel environments per rank (the go2_joystick_flat/mujoco owner sets 1024) | +| `algo.num_steps_per_env` | int | 24 | Rollout length per env; samples per iteration = num_envs × this | +| `algo.max_iterations` | int | 101 | Number of training iterations (the go2 owner sets 151) | +| `algo.save_interval` | int | 100 | Checkpoint save interval in iterations | +| `algo.algorithm.num_learning_epochs` | int | 5 | SGD epochs over each collected batch | +| `algo.algorithm.num_mini_batches` | int | 4 | Mini-batches per epoch | +| `algo.seed` | int | 1 | Base seed; rank `i` uses `seed + i` | +| `training.device` | str / null | null | Explicit single-device override; null = automatic placement | +| `training.log_interval` | int | 1 | TensorBoard/W&B scalar reporting interval in iterations | + +### Policy performance + +| Override key | Type | Default | Practical effect | +| --- | --- | --- | --- | +| `algo.algorithm.gamma` | float | 0.99 | Discount factor | +| `algo.algorithm.lam` | float | 0.95 | GAE lambda; lower trades bias for variance | +| `algo.algorithm.entropy_coef` | float | 0.01 | Entropy bonus coefficient (the go2 owner lowers it to 1e-3) | +| `algo.algorithm.learning_rate` | float | 1e-3 | Adam learning rate; initial value under the adaptive schedule (the go2 owner sets 3e-4) | +| `algo.algorithm.schedule` | str | adaptive | LR schedule; `adaptive` steers KL toward `desired_kl` | +| `algo.algorithm.desired_kl` | float | 0.01 | KL target of the adaptive LR schedule | +| `algo.algorithm.clip_param` | float | 0.2 | PPO surrogate clip range | +| `algo.algorithm.value_loss_coef` | float | 1.0 | Weight of the value loss term | +| `algo.algorithm.use_clipped_value_loss` | bool | true | Clip the value targets like the policy objective | +| `algo.algorithm.max_grad_norm` | float | 1.0 | Gradient norm clip | +| `algo.actor.hidden_dims` | list[int] | [512, 256, 128] | Policy MLP layer widths | +| `algo.actor.activation` | str | elu | Policy MLP activation | +| `algo.actor.obs_normalization` | bool | false | Normalize policy observations (the go2 owner enables it) | +| `algo.actor.distribution_cfg.init_std` | float | 1.0 | Initial Gaussian action std (the go2 owner sets 0.5) | +| `algo.critic.hidden_dims` | list[int] | [512, 256, 128] | Critic MLP layer widths | +| `algo.critic.obs_normalization` | bool | false | Normalize critic observations (the go2 owner enables it) | + +### Training runtime (`training:` section) + +| Override key | Type | Default | Practical effect | +| --- | --- | --- | --- | +| `training.no_play` | bool | false | Skip policy playback after training | +| `training.play_only` | bool | false | Skip training and only play a checkpoint | +| `training.tf32_matmul` | str | ieee | Learner FP32 matmul backend; `tf32` opts into tensor cores with automatic IEEE fallback | +| `training.nan_guard.enabled` | bool | true | Detect non-finite tensors and dump diagnostics | +| `training.nan_guard.check_interval` | int | 32 | Env steps between NaN checks | +| `training.defer_finite_validation` | bool | true | Batch finite checks into a periodic cycle | +| `training.finite_validation_interval` | int | 32 | Steps per batched finite-check cycle | +| `training.collector_tensor_device` | str | auto | Device of collector-side tensors; `auto` follows the learner | +| `training.logger` | str | tensorboard | Scalar logging backend (`tensorboard` or `wandb`) | +| `training.num_timesteps` | int / null | null | Global sample budget; overrides the derived iteration count when set | + +### Reward term weights + +Reward terms live in the task owner YAML under `reward:` — one block per term with +`func`, `weight`, and `params` (example: +`src/unilab/conf/ppo/task/go2_joystick_flat/base.yaml`). Retune a term by editing that +owner YAML or by overriding its weight on the command line: + +```bash +uv run train --algo ppo --task go2_joystick_flat --sim mujoco \ + reward.tracking_lin_vel.weight=1.5 \ + reward.action_rate.weight=-0.01 +``` + +Term names and default weights are task-specific; check the `reward:` block of the owner +YAML for the task you train (`src/unilab/conf/ppo/task//base.yaml`). diff --git a/docs/sphinx/source/en/2-user_guide/2-algorithms/2-appo.md b/docs/sphinx/source/en/2-user_guide/2-algorithms/2-appo.md index ff09be043..b7165f812 100644 --- a/docs/sphinx/source/en/2-user_guide/2-algorithms/2-appo.md +++ b/docs/sphinx/source/en/2-user_guide/2-algorithms/2-appo.md @@ -74,3 +74,87 @@ gantt The default log root is `logs/appo//`, from `algo.algo_log_name=appo` in `src/unilab/conf/appo/config.yaml`. + +## Configuration reference + +Defaults come from the typed configs in `src/unilab/structured_configs.py` +(`APPOConfig`); `src/unilab/conf/appo/config.yaml` carries the same values unless noted. +Every field can be overridden on the command line: + +```bash +uv run train --algo appo --task go2_joystick_flat --sim mujoco \ + algo.algorithm.lam=0.92 \ + algo.algorithm.vtrace_clip_rho=1.2 +``` + +### Training efficiency + +| Override key | Type | Default | Practical effect | +| --- | --- | --- | --- | +| `algo.num_envs` | int | 2048 | Environments per collector | +| `algo.steps_per_env` | int | 24 | Rollout length per env in each published rollout | +| `algo.max_iterations` | int | 150 | Number of learner iterations | +| `algo.save_interval` | int | 50 | Checkpoint save interval in iterations | +| `algo.algorithm.num_learning_epochs` | int | 5 | SGD epochs over each consumed rollout | +| `algo.algorithm.num_mini_batches` | int | 4 | Mini-batches per epoch | +| `algo.algorithm.enable_compile` | bool | true | torch.compile on the learner update; the built-in `src/unilab/conf/appo/config.yaml` ships `false` | +| `algo.seed` | int | 1 | Random seed | +| `training.collector_device` | str / null | null | Collector device; null follows the learner | +| `training.replay_queue_size` | int / null | null | Learner-side replay cache depth | +| `training.log_interval` | int | 1 | TensorBoard/W&B scalar reporting interval in iterations | + +### Policy performance + +| Override key | Type | Default | Practical effect | +| --- | --- | --- | --- | +| `algo.algorithm.gamma` | float | 0.99 | Discount factor | +| `algo.algorithm.lam` | float | 0.95 | GAE lambda; lower trades bias for variance | +| `algo.algorithm.entropy_coef` | float | 0.01 | Entropy bonus coefficient | +| `algo.algorithm.learning_rate` | float | 1e-3 | Adam learning rate; initial value under the adaptive schedule | +| `algo.algorithm.schedule` | str | adaptive | LR schedule; `adaptive` steers KL toward `desired_kl` | +| `algo.algorithm.desired_kl` | float | 0.01 | KL target of the adaptive LR schedule | +| `algo.algorithm.adaptive_kl_factor` | float | 1.2 | KL deadband: LR drops above `desired_kl` × this and rises below `desired_kl` ÷ this | +| `algo.algorithm.adaptive_lr_factor` | float | 1.1 | Multiplicative LR step of the adaptive schedule | +| `algo.algorithm.clip_param` | float | 0.2 | PPO surrogate clip range | +| `algo.algorithm.value_loss_coef` | float | 1.0 | Weight of the value loss term | +| `algo.algorithm.use_clipped_value_loss` | bool | true | Clip the value targets like the policy objective | +| `algo.algorithm.max_grad_norm` | float | 1.0 | Gradient norm clip | +| `algo.algorithm.optimizer` | str | adam | Optimizer for the learner update | +| `algo.algorithm.vtrace_clip_rho` | float | 1.0 | V-trace importance-weight clip ρ̄ | +| `algo.algorithm.vtrace_clip_c` | float | 1.0 | V-trace importance-weight clip c̄ | +| `algo.algorithm.tau` | float | 1.0 | Target-network EMA coefficient (1.0 = hard copy) | +| `algo.algorithm.target_update_freq` | int | 1 | Learner updates between target-network syncs | +| `algo.actor.hidden_dims` | list[int] | [512, 256, 128] | Policy MLP layer widths | +| `algo.actor.activation` | str | elu | Policy MLP activation | +| `algo.actor.distribution_cfg.init_std` | float | 1.0 | Initial Gaussian action std | +| `algo.critic.hidden_dims` | list[int] | [512, 256, 128] | Critic MLP layer widths | + +### Training runtime (`training:` section) + +| Override key | Type | Default | Practical effect | +| --- | --- | --- | --- | +| `training.no_play` | bool | false | Skip policy playback after training | +| `training.play_only` | bool | false | Skip training and only play a checkpoint | +| `training.tf32_matmul` | str | ieee | Learner FP32 matmul backend; `tf32` opts into tensor cores with automatic IEEE fallback | +| `training.nan_guard.enabled` | bool | true | Detect non-finite tensors and dump diagnostics | +| `training.nan_guard.check_interval` | int | 32 | Env steps between NaN checks | +| `training.defer_finite_validation` | bool | true | Batch finite checks into a periodic cycle | +| `training.finite_validation_interval` | int | 32 | Steps per batched finite-check cycle | +| `training.collector_tensor_device` | str | auto | Device of collector-side tensors; `auto` follows the learner | +| `training.logger` | str | tensorboard | Scalar logging backend (`tensorboard` or `wandb`) | + +### Reward term weights + +Reward terms live in the task owner YAML under `reward:` — one block per term with +`func`, `weight`, and `params` (example: +`src/unilab/conf/appo/task/go2_joystick_flat/base.yaml`). Retune a term by editing that +owner YAML or by overriding its weight on the command line: + +```bash +uv run train --algo appo --task go2_joystick_flat --sim mujoco \ + reward.tracking_lin_vel.weight=1.5 \ + reward.action_rate.weight=-0.01 +``` + +Term names and default weights are task-specific; check the `reward:` block of the owner +YAML for the task you train (`src/unilab/conf/appo/task//base.yaml`). diff --git a/docs/sphinx/source/en/2-user_guide/2-algorithms/3-sac.md b/docs/sphinx/source/en/2-user_guide/2-algorithms/3-sac.md index d3a6bba1e..dcb796daf 100644 --- a/docs/sphinx/source/en/2-user_guide/2-algorithms/3-sac.md +++ b/docs/sphinx/source/en/2-user_guide/2-algorithms/3-sac.md @@ -58,3 +58,97 @@ MuJoCo has a committed multi-GPU scaling benchmark. The mjwarp per-rank placemen covered by `tests/base/backend/test_process_device.py` and the off-policy runner/worker unit tests; the repository does not currently contain an mjwarp multi-GPU throughput or convergence benchmark. + +## Configuration reference + +Defaults come from the typed configs in `src/unilab/structured_configs.py` +(`SACConfig`); `src/unilab/conf/sac/config.yaml` carries the same values unless noted. +Every field can be overridden on the command line: + +```bash +uv run train --algo sac --task g1_walk_flat --sim mujoco \ + algo.gamma=0.99 \ + algo.algo_params.alpha_init=0.1 +``` + +### Training efficiency + +| Override key | Type | Default | Practical effect | +| --- | --- | --- | --- | +| `algo.num_envs` | int | 4096 | Parallel collector environments | +| `algo.batch_size` | int | 8192 | Learner batch per update | +| `algo.replay_buffer_n` | int | 512 | Replay capacity in env-vector sweeps (rows = n × num_envs) | +| `algo.updates_per_step` | int | 4 | Learner updates per env step | +| `algo.learning_starts` | int | 1 | Env-vector sweeps to collect before learner updates start | +| `algo.policy_frequency` | int | 4 | One actor/target update every N learner updates | +| `algo.max_iterations` | int | 500 | Number of training iterations | +| `algo.save_interval` | int | 500 | Checkpoint save interval in iterations | +| `algo.algo_params.use_compile` | bool | true | torch.compile on the learner update | +| `algo.seed` | int | 1 | Random seed | +| `training.use_amp` | bool | true | bf16 autocast on the learner update path | +| `training.log_interval` | int | 1 | TensorBoard/W&B scalar reporting interval in iterations | + +### Policy performance + +| Override key | Type | Default | Practical effect | +| --- | --- | --- | --- | +| `algo.gamma` | float | 0.97 | Discount factor | +| `algo.tau` | float | 0.125 | Target-critic EMA coefficient | +| `algo.actor_lr` | float | 3e-4 | Actor Adam learning rate | +| `algo.critic_lr` | float | 3e-4 | Critic Adam learning rate | +| `algo.actor_hidden_dim` | int | 512 | Actor network width | +| `algo.critic_hidden_dim` | int | 768 | Critic network width | +| `algo.num_atoms` | int | 101 | Atoms of the distributional (C51) critic | +| `algo.obs_normalization` | bool | false | Normalize observations | +| `algo.use_layer_norm` | bool | true | LayerNorm in the networks | +| `algo.algo_params.alpha_lr` | float | 3e-4 | Entropy temperature (alpha) learning rate | +| `algo.algo_params.alpha_init` | float | 0.01 | Initial entropy temperature | +| `algo.algo_params.target_entropy_ratio` | float | 0.0 | Target entropy = −action_dim × ratio (0.0 → target 0) | +| `algo.algo_params.max_grad_norm` | float | 0.0 | Gradient norm clip; 0 disables clipping | +| `algo.algo_params.amp_dtype` | str | auto | AMP dtype under `training.use_amp` | + +### Training runtime (`training:` section) + +| Override key | Type | Default | Practical effect | +| --- | --- | --- | --- | +| `training.no_play` | bool | false | Skip policy playback after training | +| `training.play_only` | bool | false | Skip training and only play a checkpoint | +| `training.export_onnx` | bool | true | Export `policy.onnx` during playback | +| `training.cuda_process_sharing` | str / null | null | null = independent CUDA contexts; `mps` requests CUDA MPS (user-managed daemon via `uni-cumps`) | +| `training.dp_gradient_wire_dtype` | str | float32 | Gradient wire dtype for multi-GPU averaging; `bfloat16` compresses | +| `training.nan_guard.enabled` | bool | true | Detect non-finite tensors and dump diagnostics | +| `training.nan_guard.check_interval` | int | 32 | Env steps between NaN checks | +| `training.defer_finite_validation` | bool | true | Batch finite checks into a periodic cycle | +| `training.finite_validation_interval` | int | 32 | Steps per batched finite-check cycle | +| `training.collector_tensor_device` | str | auto | Device of collector-side tensors; `auto` follows the learner | +| `training.env_steps_per_sync` | int | 1 | Env steps per collection sync | +| `training.inference_slot_capacity` | int | 1 | In-flight learner-inference requests; 1 keeps inference serialized | +| `training.collector_metrics_interval` | int | 32 | Collector steps between metric flushes | +| `training.replay_ingress_depth` | int | 2 | Replay-ingress chunks allowed in flight | +| `training.replay_ingress_slot_rows` | int / null | null | Rows per ingress chunk; null = one chunk per collector vector | +| `training.replay_prefetch_mode` | str | one_tick | Replay prefetch strategy | +| `training.trace_enabled` | bool | false | Chrome-trace profiling of the off-policy runtime | +| `training.trace_output_dir` | str / null | null | Trace output directory | +| `training.trace_thread_time` | bool | false | Include thread time in traces | +| `training.trace_cuda_events` | bool | true | Record CUDA events in traces | +| `training.nvtx_profile_ranges` | bool | false | Emit NVTX ranges for external profilers | +| `training.torch_threads.enabled` | bool | true | Pin torch CPU thread counts per process role | +| `training.torch_threads.learner_num_threads` | int / str | auto | Learner intra-op threads; `auto` resolves from the host CPU count | +| `training.torch_threads.collector_num_threads` | int / str | auto | Collector intra-op threads | +| `training.dp_collector_cpu_ids` | list / null | null | Explicit CPU ids per rank; null auto-partitions `cpu_count // world_size` | +| `training.logger` | str | tensorboard | Scalar logging backend (`tensorboard` or `wandb`) | + +### Reward term weights + +Reward terms live in the task owner YAML under `reward:` — one block per term with +`func`, `weight`, and `params` (example: +`src/unilab/conf/sac/task/g1_walk_flat/base.yaml`). Retune a term by editing that +owner YAML or by overriding its weight on the command line: + +```bash +uv run train --algo sac --task g1_walk_flat --sim mujoco \ + reward.tracking_lin_vel.weight=2.5 +``` + +Term names and default weights are task-specific; check the `reward:` block of the owner +YAML for the task you train (`src/unilab/conf/sac/task//base.yaml`). diff --git a/docs/sphinx/source/en/2-user_guide/2-algorithms/4-flash_sac.md b/docs/sphinx/source/en/2-user_guide/2-algorithms/4-flash_sac.md index 852c25f99..185683de3 100644 --- a/docs/sphinx/source/en/2-user_guide/2-algorithms/4-flash_sac.md +++ b/docs/sphinx/source/en/2-user_guide/2-algorithms/4-flash_sac.md @@ -73,3 +73,115 @@ closed with an error that states the minimum supported format. The only recovery path is to retrain the policy with a current release. The supported baseline starts with the first release after 1.3.5 that carries this change; it is not backported to any 1.3.5 patch release. + +## Configuration reference + +Defaults come from the typed configs in `src/unilab/structured_configs.py` +(`FlashSACConfig`); `src/unilab/conf/flashsac/config.yaml` carries the same values unless +noted. Every field can be overridden on the command line: + +```bash +uv run train --algo flashsac --task g1_walk_flat --sim mujoco \ + algo.gamma=0.99 \ + algo.algo_params.temp_initial_value=0.1 +``` + +### Training efficiency + +| Override key | Type | Default | Practical effect | +| --- | --- | --- | --- | +| `algo.num_envs` | int | 1024 | Parallel collector environments | +| `algo.batch_size` | int | 2048 | Learner batch per update | +| `algo.replay_buffer_n` | int | 512 | Replay capacity in env-vector sweeps (rows = n × num_envs) | +| `algo.updates_per_step` | int | 2 | Learner updates per env step | +| `algo.learning_starts` | int | 98 | Env-vector sweeps to collect before learner updates start | +| `algo.policy_frequency` | int | 2 | One actor/temperature update every N learner updates | +| `algo.max_iterations` | int | 5000 | Number of training iterations | +| `algo.save_interval` | int | 1000 | Checkpoint save interval in iterations | +| `algo.algo_params.use_compile` | bool | true | CUDA Inductor compilation; `false` selects the eager path and also disables the whole-cycle graph | +| `algo.algo_params.compile_full_objectives` | bool | true | Compile complete actor/critic objectives instead of loss-tensor regions | +| `algo.algo_params.use_whole_cycle_cuda_graph` | bool | true | Learner-owned whole-cycle CUDA Graph capture | +| `algo.algo_params.dp_whole_cycle_cuda_graph` | bool | true | Whole-cycle capture on the data-parallel path; `false` disables DP capture only (field set in `config.yaml`) | +| `algo.seed` | int | 1 | Random seed | +| `training.use_amp` | bool | true | bf16 autocast on the learner update path | +| `training.log_interval` | int | 1 | TensorBoard/W&B scalar reporting interval in iterations | + +### Policy performance + +| Override key | Type | Default | Practical effect | +| --- | --- | --- | --- | +| `algo.gamma` | float | 0.97 | Discount factor | +| `algo.tau` | float | 0.01 | Target-critic EMA coefficient | +| `algo.actor_lr` | float | 3e-4 | Fallback peak LR (used when `algo.algo_params.learning_rate_peak` ≤ 0) | +| `algo.critic_lr` | float | 3e-4 | Declared LR field; the effective schedule comes from `learning_rate_*` | +| `algo.actor_hidden_dim` | int | 128 | Actor network width | +| `algo.critic_hidden_dim` | int | 256 | Critic network width | +| `algo.num_atoms` | int | 101 | Atoms of the distributional critic | +| `algo.obs_normalization` | bool | false | Normalize observations | +| `algo.use_layer_norm` | bool | false | LayerNorm in the networks | +| `algo.policy_before_critic` | bool | false | Update actor (+temperature) before the critic in each update (field set in `config.yaml`) | +| `algo.algo_params.normalize_reward` | bool | true | Normalize rewards with a running return estimate | +| `algo.algo_params.normalized_g_max` | float | 5.0 | Clamp of the normalized return estimate | +| `algo.algo_params.actor_num_blocks` | int | 2 | Blocks in the block-based actor | +| `algo.algo_params.critic_num_blocks` | int | 2 | Blocks in the block-based critic | +| `algo.algo_params.actor_bc_alpha` | float | 0.0 | Behavior-cloning auxiliary loss weight; 0 disables it | +| `algo.algo_params.actor_noise_zeta_mu` | float | 2.0 | Actor noise schedule mu | +| `algo.algo_params.actor_noise_zeta_max` | int | 16 | Actor noise schedule cap | +| `algo.algo_params.critic_min_v` | float | -5.0 | Lower bound of the distributional critic support | +| `algo.algo_params.critic_max_v` | float | 5.0 | Upper bound of the distributional critic support | +| `algo.algo_params.temp_initial_value` | float | 0.01 | Initial entropy temperature | +| `algo.algo_params.temp_target_sigma` | float | 0.15 | Target sigma used by the temperature adaptation | +| `algo.algo_params.temp_target_entropy` | float / null | null | Explicit target entropy; null uses the built-in target | +| `algo.algo_params.learning_rate_init` | float | 3e-4 | Schedule start LR for actor/critic/temperature | +| `algo.algo_params.learning_rate_peak` | float | 3e-4 | Schedule peak LR | +| `algo.algo_params.learning_rate_end` | float | 1.5e-4 | Schedule end LR after decay | +| `algo.algo_params.learning_rate_warmup_steps` | int | 0 | Warmup steps from init to peak | +| `algo.algo_params.learning_rate_decay_steps` | int | 500000 | Decay steps from peak to end | +| `algo.algo_params.n_step` | int | 1 | N-step return length; only 1 is supported | +| `algo.algo_params.amp_dtype` | str | auto | AMP dtype under `training.use_amp` | + +### Training runtime (`training:` section) + +| Override key | Type | Default | Practical effect | +| --- | --- | --- | --- | +| `training.no_play` | bool | false | Skip policy playback after training | +| `training.play_only` | bool | false | Skip training and only play a checkpoint | +| `training.export_onnx` | bool | true | Export `policy.onnx` during playback | +| `training.cuda_process_sharing` | str / null | null | null = independent CUDA contexts; `mps` requests CUDA MPS (user-managed daemon via `uni-cumps`) | +| `training.dp_gradient_wire_dtype` | str | float32 | Gradient wire dtype for multi-GPU averaging; `bfloat16` compresses | +| `training.nan_guard.enabled` | bool | true | Detect non-finite tensors and dump diagnostics | +| `training.nan_guard.check_interval` | int | 32 | Env steps between NaN checks | +| `training.defer_finite_validation` | bool | true | Batch finite checks into a periodic cycle | +| `training.finite_validation_interval` | int | 32 | Steps per batched finite-check cycle | +| `training.collector_tensor_device` | str | auto | Device of collector-side tensors; `auto` follows the learner | +| `training.env_steps_per_sync` | int | 1 | Env steps per collection sync | +| `training.inference_slot_capacity` | int | 1 | In-flight learner-inference requests; 1 keeps inference serialized | +| `training.collector_metrics_interval` | int | 32 | Collector steps between metric flushes | +| `training.replay_ingress_depth` | int | 2 | Replay-ingress chunks allowed in flight | +| `training.replay_ingress_slot_rows` | int / null | null | Rows per ingress chunk; null = one chunk per collector vector | +| `training.replay_prefetch_mode` | str | one_tick | Replay prefetch strategy | +| `training.trace_enabled` | bool | false | Chrome-trace profiling of the off-policy runtime | +| `training.trace_output_dir` | str / null | null | Trace output directory | +| `training.trace_thread_time` | bool | false | Include thread time in traces | +| `training.trace_cuda_events` | bool | true | Record CUDA events in traces | +| `training.nvtx_profile_ranges` | bool | false | Emit NVTX ranges for external profilers | +| `training.torch_threads.enabled` | bool | true | Pin torch CPU thread counts per process role | +| `training.torch_threads.learner_num_threads` | int / str | auto | Learner intra-op threads; `auto` resolves from the host CPU count | +| `training.torch_threads.collector_num_threads` | int / str | auto | Collector intra-op threads | +| `training.dp_collector_cpu_ids` | list / null | null | Explicit CPU ids per rank; null auto-partitions `cpu_count // world_size` | +| `training.logger` | str | tensorboard | Scalar logging backend (`tensorboard` or `wandb`) | + +### Reward term weights + +Reward terms live in the task owner YAML under `reward:` — one block per term with +`func`, `weight`, and `params` (example: +`src/unilab/conf/flashsac/task/g1_walk_flat/base.yaml`). Retune a term by editing that +owner YAML or by overriding its weight on the command line: + +```bash +uv run train --algo flashsac --task g1_walk_flat --sim mujoco \ + reward.tracking_lin_vel.weight=2.5 +``` + +Term names and default weights are task-specific; check the `reward:` block of the owner +YAML for the task you train (`src/unilab/conf/flashsac/task//base.yaml`). diff --git a/docs/sphinx/source/en/2-user_guide/5-domain_randomization/0-index.md b/docs/sphinx/source/en/2-user_guide/5-domain_randomization/0-index.md index 4b9a7d511..1a573b4ca 100644 --- a/docs/sphinx/source/en/2-user_guide/5-domain_randomization/0-index.md +++ b/docs/sphinx/source/en/2-user_guide/5-domain_randomization/0-index.md @@ -114,6 +114,7 @@ The ownership boundary and MJWarp/CPU executor split are recorded in :hidden: 1-configuration +2-field_reference ``` ## Related Tasks diff --git a/docs/sphinx/source/en/2-user_guide/5-domain_randomization/2-field_reference.md b/docs/sphinx/source/en/2-user_guide/5-domain_randomization/2-field_reference.md new file mode 100644 index 000000000..d14d7881f --- /dev/null +++ b/docs/sphinx/source/en/2-user_guide/5-domain_randomization/2-field_reference.md @@ -0,0 +1,350 @@ +# Field Reference + +This page is the field-level configuration reference for the domain +randomization (DR) event terms in `src/unilab/envs/mdp/events.py`. It +complements {doc}`0-index`, which audits which registered tasks declare which +terms, and {doc}`1-configuration`, which explains how owner YAMLs are selected. +Every signature, default, and value hint below is taken from the source or +from owner YAMLs under `src/unilab/conf/`; nothing is inferred from design +intent. + +DR terms run under three lifecycle classes (see the lifecycle overview in +{doc}`0-index`): + +- **construction**: fixed model/tool identity, declared on + `env.fixed_model_variants`; no event term on this page belongs to it. +- **reset**: event terms with `mode: reset` sample on every episode reset and + commit through the backend-owned packed reset transaction. `mode: startup` + is an opt-in reset-lifecycle variant that samples once at initialization and + replays the frozen per-environment values afterwards. +- **interval**: event terms with `mode: interval` perturb between steps on a + sampled timer; `mode: step` fires every step and is used by terms that manage + their own force lifetime. + +## Backend Applicability + +The exact field-by-backend matrix is owned by the external `unisim-core` +package (`DomainRandomizationCapabilities.supported_reset_terms` / +`supported_interval_terms`) and cannot be confirmed field by field from this +repository, so this page deliberately does not print one. Two rules always +hold: + +1. A requested term that the selected backend does not advertise fails closed + at construction; there is no filtering or silent fallback. +2. Owner YAMLs are per-backend files + (`src/unilab/conf//task//.yaml`). A term that one + backend does not support is set to `null` in that backend's owner while the + shared base owner keeps it for the others. + +In-repo examples of this pattern: `src/unilab/conf/ppo/task/g1_walk_flat/motrix.yaml` +and `.../mjwarp.yaml` set `pd_gains: null` while the MuJoCo owner keeps kp/kd +randomization; `src/unilab/conf/sac/task/g1_motion_tracking/genesis.yaml` sets +`foot_friction: null` with a comment recording that Genesis has no +per-environment friction API. The most complete worked owner example (mass, +COM, gravity, push) is the `events:` block of +`src/unilab/conf/flashsac/task/go2_joystick_flat/mujoco.yaml`. + +## Common Parameter Family + +These parameters recur across terms. Per-term tables below only list what the +term's constructor actually accepts; unknown params are rejected at +construction. + +- `asset_cfg` (`SceneEntityCfg`): selects the entity and its components. + `name` is required (the scene entity name, e.g. `robot`); components are + chosen with `body_names`/`body_ids`, `joint_names`/`joint_ids`, + `geom_names`/`geom_ids`, or `actuator_names`/`actuator_ids`. The `*_names` + fields accept a regex string; the `*_ids` fields default to "all". Most + terms default to `SceneEntityCfg("robot")` (the whole robot). +- `distribution`: `"uniform"` (default), `"log_uniform"` (bounds must be + positive), or `"gaussian"` (the range is read as `(mean, std)` with + non-negative std). `pd_gains` accepts only `uniform`/`log_uniform`. +- `operation`: how the sample combines with the backend-declared default: + `"add"` → `default + sample`, `"scale"` → `default * sample`, `"abs"` → + the sample replaces the default. The default differs per term; see each + table. +- Range shapes: a scalar pair `[lo, hi]` (e.g. `kp_range`, + `mass_distribution_params`); an axis dict such as `{x: [lo, hi], ...}` + (`com_range`, `pose_range`, `velocity_range`; omitted keys default to + `(0, 0)`); a width-3 pair `[[lo_x, lo_y, lo_z], [hi_x, hi_y, hi_z]]` + (`gravity_distribution_params`); or, for the model-field terms, `ranges` + as a flat pair applied to all selected axes, a dict keyed by axis index, + or a dict of regex → pair matched with `fullmatch` against the selected + geom/joint names (patterns must each match at least one name and must not + overlap). + +The surrounding term block (`func`, `mode`, `interval_range_s`, +`is_global_time`, `min_step_count_between_reset`, `params`) is an +`EventTermCfg`; `interval_range_s` and `is_global_time` live on the term, not +inside `params`. `interval_range_s` is required for `mode: interval`; +`is_global_time: true` makes all environments share one timer. + +## Reset-Lifecycle Terms + +### `pd_gains` + +Randomizes actuator stiffness `kp` and damping `kd` together +(`unilab.envs.mdp.pd_gains`). Only `mode: reset` is supported; startup and +interval are rejected at construction. + +| Field | Type | Default / required | When it applies | Semantics and value hints | +| --- | --- | --- | --- | --- | +| `kp_range` | `[lo, hi]` | required | sampled each reset | Per-actuator stiffness. Finite, `lo <= hi`; must be positive with `log_uniform`. With `operation: scale` the pair is a dimensionless multiplier of the default gains (G1 owners use `[0.9, 1.1]`); with `operation: abs` it is the absolute gain (the Go2 owner uses `kp_range: [31.5, 38.5]`). | +| `kd_range` | `[lo, hi]` | required | sampled each reset | Per-actuator damping, same validation as `kp_range`. Go2 owner: `kd_range: [0.45, 0.55]` with `abs`. | +| `asset_cfg` | `SceneEntityCfg` | `SceneEntityCfg("robot")` | bound at construction | `actuator_names`/`actuator_ids` select the actuators. | +| `distribution` | `uniform` \| `log_uniform` | `uniform` | sampled each reset | `gaussian` is rejected by this term. | +| `operation` | `scale` \| `abs` | `scale` | applied each reset | `add` is rejected by this term. | + +### `randomize_rigid_body_mass` + +Randomizes body mass (`unilab.envs.mdp.randomize_rigid_body_mass`). Supports +`mode: reset` and `mode: startup`; startup samples one full-width table at +initialization and replays it. + +| Field | Type | Default / required | When it applies | Semantics and value hints | +| --- | --- | --- | --- | --- | +| `asset_cfg` | `SceneEntityCfg` | required | bound at construction | `body_names`/`body_ids` select the bodies. | +| `mass_distribution_params` | `[lo, hi]` | required | sampled each reset (or once at startup) | In the model's body-mass units: kilograms for the MuJoCo MJCF robots in this repo. The FlashSAC Go2 MuJoCo owner uses `[-1.5, 1.5]` with `operation: add` on the `base` body; the SAC G1 Genesis owner uses `[0.9, 1.1]` with `operation: scale` on `pelvis`/`torso_link` at startup. | +| `operation` | `add` \| `scale` \| `abs` | required | applied each reset | No default; must be declared. | +| `distribution` | `uniform` \| `log_uniform` \| `gaussian` | `uniform` | sampled each reset | `log_uniform` requires positive bounds. | +| `recompute_inertia` | bool | `true` | construction | `true` raises `NotImplementedError` (the reset payload has no inertia-recomputation contract); always set it explicitly to `false`. | +| `min_mass` | float | `1e-6` | applied each reset | Lower clamp for the randomized mass; must be finite and `>= 1e-6`. | + +### `randomize_body_mass_inertia` + +Scales body mass and principal inertia by one shared per-body factor +(`unilab.envs.mdp.randomize_body_mass_inertia`). Only `mode: reset` is +supported; the factor is sampled once at the first reset, cached per +environment, and reapplied on later resets, so it behaves like a startup term. +The center of mass and the principal frame are left untouched. + +| Field | Type | Default / required | When it applies | Semantics and value hints | +| --- | --- | --- | --- | --- | +| `asset_cfg` | `SceneEntityCfg` | required | bound at construction | `body_names`/`body_ids` select the bodies; mass and inertia bindings must cover the same bodies. | +| `scale_range` | `[lo, hi]` | required | sampled once at the first reset | Log-uniform factor `s = e^{2a}` with `a ~ U(ln(lo)/2, ln(hi)/2)`; bounds must be positive. Both mass and principal inertia are multiplied by `s`. | + +### `randomize_rigid_body_com` + +Additive center-of-mass randomization +(`unilab.envs.mdp.randomize_rigid_body_com`). Supports `mode: reset` and +`mode: startup`. The live `com_range` is re-read on every apply, so a +curriculum can widen the range between resets. + +| Field | Type | Default / required | When it applies | Semantics and value hints | +| --- | --- | --- | --- | --- | +| `com_range` | `{x, y, z: [lo, hi]}` | required | sampled each reset (or once at startup) | Offset added to the default body COM position, in meters in the body frame. Omitted axes default to `(0, 0)`. The SAC G1 MuJoCo owner uses `x: [-0.025, 0.025]`, `y/z: [-0.05, 0.05]` on `torso_link`; the FlashSAC Go2 owner uses `x: [-0.05, 0.05]` on `base`. | +| `asset_cfg` | `SceneEntityCfg` | required | bound at construction | `body_names`/`body_ids` select the bodies. | + +### `randomize_physics_scene_gravity` + +Randomizes the full gravity vector per environment +(`unilab.envs.mdp.randomize_physics_scene_gravity`). Only `mode: reset` is +supported. Prefer a small tilt range: large horizontal gravity can make an +early task unlearnable. + +| Field | Type | Default / required | When it applies | Semantics and value hints | +| --- | --- | --- | --- | --- | +| `gravity_distribution_params` | `[[lo_x, lo_y, lo_z], [hi_x, hi_y, hi_z]]` | required | sampled each reset | Per-axis bounds for `(gx, gy, gz)` in m/s^2. With `operation: abs` the sampled vector replaces the default; the FlashSAC Go2 MuJoCo owner declares `[[0.0, 0.0, -9.81], [0.0, 0.0, -9.81]]` with `abs` (a zero-width placeholder showing the shape). | +| `operation` | `add` \| `scale` \| `abs` | required | applied each reset | No default; must be declared. | +| `distribution` | `uniform` \| `log_uniform` \| `gaussian` | `uniform` | sampled each reset | | + +### `geom_friction` + +Randomizes the 3-component geom friction triple (tangential, torsional, +rolling), matching the pinned mjlab semantics +(`unilab.envs.mdp.geom_friction`). Only `mode: reset` is supported. Final +values must stay non-negative and finite. + +| Field | Type | Default / required | When it applies | Semantics and value hints | +| --- | --- | --- | --- | --- | +| `ranges` | pair / axis dict / regex dict | required | sampled each reset | Flat `[lo, hi]` applies to all selected axes; `{0: [lo, hi]}` targets one axis; `{".*_foot.*": [lo, hi]}` applies per-name ranges via regex `fullmatch`. The SAC G1 MuJoCo owner uses `[0.3, 1.2]` with `operation: abs` on the foot collision geoms. | +| `asset_cfg` | `SceneEntityCfg` | `SceneEntityCfg("robot")` | bound at construction | `geom_names`/`geom_ids` select the geoms. | +| `axes` | list of int | `(0,)` | sampled each reset | Valid axes are `0, 1, 2`; the default touches only the tangential component. | +| `distribution` | `uniform` \| `log_uniform` \| `gaussian` | `uniform` | sampled each reset | | +| `operation` | `add` \| `scale` \| `abs` | `abs` | applied each reset | | +| `shared_random` | bool | `false` | sampled each reset | `true` gives geoms that share identical bounds one shared sample per environment (used by the SAC G1 MuJoCo owner). | + +### `dof_armature` + +Randomizes joint armature — the added rotor inertia that models geared +transmissions (`unilab.envs.mdp.dof_armature`). Same parameter family as +`geom_friction`, with a scalar field: only axis `0` is valid. Only +`mode: reset` is supported; final values must stay non-negative and finite. + +| Field | Type | Default / required | When it applies | Semantics and value hints | +| --- | --- | --- | --- | --- | +| `ranges` | pair / `{0: [lo, hi]}` / regex dict | required | sampled each reset | Regex keys are matched with `fullmatch` against the selected joint names. | +| `asset_cfg` | `SceneEntityCfg` | `SceneEntityCfg("robot")` | bound at construction | `joint_names`/`joint_ids` select the joints. | +| `axes` | list of int | `(0,)` | sampled each reset | Only `0` is valid for this scalar field. | +| `distribution` | `uniform` \| `log_uniform` \| `gaussian` | `uniform` | sampled each reset | | +| `operation` | `add` \| `scale` \| `abs` | `abs` | applied each reset | | +| `shared_random` | bool | `false` | sampled each reset | | + +### `randomize_encoder_bias` + +Adds a constant per-joint bias to the encoder position readings +(`unilab.envs.mdp.randomize_encoder_bias`). Only `mode: reset` is supported. +This writes to the entity data surface (an observation bias), not to a +physics model field, and requires the Torch control plane. + +| Field | Type | Default / required | When it applies | Semantics and value hints | +| --- | --- | --- | --- | --- | +| `bias_range` | `[lo, hi]` | required | sampled each reset | Uniform bias in radians (joint position units). The SAC G1 MuJoCo owner uses `[-0.01, 0.01]` on all joints. | +| `asset_cfg` | `SceneEntityCfg` | required | bound at construction | `joint_names`/`joint_ids` select the joints; at least one joint must be selected. | + +## Reset-State Terms + +### `reset_root_state_uniform` + +Resets a floating root to the backend default state plus uniformly sampled +SE(3) offsets (`unilab.envs.mdp.reset_root_state_uniform`). Fixed-base or +mocap roots fail closed. + +| Field | Type | Default / required | When it applies | Semantics and value hints | +| --- | --- | --- | --- | --- | +| `pose_range` | `{x, y, z, roll, pitch, yaw: [lo, hi]}` | required | sampled each reset | Position offsets in meters are added on top of the default root position and the environment origin; `roll`/`pitch`/`yaw` offsets in radians compose with the default orientation. Omitted keys default to `(0, 0)`. The Go2/G1 base owners use `x/y: [-0.5, 0.5]` and `yaw: [-pi, pi]`. | +| `velocity_range` | same keys | optional (no offset) | sampled each reset | Linear (m/s) and angular (rad/s) velocity offsets added to the default root velocity. The Go2/G1 base owners use `[-0.5, 0.5]` on all six axes. | +| `asset_cfg` | `SceneEntityCfg` | `SceneEntityCfg("robot")` | bound at construction | | + +### `reset_root_state_uniform_fused` + +Same parameters and semantics as `reset_root_state_uniform` +(`unilab.envs.mdp.reset_root_state_uniform_fused`), but samples all +environments in one compiled Torch graph and requires the Manager RNG on a +CUDA tensor runtime. Backend owners swap the `func` of the existing +`reset_root_state_uniform` term, e.g. `src/unilab/conf/ppo/task/g1_walk_flat/mjwarp.yaml`. + +### `reset_scene_to_default` + +`unilab.envs.mdp.reset_scene_to_default`. No params. Resets all materialized +scene entities to the backend default `qpos`/`qvel`; base owners declare it as +the first reset term. + +### `skip_scene_reset` + +`unilab.envs.mdp.skip_scene_reset`. No params. No-op reset hook used as the +canonical MJWarp reset scene term: backend owners swap the `func` of the +inherited `reset_scene_to_default` term to it (e.g. +`src/unilab/conf/ppo/task/g1_walk_flat/mjwarp.yaml`, +`src/unilab/conf/ppo/task/go2_joystick_flat/superdex.yaml`). + +## Interval And Step Terms + +### `push_by_setting_velocity` + +Applies a world-frame velocity kick to the root +(`unilab.envs.mdp.push_by_setting_velocity`). Only `mode: interval` is +supported; `interval_range_s` and `is_global_time` are declared on the term, +not in `params`. The live `velocity_range` is re-read on every apply so +curriculum range stages take effect. + +| Field | Type | Default / required | When it applies | Semantics and value hints | +| --- | --- | --- | --- | --- | +| `velocity_range` | `{x, y, z, roll, pitch, yaw: [lo, hi]}` | required | sampled at each interval trigger | Linear kicks (`x`/`y`/`z`, m/s) and angular kicks (`roll`/`pitch`/`yaw`, rad/s) are world-frame deltas; omitted keys default to `(0, 0)`. Angular axes require the backend's interval angular-velocity capability, bound at construction from the initial ranges. The FlashSAC Go2 MuJoCo owner uses `x/y: [-1.0, 1.0]`, `z: [-0.5, 0.5]`, linear only, every 15 s; the SAC G1 owners add angular ranges (`roll/pitch: [-0.52, 0.52]`, `yaw: [-0.78, 0.78]`) every 1-3 s. | +| `asset_cfg` | `SceneEntityCfg` | `SceneEntityCfg("robot")` | bound at construction | | + +### `apply_body_impulse` + +Applies transient random body wrenches with a per-environment +cooldown → trigger → sustain → expire lifecycle +(`unilab.envs.mdp.apply_body_impulse`). Only `mode: step` is supported; the +term runs every step and manages its own timers. Torque channels (a non-zero +`torque_range` or a `body_point_offset`) require the backend's interval +body-torque capability. + +| Field | Type | Default / required | When it applies | Semantics and value hints | +| --- | --- | --- | --- | --- | +| `force_range` | `[lo, hi]` | required | sampled at trigger | Uniform magnitude per force component, applied to the 3 world-frame axes of every selected body; model units (N). | +| `torque_range` | `[lo, hi]` | required | sampled at trigger | Same shape convention for the world-frame torque (N*m). Use `[0.0, 0.0]` for force-only impulses. | +| `duration_s` | `[min, max]` | required | sampled at trigger | How long the wrench is sustained, in seconds; must satisfy `0 <= min <= max`. The wrench is re-staged every step while active. | +| `cooldown_s` | `[min, max]` | required | sampled after each trigger/reset | Idle time before the next impulse, in seconds; `0 <= min <= max`. The initial cooldown is pre-sampled so no impulse fires at `t=0`. | +| `asset_cfg` | `SceneEntityCfg` | `SceneEntityCfg("robot")` | bound at construction | `body_names`/`body_ids` select the bodies. | +| `body_point_offset` | `(x, y, z)` | optional (none) | applied at trigger | Application point in the body link frame; adds the induced `offset x force` torque. Requires the packed body-state read plan. | + +## Minimal Configuration Example + +DR terms are declared in the `events:` block of the selected backend owner, +`src/unilab/conf//task//.yaml` — for example +`src/unilab/conf/flashsac/task/go2_joystick_flat/mujoco.yaml`, whose block is +adapted here: + +```yaml +env: + events: + randomize_rigid_body_mass: + func: unilab.envs.mdp.randomize_rigid_body_mass + mode: reset + params: + asset_cfg: + _target_: SceneEntityCfg + name: robot + body_names: base + mass_distribution_params: [-1.5, 1.5] + operation: add + recompute_inertia: false + randomize_rigid_body_com: + func: unilab.envs.mdp.randomize_rigid_body_com + mode: reset + params: + asset_cfg: + _target_: SceneEntityCfg + name: robot + body_names: base + com_range: + x: [-0.05, 0.05] + push_by_setting_velocity: + func: unilab.envs.mdp.push_by_setting_velocity + mode: interval + interval_range_s: [15.0, 15.0] + is_global_time: true + params: + velocity_range: + x: [-1.0, 1.0] + y: [-1.0, 1.0] + z: [-0.5, 0.5] + roll: [0.0, 0.0] + pitch: [0.0, 0.0] + yaw: [0.0, 0.0] +``` + +Select the backend with the CLI — `uv run train --algo --task +--sim ` — and keep terms a backend does not advertise out of its +owner (or set them to `null`), because unsupported terms fail closed at +construction. + +## FAQ + +**How do I add friction randomization for Go2?** +Declare a `geom_friction` term with `mode: reset` in the backend owner that +supports it, selecting the foot geoms with a `geom_names` regex. The SAC G1 +MuJoCo owner is the in-repo template: +`src/unilab/conf/sac/task/g1_motion_tracking/mujoco.yaml` selects +`geom_names: "^(left|right)_foot[1-7]_collision$"` and uses +`ranges: [0.3, 1.2]`, `operation: abs`, `shared_random: true`. Check the Go2 +model's geom names for the equivalent pattern. Geom friction is a +backend-advertised capability: the Genesis owner for the same task sets +`foot_friction: null` because Genesis has no per-environment friction API, so +do not enable the term on backends that do not advertise it. + +**What unit are the mass parameters in?** +The model's body-mass units — kilograms for the MuJoCo MJCF robots in this +repo. The sampled value is combined with the backend-declared default mass by +`operation`: `add` adds kilograms (the FlashSAC Go2 owner adds ±1.5 kg to the +`base` body), `scale` multiplies by a dimensionless factor (the SAC G1 +Genesis owner scales `pelvis`/`torso_link` by `[0.9, 1.1]`), and `abs` sets +absolute kilograms. The result is clamped below at `min_mass` (default +`1e-6`). + +**How do I keep one task runnable on Motrix and MuJoCo when Motrix has no gravity reset?** +Per {doc}`1-configuration`, gravity reset randomization is MuJoCo-only today; +`MotrixBackend` advertises `base_mass_delta`, `base_com_offset`, `kp`, `kd`, +and interval push only (see {doc}`0-index`). Keep shared terms in the task +`base.yaml`, declare `randomize_physics_scene_gravity` only in `mujoco.yaml`, +and if the shared base declares it, set `randomize_physics_scene_gravity: null` +in `motrix.yaml` — the same pattern as `pd_gains: null` in +`src/unilab/conf/ppo/task/g1_walk_flat/motrix.yaml` and `.../mjwarp.yaml`. +Leaving the term enabled for Motrix would fail closed at construction instead +of being skipped. + +For the payload-level contract behind these terms, see +{doc}`Domain Randomization Contract `. diff --git a/docs/sphinx/source/en/5-reference/0-index.md b/docs/sphinx/source/en/5-reference/0-index.md index 7f770ba68..525dddced 100644 --- a/docs/sphinx/source/en/5-reference/0-index.md +++ b/docs/sphinx/source/en/5-reference/0-index.md @@ -35,6 +35,12 @@ Generated backend × task evidence and selection notes. Shared architecture decisions and their status. ::: +:::{grid-item-card} Manager terms +:link: 6-manager_terms/0-index +:link-type: doc +Built-in reward, observation, and termination terms with parameters. +::: + :::: ```{toctree} @@ -46,4 +52,5 @@ Shared architecture decisions and their status. 3-changelog 4-adr 5-support_matrix +6-manager_terms/0-index ``` diff --git a/docs/sphinx/source/en/5-reference/6-manager_terms/0-index.md b/docs/sphinx/source/en/5-reference/6-manager_terms/0-index.md new file mode 100644 index 000000000..7a2691c61 --- /dev/null +++ b/docs/sphinx/source/en/5-reference/6-manager_terms/0-index.md @@ -0,0 +1,128 @@ +# Manager Term Reference + +Lookup tables for the built-in Manager-Based reward, observation, and +termination terms shipped with UniLab, organized by owner layer: +framework-level terms in `unilab.envs.mdp` and task-owned terms under +`src/unilab/tasks/`. Every name, parameter, and default in these tables is +compiled from the linked source modules; the authoritative signatures are the +API reference. + +:::{only} api_ref +```{raw} html + +``` +::: + +:::{only} prose_only +The API reference is skipped in prose-only preview builds. Authoritative +signatures live in the source tree: +[unilab/envs/mdp](https://github.com/unilabsim/UniLab/tree/main/src/unilab/envs/mdp), +[unilab/tasks](https://github.com/unilabsim/UniLab/tree/main/src/unilab/tasks). +::: + +## How terms are referenced + +Terms are declared in the task owner YAML (for example +`src/unilab/conf/ppo/task/go2_joystick_flat/base.yaml`) as `func + params` +entries. `func` is an importable `module.path.name`; all framework-level terms +are re-exported at `unilab.envs.mdp.`, which is the path the owner YAMLs +use. Class terms are instantiated once with `(cfg, env)` at manager +construction and then called per step; plain functions are called per step. In +both cases the manager invokes the term as `func(env, **params)` +(`src/unilab/managers/manager_base.py`, `src/unilab/managers/reward_manager.py`), +so the `params:` mapping must match the term's accepted parameters. + +- Reward terms live under the top-level `reward:` mapping with `func`, + `weight`, and optional `params`: + ```yaml + reward: + action_rate: + func: unilab.envs.mdp.action_rate_l2 + weight: -0.005 + ``` +- Observation terms live under `env.observations..terms.` with + `func` and optional `params`: + ```yaml + env: + observations: + policy: + terms: + joint_pos: + func: unilab.envs.mdp.joint_pos_rel + ``` +- Termination terms live under `env.terminations.` with `func` and + optional `params`; adding `time_out: true` marks the term as an episodic + timeout (truncation) instead of a failure (`TerminationTermCfg.time_out` in + `src/unilab/managers/termination_manager.py`): + ```yaml + env: + terminations: + time_out: + func: unilab.envs.mdp.time_out + time_out: true + ``` + +Terms that need a backend capability — named sensors, ray casters, reset +fields — bind that capability at construction and raise when the backend +cannot provide it; they fail closed instead of silently degrading (see the +`bind_sensor_data` binding in `src/unilab/envs/mdp/observations.py` and the +`_ResetRandomizationTerm` contract in `src/unilab/envs/mdp/events.py`). + +## Scope + +These tables cover reward, observation, and termination terms. Task packages +also own term kinds configured in other YAML sections, which are not listed in +the tables: + +- Command and action terms, for example + `unilab.envs.mdp.commands.UniformVelocityCommandCfg`, + `unilab.tasks.locomotion.g1.manager_terms.G1VelocityCommandCfg` (G1 velocity + command with a 0.2 planar-norm dead zone), and + `unilab.tasks.motion_tracking.common.manager_terms.MotionCommandCfg` / + `MotionJointPositionActionCfg` (motion reference command and its joint + position action). +- Curriculum terms, for example + `unilab.tasks.locomotion.g1.manager_terms.G1PenaltyCurriculum` (declared under + `curriculum:`; scales negative-weight reward terms between + `initial_scale`/`min_scale` and `max_scale` as the tracked average episode + length crosses `level_down_threshold`/`level_up_threshold`, by `degree` per + update). +- Event (domain-randomization and reset) terms in + `unilab.envs.mdp.events`, declared under `env.events`. +- Reset owners, for example + `unilab.tasks.locomotion.g1.manager_terms.G1WalkResetOwner`. + +::::{grid} 1 1 2 2 +:gutter: 3 + +:::{grid-item-card} Reward terms +:link: 1-rewards +:link-type: doc +Framework, locomotion, G1, and motion-tracking reward terms with parameters. +::: + +:::{grid-item-card} Observation terms +:link: 2-observations +:link-type: doc +State, sensor, command, gait-phase, and ray-scan observation terms. +::: + +:::{grid-item-card} Termination terms +:link: 3-terminations +:link-type: doc +Timeout and failure conditions, including fused termination packs. +::: + +:::: + +```{toctree} +:hidden: + +1-rewards +2-observations +3-terminations +``` diff --git a/docs/sphinx/source/en/5-reference/6-manager_terms/1-rewards.md b/docs/sphinx/source/en/5-reference/6-manager_terms/1-rewards.md new file mode 100644 index 000000000..3b81ebee8 --- /dev/null +++ b/docs/sphinx/source/en/5-reference/6-manager_terms/1-rewards.md @@ -0,0 +1,439 @@ +# Reward Terms + +Built-in reward terms, compiled from source. All terms return one float32 +value per environment; the reward manager multiplies by the YAML `weight`. +Framework-level terms are re-exported at `unilab.envs.mdp.`. + +:::{only} api_ref +```{raw} html + +``` +::: + +:::{only} prose_only +Authoritative signatures live in the source tree: +[unilab/envs/mdp](https://github.com/unilabsim/UniLab/tree/main/src/unilab/envs/mdp), +[unilab/tasks](https://github.com/unilabsim/UniLab/tree/main/src/unilab/tasks). +::: + +## Framework (`unilab.envs.mdp.rewards`) + +Source: `src/unilab/envs/mdp/rewards.py`. + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - Function / class + - Parameters + - Semantics +* - `is_alive` + - `unilab.envs.mdp.rewards.is_alive` (function) + - — + - 1.0 while the environment has not reached a non-timeout termination. +* - `is_terminated` + - `unilab.envs.mdp.rewards.is_terminated` (function) + - — + - 1.0 on non-timeout terminations; pair with a negative weight to penalize + failure. +* - `root_height` + - `unilab.envs.mdp.rewards.root_height` (function) + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - World-frame root height (z) as a scalar reward or metric. +* - `joint_vel_l2` + - `unilab.envs.mdp.rewards.joint_vel_l2` (function) + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - Sum of squared velocities over the selected joints. +* - `action_rate_l2` + - `unilab.envs.mdp.rewards.action_rate_l2` (function) + - — + - Sum of squares of the first difference of raw policy actions. +* - `action_acc_l2` + - `unilab.envs.mdp.rewards.action_acc_l2` (function) + - — + - Sum of squares of the second difference of raw policy actions. +* - `flat_orientation_l2` + - `unilab.envs.mdp.rewards.flat_orientation_l2` (function) + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - Squared xy components of the world gravity vector expressed in the root + body frame; penalizes non-flat base orientation. +* - `upright` + - `unilab.envs.mdp.rewards.upright` (function) + - `std: float` (required, > 0); + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - Gaussian reward `exp(-||pg_xy||^2 / std^2)` for keeping one selected body + upright; requires exactly one selected body. Flat-ground form only. +* - `track_linear_velocity` + - `unilab.envs.mdp.rewards.track_linear_velocity` (function) + - `std: float` (required, > 0); `command_name: str` (required); + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - `exp(-(||cmd_xy - v_xy||^2 + v_z^2) / std^2)` with body-frame velocity; + assumes commanded z is zero. +* - `track_angular_velocity` + - `unilab.envs.mdp.rewards.track_angular_velocity` (function) + - `std: float` (required, > 0); `command_name: str` (required); + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - `exp(-((cmd_yaw - w_z)^2 + ||w_xy||^2) / std^2)`; rewards commanded yaw + rate while keeping roll/pitch rates near zero. +* - `body_angular_velocity_penalty` + - `unilab.envs.mdp.rewards.body_angular_velocity_penalty` (function) + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - Squared roll/pitch angular velocity of one selected body; requires exactly + one selected body. +* - `joint_pos_limits` + - `unilab.envs.mdp.rewards.joint_pos_limits` (function) + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - Sum of joint position violations beyond the soft limits (linear, not + squared). +* - `posture` + - `unilab.envs.mdp.rewards.posture` (class) + - `std: dict[str, float]` (required; joint-name regex to per-joint std); + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - `exp(-mean((q - q_default)^2 / std^2))` over the selected joints. +* - `variable_posture` + - `unilab.envs.mdp.rewards.variable_posture` (class) + - `std_standing` / `std_walking` / `std_running: dict[str, float]` + (required); `command_name: str` (required); + `walking_threshold: float = 0.5`; `running_threshold: float = 1.5`; + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - `posture` with speed-dependent tolerance: the std map is selected from the + total command speed (planar norm plus yaw magnitude). +``` + +## Locomotion — shared terms + +### `unilab.tasks.locomotion.common.manager_terms` + +Source: `src/unilab/tasks/locomotion/common/manager_terms.py`. + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - Function / class + - Parameters + - Semantics +* - `base_height_l2` + - `unilab.tasks.locomotion.common.manager_terms.base_height_l2` (class) + - `target_height: float` (required, m); + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - Squared error between world-frame root height and `target_height`. +* - `alive` + - `unilab.tasks.locomotion.common.manager_terms.alive` (function) + - — + - Constant 1.0 reward every step, unconditional. +* - `joint_deviation_l1` + - `unilab.tasks.locomotion.common.manager_terms.joint_deviation_l1` (class) + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - Sum of absolute joint displacement from the default pose over the selected + joints. +* - `stand_still_l1` + - `unilab.tasks.locomotion.common.manager_terms.stand_still_l1` (class) + - `command_name: str` (required); `command_threshold: float = 0.1`; + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - `joint_deviation_l1` masked to environments whose command norm is below + `command_threshold`. +* - `feet_phase_contact` + - `unilab.tasks.locomotion.common.manager_terms.feet_phase_contact` (class) + - `sensor_names: list[str]` (required; 4 unique contact sensors); + `frequency: float = 2.0` (Hz); + `phase_offsets: list[float] = (0.0, 0.5, 0.5, 0.0)`; + `contact_threshold: float = 0.1`; `stance_threshold: float = 0.6`; + `command_name: str | None = None`; `command_threshold: float = 0.0` + - Fraction of feet whose measured contact matches the expected stance + portion of the gait phase. +* - `feet_phase_swing_height` + - `unilab.tasks.locomotion.common.manager_terms.feet_phase_swing_height` + (class) + - `sensor_names: list[str]` (required; 4 3-D position sensors); + `frequency: float = 2.0`; `phase_offsets = (0.0, 0.5, 0.5, 0.0)`; + `target_height: float = 0.1` (m); `kernel: float = 0.01` (> 0); + `swing_start: float = 0.6`; `command_name: str | None = None`; + `command_threshold: float = 0.0` + - `exp(-(z - target_height)^2 / kernel)` for each foot during its swing + phase, averaged over feet. +* - `feet_air_while_standing` + - `unilab.tasks.locomotion.common.manager_terms.feet_air_while_standing` + (class) + - `sensor_names: list[str]` (required; 4 contact sensors); + `command_name: str` (required); `command_threshold: float = 0.1`; + `contact_threshold: float = 0.1` + - Counts feet without contact while the command norm is at or below + `command_threshold`. +``` + +### `unilab.tasks.locomotion.common.gait_terms` + +Source: `src/unilab/tasks/locomotion/common/gait_terms.py`. Terms with +`sensor_groups` bind one named contact-sensor group per foot; any sensor in +the group above `contact_threshold` counts as foot contact. + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - Function / class + - Parameters + - Semantics +* - `feet_air_time` + - `unilab.tasks.locomotion.common.gait_terms.feet_air_time` (class) + - `sensor_groups: list[list[str]]` (required; one group per foot); + `contact_threshold: float = 0.0`; `threshold_min: float = 0.05` (s); + `threshold_max: float = 0.5` (s, > `threshold_min`); + `command_name: str | None = None`; `command_threshold: float = 0.5` + - Counts feet whose current air time sits inside the rewarded window; + command-gated when `command_name` is set. +* - `feet_swing_height` + - `unilab.tasks.locomotion.common.gait_terms.feet_swing_height` (class) + - `sensor_groups: list[list[str]]` (required); `target_height: float` + (required, > 0, m); `command_name: str` (required); + `command_threshold: float` (required); `contact_threshold: float = 0.0`; + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` (one foot body per + sensor group) + - Tracks the peak foot height while airborne; at first contact charges + `(peak / target_height - 1)^2` per landing foot. +* - `feet_slip` + - `unilab.tasks.locomotion.common.gait_terms.feet_slip` (class) + - `sensor_groups: list[list[str]]` (required); `command_name: str` + (required); `command_threshold: float = 0.01`; + `contact_threshold: float = 0.0`; + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - Sum of squared xy foot velocity while the foot is in contact. +* - `feet_clearance` + - `unilab.tasks.locomotion.common.gait_terms.feet_clearance` (class) + - `target_height: float` (required, m); `command_name: str | None = None`; + `command_threshold: float = 0.01`; + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` (foot bodies) + - Sum over feet of `|z - target_height|` weighted by planar foot speed. +* - `self_collision_cost` + - `unilab.tasks.locomotion.common.gait_terms.self_collision_cost` (class) + - `sensor_groups: list[list[str]]` (required); + `contact_threshold: float = 0.0` + - Count of monitored self-collision sensor groups reporting contact. +* - `angular_momentum_penalty` + - `unilab.tasks.locomotion.common.gait_terms.angular_momentum_penalty` + (class) + - `sensor_name: str` (required; one 3-D sensor) + - Squared norm of the whole-body angular momentum read from the named + sensor. +``` + +### `unilab.tasks.locomotion.common.sensor_reward_terms` + +Source: `src/unilab/tasks/locomotion/common/sensor_reward_terms.py`. Each +term binds one named 3-D XML sensor of the robot. + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - Function / class + - Parameters + - Semantics +* - `track_lin_vel` + - `unilab.tasks.locomotion.common.sensor_reward_terms.track_lin_vel` (class) + - `sensor_name: str` (required; linear-velocity sensor); + `tracking_sigma: float = 0.25` (> 0); `command_name: str = "twist"` + - `exp(-||cmd_xy - v_xy||^2 / tracking_sigma)`. +* - `track_ang_vel` + - `unilab.tasks.locomotion.common.sensor_reward_terms.track_ang_vel` (class) + - `sensor_name: str` (required; gyro sensor); + `tracking_sigma: float = 0.25` (> 0); `command_name: str = "twist"` + - `exp(-(cmd_yaw - w_z)^2 / tracking_sigma)`. +* - `lin_vel_z` + - `unilab.tasks.locomotion.common.sensor_reward_terms.lin_vel_z` (class) + - `sensor_name: str` (required) + - Squared vertical (z) linear velocity. +* - `ang_vel_xy` + - `unilab.tasks.locomotion.common.sensor_reward_terms.ang_vel_xy` (class) + - `sensor_name: str` (required) + - Squared roll/pitch angular velocity sum. +* - `orientation` + - `unilab.tasks.locomotion.common.sensor_reward_terms.orientation` (class) + - `sensor_name: str` (required; up-vector sensor) + - Squared horizontal up-vector components; penalizes deviation from + upright. +``` + +## Locomotion — G1 (`unilab.tasks.locomotion.g1.manager_terms`) + +Source: `src/unilab/tasks/locomotion/g1/manager_terms.py`. The gait terms +share the parameters `frequency: float = 1.5` (Hz), +`init_mode: str = "offset_phase"` (`"offset_phase"` or `"independent"`), +`swing_height: float = 0.09` (m), `tracking_sigma: float = 0.008` (> 0), +`min_forward_speed: float = 0.0`, and `command_name: str = "twist"`; they bind +the named sensors `left_foot_pos` / `right_foot_pos` and +`pelvis_local_linvel` (contact terms additionally bind +`left_foot_contact_0..3` / `right_foot_contact_0..3`). Gait terms on one env +must share `frequency` and `init_mode`. + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - Function / class + - Parameters + - Semantics +* - `feet_phase` + - `unilab.tasks.locomotion.g1.manager_terms.feet_phase` (class) + - Shared gait parameters (see above) + - `exp(-(err_left^2 + err_right^2) / tracking_sigma)` against cubic-Bézier + per-foot height targets, gated by forward speed. +* - `feet_phase_contrast` + - `unilab.tasks.locomotion.g1.manager_terms.feet_phase_contrast` (class) + - Shared gait parameters + - `exp(-((h_left - h_right) - (t_left - t_right))^2 / tracking_sigma)`; + rewards left/right foot-height contrast against the phase targets. +* - `feet_phase_contact` + - `unilab.tasks.locomotion.g1.manager_terms.feet_phase_contact` (class) + - Shared gait parameters (see above) + - Mean per-foot match between measured contact and the expected stance + phase, gated by forward speed. +* - `feet_double_stance` + - `unilab.tasks.locomotion.g1.manager_terms.feet_double_stance` (class) + - Shared gait parameters (see above) + - Penalizes double-stance contact while a forward command is active. +* - `feet_air_time` + - `unilab.tasks.locomotion.g1.manager_terms.feet_air_time` (class) + - Shared gait parameters (see above) + - Counts feet whose current air time sits in the hardcoded (0.05, 0.5) s + window. +* - `forward_progress` + - `unilab.tasks.locomotion.g1.manager_terms.forward_progress` (class) + - `command_name: str = "twist"` + - `min(clamp(v_x, min=0) / clamp(cmd_x, min=1e-6), 1)`; forward progress + relative to commanded speed. +* - `under_speed` + - `unilab.tasks.locomotion.g1.manager_terms.under_speed` (class) + - `command_name: str = "twist"` + - Relative shortfall `clamp(cmd_x - clamp(v_x, min=0), min=0) / + clamp(cmd_x, min=1e-6)`. +* - `penalty_feet_ori` + - `unilab.tasks.locomotion.g1.manager_terms.penalty_feet_ori` (class) + - — + - Sum of the squared x and y quaternion components of both foot frames; + penalizes non-flat foot orientations. +* - `penalty_close_feet_xy` + - `unilab.tasks.locomotion.g1.manager_terms.penalty_close_feet_xy` (class) + - `threshold: float = 0.15` (m) + - `(dist_xy - threshold)^2` while the feet are closer than `threshold` in + the horizontal plane. +* - `weighted_pose` + - `unilab.tasks.locomotion.g1.manager_terms.weighted_pose` (class) + - `pose_weights: list[float]` (required; one per selected joint); + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - Weighted sum of squared joint deviation from the default pose. +* - `upper_body_pose` + - `unilab.tasks.locomotion.g1.manager_terms.upper_body_pose` (class) + - `pose_weights: list[float]` (required, at least 12 entries); + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - `weighted_pose` with the twelve leg-joint weights forced to zero. +* - `G1WalkRewardPack` + - `unilab.tasks.locomotion.g1.manager_terms.G1WalkRewardPack` (class; cfg + `G1WalkRewardPackCfg`) + - `command_name: str = "twist"`; `entity_name: str = "robot"`; + `tracking_lin_sigma: float = 0.25`; `tracking_ang_sigma: float = 0.25`; + `gait_frequency: float = 1.5`; `gait_init_mode: str = "offset_phase"`; + `swing_height: float = 0.09`; `feet_phase_sigma: float = 0.04`; + `min_forward_speed: float = 0.0`; + `pose_weights: tuple[float, ...] = 29 x 1.0` + - Fused canonical G1 walk reward family evaluated in one tensor carrier + read: linear/angular tracking, angular-xy and orientation penalties, + action rate, weighted pose, foot-orientation penalty, gated phase + tracking, and alive. Declared with `_target_` + + `func` and wired to `reward_pack_names` (see + `src/unilab/conf/sac/task/g1_walk_flat/mjwarp.yaml`). +``` + +## Motion tracking (`unilab.tasks.motion_tracking`) + +Sources: `src/unilab/tasks/motion_tracking/common/manager_terms.py` and +`src/unilab/tasks/motion_tracking/g1/manager_terms.py`. All terms resolve a +`MotionCommand` term by `command_name`; `std` parameters must be positive. + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - Function / class + - Parameters + - Semantics +* - `motion_global_anchor_position_error_exp` + - `unilab.tasks.motion_tracking.common.manager_terms.motion_global_anchor_position_error_exp` + (function) + - `command_name: str` (required); `std: float` (required, > 0) + - `exp(-||p_anchor_motion - p_anchor_robot||^2 / std^2)` in the world frame. +* - `motion_global_anchor_orientation_error_exp` + - `unilab.tasks.motion_tracking.common.manager_terms.motion_global_anchor_orientation_error_exp` + (function) + - `command_name: str` (required); `std: float` (required, > 0) + - `exp(-theta^2 / std^2)` with the shortest-path quaternion angle between + motion and robot anchor orientation. +* - `MotionRelativeBodyPositionErrorExp` + - `unilab.tasks.motion_tracking.common.manager_terms.MotionRelativeBodyPositionErrorExp` (class) + - `command_name: str` (required); `std: float` (required, > 0); + `body_names: tuple[str, ...] | None = None` (subset of the command's + tracked bodies) + - `exp(-sum_b ||dp_b||^2 / (N * std^2))` over anchor-relative body + positions. +* - `MotionRelativeBodyOrientationErrorExp` + - `unilab.tasks.motion_tracking.common.manager_terms.MotionRelativeBodyOrientationErrorExp` (class) + - `command_name: str` (required); `std: float` (required, > 0); + `body_names: tuple[str, ...] | None = None` + - `exp(-sum_b theta_b^2 / (N * std^2))` over anchor-relative body + orientations. +* - `MotionGlobalBodyLinearVelocityErrorExp` + - `unilab.tasks.motion_tracking.common.manager_terms.MotionGlobalBodyLinearVelocityErrorExp` (class) + - `command_name: str` (required); `std: float` (required, > 0); + `body_names: tuple[str, ...] | None = None` + - `exp(-sum_b ||dv_b||^2 / (N * std^2))` over world-frame body linear + velocities. +* - `MotionGlobalBodyAngularVelocityErrorExp` + - `unilab.tasks.motion_tracking.common.manager_terms.MotionGlobalBodyAngularVelocityErrorExp` (class) + - `command_name: str` (required); `std: float` (required, > 0); + `body_names: tuple[str, ...] | None = None` + - `exp(-sum_b ||dw_b||^2 / (N * std^2))` over world-frame body angular + velocities. +* - `motion_joint_position_error_exp` + - `unilab.tasks.motion_tracking.common.manager_terms.motion_joint_position_error_exp` (function) + - `command_name: str` (required); `std: float` (required, > 0) + - `exp(-mean((q_motion - q_robot)^2) / std^2)`. +* - `motion_joint_velocity_error_exp` + - `unilab.tasks.motion_tracking.common.manager_terms.motion_joint_velocity_error_exp` (function) + - `command_name: str` (required); `std: float` (required, > 0) + - `exp(-mean((dq_motion - dq_robot)^2) / std^2)`. +* - `joint_pos_limits` + - `unilab.tasks.motion_tracking.common.manager_terms.joint_pos_limits` (function) + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - Sum of squared soft-limit violations; task-local squared variant, distinct + from the linear framework term of the same name. +* - `UndesiredBodyContacts` + - `unilab.tasks.motion_tracking.common.manager_terms.UndesiredBodyContacts` (class) + - `command_name: str` (required); `threshold: float` (required, m); + `body_names: tuple[str, ...] | None = None` + - Count of selected bodies whose world-frame z drops below `threshold`. +* - `joint_acc_l2` + - `unilab.tasks.motion_tracking.g1.manager_terms.joint_acc_l2` (class) + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - Squared finite-difference joint acceleration `(dq_t - dq_t-1) / + env.step_dt`, summed; reset-aware. Not referenced by the shipped owner + YAMLs. +* - `joint_torque_l2` + - `unilab.tasks.motion_tracking.g1.manager_terms.joint_torque_l2` (class) + - `action_name: str = "joint_pos"`; + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - Squared position-controller torque estimate `kp * (target - q) - + kd * dq`, summed; requires a `MotionJointPositionAction` whose target + order matches the actuator order. Not referenced by the shipped owner + YAMLs. +``` diff --git a/docs/sphinx/source/en/5-reference/6-manager_terms/2-observations.md b/docs/sphinx/source/en/5-reference/6-manager_terms/2-observations.md new file mode 100644 index 000000000..82191957e --- /dev/null +++ b/docs/sphinx/source/en/5-reference/6-manager_terms/2-observations.md @@ -0,0 +1,308 @@ +# Observation Terms + +Built-in observation terms, compiled from source. Framework-level terms are +re-exported at `unilab.envs.mdp.`. + +:::{only} api_ref +```{raw} html + +``` +::: + +:::{only} prose_only +Authoritative signatures live in the source tree: +[unilab/envs/mdp](https://github.com/unilabsim/UniLab/tree/main/src/unilab/envs/mdp), +[unilab/tasks](https://github.com/unilabsim/UniLab/tree/main/src/unilab/tasks). +::: + +## Framework (`unilab.envs.mdp.observations`) + +Source: `src/unilab/envs/mdp/observations.py`. Terms that read the physics +state go through the Manager packed Torch read plan; sensor terms bind one +existing named backend sensor at construction and fail closed when the backend +cannot provide it. + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - Function / class + - Parameters + - Semantics +* - `base_lin_vel` + - `unilab.envs.mdp.observations.base_lin_vel` (function) + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - Root linear velocity expressed in the root body frame (3). +* - `base_ang_vel` + - `unilab.envs.mdp.observations.base_ang_vel` (function) + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - Root angular velocity expressed in the root body frame (3). +* - `projected_gravity` + - `unilab.envs.mdp.observations.projected_gravity` (function) + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - World gravity vector expressed in the root body frame (3). +* - `joint_pos_rel` + - `unilab.envs.mdp.observations.joint_pos_rel` (function) + - `biased: bool = False`; + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - Joint positions relative to the default pose; `biased=True` additionally + adds the per-joint encoder bias. +* - `joint_vel_rel` + - `unilab.envs.mdp.observations.joint_vel_rel` (function) + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - Joint velocities relative to the default joint velocities. +* - `last_action` + - `unilab.envs.mdp.observations.last_action` (function) + - `action_name: str | None = None` + - Raw actions of the action manager, or of one named action term when + `action_name` is set. +* - `generated_commands` + - `unilab.envs.mdp.observations.generated_commands` (function) + - `command_name: str` (required) + - Current command vector of the named command term. +* - `builtin_sensor` + - `unilab.envs.mdp.observations.builtin_sensor` (class) + - `sensor_name: str` (required); `entity_name: str = "robot"` + - Reads one existing backend sensor through a cached view. +* - `projected_gravity_from_sensor` + - `unilab.envs.mdp.observations.projected_gravity_from_sensor` (class) + - `sensor_name: str` (required; 3-D up-vector sensor); + `entity_name: str = "robot"` + - Negated up-vector sensor reading, yielding projected gravity (3). +* - `base_ang_vel_imu_misaligned` + - `unilab.envs.mdp.observations.base_ang_vel_imu_misaligned` (class) + - `max_angle_deg: float = 1.0` (deg); + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - Base angular velocity rotated by a per-env constant IMU mounting + misalignment, sampled once per env from the Manager RNG and fixed for the + whole run. +* - `projected_gravity_imu_misaligned` + - `unilab.envs.mdp.observations.projected_gravity_imu_misaligned` (class) + - `max_angle_deg: float = 1.0` (deg); + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - Projected gravity rotated by the same per-env IMU misalignment as + `base_ang_vel_imu_misaligned` of that env and angle. +``` + +## Ray scans (`unilab.envs.mdp.ray_observation`) + +Source: `src/unilab/envs/mdp/ray_observation.py`. `RayScan` binds a UniSim +tensor ray caster (factory name in `RayPatternCfg.caster`, default +`"uni_ray"`) attached to a scene body or site, traces the declared pattern +every step, and returns per-ray distances; `include_normal` / +`include_hit_point` append 3 values per ray each, flattened. Caster +capabilities are checked at read-phase binding and unsupported features fail +closed. No shipped owner YAML references `RayScan` yet; coverage lives in +`tests/envs/mdp/test_ray_observation.py`. + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - Function / class + - Parameters + - Semantics +* - `RayScan` + - `unilab.envs.mdp.ray_observation.RayScan` (class) + - `ray_pattern: RayPatternCfg` (required) + - Per-ray distances from the configured pattern + (`ray_count` values, plus normals / hit points when enabled). +``` + +`RayPatternCfg` (frozen dataclass) declares the attachment and the pattern: + +```{list-table} +:header-rows: 1 +:widths: 30 70 + +* - Field + - Type / default +* - `entity_name` + - `str` (required); owner entity of the attachment body +* - `pattern` + - `RayRingPatternCfg | RayGridPatternCfg | RayCameraPatternCfg` (required) +* - `max_distance` + - `float` (required, > 0) +* - `attachment_body_name` / `attachment_site_name` + - `str | None`; exactly one of the two must be set +* - `alignment` + - `"body" | "yaw" | "world"`, default `"body"` +* - `caster` + - `str`, default `"uni_ray"` (public UniSim caster factory name) +* - `sensor_offset` + - 3-float sequence, default `(0.0, 0.0, 0.0)` +* - `min_distance` + - `float = 0.0` (`0 <= min_distance < max_distance`) +* - `include_normal` / `include_hit_point` + - `bool = False` +* - `sync_geometry` + - `bool = False`; re-sync backend-owned geometry on reset +* - `allowed_geom_names` / `allowed_tags` / `excluded_body_names` / + `excluded_ancestor_body_names` + - `Sequence[str] | None = None`; static ray filters resolved on the cold + path +* - `caster_options` + - `dict`, default `{}`; forwarded to the caster factory +``` + +Pattern configs (all frozen dataclasses; angles in radians unless noted): + +```{list-table} +:header-rows: 1 +:widths: 22 40 38 + +* - Pattern + - Fields + - Semantics +* - `RayRingPatternCfg` + - `num_rays: int` (required, > 0); `start: float` (required); + `stop: float` (required); `pitch: float = 0.0` + - Rays on a body-local yaw arc with one shared pitch. +* - `RayGridPatternCfg` + - `size: (float, float)` (required, > 0); + `resolution: float | (float, float)` (required, > 0); + `offset: (float, float) = (0.0, 0.0)`; + `direction: (float, float, float) = (0.0, 0.0, -1.0)` (nonzero) + - Parallel rays on an inclusive two-dimensional local grid. +* - `RayCameraPatternCfg` + - `width: int` (required, > 0); `height: int` (required, > 0); + `fovy: float` (required, degrees in (0, 180)); + `forward: (float, float, float) = (1.0, 0.0, 0.0)`; + `up: (float, float, float) = (0.0, 0.0, 1.0)` (orthogonal to `forward`) + - Unit ray directions for a pinhole camera in an explicit local frame. +``` + +## Locomotion — shared terms + +`quadruped_gait_phase` is defined in +`src/unilab/tasks/locomotion/common/manager_terms.py`; the four privileged +per-foot terms are defined in +`src/unilab/tasks/locomotion/common/gait_terms.py` and mirror the mjlab +velocity-task critic observations. + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - Function / class + - Parameters + - Semantics +* - `quadruped_gait_phase` + - `unilab.tasks.locomotion.common.manager_terms.quadruped_gait_phase` + (class) + - `frequency: float = 2.0` (Hz); + `phase_offsets: list[float] = (0.0, 0.5, 0.5, 0.0)`; + `command_name: str | None = None`; `command_threshold: float = 0.0` + - Four-foot phase observation in [0, 1) with the diagonal-trot ordering; + advances only while the command norm exceeds `command_threshold` when + `command_name` is set. +* - `foot_height` + - `unilab.tasks.locomotion.common.gait_terms.foot_height` (class) + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` (foot bodies) + - World-frame foot body z, shape `(num_envs, num_feet)`; exact on flat + terrain. +* - `foot_air_time` + - `unilab.tasks.locomotion.common.gait_terms.foot_air_time` (class) + - `sensor_groups: list[list[str]]` (required); + `contact_threshold: float = 0.0` + - Current per-foot air time in seconds, shape `(num_envs, num_feet)`. +* - `foot_contact` + - `unilab.tasks.locomotion.common.gait_terms.foot_contact` (class) + - `sensor_groups: list[list[str]]` (required); + `contact_threshold: float = 0.0` + - Per-foot contact flag as float, shape `(num_envs, num_feet)`. +* - `foot_contact_forces` + - `unilab.tasks.locomotion.common.gait_terms.foot_contact_forces` (class) + - `sensor_groups: list[list[str]]` (required; 3-D force sensors); + `contact_threshold: float = 0.0` + - Log-compressed per-foot contact forces `sign(f) * log1p(|f|)`, shape + `(num_envs, 3 * num_feet)`. +``` + +## Locomotion — G1 (`unilab.tasks.locomotion.g1.manager_terms`) + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - Function / class + - Parameters + - Semantics +* - `G1GaitPhase` + - `unilab.tasks.locomotion.g1.manager_terms.G1GaitPhase` (class) + - `frequency: float = 1.5` (Hz); `init_mode: str = "offset_phase"` + (`"offset_phase"` or `"independent"`) + - Two-foot gait phase in radians in [0, 2 pi), shape `(num_envs, 2)`; + phases are resampled per reset and shared with the G1 gait reward terms. +``` + +## Motion tracking (`unilab.tasks.motion_tracking.common.manager_terms`) + +Source: `src/unilab/tasks/motion_tracking/common/manager_terms.py`. All terms +resolve the `MotionCommand` term named by `command_name` and read its +device-resident carriers. + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - Function / class + - Parameters + - Semantics +* - `motion_anchor_pos_b` + - `unilab.tasks.motion_tracking.common.manager_terms.motion_anchor_pos_b` + (function) + - `command_name: str` (required) + - Motion anchor position relative to the robot anchor, expressed in the + robot anchor frame (3). +* - `motion_anchor_ori_b` + - `unilab.tasks.motion_tracking.common.manager_terms.motion_anchor_ori_b` + (function) + - `command_name: str` (required) + - Relative anchor orientation (motion w.r.t. robot) as a 6-D rotation + representation. +* - `robot_body_pos_b` + - `unilab.tasks.motion_tracking.common.manager_terms.robot_body_pos_b` + (function) + - `command_name: str` (required) + - Robot body positions in the robot anchor frame, flattened to + `(num_envs, 3 * num_bodies)`. +* - `robot_body_ori_b` + - `unilab.tasks.motion_tracking.common.manager_terms.robot_body_ori_b` + (function) + - `command_name: str` (required) + - Robot body orientations in the robot anchor frame as 6-D rotations, + flattened to `(num_envs, 6 * num_bodies)`. +* - `motion_joint_pos_rel` + - `unilab.tasks.motion_tracking.common.manager_terms.motion_joint_pos_rel` + (function) + - `command_name: str` (required) + - Joint positions relative to the default pose and the per-episode + `joint_default_bias`. +* - `motion_joint_pos_rel_biased` + - `unilab.tasks.motion_tracking.common.manager_terms.motion_joint_pos_rel_biased` + (function) + - `command_name: str` (required) + - `motion_joint_pos_rel` plus the per-joint encoder bias. +* - `MotionAnchorPositionObservation` + - `unilab.tasks.motion_tracking.common.manager_terms.MotionAnchorPositionObservation` + (class) + - `command_name: str = "motion"` + - Tensor-native anchor-relative motion position (3), served by the packed + scene body read. +* - `MotionAnchorOrientationObservation` + - `unilab.tasks.motion_tracking.common.manager_terms.MotionAnchorOrientationObservation` + (class) + - `command_name: str = "motion"` + - Tensor-native anchor-relative motion orientation as 6-D rotation, served + by the packed scene body read. +``` diff --git a/docs/sphinx/source/en/5-reference/6-manager_terms/3-terminations.md b/docs/sphinx/source/en/5-reference/6-manager_terms/3-terminations.md new file mode 100644 index 000000000..1f30c817e --- /dev/null +++ b/docs/sphinx/source/en/5-reference/6-manager_terms/3-terminations.md @@ -0,0 +1,138 @@ +# Termination Terms + +Built-in termination terms, compiled from source. All terms return one boolean +per environment. Framework-level terms are re-exported at +`unilab.envs.mdp.`. + +:::{only} api_ref +```{raw} html + +``` +::: + +:::{only} prose_only +Authoritative signatures live in the source tree: +[unilab/envs/mdp](https://github.com/unilabsim/UniLab/tree/main/src/unilab/envs/mdp), +[unilab/tasks](https://github.com/unilabsim/UniLab/tree/main/src/unilab/tasks). +::: + +Terms declared with `time_out: true` in the owner YAML count as episodic +timeouts (truncations) rather than failures; see `TerminationTermCfg.time_out` +in `src/unilab/managers/termination_manager.py`. + +## Framework (`unilab.envs.mdp.terminations`) + +Source: `src/unilab/envs/mdp/terminations.py`. + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - Function / class + - Parameters + - Semantics +* - `time_out` + - `unilab.envs.mdp.terminations.time_out` (function) + - — + - Terminates when the episode length reaches its maximum; declare with + `time_out: true`. +* - `bad_orientation` + - `unilab.envs.mdp.terminations.bad_orientation` (function) + - `limit_angle: float` (required, rad); + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - Terminates when the root-body tilt angle from upright exceeds + `limit_angle`. +* - `root_height_below_minimum` + - `unilab.envs.mdp.terminations.root_height_below_minimum` (function) + - `minimum_height: float` (required, m); + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - Terminates when the world-frame root height drops below + `minimum_height`. +* - `nan_detection` + - `unilab.envs.mdp.terminations.nan_detection` (function) + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - Terminates environments whose root pose/velocity or joint + position/velocity contains NaN or Inf. +``` + +## Locomotion — G1 (`unilab.tasks.locomotion.g1.manager_terms`) + +Source: `src/unilab/tasks/locomotion/g1/manager_terms.py`. + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - Function / class + - Parameters + - Semantics +* - `g1_tilt_exceeded` + - `unilab.tasks.locomotion.g1.manager_terms.g1_tilt_exceeded` (class) + - `max_tilt_deg: float` (required, deg) + - Terminates when the base tilt from upright exceeds `max_tilt_deg`, read + from the named `torso_upvector` sensor. +* - `g1_base_height_below_minimum` + - `unilab.tasks.locomotion.g1.manager_terms.g1_base_height_below_minimum` + (class) + - `minimum_height: float` (required, m); + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - Terminates when the root body height drops below `minimum_height`, read + from the packed root-body read plan. +* - `G1WalkTerminationPack` + - `unilab.tasks.locomotion.g1.manager_terms.G1WalkTerminationPack` (class; + cfg `G1WalkTerminationPackCfg`) + - `entity_name: str = "robot"`; `root_body_name: str = "pelvis"`; + `max_tilt_deg: float = 65.0`; `minimum_height: float = 0.3` + - Fused canonical G1 walk failure terms evaluated in one carrier read: + tilt beyond `max_tilt_deg` OR root height below `minimum_height`. + Declared with `_target_` + `func` (see + `src/unilab/conf/sac/task/g1_walk_flat/mjwarp.yaml`). +``` + +## Motion tracking (`unilab.tasks.motion_tracking.common.manager_terms`) + +Source: `src/unilab/tasks/motion_tracking/common/manager_terms.py`. All terms +resolve the `MotionCommand` term named by `command_name`. + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - Function / class + - Parameters + - Semantics +* - `BadAnchorPosZOnly` + - `unilab.tasks.motion_tracking.common.manager_terms.BadAnchorPosZOnly` + (class) + - `command_name: str` (required); `threshold: float` (required, m) + - Terminates when the anchor height gap `|z_motion - z_robot|` exceeds + `threshold`. +* - `bad_anchor_ori` + - `unilab.tasks.motion_tracking.common.manager_terms.bad_anchor_ori` + (function) + - `command_name: str` (required); `threshold: float` (required); + `asset_cfg` accepted but unused + - Terminates when the quaternion-derived z-axis difference between motion + and robot anchor orientation exceeds `threshold`. +* - `BadMotionBodyPosZOnly` + - `unilab.tasks.motion_tracking.common.manager_terms.BadMotionBodyPosZOnly` + (class) + - `command_name: str` (required); `threshold: float` (required, m); + `body_names: tuple[str, ...] | None = None` + - Terminates when any selected body's anchor-relative z error exceeds + `threshold`. +* - `motion_clip_end` + - `unilab.tasks.motion_tracking.common.manager_terms.motion_clip_end` + (function) + - `command_name: str` (required) + - Terminates when the motion frame counter reaches the clip end; declare + with `time_out: true` (see + `src/unilab/conf/sac/task/g1_motion_tracking/base.yaml`). +``` diff --git a/docs/sphinx/source/zh_CN/2-user_guide/1-training/2-hydra_config.md b/docs/sphinx/source/zh_CN/2-user_guide/1-training/2-hydra_config.md index eeffb78a6..b38e460fa 100644 --- a/docs/sphinx/source/zh_CN/2-user_guide/1-training/2-hydra_config.md +++ b/docs/sphinx/source/zh_CN/2-user_guide/1-training/2-hydra_config.md @@ -66,3 +66,13 @@ uv run train --algo ppo --task go2_joystick_flat --sim mujoco --cfg job 换开关。请使用统一 CLI 的 `--sim` flag 来选择后端。 参见 {doc}`../../4-developer_guide/2-contracts/3-task_owner` 中的开发者 contract。 + +## 按算法的配置参考 + +逐字段表格 —— 训练效率、策略性能（GAE lambda、熵系数、学习率、网络结构）、训练 +运行与 reward 权重 —— 见各算法页： + +- {doc}`../2-algorithms/1-ppo` +- {doc}`../2-algorithms/2-appo` +- {doc}`../2-algorithms/3-sac` +- {doc}`../2-algorithms/4-flash_sac` diff --git a/docs/sphinx/source/zh_CN/2-user_guide/2-algorithms/1-ppo.md b/docs/sphinx/source/zh_CN/2-user_guide/2-algorithms/1-ppo.md index aa60ed7f1..272d9c228 100644 --- a/docs/sphinx/source/zh_CN/2-user_guide/2-algorithms/1-ppo.md +++ b/docs/sphinx/source/zh_CN/2-user_guide/2-algorithms/1-ppo.md @@ -82,3 +82,80 @@ RSL-RL 启动后不会同步 observation normalizer buffer 或环境 curriculum `g1_motion_tracking` 的双卡 MuJoCo smoke；这不构成多机或 Motrix 多卡 support 声明。当前验证的 RTX 6000D 主机沿用 production NCCL 兼容默认值 `NCCL_P2P_DISABLE=1`、`NCCL_SHM_DISABLE=1`，用户显式环境变量优先。 + +## 配置参考 + +默认值以 `src/unilab/structured_configs.py` 中的类型化配置（`PPOConfig`）为准； +`src/unilab/conf/ppo/config.yaml` 中的取值与之相同，除非另行注明。所有字段都可以在 +命令行上 override： + +```bash +uv run train --algo ppo --task go2_joystick_flat --sim mujoco \ + algo.algorithm.lam=0.92 \ + algo.algorithm.entropy_coef=0.005 +``` + +### 训练效率 + +| Override key | 类型 | 默认值 | 实际影响 | +| --- | --- | --- | --- | +| `algo.num_envs` | int | 4096 | 每个 rank 的并行环境数（go2_joystick_flat/mujoco owner 设为 1024） | +| `algo.num_steps_per_env` | int | 24 | 每个环境的 rollout 长度；每轮迭代样本数 = num_envs × 该值 | +| `algo.max_iterations` | int | 101 | 训练迭代数（go2 owner 设为 151） | +| `algo.save_interval` | int | 100 | checkpoint 保存间隔（迭代数） | +| `algo.algorithm.num_learning_epochs` | int | 5 | 每批采集数据上的 SGD epoch 数 | +| `algo.algorithm.num_mini_batches` | int | 4 | 每个 epoch 的 mini-batch 数 | +| `algo.seed` | int | 1 | 基础随机种子；rank `i` 使用 `seed + i` | +| `training.device` | str / null | null | 显式单设备覆盖；null = 自动放置 | +| `training.log_interval` | int | 1 | TensorBoard/W&B 标量上报间隔（迭代数） | + +### 策略性能 + +| Override key | 类型 | 默认值 | 实际影响 | +| --- | --- | --- | --- | +| `algo.algorithm.gamma` | float | 0.99 | 折扣因子 | +| `algo.algorithm.lam` | float | 0.95 | GAE lambda；调小以偏差换方差 | +| `algo.algorithm.entropy_coef` | float | 0.01 | 熵奖励系数（go2 owner 降为 1e-3） | +| `algo.algorithm.learning_rate` | float | 1e-3 | Adam 学习率；adaptive schedule 下为初始值（go2 owner 设为 3e-4） | +| `algo.algorithm.schedule` | str | adaptive | 学习率 schedule；`adaptive` 将 KL 拉向 `desired_kl` | +| `algo.algorithm.desired_kl` | float | 0.01 | adaptive 学习率 schedule 的 KL 目标 | +| `algo.algorithm.clip_param` | float | 0.2 | PPO surrogate 裁剪范围 | +| `algo.algorithm.value_loss_coef` | float | 1.0 | value loss 项的权重 | +| `algo.algorithm.use_clipped_value_loss` | bool | true | 像策略目标一样裁剪 value target | +| `algo.algorithm.max_grad_norm` | float | 1.0 | 梯度范数裁剪 | +| `algo.actor.hidden_dims` | list[int] | [512, 256, 128] | 策略 MLP 各层宽度 | +| `algo.actor.activation` | str | elu | 策略 MLP 激活函数 | +| `algo.actor.obs_normalization` | bool | false | 归一化策略观测（go2 owner 开启） | +| `algo.actor.distribution_cfg.init_std` | float | 1.0 | 高斯动作初始标准差（go2 owner 设为 0.5） | +| `algo.critic.hidden_dims` | list[int] | [512, 256, 128] | Critic MLP 各层宽度 | +| `algo.critic.obs_normalization` | bool | false | 归一化 critic 观测（go2 owner 开启） | + +### 训练运行（`training:` 段） + +| Override key | 类型 | 默认值 | 实际影响 | +| --- | --- | --- | --- | +| `training.no_play` | bool | false | 训练结束后跳过策略回放 | +| `training.play_only` | bool | false | 跳过训练，只回放已有 checkpoint | +| `training.tf32_matmul` | str | ieee | learner FP32 矩阵乘后端；`tf32` 启用 tensor core 并自动回退 IEEE | +| `training.nan_guard.enabled` | bool | true | 检测非有限张量并转储诊断信息 | +| `training.nan_guard.check_interval` | int | 32 | NaN 检查之间的 env step 数 | +| `training.defer_finite_validation` | bool | true | 将有限性检查合并为周期性批次 | +| `training.finite_validation_interval` | int | 32 | 每个批量有限性检查周期的步数 | +| `training.collector_tensor_device` | str | auto | collector 侧张量设备；`auto` 跟随 learner | +| `training.logger` | str | tensorboard | 标量日志后端（`tensorboard` 或 `wandb`） | +| `training.num_timesteps` | int / null | null | 全局样本预算；设置后覆盖推导出的迭代数 | + +### Reward 权重 + +Reward 项位于任务 owner YAML 的 `reward:` 段 —— 每项一个块，含 `func`、`weight` +与 `params`（示例：`src/unilab/conf/ppo/task/go2_joystick_flat/base.yaml`）。调整某个 +项可以编辑该 owner YAML，也可以在命令行覆盖其权重： + +```bash +uv run train --algo ppo --task go2_joystick_flat --sim mujoco \ + reward.tracking_lin_vel.weight=1.5 \ + reward.action_rate.weight=-0.01 +``` + +各项名称与默认权重因任务而异；请查看所训练任务 owner YAML 的 `reward:` 段 +（`src/unilab/conf/ppo/task//base.yaml`）。 diff --git a/docs/sphinx/source/zh_CN/2-user_guide/2-algorithms/2-appo.md b/docs/sphinx/source/zh_CN/2-user_guide/2-algorithms/2-appo.md index 36c613a39..a04d529dd 100644 --- a/docs/sphinx/source/zh_CN/2-user_guide/2-algorithms/2-appo.md +++ b/docs/sphinx/source/zh_CN/2-user_guide/2-algorithms/2-appo.md @@ -72,3 +72,86 @@ gantt 默认日志根目录为 `logs/appo//`，来自 `src/unilab/conf/appo/config.yaml` 中的 `algo.algo_log_name=appo`。 + +## 配置参考 + +默认值以 `src/unilab/structured_configs.py` 中的类型化配置（`APPOConfig`）为准； +`src/unilab/conf/appo/config.yaml` 中的取值与之相同，除非另行注明。所有字段都可以在 +命令行上 override： + +```bash +uv run train --algo appo --task go2_joystick_flat --sim mujoco \ + algo.algorithm.lam=0.92 \ + algo.algorithm.vtrace_clip_rho=1.2 +``` + +### 训练效率 + +| Override key | 类型 | 默认值 | 实际影响 | +| --- | --- | --- | --- | +| `algo.num_envs` | int | 2048 | 每个 collector 的环境数 | +| `algo.steps_per_env` | int | 24 | 每份发布的 rollout 中单个环境的长度 | +| `algo.max_iterations` | int | 150 | learner 迭代数 | +| `algo.save_interval` | int | 50 | checkpoint 保存间隔（迭代数） | +| `algo.algorithm.num_learning_epochs` | int | 5 | 每份消费的 rollout 上的 SGD epoch 数 | +| `algo.algorithm.num_mini_batches` | int | 4 | 每个 epoch 的 mini-batch 数 | +| `algo.algorithm.enable_compile` | bool | true | learner 更新的 torch.compile；内置 `src/unilab/conf/appo/config.yaml` 出货为 `false` | +| `algo.seed` | int | 1 | 随机种子 | +| `training.collector_device` | str / null | null | collector 设备；null 跟随 learner | +| `training.replay_queue_size` | int / null | null | learner 侧 replay 缓存深度 | +| `training.log_interval` | int | 1 | TensorBoard/W&B 标量上报间隔（迭代数） | + +### 策略性能 + +| Override key | 类型 | 默认值 | 实际影响 | +| --- | --- | --- | --- | +| `algo.algorithm.gamma` | float | 0.99 | 折扣因子 | +| `algo.algorithm.lam` | float | 0.95 | GAE lambda；调小以偏差换方差 | +| `algo.algorithm.entropy_coef` | float | 0.01 | 熵奖励系数 | +| `algo.algorithm.learning_rate` | float | 1e-3 | Adam 学习率；adaptive schedule 下为初始值 | +| `algo.algorithm.schedule` | str | adaptive | 学习率 schedule；`adaptive` 将 KL 拉向 `desired_kl` | +| `algo.algorithm.desired_kl` | float | 0.01 | adaptive 学习率 schedule 的 KL 目标 | +| `algo.algorithm.adaptive_kl_factor` | float | 1.2 | KL 死区：高于 `desired_kl` × 该值时 LR 下降，低于 `desired_kl` ÷ 该值时 LR 上升 | +| `algo.algorithm.adaptive_lr_factor` | float | 1.1 | adaptive schedule 的 LR 乘性步长 | +| `algo.algorithm.clip_param` | float | 0.2 | PPO surrogate 裁剪范围 | +| `algo.algorithm.value_loss_coef` | float | 1.0 | value loss 项的权重 | +| `algo.algorithm.use_clipped_value_loss` | bool | true | 像策略目标一样裁剪 value target | +| `algo.algorithm.max_grad_norm` | float | 1.0 | 梯度范数裁剪 | +| `algo.algorithm.optimizer` | str | adam | learner 更新使用的优化器 | +| `algo.algorithm.vtrace_clip_rho` | float | 1.0 | V-trace 重要性权重裁剪 ρ̄ | +| `algo.algorithm.vtrace_clip_c` | float | 1.0 | V-trace 重要性权重裁剪 c̄ | +| `algo.algorithm.tau` | float | 1.0 | target network 的 EMA 系数（1.0 = 硬拷贝） | +| `algo.algorithm.target_update_freq` | int | 1 | 两次 target network 同步之间的 learner 更新数 | +| `algo.actor.hidden_dims` | list[int] | [512, 256, 128] | 策略 MLP 各层宽度 | +| `algo.actor.activation` | str | elu | 策略 MLP 激活函数 | +| `algo.actor.distribution_cfg.init_std` | float | 1.0 | 高斯动作初始标准差 | +| `algo.critic.hidden_dims` | list[int] | [512, 256, 128] | Critic MLP 各层宽度 | + +### 训练运行（`training:` 段） + +| Override key | 类型 | 默认值 | 实际影响 | +| --- | --- | --- | --- | +| `training.no_play` | bool | false | 训练结束后跳过策略回放 | +| `training.play_only` | bool | false | 跳过训练，只回放已有 checkpoint | +| `training.tf32_matmul` | str | ieee | learner FP32 矩阵乘后端；`tf32` 启用 tensor core 并自动回退 IEEE | +| `training.nan_guard.enabled` | bool | true | 检测非有限张量并转储诊断信息 | +| `training.nan_guard.check_interval` | int | 32 | NaN 检查之间的 env step 数 | +| `training.defer_finite_validation` | bool | true | 将有限性检查合并为周期性批次 | +| `training.finite_validation_interval` | int | 32 | 每个批量有限性检查周期的步数 | +| `training.collector_tensor_device` | str | auto | collector 侧张量设备；`auto` 跟随 learner | +| `training.logger` | str | tensorboard | 标量日志后端（`tensorboard` 或 `wandb`） | + +### Reward 权重 + +Reward 项位于任务 owner YAML 的 `reward:` 段 —— 每项一个块，含 `func`、`weight` +与 `params`（示例：`src/unilab/conf/appo/task/go2_joystick_flat/base.yaml`）。调整某个 +项可以编辑该 owner YAML，也可以在命令行覆盖其权重： + +```bash +uv run train --algo appo --task go2_joystick_flat --sim mujoco \ + reward.tracking_lin_vel.weight=1.5 \ + reward.action_rate.weight=-0.01 +``` + +各项名称与默认权重因任务而异；请查看所训练任务 owner YAML 的 `reward:` 段 +（`src/unilab/conf/appo/task//base.yaml`）。 diff --git a/docs/sphinx/source/zh_CN/2-user_guide/2-algorithms/3-sac.md b/docs/sphinx/source/zh_CN/2-user_guide/2-algorithms/3-sac.md index 85721b109..d84e0d179 100644 --- a/docs/sphinx/source/zh_CN/2-user_guide/2-algorithms/3-sac.md +++ b/docs/sphinx/source/zh_CN/2-user_guide/2-algorithms/3-sac.md @@ -102,3 +102,96 @@ MuJoCo worker 线程逐核绑定外，collector 进程及其已启动的本地 collector `Steps/s` 与 learner `Samples/s` 各自相对单卡的 scaling 基准见 `scripts/benchmark/rl/benchmark_offpolicy_dp_scaling.py`（issue #968，真实运行不进 CI）。 + +## 配置参考 + +默认值以 `src/unilab/structured_configs.py` 中的类型化配置（`SACConfig`）为准； +`src/unilab/conf/sac/config.yaml` 中的取值与之相同，除非另行注明。所有字段都可以在 +命令行上 override： + +```bash +uv run train --algo sac --task g1_walk_flat --sim mujoco \ + algo.gamma=0.99 \ + algo.algo_params.alpha_init=0.1 +``` + +### 训练效率 + +| Override key | 类型 | 默认值 | 实际影响 | +| --- | --- | --- | --- | +| `algo.num_envs` | int | 4096 | 并行采集环境数 | +| `algo.batch_size` | int | 8192 | 每次 learner 更新的 batch | +| `algo.replay_buffer_n` | int | 512 | replay 容量（以整轮 env 向量为单位；行数 = n × num_envs） | +| `algo.updates_per_step` | int | 4 | 每个 env step 的 learner 更新数 | +| `algo.learning_starts` | int | 1 | learner 开始更新前需采集的整轮 env 向量数 | +| `algo.policy_frequency` | int | 4 | 每 N 次 learner 更新做一次 actor/target 更新 | +| `algo.max_iterations` | int | 500 | 训练迭代数 | +| `algo.save_interval` | int | 500 | checkpoint 保存间隔（迭代数） | +| `algo.algo_params.use_compile` | bool | true | learner 更新的 torch.compile | +| `algo.seed` | int | 1 | 随机种子 | +| `training.use_amp` | bool | true | learner 更新路径上的 bf16 autocast | +| `training.log_interval` | int | 1 | TensorBoard/W&B 标量上报间隔（迭代数） | + +### 策略性能 + +| Override key | 类型 | 默认值 | 实际影响 | +| --- | --- | --- | --- | +| `algo.gamma` | float | 0.97 | 折扣因子 | +| `algo.tau` | float | 0.125 | target critic 的 EMA 系数 | +| `algo.actor_lr` | float | 3e-4 | actor 的 Adam 学习率 | +| `algo.critic_lr` | float | 3e-4 | critic 的 Adam 学习率 | +| `algo.actor_hidden_dim` | int | 512 | actor 网络宽度 | +| `algo.critic_hidden_dim` | int | 768 | critic 网络宽度 | +| `algo.num_atoms` | int | 101 | 分布式（C51）critic 的 atom 数 | +| `algo.obs_normalization` | bool | false | 归一化观测 | +| `algo.use_layer_norm` | bool | true | 网络中使用 LayerNorm | +| `algo.algo_params.alpha_lr` | float | 3e-4 | 熵温度（alpha）的学习率 | +| `algo.algo_params.alpha_init` | float | 0.01 | 初始熵温度 | +| `algo.algo_params.target_entropy_ratio` | float | 0.0 | 目标熵 = −action_dim × 该比例（0.0 → 目标为 0） | +| `algo.algo_params.max_grad_norm` | float | 0.0 | 梯度范数裁剪；0 表示不裁剪 | +| `algo.algo_params.amp_dtype` | str | auto | `training.use_amp` 下的 AMP dtype | + +### 训练运行（`training:` 段） + +| Override key | 类型 | 默认值 | 实际影响 | +| --- | --- | --- | --- | +| `training.no_play` | bool | false | 训练结束后跳过策略回放 | +| `training.play_only` | bool | false | 跳过训练，只回放已有 checkpoint | +| `training.export_onnx` | bool | true | 回放时导出 `policy.onnx` | +| `training.cuda_process_sharing` | str / null | null | null = 独立 CUDA context；`mps` 请求 CUDA MPS（守护进程由用户经 `uni-cumps` 管理） | +| `training.dp_gradient_wire_dtype` | str | float32 | 多卡梯度平均的传输 dtype；`bfloat16` 会压缩 | +| `training.nan_guard.enabled` | bool | true | 检测非有限张量并转储诊断信息 | +| `training.nan_guard.check_interval` | int | 32 | NaN 检查之间的 env step 数 | +| `training.defer_finite_validation` | bool | true | 将有限性检查合并为周期性批次 | +| `training.finite_validation_interval` | int | 32 | 每个批量有限性检查周期的步数 | +| `training.collector_tensor_device` | str | auto | collector 侧张量设备；`auto` 跟随 learner | +| `training.env_steps_per_sync` | int | 1 | 每次采集同步的 env step 数 | +| `training.inference_slot_capacity` | int | 1 | 在途 learner 推理请求数；1 表示严格串行 | +| `training.collector_metrics_interval` | int | 32 | collector 指标刷写间隔（步数） | +| `training.replay_ingress_depth` | int | 2 | 允许在途的 replay ingress 块数 | +| `training.replay_ingress_slot_rows` | int / null | null | 每个 ingress 块的行数；null = 每个 collector 向量一块 | +| `training.replay_prefetch_mode` | str | one_tick | replay 预取策略 | +| `training.trace_enabled` | bool | false | off-policy 运行时的 Chrome trace 剖析 | +| `training.trace_output_dir` | str / null | null | trace 输出目录 | +| `training.trace_thread_time` | bool | false | trace 中包含线程时间 | +| `training.trace_cuda_events` | bool | true | trace 中记录 CUDA event | +| `training.nvtx_profile_ranges` | bool | false | 为外部剖析器发出 NVTX range | +| `training.torch_threads.enabled` | bool | true | 按进程角色固定 torch CPU 线程数 | +| `training.torch_threads.learner_num_threads` | int / str | auto | learner 的 intra-op 线程数；`auto` 按主机 CPU 数解析 | +| `training.torch_threads.collector_num_threads` | int / str | auto | collector 的 intra-op 线程数 | +| `training.dp_collector_cpu_ids` | list / null | null | 每个 rank 的显式 CPU id；null 按 `cpu_count // world_size` 自动划分 | +| `training.logger` | str | tensorboard | 标量日志后端（`tensorboard` 或 `wandb`） | + +### Reward 权重 + +Reward 项位于任务 owner YAML 的 `reward:` 段 —— 每项一个块，含 `func`、`weight` +与 `params`（示例：`src/unilab/conf/sac/task/g1_walk_flat/base.yaml`）。调整某个 +项可以编辑该 owner YAML，也可以在命令行覆盖其权重： + +```bash +uv run train --algo sac --task g1_walk_flat --sim mujoco \ + reward.tracking_lin_vel.weight=2.5 +``` + +各项名称与默认权重因任务而异；请查看所训练任务 owner YAML 的 `reward:` 段 +（`src/unilab/conf/sac/task//base.yaml`）。 diff --git a/docs/sphinx/source/zh_CN/2-user_guide/2-algorithms/4-flash_sac.md b/docs/sphinx/source/zh_CN/2-user_guide/2-algorithms/4-flash_sac.md index e71186e46..0fba8602f 100644 --- a/docs/sphinx/source/zh_CN/2-user_guide/2-algorithms/4-flash_sac.md +++ b/docs/sphinx/source/zh_CN/2-user_guide/2-algorithms/4-flash_sac.md @@ -77,3 +77,114 @@ export CUDA_VISIBLE_DEVICES=, uv run train --algo flashsac --task g1_walk_flat --sim mujoco \ training.no_play=true ``` + +## 配置参考 + +默认值以 `src/unilab/structured_configs.py` 中的类型化配置（`FlashSACConfig`）为准； +`src/unilab/conf/flashsac/config.yaml` 中的取值与之相同，除非另行注明。所有字段都可以 +在命令行上 override： + +```bash +uv run train --algo flashsac --task g1_walk_flat --sim mujoco \ + algo.gamma=0.99 \ + algo.algo_params.temp_initial_value=0.1 +``` + +### 训练效率 + +| Override key | 类型 | 默认值 | 实际影响 | +| --- | --- | --- | --- | +| `algo.num_envs` | int | 1024 | 并行采集环境数 | +| `algo.batch_size` | int | 2048 | 每次 learner 更新的 batch | +| `algo.replay_buffer_n` | int | 512 | replay 容量（以整轮 env 向量为单位；行数 = n × num_envs） | +| `algo.updates_per_step` | int | 2 | 每个 env step 的 learner 更新数 | +| `algo.learning_starts` | int | 98 | learner 开始更新前需采集的整轮 env 向量数 | +| `algo.policy_frequency` | int | 2 | 每 N 次 learner 更新做一次 actor/温度更新 | +| `algo.max_iterations` | int | 5000 | 训练迭代数 | +| `algo.save_interval` | int | 1000 | checkpoint 保存间隔（迭代数） | +| `algo.algo_params.use_compile` | bool | true | CUDA Inductor 编译；`false` 走 eager 路径并同时关闭 whole-cycle graph | +| `algo.algo_params.compile_full_objectives` | bool | true | 编译完整 actor/critic objective，而非较小的 loss-tensor 区域 | +| `algo.algo_params.use_whole_cycle_cuda_graph` | bool | true | learner 侧整轮 CUDA Graph 捕获 | +| `algo.algo_params.dp_whole_cycle_cuda_graph` | bool | true | 数据并行路径上的整轮捕获；`false` 只关闭 DP 捕获（该字段设于 `config.yaml`） | +| `algo.seed` | int | 1 | 随机种子 | +| `training.use_amp` | bool | true | learner 更新路径上的 bf16 autocast | +| `training.log_interval` | int | 1 | TensorBoard/W&B 标量上报间隔（迭代数） | + +### 策略性能 + +| Override key | 类型 | 默认值 | 实际影响 | +| --- | --- | --- | --- | +| `algo.gamma` | float | 0.97 | 折扣因子 | +| `algo.tau` | float | 0.01 | target critic 的 EMA 系数 | +| `algo.actor_lr` | float | 3e-4 | 兜底峰值 LR（当 `algo.algo_params.learning_rate_peak` ≤ 0 时使用） | +| `algo.critic_lr` | float | 3e-4 | 声明的 LR 字段；实际 schedule 由 `learning_rate_*` 决定 | +| `algo.actor_hidden_dim` | int | 128 | actor 网络宽度 | +| `algo.critic_hidden_dim` | int | 256 | critic 网络宽度 | +| `algo.num_atoms` | int | 101 | 分布式 critic 的 atom 数 | +| `algo.obs_normalization` | bool | false | 归一化观测 | +| `algo.use_layer_norm` | bool | false | 网络中使用 LayerNorm | +| `algo.policy_before_critic` | bool | false | 每次更新先更新 actor（+温度）再更新 critic（该字段设于 `config.yaml`） | +| `algo.algo_params.normalize_reward` | bool | true | 用运行中的回报估计归一化 reward | +| `algo.algo_params.normalized_g_max` | float | 5.0 | 归一化回报估计的截断上限 | +| `algo.algo_params.actor_num_blocks` | int | 2 | block 结构 actor 的块数 | +| `algo.algo_params.critic_num_blocks` | int | 2 | block 结构 critic 的块数 | +| `algo.algo_params.actor_bc_alpha` | float | 0.0 | 行为克隆辅助损失权重；0 表示关闭 | +| `algo.algo_params.actor_noise_zeta_mu` | float | 2.0 | actor 噪声调度的 mu | +| `algo.algo_params.actor_noise_zeta_max` | int | 16 | actor 噪声调度的上限 | +| `algo.algo_params.critic_min_v` | float | -5.0 | 分布式 critic 值域下界 | +| `algo.algo_params.critic_max_v` | float | 5.0 | 分布式 critic 值域上界 | +| `algo.algo_params.temp_initial_value` | float | 0.01 | 初始熵温度 | +| `algo.algo_params.temp_target_sigma` | float | 0.15 | 温度自适应使用的目标 sigma | +| `algo.algo_params.temp_target_entropy` | float / null | null | 显式目标熵；null 使用内置目标 | +| `algo.algo_params.learning_rate_init` | float | 3e-4 | actor/critic/温度调度起始 LR | +| `algo.algo_params.learning_rate_peak` | float | 3e-4 | 调度峰值 LR | +| `algo.algo_params.learning_rate_end` | float | 1.5e-4 | 衰减结束后的 LR | +| `algo.algo_params.learning_rate_warmup_steps` | int | 0 | 从起始值到峰值的 warmup 步数 | +| `algo.algo_params.learning_rate_decay_steps` | int | 500000 | 从峰值到结束值的衰减步数 | +| `algo.algo_params.n_step` | int | 1 | N-step 回报长度；当前仅支持 1 | +| `algo.algo_params.amp_dtype` | str | auto | `training.use_amp` 下的 AMP dtype | + +### 训练运行（`training:` 段） + +| Override key | 类型 | 默认值 | 实际影响 | +| --- | --- | --- | --- | +| `training.no_play` | bool | false | 训练结束后跳过策略回放 | +| `training.play_only` | bool | false | 跳过训练，只回放已有 checkpoint | +| `training.export_onnx` | bool | true | 回放时导出 `policy.onnx` | +| `training.cuda_process_sharing` | str / null | null | null = 独立 CUDA context；`mps` 请求 CUDA MPS（守护进程由用户经 `uni-cumps` 管理） | +| `training.dp_gradient_wire_dtype` | str | float32 | 多卡梯度平均的传输 dtype；`bfloat16` 会压缩 | +| `training.nan_guard.enabled` | bool | true | 检测非有限张量并转储诊断信息 | +| `training.nan_guard.check_interval` | int | 32 | NaN 检查之间的 env step 数 | +| `training.defer_finite_validation` | bool | true | 将有限性检查合并为周期性批次 | +| `training.finite_validation_interval` | int | 32 | 每个批量有限性检查周期的步数 | +| `training.collector_tensor_device` | str | auto | collector 侧张量设备；`auto` 跟随 learner | +| `training.env_steps_per_sync` | int | 1 | 每次采集同步的 env step 数 | +| `training.inference_slot_capacity` | int | 1 | 在途 learner 推理请求数；1 表示严格串行 | +| `training.collector_metrics_interval` | int | 32 | collector 指标刷写间隔（步数） | +| `training.replay_ingress_depth` | int | 2 | 允许在途的 replay ingress 块数 | +| `training.replay_ingress_slot_rows` | int / null | null | 每个 ingress 块的行数；null = 每个 collector 向量一块 | +| `training.replay_prefetch_mode` | str | one_tick | replay 预取策略 | +| `training.trace_enabled` | bool | false | off-policy 运行时的 Chrome trace 剖析 | +| `training.trace_output_dir` | str / null | null | trace 输出目录 | +| `training.trace_thread_time` | bool | false | trace 中包含线程时间 | +| `training.trace_cuda_events` | bool | true | trace 中记录 CUDA event | +| `training.nvtx_profile_ranges` | bool | false | 为外部剖析器发出 NVTX range | +| `training.torch_threads.enabled` | bool | true | 按进程角色固定 torch CPU 线程数 | +| `training.torch_threads.learner_num_threads` | int / str | auto | learner 的 intra-op 线程数；`auto` 按主机 CPU 数解析 | +| `training.torch_threads.collector_num_threads` | int / str | auto | collector 的 intra-op 线程数 | +| `training.dp_collector_cpu_ids` | list / null | null | 每个 rank 的显式 CPU id；null 按 `cpu_count // world_size` 自动划分 | +| `training.logger` | str | tensorboard | 标量日志后端（`tensorboard` 或 `wandb`） | + +### Reward 权重 + +Reward 项位于任务 owner YAML 的 `reward:` 段 —— 每项一个块，含 `func`、`weight` +与 `params`（示例：`src/unilab/conf/flashsac/task/g1_walk_flat/base.yaml`）。调整某个 +项可以编辑该 owner YAML，也可以在命令行覆盖其权重： + +```bash +uv run train --algo flashsac --task g1_walk_flat --sim mujoco \ + reward.tracking_lin_vel.weight=2.5 +``` + +各项名称与默认权重因任务而异；请查看所训练任务 owner YAML 的 `reward:` 段 +（`src/unilab/conf/flashsac/task//base.yaml`）。 diff --git a/docs/sphinx/source/zh_CN/2-user_guide/5-domain_randomization/0-index.md b/docs/sphinx/source/zh_CN/2-user_guide/5-domain_randomization/0-index.md index 79c6f15db..bafe439ba 100644 --- a/docs/sphinx/source/zh_CN/2-user_guide/5-domain_randomization/0-index.md +++ b/docs/sphinx/source/zh_CN/2-user_guide/5-domain_randomization/0-index.md @@ -101,6 +101,7 @@ Reset-time model-field DR 保持在已选 identity 内。其 canonical 或 per-e :hidden: 1-configuration +2-field_reference ``` ## 相关任务 diff --git a/docs/sphinx/source/zh_CN/2-user_guide/5-domain_randomization/2-field_reference.md b/docs/sphinx/source/zh_CN/2-user_guide/5-domain_randomization/2-field_reference.md new file mode 100644 index 000000000..7161e74de --- /dev/null +++ b/docs/sphinx/source/zh_CN/2-user_guide/5-domain_randomization/2-field_reference.md @@ -0,0 +1,327 @@ +# 字段参考 + +本页是 `src/unilab/envs/mdp/events.py` 中域随机化（DR）event term 的字段级 +配置参考，与 {doc}`0-index`（审计各注册任务声明了哪些 term）和 +{doc}`1-configuration`（说明 owner YAML 的选择方式）互补。下文所有签名、 +默认值与取值线索均来自源码或 `src/unilab/conf/` 下的 owner YAML，不从设计 +意图推断任何内容。 + +DR term 运行在三个生命周期类别下（见 {doc}`0-index` 的生命周期概述）： + +- **construction**：固定 model/tool identity，在 `env.fixed_model_variants` + 上声明；本页没有任何 event term 属于该类别。 +- **reset**：`mode: reset` 的 event term 在每次 episode reset 时采样，并经 + backend 持有的 packed reset transaction 一次性提交。`mode: startup` 是 + reset 生命周期的可选变体：初始化时采样一次，之后回放冻结的逐环境取值。 +- **interval**：`mode: interval` 的 event term 在 step 之间按采样计时器施加 + 扰动；`mode: step` 每个 step 都触发，由 term 自行管理力的生命周期。 + +## Backend 适用性 + +字段 × backend 的精确矩阵由外部包 `unisim-core` 持有 +（`DomainRandomizationCapabilities.supported_reset_terms` / +`supported_interval_terms`），无法从本仓库逐字段确认，因此本页刻意不给出 +该矩阵。有两条规则始终成立： + +1. 所选 backend 未声明支持的 term 在构造期 fail closed；不存在过滤或静默 + 回退。 +2. Owner YAML 按 backend 分文件 + （`src/unilab/conf//task//.yaml`）。某个 backend + 不支持的 term 在该 backend 的 owner 中置为 `null`，共享 base owner 为 + 其余 backend 保留它。 + +仓库内的该模式示例：`src/unilab/conf/ppo/task/g1_walk_flat/motrix.yaml` 与 +`.../mjwarp.yaml` 把 `pd_gains` 置为 `null`，而 MuJoCo owner 保留 kp/kd +随机化；`src/unilab/conf/sac/task/g1_motion_tracking/genesis.yaml` 把 +`foot_friction` 置为 `null`，注释记录了 Genesis 没有逐环境 friction API。 +最完整的 owner 实例（mass、COM、gravity、push）是 +`src/unilab/conf/flashsac/task/go2_joystick_flat/mujoco.yaml` 的 `events:` 段。 + +## 通用参数族 + +下列参数在多个 term 间复用。下方各 term 的表只列出其构造函数实际接受的 +参数；未知 param 会在构造期被拒绝。 + +- `asset_cfg`（`SceneEntityCfg`）：选择 entity 及其组件。`name` 必填（场景 + 实体名，例如 `robot`）；组件通过 `body_names`/`body_ids`、 + `joint_names`/`joint_ids`、`geom_names`/`geom_ids` 或 + `actuator_names`/`actuator_ids` 选择。`*_names` 字段接受正则字符串； + `*_ids` 字段默认为"全部"。多数 term 默认为 `SceneEntityCfg("robot")` + （整台机器人）。 +- `distribution`：`"uniform"`（默认）、`"log_uniform"`（bounds 必须为正） + 或 `"gaussian"`（range 读作 `(mean, std)`，std 非负）。`pd_gains` 只接受 + `uniform`/`log_uniform`。 +- `operation`：采样值与 backend 声明的默认值的组合方式：`"add"` → + `default + sample`，`"scale"` → `default * sample`，`"abs"` → 采样值 + 直接覆盖默认值。默认值因 term 而异，见各表。 +- range 形状约定：标量对 `[lo, hi]`（如 `kp_range`、 + `mass_distribution_params`）；按轴字典 `{x: [lo, hi], ...}` + （`com_range`、`pose_range`、`velocity_range`，省略的键默认为 `(0, 0)`）； + 宽度为 3 的 `[[lo_x, lo_y, lo_z], [hi_x, hi_y, hi_z]]` + （`gravity_distribution_params`）；model-field term 的 `ranges` 还可以是 + 作用于所有选中轴的扁平 pair、按轴索引的字典，或 regex → pair 的字典 + （对选中的 geom/joint 名做 `fullmatch`；每个 pattern 至少匹配一个名字且 + 不得重叠）。 + +外层的 term 块（`func`、`mode`、`interval_range_s`、`is_global_time`、 +`min_step_count_between_reset`、`params`）是一个 `EventTermCfg`； +`interval_range_s` 与 `is_global_time` 位于 term 上，不在 `params` 内。 +`mode: interval` 时 `interval_range_s` 必填；`is_global_time: true` 让所有 +环境共用一个计时器。 + +## Reset 生命周期 Term + +### `pd_gains` + +同时随机化 actuator 刚度 `kp` 与阻尼 `kd`（`unilab.envs.mdp.pd_gains`）。 +仅支持 `mode: reset`；startup 与 interval 在构造期被拒绝。 + +| 字段 | 类型 | 默认值/必填 | 生效时机 | 语义与合理取值线索 | +| --- | --- | --- | --- | --- | +| `kp_range` | `[lo, hi]` | 必填 | 每次 reset 采样 | 逐 actuator 刚度。有限且 `lo <= hi`；`log_uniform` 时必须为正。`operation: scale` 时该 pair 是默认增益的无量纲乘子（G1 owner 用 `[0.9, 1.1]`）；`operation: abs` 时是绝对增益（Go2 owner 用 `kp_range: [31.5, 38.5]`）。 | +| `kd_range` | `[lo, hi]` | 必填 | 每次 reset 采样 | 逐 actuator 阻尼，校验同 `kp_range`。Go2 owner：`kd_range: [0.45, 0.55]`，配合 `abs`。 | +| `asset_cfg` | `SceneEntityCfg` | `SceneEntityCfg("robot")` | 构造期绑定 | 用 `actuator_names`/`actuator_ids` 选择 actuator。 | +| `distribution` | `uniform` \| `log_uniform` | `uniform` | 每次 reset 采样 | 该 term 拒绝 `gaussian`。 | +| `operation` | `scale` \| `abs` | `scale` | 每次 reset 应用 | 该 term 拒绝 `add`。 | + +### `randomize_rigid_body_mass` + +随机化 body 质量（`unilab.envs.mdp.randomize_rigid_body_mass`）。支持 +`mode: reset` 与 `mode: startup`；startup 在初始化时采样一张全宽表并回放。 + +| 字段 | 类型 | 默认值/必填 | 生效时机 | 语义与合理取值线索 | +| --- | --- | --- | --- | --- | +| `asset_cfg` | `SceneEntityCfg` | 必填 | 构造期绑定 | 用 `body_names`/`body_ids` 选择 body。 | +| `mass_distribution_params` | `[lo, hi]` | 必填 | 每次 reset 采样（startup 时一次） | 使用模型的 body 质量单位：本仓库 MuJoCo MJCF 机器人为千克。FlashSAC Go2 MuJoCo owner 在 `base` 上用 `[-1.5, 1.5]` 配合 `operation: add`；SAC G1 Genesis owner 在 startup 下对 `pelvis`/`torso_link` 用 `[0.9, 1.1]` 配合 `operation: scale`。 | +| `operation` | `add` \| `scale` \| `abs` | 必填 | 每次 reset 应用 | 无默认值，必须显式声明。 | +| `distribution` | `uniform` \| `log_uniform` \| `gaussian` | `uniform` | 每次 reset 采样 | `log_uniform` 要求 bounds 为正。 | +| `recompute_inertia` | bool | `true` | 构造期 | 取 `true` 会抛 `NotImplementedError`（reset payload 没有 inertia 重算契约）；必须显式置 `false`。 | +| `min_mass` | float | `1e-6` | 每次 reset 应用 | 随机化质量的下限截断；必须有限且 `>= 1e-6`。 | + +### `randomize_body_mass_inertia` + +用一个共享的逐 body 因子同时缩放 body 质量与主惯性矩 +（`unilab.envs.mdp.randomize_body_mass_inertia`）。仅支持 `mode: reset`； +因子在首次 reset 时采样一次、按环境缓存，并在之后的 reset 重放，因此行为 +等同于 startup term。质心与主轴坐标系保持不变。 + +| 字段 | 类型 | 默认值/必填 | 生效时机 | 语义与合理取值线索 | +| --- | --- | --- | --- | --- | +| `asset_cfg` | `SceneEntityCfg` | 必填 | 构造期绑定 | 用 `body_names`/`body_ids` 选择 body；mass 与 inertia 的绑定必须覆盖同一组 body。 | +| `scale_range` | `[lo, hi]` | 必填 | 首次 reset 时采样一次 | 对数均匀因子 `s = e^{2a}`，其中 `a ~ U(ln(lo)/2, ln(hi)/2)`；bounds 必须为正。质量与主惯性矩都乘以 `s`。 | + +### `randomize_rigid_body_com` + +加性质心随机化（`unilab.envs.mdp.randomize_rigid_body_com`）。支持 +`mode: reset` 与 `mode: startup`。每次应用都会重新读取实时的 +`com_range`，因此 curriculum 可以在两次 reset 之间放宽范围。 + +| 字段 | 类型 | 默认值/必填 | 生效时机 | 语义与合理取值线索 | +| --- | --- | --- | --- | --- | +| `com_range` | `{x, y, z: [lo, hi]}` | 必填 | 每次 reset 采样（startup 时一次） | 加到默认 body 质心位置上的偏移，单位为米、在 body 坐标系下。省略的轴默认为 `(0, 0)`。SAC G1 MuJoCo owner 在 `torso_link` 上用 `x: [-0.025, 0.025]`、`y/z: [-0.05, 0.05]`；FlashSAC Go2 owner 在 `base` 上用 `x: [-0.05, 0.05]`。 | +| `asset_cfg` | `SceneEntityCfg` | 必填 | 构造期绑定 | 用 `body_names`/`body_ids` 选择 body。 | + +### `randomize_physics_scene_gravity` + +按环境随机化完整 gravity 向量 +（`unilab.envs.mdp.randomize_physics_scene_gravity`）。仅支持 +`mode: reset`。建议从较小倾斜范围开始：过大的水平重力会让早期任务不可 +学习。 + +| 字段 | 类型 | 默认值/必填 | 生效时机 | 语义与合理取值线索 | +| --- | --- | --- | --- | --- | +| `gravity_distribution_params` | `[[lo_x, lo_y, lo_z], [hi_x, hi_y, hi_z]]` | 必填 | 每次 reset 采样 | `(gx, gy, gz)` 的逐轴 bounds，单位 m/s^2。`operation: abs` 时采样向量直接覆盖默认值；FlashSAC Go2 MuJoCo owner 声明 `[[0.0, 0.0, -9.81], [0.0, 0.0, -9.81]]` 配合 `abs`（零宽度占位，展示形状）。 | +| `operation` | `add` \| `scale` \| `abs` | 必填 | 每次 reset 应用 | 无默认值，必须显式声明。 | +| `distribution` | `uniform` \| `log_uniform` \| `gaussian` | `uniform` | 每次 reset 采样 | | + +### `geom_friction` + +随机化三分量 geom friction 三元组（切向、扭转、滚动），与 pin 定的 mjlab +语义一致（`unilab.envs.mdp.geom_friction`）。仅支持 `mode: reset`。最终 +取值必须保持非负且有限。 + +| 字段 | 类型 | 默认值/必填 | 生效时机 | 语义与合理取值线索 | +| --- | --- | --- | --- | --- | +| `ranges` | pair / 按轴字典 / regex 字典 | 必填 | 每次 reset 采样 | 扁平 `[lo, hi]` 作用于所有选中轴；`{0: [lo, hi]}` 只针对一个轴；`{".*_foot.*": [lo, hi]}` 按名字用 regex `fullmatch` 分别给范围。SAC G1 MuJoCo owner 在脚部碰撞 geom 上用 `[0.3, 1.2]` 配合 `operation: abs`。 | +| `asset_cfg` | `SceneEntityCfg` | `SceneEntityCfg("robot")` | 构造期绑定 | 用 `geom_names`/`geom_ids` 选择 geom。 | +| `axes` | int 列表 | `(0,)` | 每次 reset 采样 | 合法轴为 `0, 1, 2`；默认只动切向分量。 | +| `distribution` | `uniform` \| `log_uniform` \| `gaussian` | `uniform` | 每次 reset 采样 | | +| `operation` | `add` \| `scale` \| `abs` | `abs` | 每次 reset 应用 | | +| `shared_random` | bool | `false` | 每次 reset 采样 | 为 `true` 时，bounds 相同的 geom 在每个环境内共享一次采样（SAC G1 MuJoCo owner 使用）。 | + +### `dof_armature` + +随机化关节 armature —— 用于建模齿轮传动的附加转子惯量 +（`unilab.envs.mdp.dof_armature`）。参数族与 `geom_friction` 相同，但字段 +为标量：只有轴 `0` 合法。仅支持 `mode: reset`；最终取值必须保持非负且 +有限。 + +| 字段 | 类型 | 默认值/必填 | 生效时机 | 语义与合理取值线索 | +| --- | --- | --- | --- | --- | +| `ranges` | pair / `{0: [lo, hi]}` / regex 字典 | 必填 | 每次 reset 采样 | regex 键对选中的 joint 名做 `fullmatch`。 | +| `asset_cfg` | `SceneEntityCfg` | `SceneEntityCfg("robot")` | 构造期绑定 | 用 `joint_names`/`joint_ids` 选择 joint。 | +| `axes` | int 列表 | `(0,)` | 每次 reset 采样 | 标量字段只有 `0` 合法。 | +| `distribution` | `uniform` \| `log_uniform` \| `gaussian` | `uniform` | 每次 reset 采样 | | +| `operation` | `add` \| `scale` \| `abs` | `abs` | 每次 reset 应用 | | +| `shared_random` | bool | `false` | 每次 reset 采样 | | + +### `randomize_encoder_bias` + +给 encoder 位置读数加入逐关节常值偏置 +（`unilab.envs.mdp.randomize_encoder_bias`）。仅支持 `mode: reset`。写入 +的是 entity 数据面（观测偏置），不是物理模型字段，且要求 Torch 控制平面。 + +| 字段 | 类型 | 默认值/必填 | 生效时机 | 语义与合理取值线索 | +| --- | --- | --- | --- | --- | +| `bias_range` | `[lo, hi]` | 必填 | 每次 reset 采样 | 均匀偏置，单位为弧度（关节位置单位）。SAC G1 MuJoCo owner 在全部关节上用 `[-0.01, 0.01]`。 | +| `asset_cfg` | `SceneEntityCfg` | 必填 | 构造期绑定 | 用 `joint_names`/`joint_ids` 选择 joint；至少选中一个关节。 | + +## Reset 状态 Term + +### `reset_root_state_uniform` + +把浮动基 root 重置为 backend 默认状态叠加均匀采样的 SE(3) 偏移 +（`unilab.envs.mdp.reset_root_state_uniform`）。固定基或 mocap root 会 +fail closed。 + +| 字段 | 类型 | 默认值/必填 | 生效时机 | 语义与合理取值线索 | +| --- | --- | --- | --- | --- | +| `pose_range` | `{x, y, z, roll, pitch, yaw: [lo, hi]}` | 必填 | 每次 reset 采样 | 位置偏移（米）叠加在默认 root 位置与 environment 原点之上；`roll`/`pitch`/`yaw` 偏移（弧度）与默认朝向复合。省略的键默认为 `(0, 0)`。Go2/G1 base owner 用 `x/y: [-0.5, 0.5]` 与 `yaw: [-pi, pi]`。 | +| `velocity_range` | 同上 | 可选（无偏移） | 每次 reset 采样 | 叠加在默认 root 速度上的线速度（m/s）与角速度（rad/s）偏移。Go2/G1 base owner 六个轴都用 `[-0.5, 0.5]`。 | +| `asset_cfg` | `SceneEntityCfg` | `SceneEntityCfg("robot")` | 构造期绑定 | | + +### `reset_root_state_uniform_fused` + +参数与语义同 `reset_root_state_uniform` +（`unilab.envs.mdp.reset_root_state_uniform_fused`），但在一张编译后的 +Torch 图中为所有环境采样，并要求 CUDA tensor runtime 上的 Manager RNG。 +backend owner 通过替换现有 `reset_root_state_uniform` term 的 `func` 来 +启用，例如 `src/unilab/conf/ppo/task/g1_walk_flat/mjwarp.yaml`。 + +### `reset_scene_to_default` + +`unilab.envs.mdp.reset_scene_to_default`。无参数。把所有已物化的场景实体 +重置为 backend 默认 `qpos`/`qvel`；base owner 将其声明为第一个 reset +term。 + +### `skip_scene_reset` + +`unilab.envs.mdp.skip_scene_reset`。无参数。空操作 reset 钩子，作为 +MJWarp 的规范 reset 场景 term：backend owner 把继承来的 +`reset_scene_to_default` term 的 `func` 换成它（例如 +`src/unilab/conf/ppo/task/g1_walk_flat/mjwarp.yaml`、 +`src/unilab/conf/ppo/task/go2_joystick_flat/superdex.yaml`）。 + +## Interval 与 Step Term + +### `push_by_setting_velocity` + +对 root 施加世界系速度冲击（`unilab.envs.mdp.push_by_setting_velocity`）。 +仅支持 `mode: interval`；`interval_range_s` 与 `is_global_time` 声明在 +term 上而非 `params` 内。每次应用都会重新读取实时的 `velocity_range`， +curriculum 的范围阶段因此生效。 + +| 字段 | 类型 | 默认值/必填 | 生效时机 | 语义与合理取值线索 | +| --- | --- | --- | --- | --- | +| `velocity_range` | `{x, y, z, roll, pitch, yaw: [lo, hi]}` | 必填 | 每次 interval 触发时采样 | 线速度冲击（`x`/`y`/`z`，m/s）与角速度冲击（`roll`/`pitch`/`yaw`，rad/s）都是世界系增量；省略的键默认为 `(0, 0)`。角速度轴要求 backend 的 interval 角速度能力，在构造期按初始范围绑定。FlashSAC Go2 MuJoCo owner 每 15 s 用 `x/y: [-1.0, 1.0]`、`z: [-0.5, 0.5]`（仅线速度）；SAC G1 owner 每 1-3 s 另加角速度范围（`roll/pitch: [-0.52, 0.52]`、`yaw: [-0.78, 0.78]`）。 | +| `asset_cfg` | `SceneEntityCfg` | `SceneEntityCfg("robot")` | 构造期绑定 | | + +### `apply_body_impulse` + +按逐环境的"冷却 → 触发 → 持续 → 失效"生命周期施加瞬态随机 body 力 wrench +（`unilab.envs.mdp.apply_body_impulse`）。仅支持 `mode: step`；term 每个 +step 运行并自行管理计时器。力矩通道（非零 `torque_range` 或 +`body_point_offset`）要求 backend 的 interval body-torque 能力。 + +| 字段 | 类型 | 默认值/必填 | 生效时机 | 语义与合理取值线索 | +| --- | --- | --- | --- | --- | +| `force_range` | `[lo, hi]` | 必填 | 触发时采样 | 每个力分量的均匀幅值，作用于每个选中 body 的 3 个世界系轴；模型单位（N）。 | +| `torque_range` | `[lo, hi]` | 必填 | 触发时采样 | 世界系力矩（N*m）的同样形状约定。纯力冲击用 `[0.0, 0.0]`。 | +| `duration_s` | `[min, max]` | 必填 | 触发时采样 | wrench 持续时长（秒）；必须满足 `0 <= min <= max`。激活期间每个 step 重新提交。 | +| `cooldown_s` | `[min, max]` | 必填 | 每次触发/reset 后采样 | 下一次冲击前的空闲时长（秒）；`0 <= min <= max`。初始冷却已预采样，因此 `t=0` 不会立即触发。 | +| `asset_cfg` | `SceneEntityCfg` | `SceneEntityCfg("robot")` | 构造期绑定 | 用 `body_names`/`body_ids` 选择 body。 | +| `body_point_offset` | `(x, y, z)` | 可选（无） | 触发时应用 | body 连杆坐标系下的施力点；附加 `offset x force` 诱导力矩。要求 packed body-state 读取计划。 | + +## 最小配置示例 + +DR term 声明在所选 backend owner 的 `events:` 段中，文件位于 +`src/unilab/conf//task//.yaml` —— 例如 +`src/unilab/conf/flashsac/task/go2_joystick_flat/mujoco.yaml`，以下示例 +即改编自该文件的配置段： + +```yaml +env: + events: + randomize_rigid_body_mass: + func: unilab.envs.mdp.randomize_rigid_body_mass + mode: reset + params: + asset_cfg: + _target_: SceneEntityCfg + name: robot + body_names: base + mass_distribution_params: [-1.5, 1.5] + operation: add + recompute_inertia: false + randomize_rigid_body_com: + func: unilab.envs.mdp.randomize_rigid_body_com + mode: reset + params: + asset_cfg: + _target_: SceneEntityCfg + name: robot + body_names: base + com_range: + x: [-0.05, 0.05] + push_by_setting_velocity: + func: unilab.envs.mdp.push_by_setting_velocity + mode: interval + interval_range_s: [15.0, 15.0] + is_global_time: true + params: + velocity_range: + x: [-1.0, 1.0] + y: [-1.0, 1.0] + z: [-0.5, 0.5] + roll: [0.0, 0.0] + pitch: [0.0, 0.0] + yaw: [0.0, 0.0] +``` + +用 CLI 选择 backend —— `uv run train --algo --task --sim +` —— 并把 backend 未声明支持的 term 从其 owner 中移除（或置为 +`null`），因为不支持的 term 会在构造期 fail closed。 + +## 常见问题 + +**如何给 Go2 加 friction 随机化？** +在支持该能力的 backend owner 中声明一个 `mode: reset` 的 `geom_friction` +term，用 `geom_names` 正则选中足部 geom。仓库内的模板是 SAC G1 MuJoCo +owner：`src/unilab/conf/sac/task/g1_motion_tracking/mujoco.yaml` 选中 +`geom_names: "^(left|right)_foot[1-7]_collision$"`，并用 +`ranges: [0.3, 1.2]`、`operation: abs`、`shared_random: true`。请对照 Go2 +模型的 geom 名写出等价 pattern。geom friction 是 backend 声明的能力：同一 +任务的 Genesis owner 因 Genesis 没有逐环境 friction API 而把 +`foot_friction` 置为 `null`，因此不要在未声明该能力的 backend 上启用此 +term。 + +**mass 参数的单位是什么？** +使用模型的 body 质量单位 —— 本仓库的 MuJoCo MJCF 机器人为千克。采样值 +经 `operation` 与 backend 声明的默认质量组合：`add` 加千克（FlashSAC Go2 +owner 给 `base` 加 ±1.5 kg），`scale` 乘无量纲因子（SAC G1 Genesis owner +把 `pelvis`/`torso_link` 缩放 `[0.9, 1.1]`），`abs` 直接设绝对千克数。 +结果按 `min_mass`（默认 `1e-6`）做下限截断。 + +**Motrix 不支持 gravity 时，如何写一份 Motrix 与 MuJoCo 都能跑的 owner YAML？** +按 {doc}`1-configuration`，gravity reset 随机化目前仅 MuJoCo 可用； +`MotrixBackend` 只声明支持 `base_mass_delta`、`base_com_offset`、`kp`、 +`kd` 与 interval push（见 {doc}`0-index`）。共享 term 留在任务的 +`base.yaml`，只在 `mujoco.yaml` 声明 +`randomize_physics_scene_gravity`；如果共享 base 已声明它，就在 +`motrix.yaml` 中设置 `randomize_physics_scene_gravity: null` —— 与 +`src/unilab/conf/ppo/task/g1_walk_flat/motrix.yaml` 和 `.../mjwarp.yaml` +中 `pd_gains: null` 的模式相同。若让 Motrix 保留该 term，构造期会 fail +closed 而不是被跳过。 + +这些 term 背后的 payload 级契约见 +{doc}`Domain Randomization Contract `。 diff --git a/docs/sphinx/source/zh_CN/5-reference/0-index.md b/docs/sphinx/source/zh_CN/5-reference/0-index.md index f53f6f88d..5059c0ac8 100644 --- a/docs/sphinx/source/zh_CN/5-reference/0-index.md +++ b/docs/sphinx/source/zh_CN/5-reference/0-index.md @@ -35,6 +35,12 @@ 共享的架构决策及其状态。 ::: +:::{grid-item-card} Manager Term 参考 +:link: 6-manager_terms/0-index +:link-type: doc +内置 reward / observation / termination term 及其参数。 +::: + :::: ```{toctree} @@ -46,4 +52,5 @@ 3-changelog 4-adr 5-support_matrix +6-manager_terms/0-index ``` diff --git a/docs/sphinx/source/zh_CN/5-reference/6-manager_terms/0-index.md b/docs/sphinx/source/zh_CN/5-reference/6-manager_terms/0-index.md new file mode 100644 index 000000000..778b60900 --- /dev/null +++ b/docs/sphinx/source/zh_CN/5-reference/6-manager_terms/0-index.md @@ -0,0 +1,122 @@ +# Manager Term 参考 + +UniLab 内置 Manager-Based reward、observation、termination term 的速查表， +按 owner 层组织：`unilab.envs.mdp` 下的框架级 term，以及 +`src/unilab/tasks/` 下的任务层 term。表中的名称、参数与默认值均以所引 +源码模块为准整理；权威签名以 API Reference 为准。 + +:::{only} api_ref +```{raw} html + +``` +::: + +:::{only} prose_only +纯散文预览构建会跳过 API Reference。权威签名见源码树： +[unilab/envs/mdp](https://github.com/unilabsim/UniLab/tree/main/src/unilab/envs/mdp)、 +[unilab/tasks](https://github.com/unilabsim/UniLab/tree/main/src/unilab/tasks)。 +::: + +## 如何引用 term + +term 在任务 owner YAML（例如 +`src/unilab/conf/ppo/task/go2_joystick_flat/base.yaml`）中以 +`func + params` 形式声明。`func` 是可导入的 `模块.路径.名字`；所有框架级 +term 都在 `unilab.envs.mdp.` 处重导出，owner YAML 使用的正是这个短 +路径。类形式的 term 在 manager 构建时用 `(cfg, env)` 实例化一次，之后每步 +调用；普通函数则每步直接调用。两种形式下 manager 都以 +`func(env, **params)` 调用 term +（见 `src/unilab/managers/manager_base.py`、 +`src/unilab/managers/reward_manager.py`），因此 `params:` 必须与 term 接受 +的参数一致。 + +- Reward term 声明在顶层 `reward:` 映射下，含 `func`、`weight` 与可选的 + `params`： + ```yaml + reward: + action_rate: + func: unilab.envs.mdp.action_rate_l2 + weight: -0.005 + ``` +- Observation term 声明在 `env.observations.<组名>.terms.<名称>` 下，含 + `func` 与可选的 `params`： + ```yaml + env: + observations: + policy: + terms: + joint_pos: + func: unilab.envs.mdp.joint_pos_rel + ``` +- Termination term 声明在 `env.terminations.<名称>` 下，含 `func` 与可选 + `params`；加上 `time_out: true` 表示该 term 计入回合超时（截断）而非失败 + （见 `src/unilab/managers/termination_manager.py` 的 + `TerminationTermCfg.time_out`）： + ```yaml + env: + terminations: + time_out: + func: unilab.envs.mdp.time_out + time_out: true + ``` + +依赖后端能力的 term——具名传感器、射线投射器、reset 字段——在构造期绑定 +该能力，后端不具备时直接抛错（fail-closed），不会静默降级（机制见 +`src/unilab/envs/mdp/observations.py` 的 `bind_sensor_data` 绑定与 +`src/unilab/envs/mdp/events.py` 的 `_ResetRandomizationTerm` 约定）。 + +## 范围说明 + +本表只覆盖 reward、observation、termination 三类 term。任务包还在其他 +YAML 段中配置了下列 term 类型，未列入表中： + +- Command 与 action term，例如 + `unilab.envs.mdp.commands.UniformVelocityCommandCfg`、 + `unilab.tasks.locomotion.g1.manager_terms.G1VelocityCommandCfg`（带 0.2 + 平面模长死区的 G1 速度指令）、 + `unilab.tasks.motion_tracking.common.manager_terms.MotionCommandCfg` / + `MotionJointPositionActionCfg`（运动参考指令及其关节位置动作）。 +- Curriculum term，例如 + `unilab.tasks.locomotion.g1.manager_terms.G1PenaltyCurriculum`（声明在 + `curriculum:` 段；当被跟踪的平均回合长度越过 + `level_down_threshold` / `level_up_threshold` 时，按 `degree` 步进在 + `initial_scale`/`min_scale` 与 `max_scale` 之间缩放负权重 reward 项）。 +- 事件（域随机化与 reset）term，见 `unilab.envs.mdp.events`，声明在 + `env.events` 段。 +- Reset owner，例如 + `unilab.tasks.locomotion.g1.manager_terms.G1WalkResetOwner`。 + +::::{grid} 1 1 2 2 +:gutter: 3 + +:::{grid-item-card} Reward term +:link: 1-rewards +:link-type: doc +框架级、locomotion、G1 与 motion tracking 的 reward term 及参数。 +::: + +:::{grid-item-card} Observation term +:link: 2-observations +:link-type: doc +状态、传感器、指令、步态相位与射线扫描观测 term。 +::: + +:::{grid-item-card} Termination term +:link: 3-terminations +:link-type: doc +超时与失败终止条件，含融合终止包。 +::: + +:::: + +```{toctree} +:hidden: + +1-rewards +2-observations +3-terminations +``` diff --git a/docs/sphinx/source/zh_CN/5-reference/6-manager_terms/1-rewards.md b/docs/sphinx/source/zh_CN/5-reference/6-manager_terms/1-rewards.md new file mode 100644 index 000000000..334a0c7b3 --- /dev/null +++ b/docs/sphinx/source/zh_CN/5-reference/6-manager_terms/1-rewards.md @@ -0,0 +1,424 @@ +# Reward Term + +内置 reward term，由源码整理。所有 term 每个环境返回一个 float32 标量； +reward manager 会乘上 YAML 中的 `weight`。框架级 term 在 +`unilab.envs.mdp.` 处重导出。 + +:::{only} api_ref +```{raw} html + +``` +::: + +:::{only} prose_only +权威签名见源码树： +[unilab/envs/mdp](https://github.com/unilabsim/UniLab/tree/main/src/unilab/envs/mdp)、 +[unilab/tasks](https://github.com/unilabsim/UniLab/tree/main/src/unilab/tasks)。 +::: + +## 框架级（`unilab.envs.mdp.rewards`） + +源码：`src/unilab/envs/mdp/rewards.py`。 + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - 函数 / 类 + - 参数 + - 语义 +* - `is_alive` + - `unilab.envs.mdp.rewards.is_alive`（函数） + - — + - 环境未发生非超时终止时恒为 1.0。 +* - `is_terminated` + - `unilab.envs.mdp.rewards.is_terminated`（函数） + - — + - 非超时终止时为 1.0；配合负权重惩罚失败。 +* - `root_height` + - `unilab.envs.mdp.rewards.root_height`（函数） + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 世界系根高度（z），作为标量奖励或度量。 +* - `joint_vel_l2` + - `unilab.envs.mdp.rewards.joint_vel_l2`（函数） + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 所选关节速度的平方和。 +* - `action_rate_l2` + - `unilab.envs.mdp.rewards.action_rate_l2`（函数） + - — + - 原始策略动作一阶差分的平方和。 +* - `action_acc_l2` + - `unilab.envs.mdp.rewards.action_acc_l2`（函数） + - — + - 原始策略动作二阶差分的平方和。 +* - `flat_orientation_l2` + - `unilab.envs.mdp.rewards.flat_orientation_l2`（函数） + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 世界重力向量在根 body 系下 xy 分量的平方和；惩罚机身不水平。 +* - `upright` + - `unilab.envs.mdp.rewards.upright`（函数） + - `std: float`（必填，> 0）； + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 高斯奖励 `exp(-||pg_xy||^2 / std^2)`，用于保持单个所选 body 竖直； + 要求恰好选中一个 body。仅平地形式。 +* - `track_linear_velocity` + - `unilab.envs.mdp.rewards.track_linear_velocity`（函数） + - `std: float`（必填，> 0）；`command_name: str`（必填）； + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - `exp(-(||cmd_xy - v_xy||^2 + v_z^2) / std^2)`，速度取机体系；假设 + z 向指令为零。 +* - `track_angular_velocity` + - `unilab.envs.mdp.rewards.track_angular_velocity`（函数） + - `std: float`（必填，> 0）；`command_name: str`（必填）； + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - `exp(-((cmd_yaw - w_z)^2 + ||w_xy||^2) / std^2)`；奖励 yaw 角速度 + 跟踪，同时抑制 roll/pitch 角速度。 +* - `body_angular_velocity_penalty` + - `unilab.envs.mdp.rewards.body_angular_velocity_penalty`（函数） + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 单个所选 body 的 roll/pitch 角速度平方和；要求恰好选中一个 body。 +* - `joint_pos_limits` + - `unilab.envs.mdp.rewards.joint_pos_limits`（函数） + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 所选关节超出软限位的线性（未平方）违例量之和。 +* - `posture` + - `unilab.envs.mdp.rewards.posture`（类） + - `std: dict[str, float]`（必填；关节名正则 → 逐关节 std）； + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 对所选关节计算 `exp(-mean((q - q_default)^2 / std^2))`。 +* - `variable_posture` + - `unilab.envs.mdp.rewards.variable_posture`（类） + - `std_standing` / `std_walking` / `std_running: dict[str, float]` + （必填）；`command_name: str`（必填）； + `walking_threshold: float = 0.5`；`running_threshold: float = 1.5`； + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 带速度自适应容差的 `posture`：按总指令速度（平面模长加 yaw 绝对值） + 在站立/行走/奔跑三档 std 表之间选择。 +``` + +## Locomotion —— 通用 term + +### `unilab.tasks.locomotion.common.manager_terms` + +源码：`src/unilab/tasks/locomotion/common/manager_terms.py`。 + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - 函数 / 类 + - 参数 + - 语义 +* - `base_height_l2` + - `unilab.tasks.locomotion.common.manager_terms.base_height_l2`（类） + - `target_height: float`（必填，m）； + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 世界系根高度与 `target_height` 的平方误差。 +* - `alive` + - `unilab.tasks.locomotion.common.manager_terms.alive`（函数） + - — + - 每步恒定 1.0，无条件。 +* - `joint_deviation_l1` + - `unilab.tasks.locomotion.common.manager_terms.joint_deviation_l1`（类） + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 所选关节相对默认位姿的绝对偏差之和（L1）。 +* - `stand_still_l1` + - `unilab.tasks.locomotion.common.manager_terms.stand_still_l1`（类） + - `command_name: str`（必填）；`command_threshold: float = 0.1`； + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 仅在指令模长低于 `command_threshold` 时计入的 + `joint_deviation_l1`。 +* - `feet_phase_contact` + - `unilab.tasks.locomotion.common.manager_terms.feet_phase_contact` + （类） + - `sensor_names: list[str]`（必填；4 个互不相同的接触传感器）； + `frequency: float = 2.0`（Hz）； + `phase_offsets: list[float] = (0.0, 0.5, 0.5, 0.0)`； + `contact_threshold: float = 0.1`；`stance_threshold: float = 0.6`； + `command_name: str | None = None`；`command_threshold: float = 0.0` + - 实测接触与步态相位期望站立段一致的脚所占比例。 +* - `feet_phase_swing_height` + - `unilab.tasks.locomotion.common.manager_terms.feet_phase_swing_height` + （类） + - `sensor_names: list[str]`（必填；4 个 3 维位置传感器）； + `frequency: float = 2.0`；`phase_offsets = (0.0, 0.5, 0.5, 0.0)`； + `target_height: float = 0.1`（m）；`kernel: float = 0.01`（> 0）； + `swing_start: float = 0.6`；`command_name: str | None = None`； + `command_threshold: float = 0.0` + - 摆动相内对每只脚给 `exp(-(z - target_height)^2 / kernel)`，再对脚 + 取平均。 +* - `feet_air_while_standing` + - `unilab.tasks.locomotion.common.manager_terms.feet_air_while_standing` + （类） + - `sensor_names: list[str]`（必填；4 个接触传感器）； + `command_name: str`（必填）；`command_threshold: float = 0.1`； + `contact_threshold: float = 0.1` + - 指令模长不超过 `command_threshold`（站立）时统计无接触的脚数。 +``` + +### `unilab.tasks.locomotion.common.gait_terms` + +源码：`src/unilab/tasks/locomotion/common/gait_terms.py`。带 +`sensor_groups` 的 term 为每只脚绑定一组具名接触传感器；组内任一传感器 +超过 `contact_threshold` 即视为该脚接触。 + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - 函数 / 类 + - 参数 + - 语义 +* - `feet_air_time` + - `unilab.tasks.locomotion.common.gait_terms.feet_air_time`（类） + - `sensor_groups: list[list[str]]`（必填；每脚一组）； + `contact_threshold: float = 0.0`；`threshold_min: float = 0.05`（s）； + `threshold_max: float = 0.5`（s，须大于 `threshold_min`）； + `command_name: str | None = None`；`command_threshold: float = 0.5` + - 统计当前腾空时间落在奖励窗口内的脚数；设置 `command_name` 时按指令 + 门控。 +* - `feet_swing_height` + - `unilab.tasks.locomotion.common.gait_terms.feet_swing_height`（类） + - `sensor_groups: list[list[str]]`（必填）；`target_height: float` + （必填，> 0，m）；`command_name: str`（必填）； + `command_threshold: float`（必填）；`contact_threshold: float = 0.0`； + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")`（每个传感器组 + 对应一个脚部 body） + - 腾空期间跟踪峰值脚高；首次着地时对每只脚计 + `(peak / target_height - 1)^2`。 +* - `feet_slip` + - `unilab.tasks.locomotion.common.gait_terms.feet_slip`（类） + - `sensor_groups: list[list[str]]`（必填）；`command_name: str` + （必填）；`command_threshold: float = 0.01`； + `contact_threshold: float = 0.0`； + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 接触状态下脚部 xy 速度平方和（打滑惩罚）。 +* - `feet_clearance` + - `unilab.tasks.locomotion.common.gait_terms.feet_clearance`（类） + - `target_height: float`（必填，m）； + `command_name: str | None = None`； + `command_threshold: float = 0.01`； + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")`（脚部 body） + - 对每只脚累加 `|z - target_height|` 并按平面脚速加权。 +* - `self_collision_cost` + - `unilab.tasks.locomotion.common.gait_terms.self_collision_cost`（类） + - `sensor_groups: list[list[str]]`（必填）； + `contact_threshold: float = 0.0` + - 报告接触的自碰撞传感器组数量。 +* - `angular_momentum_penalty` + - `unilab.tasks.locomotion.common.gait_terms.angular_momentum_penalty` + （类） + - `sensor_name: str`（必填；一个 3 维传感器） + - 从该具名传感器读取的整机角动量平方范数。 +``` + +### `unilab.tasks.locomotion.common.sensor_reward_terms` + +源码：`src/unilab/tasks/locomotion/common/sensor_reward_terms.py`。每个 +term 绑定机器人的一个具名 3 维 XML 传感器。 + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - 函数 / 类 + - 参数 + - 语义 +* - `track_lin_vel` + - `unilab.tasks.locomotion.common.sensor_reward_terms.track_lin_vel` + （类） + - `sensor_name: str`（必填；线速度传感器）； + `tracking_sigma: float = 0.25`（> 0）；`command_name: str = "twist"` + - `exp(-||cmd_xy - v_xy||^2 / tracking_sigma)`。 +* - `track_ang_vel` + - `unilab.tasks.locomotion.common.sensor_reward_terms.track_ang_vel` + （类） + - `sensor_name: str`（必填；陀螺仪传感器）； + `tracking_sigma: float = 0.25`（> 0）；`command_name: str = "twist"` + - `exp(-(cmd_yaw - w_z)^2 / tracking_sigma)`。 +* - `lin_vel_z` + - `unilab.tasks.locomotion.common.sensor_reward_terms.lin_vel_z`（类） + - `sensor_name: str`（必填） + - 竖直方向（z）线速度平方。 +* - `ang_vel_xy` + - `unilab.tasks.locomotion.common.sensor_reward_terms.ang_vel_xy`（类） + - `sensor_name: str`（必填） + - roll/pitch 角速度平方和。 +* - `orientation` + - `unilab.tasks.locomotion.common.sensor_reward_terms.orientation`（类） + - `sensor_name: str`（必填；朝上向量传感器） + - 朝上向量水平分量平方和；惩罚偏离竖直。 +``` + +## Locomotion —— G1（`unilab.tasks.locomotion.g1.manager_terms`） + +源码：`src/unilab/tasks/locomotion/g1/manager_terms.py`。步态类 term 共享 +参数 `frequency: float = 1.5`（Hz）、`init_mode: str = "offset_phase"` +（`"offset_phase"` 或 `"independent"`）、`swing_height: float = 0.09` +（m）、`tracking_sigma: float = 0.008`（> 0）、 +`min_forward_speed: float = 0.0`、`command_name: str = "twist"`；并绑定具名 +传感器 `left_foot_pos` / `right_foot_pos` 与 `pelvis_local_linvel`（接触类 +term 额外绑定 `left_foot_contact_0..3` / `right_foot_contact_0..3`）。同一 +env 上的步态 term 必须共享 `frequency` 与 `init_mode`。 + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - 函数 / 类 + - 参数 + - 语义 +* - `feet_phase` + - `unilab.tasks.locomotion.g1.manager_terms.feet_phase`（类） + - 共享步态参数（见上） + - 相对三次 Bézier 逐脚高度目标计算 + `exp(-(err_left^2 + err_right^2) / tracking_sigma)`，并按前向速度 + 门控。 +* - `feet_phase_contrast` + - `unilab.tasks.locomotion.g1.manager_terms.feet_phase_contrast`（类） + - 共享步态参数 + - `exp(-((h_left - h_right) - (t_left - t_right))^2 / tracking_sigma)`； + 奖励左右脚高差与相位目标的对比一致。 +* - `feet_phase_contact` + - `unilab.tasks.locomotion.g1.manager_terms.feet_phase_contact`（类） + - 共享步态参数（见上） + - 实测接触与期望站立相位逐脚匹配率的均值，按前向速度门控。 +* - `feet_double_stance` + - `unilab.tasks.locomotion.g1.manager_terms.feet_double_stance`（类） + - 共享步态参数（见上） + - 前向指令激活时惩罚双脚同时着地。 +* - `feet_air_time` + - `unilab.tasks.locomotion.g1.manager_terms.feet_air_time`（类） + - 共享步态参数（见上） + - 统计当前腾空时间落在硬编码 (0.05, 0.5) s 窗口内的脚数。 +* - `forward_progress` + - `unilab.tasks.locomotion.g1.manager_terms.forward_progress`（类） + - `command_name: str = "twist"` + - `min(clamp(v_x, min=0) / clamp(cmd_x, min=1e-6), 1)`；相对指令速度的 + 前向进度。 +* - `under_speed` + - `unilab.tasks.locomotion.g1.manager_terms.under_speed`（类） + - `command_name: str = "twist"` + - 相对欠速量 `clamp(cmd_x - clamp(v_x, min=0), min=0) / + clamp(cmd_x, min=1e-6)`。 +* - `penalty_feet_ori` + - `unilab.tasks.locomotion.g1.manager_terms.penalty_feet_ori`（类） + - — + - 两只脚坐标系四元数 x、y 分量的平方和；惩罚脚底不水平。 +* - `penalty_close_feet_xy` + - `unilab.tasks.locomotion.g1.manager_terms.penalty_close_feet_xy`（类） + - `threshold: float = 0.15`（m） + - 双脚水平距离小于 `threshold` 时计 `(dist_xy - threshold)^2`。 +* - `weighted_pose` + - `unilab.tasks.locomotion.g1.manager_terms.weighted_pose`（类） + - `pose_weights: list[float]`（必填；与所选关节一一对应）； + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 关节相对默认位姿的加权平方偏差和。 +* - `upper_body_pose` + - `unilab.tasks.locomotion.g1.manager_terms.upper_body_pose`（类） + - `pose_weights: list[float]`（必填，至少 12 项）； + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 将前 12 个（腿部）权重置零后的 `weighted_pose`。 +* - `G1WalkRewardPack` + - `unilab.tasks.locomotion.g1.manager_terms.G1WalkRewardPack`（类；配置 + `G1WalkRewardPackCfg`） + - `command_name: str = "twist"`；`entity_name: str = "robot"`； + `tracking_lin_sigma: float = 0.25`； + `tracking_ang_sigma: float = 0.25`；`gait_frequency: float = 1.5`； + `gait_init_mode: str = "offset_phase"`； + `swing_height: float = 0.09`；`feet_phase_sigma: float = 0.04`； + `min_forward_speed: float = 0.0`； + `pose_weights: tuple[float, ...] = 29 个 1.0` + - 一次张量读取内融合计算 G1 行走奖励族：线速度/角速度跟踪、角速度 xy + 与姿态惩罚、动作变化率、加权位姿、脚部姿态惩罚、门控相位跟踪与 + alive。以 `_target_` + `func` 声明并经 `reward_pack_names` 接线 + （见 `src/unilab/conf/sac/task/g1_walk_flat/mjwarp.yaml`）。 +``` + +## Motion tracking（`unilab.tasks.motion_tracking`） + +源码：`src/unilab/tasks/motion_tracking/common/manager_terms.py` 与 +`src/unilab/tasks/motion_tracking/g1/manager_terms.py`。所有 term 按 +`command_name` 解析一个 `MotionCommand` term；`std` 参数必须为正。 + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - 函数 / 类 + - 参数 + - 语义 +* - `motion_global_anchor_position_error_exp` + - `unilab.tasks.motion_tracking.common.manager_terms.motion_global_anchor_position_error_exp`（函数） + - `command_name: str`（必填）；`std: float`（必填，> 0） + - 世界系下 `exp(-||p_anchor_motion - p_anchor_robot||^2 / std^2)`。 +* - `motion_global_anchor_orientation_error_exp` + - `unilab.tasks.motion_tracking.common.manager_terms.motion_global_anchor_orientation_error_exp`（函数） + - `command_name: str`（必填）；`std: float`（必填，> 0） + - `exp(-theta^2 / std^2)`，theta 为动作与机器人 anchor 姿态间的最短 + 路径四元数夹角。 +* - `MotionRelativeBodyPositionErrorExp` + - `unilab.tasks.motion_tracking.common.manager_terms.MotionRelativeBodyPositionErrorExp`（类） + - `command_name: str`（必填）；`std: float`（必填，> 0）； + `body_names: tuple[str, ...] | None = None`（指令所跟踪 body 的 + 子集） + - 对 anchor 相对系下各 body 位置误差计算 + `exp(-sum_b ||dp_b||^2 / (N * std^2))`。 +* - `MotionRelativeBodyOrientationErrorExp` + - `unilab.tasks.motion_tracking.common.manager_terms.MotionRelativeBodyOrientationErrorExp`（类） + - `command_name: str`（必填）；`std: float`（必填，> 0）； + `body_names: tuple[str, ...] | None = None` + - 对 anchor 相对系下各 body 姿态误差计算 + `exp(-sum_b theta_b^2 / (N * std^2))`。 +* - `MotionGlobalBodyLinearVelocityErrorExp` + - `unilab.tasks.motion_tracking.common.manager_terms.MotionGlobalBodyLinearVelocityErrorExp`（类） + - `command_name: str`（必填）；`std: float`（必填，> 0）； + `body_names: tuple[str, ...] | None = None` + - 对世界系各 body 线速度误差计算 + `exp(-sum_b ||dv_b||^2 / (N * std^2))`。 +* - `MotionGlobalBodyAngularVelocityErrorExp` + - `unilab.tasks.motion_tracking.common.manager_terms.MotionGlobalBodyAngularVelocityErrorExp`（类） + - `command_name: str`（必填）；`std: float`（必填，> 0）； + `body_names: tuple[str, ...] | None = None` + - 对世界系各 body 角速度误差计算 + `exp(-sum_b ||dw_b||^2 / (N * std^2))`。 +* - `motion_joint_position_error_exp` + - `unilab.tasks.motion_tracking.common.manager_terms.motion_joint_position_error_exp`（函数） + - `command_name: str`（必填）；`std: float`（必填，> 0） + - `exp(-mean((q_motion - q_robot)^2) / std^2)`。 +* - `motion_joint_velocity_error_exp` + - `unilab.tasks.motion_tracking.common.manager_terms.motion_joint_velocity_error_exp`（函数） + - `command_name: str`（必填）；`std: float`（必填，> 0） + - `exp(-mean((dq_motion - dq_robot)^2) / std^2)`。 +* - `joint_pos_limits` + - `unilab.tasks.motion_tracking.common.manager_terms.joint_pos_limits`（函数） + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 软限位违例量的平方和；任务局部实现，与框架级同名线性项不同。 +* - `UndesiredBodyContacts` + - `unilab.tasks.motion_tracking.common.manager_terms.UndesiredBodyContacts`（类） + - `command_name: str`（必填）；`threshold: float`（必填，m）； + `body_names: tuple[str, ...] | None = None` + - 统计世界系 z 低于 `threshold` 的所选 body 数量。 +* - `joint_acc_l2` + - `unilab.tasks.motion_tracking.g1.manager_terms.joint_acc_l2`（类） + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 有限差分关节加速度 `(dq_t - dq_t-1) / env.step_dt` 的平方和；带 + reset 感知。当前未被随仓库发布的 owner YAML 引用。 +* - `joint_torque_l2` + - `unilab.tasks.motion_tracking.g1.manager_terms.joint_torque_l2`（类） + - `action_name: str = "joint_pos"`； + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 位置控制器力矩估计 `kp * (target - q) - kd * dq` 的平方和；要求 + 存在目标顺序与执行器顺序一致的 `MotionJointPositionAction`。当前 + 未被随仓库发布的 owner YAML 引用。 +``` diff --git a/docs/sphinx/source/zh_CN/5-reference/6-manager_terms/2-observations.md b/docs/sphinx/source/zh_CN/5-reference/6-manager_terms/2-observations.md new file mode 100644 index 000000000..8f9e0f7ce --- /dev/null +++ b/docs/sphinx/source/zh_CN/5-reference/6-manager_terms/2-observations.md @@ -0,0 +1,290 @@ +# Observation Term + +内置 observation term，由源码整理。框架级 term 在 +`unilab.envs.mdp.` 处重导出。 + +:::{only} api_ref +```{raw} html + +``` +::: + +:::{only} prose_only +权威签名见源码树： +[unilab/envs/mdp](https://github.com/unilabsim/UniLab/tree/main/src/unilab/envs/mdp)、 +[unilab/tasks](https://github.com/unilabsim/UniLab/tree/main/src/unilab/tasks)。 +::: + +## 框架级（`unilab.envs.mdp.observations`） + +源码：`src/unilab/envs/mdp/observations.py`。读取物理状态的 term 走 +Manager 打包的 Torch read plan；传感器 term 在构造期绑定一个已存在的具名 +后端传感器，后端不具备该能力时直接抛错（fail-closed）。 + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - 函数 / 类 + - 参数 + - 语义 +* - `base_lin_vel` + - `unilab.envs.mdp.observations.base_lin_vel`（函数） + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 根线速度在根 body 系下的表示（3 维）。 +* - `base_ang_vel` + - `unilab.envs.mdp.observations.base_ang_vel`（函数） + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 根角速度在根 body 系下的表示（3 维）。 +* - `projected_gravity` + - `unilab.envs.mdp.observations.projected_gravity`（函数） + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 世界重力向量在根 body 系下的表示（3 维）。 +* - `joint_pos_rel` + - `unilab.envs.mdp.observations.joint_pos_rel`（函数） + - `biased: bool = False`； + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 关节位置相对默认位姿；`biased=True` 时额外加上逐关节编码器偏置。 +* - `joint_vel_rel` + - `unilab.envs.mdp.observations.joint_vel_rel`（函数） + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 关节速度相对默认关节速度。 +* - `last_action` + - `unilab.envs.mdp.observations.last_action`（函数） + - `action_name: str | None = None` + - 动作管理器的原始动作；设置 `action_name` 时只取该具名动作 term。 +* - `generated_commands` + - `unilab.envs.mdp.observations.generated_commands`（函数） + - `command_name: str`（必填） + - 具名指令 term 的当前指令向量。 +* - `builtin_sensor` + - `unilab.envs.mdp.observations.builtin_sensor`（类） + - `sensor_name: str`（必填）；`entity_name: str = "robot"` + - 通过缓存视图读取一个已存在的后端传感器。 +* - `projected_gravity_from_sensor` + - `unilab.envs.mdp.observations.projected_gravity_from_sensor`（类） + - `sensor_name: str`（必填；3 维朝上向量传感器）； + `entity_name: str = "robot"` + - 朝上向量传感器读数取负，得到投影重力（3 维）。 +* - `base_ang_vel_imu_misaligned` + - `unilab.envs.mdp.observations.base_ang_vel_imu_misaligned`（类） + - `max_angle_deg: float = 1.0`（度）； + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 基座角速度叠加逐 env 恒定的 IMU 安装偏差旋转；偏差由 Manager RNG + 采样一次并在整个运行期间固定。 +* - `projected_gravity_imu_misaligned` + - `unilab.envs.mdp.observations.projected_gravity_imu_misaligned`（类） + - `max_angle_deg: float = 1.0`（度）； + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 投影重力叠加与同一 env、同一角度下 + `base_ang_vel_imu_misaligned` 相同的 IMU 安装偏差。 +``` + +## 射线扫描（`unilab.envs.mdp.ray_observation`） + +源码：`src/unilab/envs/mdp/ray_observation.py`。`RayScan` 绑定一个 +UniSim 张量射线投射器（工厂名见 `RayPatternCfg.caster`，默认 +`"uni_ray"`），挂接在场景 body 或 site 上，每步按声明的图样投射并返回逐 +射线距离；`include_normal` / `include_hit_point` 各自为每条射线追加 3 个 +值并展平。投射器能力在读阶段绑定时检查，不支持的特性 fail-closed。当前 +没有随仓库发布的 owner YAML 引用 `RayScan`；覆盖见 +`tests/envs/mdp/test_ray_observation.py`。 + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - 函数 / 类 + - 参数 + - 语义 +* - `RayScan` + - `unilab.envs.mdp.ray_observation.RayScan`（类） + - `ray_pattern: RayPatternCfg`（必填） + - 按配置图样返回逐射线距离（`ray_count` 个值；启用时追加法向 / 命中 + 点）。 +``` + +`RayPatternCfg`（冻结 dataclass）声明挂接方式与图样： + +```{list-table} +:header-rows: 1 +:widths: 30 70 + +* - 字段 + - 类型 / 默认值 +* - `entity_name` + - `str`（必填）；挂接 body 所属的 entity +* - `pattern` + - `RayRingPatternCfg | RayGridPatternCfg | RayCameraPatternCfg`（必填） +* - `max_distance` + - `float`（必填，> 0） +* - `attachment_body_name` / `attachment_site_name` + - `str | None`；两者必须恰好设置其一 +* - `alignment` + - `"body" | "yaw" | "world"`，默认 `"body"` +* - `caster` + - `str`，默认 `"uni_ray"`（UniSim 公开投射器工厂名） +* - `sensor_offset` + - 3 元浮点序列，默认 `(0.0, 0.0, 0.0)` +* - `min_distance` + - `float = 0.0`（满足 `0 <= min_distance < max_distance`） +* - `include_normal` / `include_hit_point` + - `bool = False` +* - `sync_geometry` + - `bool = False`；reset 时重新同步后端持有的几何 +* - `allowed_geom_names` / `allowed_tags` / `excluded_body_names` / + `excluded_ancestor_body_names` + - `Sequence[str] | None = None`；冷路径解析的静态射线过滤器 +* - `caster_options` + - `dict`，默认 `{}`；透传给投射器工厂 +``` + +图样配置（均为冻结 dataclass；除注明外角度均为弧度）： + +```{list-table} +:header-rows: 1 +:widths: 22 40 38 + +* - 图样 + - 字段 + - 语义 +* - `RayRingPatternCfg` + - `num_rays: int`（必填，> 0）；`start: float`（必填）； + `stop: float`（必填）；`pitch: float = 0.0` + - body 局部系下一段 yaw 圆弧上的射线，共享同一俯仰角。 +* - `RayGridPatternCfg` + - `size: (float, float)`（必填，> 0）； + `resolution: float | (float, float)`（必填，> 0）； + `offset: (float, float) = (0.0, 0.0)`； + `direction: (float, float, float) = (0.0, 0.0, -1.0)`（非零） + - 局部二维含端点网格上的平行射线。 +* - `RayCameraPatternCfg` + - `width: int`（必填，> 0）；`height: int`（必填，> 0）； + `fovy: float`（必填，角度制，范围 (0, 180)）； + `forward: (float, float, float) = (1.0, 0.0, 0.0)`； + `up: (float, float, float) = (0.0, 0.0, 1.0)`（须与 `forward` + 正交） + - 显式局部系下针孔相机的单位射线方向。 +``` + +## Locomotion —— 通用 term + +`quadruped_gait_phase` 定义于 +`src/unilab/tasks/locomotion/common/manager_terms.py`；四个特权（critic） +逐脚观测 term 定义于 `src/unilab/tasks/locomotion/common/gait_terms.py`， +对应 mjlab velocity 任务 critic 的观测。 + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - 函数 / 类 + - 参数 + - 语义 +* - `quadruped_gait_phase` + - `unilab.tasks.locomotion.common.manager_terms.quadruped_gait_phase` + （类） + - `frequency: float = 2.0`（Hz）； + `phase_offsets: list[float] = (0.0, 0.5, 0.5, 0.0)`； + `command_name: str | None = None`；`command_threshold: float = 0.0` + - 四脚相位观测，取值 [0, 1)，对角小跑步态顺序；设置 `command_name` + 后仅当指令模长超过 `command_threshold` 时推进。 +* - `foot_height` + - `unilab.tasks.locomotion.common.gait_terms.foot_height`（类） + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")`（脚部 body） + - 世界系脚部 body 高度 z，形状 `(num_envs, num_feet)`；平地上精确。 +* - `foot_air_time` + - `unilab.tasks.locomotion.common.gait_terms.foot_air_time`（类） + - `sensor_groups: list[list[str]]`（必填）； + `contact_threshold: float = 0.0` + - 当前逐脚腾空时间（秒），形状 `(num_envs, num_feet)`。 +* - `foot_contact` + - `unilab.tasks.locomotion.common.gait_terms.foot_contact`（类） + - `sensor_groups: list[list[str]]`（必填）； + `contact_threshold: float = 0.0` + - 逐脚接触标志（浮点），形状 `(num_envs, num_feet)`。 +* - `foot_contact_forces` + - `unilab.tasks.locomotion.common.gait_terms.foot_contact_forces`（类） + - `sensor_groups: list[list[str]]`（必填；3 维力传感器）； + `contact_threshold: float = 0.0` + - 对数压缩的逐脚接触力 `sign(f) * log1p(|f|)`，形状 + `(num_envs, 3 * num_feet)`。 +``` + +## Locomotion —— G1（`unilab.tasks.locomotion.g1.manager_terms`） + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - 函数 / 类 + - 参数 + - 语义 +* - `G1GaitPhase` + - `unilab.tasks.locomotion.g1.manager_terms.G1GaitPhase`（类） + - `frequency: float = 1.5`（Hz）； + `init_mode: str = "offset_phase"`（`"offset_phase"` 或 + `"independent"`） + - 双脚步态相位（弧度，[0, 2 pi)），形状 `(num_envs, 2)`；每次 reset + 重采样，并与 G1 步态奖励 term 共享。 +``` + +## Motion tracking（`unilab.tasks.motion_tracking.common.manager_terms`） + +源码：`src/unilab/tasks/motion_tracking/common/manager_terms.py`。所有 +term 按 `command_name` 解析 `MotionCommand` term，并读取其驻留设备的 +carrier。 + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - 函数 / 类 + - 参数 + - 语义 +* - `motion_anchor_pos_b` + - `unilab.tasks.motion_tracking.common.manager_terms.motion_anchor_pos_b`（函数） + - `command_name: str`（必填） + - 动作 anchor 相对机器人 anchor 的位置，在机器人 anchor 系下表示 + （3 维）。 +* - `motion_anchor_ori_b` + - `unilab.tasks.motion_tracking.common.manager_terms.motion_anchor_ori_b`（函数） + - `command_name: str`（必填） + - anchor 相对姿态（动作相对机器人）的 6 维旋转表示。 +* - `robot_body_pos_b` + - `unilab.tasks.motion_tracking.common.manager_terms.robot_body_pos_b`（函数） + - `command_name: str`（必填） + - 机器人各 body 位置在机器人 anchor 系下表示，展平为 + `(num_envs, 3 * num_bodies)`。 +* - `robot_body_ori_b` + - `unilab.tasks.motion_tracking.common.manager_terms.robot_body_ori_b`（函数） + - `command_name: str`（必填） + - 机器人各 body 姿态在机器人 anchor 系下的 6 维旋转表示，展平为 + `(num_envs, 6 * num_bodies)`。 +* - `motion_joint_pos_rel` + - `unilab.tasks.motion_tracking.common.manager_terms.motion_joint_pos_rel`（函数） + - `command_name: str`（必填） + - 关节位置相对默认位姿与逐回合 `joint_default_bias`。 +* - `motion_joint_pos_rel_biased` + - `unilab.tasks.motion_tracking.common.manager_terms.motion_joint_pos_rel_biased`（函数） + - `command_name: str`（必填） + - 在 `motion_joint_pos_rel` 基础上加入逐关节编码器偏置。 +* - `MotionAnchorPositionObservation` + - `unilab.tasks.motion_tracking.common.manager_terms.MotionAnchorPositionObservation`（类） + - `command_name: str = "motion"` + - 张量原生 anchor 相对动作位置（3 维），由打包场景 body 读取提供。 +* - `MotionAnchorOrientationObservation` + - `unilab.tasks.motion_tracking.common.manager_terms.MotionAnchorOrientationObservation`（类） + - `command_name: str = "motion"` + - 张量原生 anchor 相对动作姿态（6 维旋转表示），由打包场景 body + 读取提供。 +``` diff --git a/docs/sphinx/source/zh_CN/5-reference/6-manager_terms/3-terminations.md b/docs/sphinx/source/zh_CN/5-reference/6-manager_terms/3-terminations.md new file mode 100644 index 000000000..c8c4e8bf6 --- /dev/null +++ b/docs/sphinx/source/zh_CN/5-reference/6-manager_terms/3-terminations.md @@ -0,0 +1,123 @@ +# Termination Term + +内置 termination term，由源码整理。所有 term 每个环境返回一个布尔值。 +框架级 term 在 `unilab.envs.mdp.` 处重导出。 + +:::{only} api_ref +```{raw} html + +``` +::: + +:::{only} prose_only +权威签名见源码树： +[unilab/envs/mdp](https://github.com/unilabsim/UniLab/tree/main/src/unilab/envs/mdp)、 +[unilab/tasks](https://github.com/unilabsim/UniLab/tree/main/src/unilab/tasks)。 +::: + +在 owner YAML 中以 `time_out: true` 声明的 term 计入回合超时（截断）而非 +失败；见 `src/unilab/managers/termination_manager.py` 的 +`TerminationTermCfg.time_out`。 + +## 框架级（`unilab.envs.mdp.terminations`） + +源码：`src/unilab/envs/mdp/terminations.py`。 + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - 函数 / 类 + - 参数 + - 语义 +* - `time_out` + - `unilab.envs.mdp.terminations.time_out`（函数） + - — + - 回合长度达到上限时终止；声明时加 `time_out: true`。 +* - `bad_orientation` + - `unilab.envs.mdp.terminations.bad_orientation`（函数） + - `limit_angle: float`（必填，弧度）； + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 根 body 相对竖直方向的倾角超过 `limit_angle` 时终止。 +* - `root_height_below_minimum` + - `unilab.envs.mdp.terminations.root_height_below_minimum`（函数） + - `minimum_height: float`（必填，m）； + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 世界系根高度低于 `minimum_height` 时终止。 +* - `nan_detection` + - `unilab.envs.mdp.terminations.nan_detection`（函数） + - `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 根位姿/速度或关节位置/速度出现 NaN 或 Inf 的环境终止。 +``` + +## Locomotion —— G1（`unilab.tasks.locomotion.g1.manager_terms`） + +源码：`src/unilab/tasks/locomotion/g1/manager_terms.py`。 + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - 函数 / 类 + - 参数 + - 语义 +* - `g1_tilt_exceeded` + - `unilab.tasks.locomotion.g1.manager_terms.g1_tilt_exceeded`（类） + - `max_tilt_deg: float`（必填，角度制） + - 从具名 `torso_upvector` 传感器读得基座倾角超过 `max_tilt_deg` 时 + 终止。 +* - `g1_base_height_below_minimum` + - `unilab.tasks.locomotion.g1.manager_terms.g1_base_height_below_minimum`（类） + - `minimum_height: float`（必填，m）； + `asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")` + - 经打包根 body 读计划读得根 body 高度低于 `minimum_height` 时终止。 +* - `G1WalkTerminationPack` + - `unilab.tasks.locomotion.g1.manager_terms.G1WalkTerminationPack` + （类；配置 `G1WalkTerminationPackCfg`） + - `entity_name: str = "robot"`；`root_body_name: str = "pelvis"`； + `max_tilt_deg: float = 65.0`；`minimum_height: float = 0.3` + - 一次载体读取内融合计算 G1 行走失败条件：倾角超过 `max_tilt_deg` + 或根高度低于 `minimum_height`。以 `_target_` + `func` 声明（见 + `src/unilab/conf/sac/task/g1_walk_flat/mjwarp.yaml`）。 +``` + +## Motion tracking（`unilab.tasks.motion_tracking.common.manager_terms`） + +源码：`src/unilab/tasks/motion_tracking/common/manager_terms.py`。所有 +term 按 `command_name` 解析 `MotionCommand` term。 + +```{list-table} +:header-rows: 1 +:widths: 15 27 33 25 + +* - Term + - 函数 / 类 + - 参数 + - 语义 +* - `BadAnchorPosZOnly` + - `unilab.tasks.motion_tracking.common.manager_terms.BadAnchorPosZOnly`（类） + - `command_name: str`（必填）；`threshold: float`（必填，m） + - anchor 高度差 `|z_motion - z_robot|` 超过 `threshold` 时终止。 +* - `bad_anchor_ori` + - `unilab.tasks.motion_tracking.common.manager_terms.bad_anchor_ori`（函数） + - `command_name: str`（必填）；`threshold: float`（必填）； + `asset_cfg` 可传入但不使用 + - 动作与机器人 anchor 姿态的四元数 z 轴分量差超过 `threshold` 时 + 终止。 +* - `BadMotionBodyPosZOnly` + - `unilab.tasks.motion_tracking.common.manager_terms.BadMotionBodyPosZOnly`（类） + - `command_name: str`（必填）；`threshold: float`（必填，m）； + `body_names: tuple[str, ...] | None = None` + - 任一所选 body 的 anchor 相对系 z 误差超过 `threshold` 时终止。 +* - `motion_clip_end` + - `unilab.tasks.motion_tracking.common.manager_terms.motion_clip_end`（函数） + - `command_name: str`（必填） + - 动作帧计数到达片段末尾时终止；声明时加 `time_out: true`（见 + `src/unilab/conf/sac/task/g1_motion_tracking/base.yaml`）。 +``` From f3113e7eab41d28cdb6ddfca54203c8284f6d4f2 Mon Sep 17 00:00:00 2001 From: LeeLeno <1846301824@qq.com> Date: Fri, 6 Nov 2026 00:34:10 +0800 Subject: [PATCH 2/8] docs: P1 walkthroughs + training diagnostics guide (#706) - extending/5-onboard_new_robot: end-to-end walkthrough from URDF/MJCF to first training run (assets, keyframe checklist, minimal owner YAML, registry, per-step verification) - extending/6-custom_reward: from function definition to the term showing up in training logs (owner YAML injection, info["log"] reward/ keys, weight tuning) - training/8-training_diagnostics: NaN guard -> viz chain, info["log"] field reference, common failure modes and triage paths - fix stale references: 1-new_task no longer points at the deleted go2/joystick.py; reward_porting uses the real sensor_reward_terms track_lin_vel and drops the outdated NumPy wording - bilingual (en/zh_CN mirror), zero new sphinx warnings --- .../en/2-user_guide/1-training/0-index.md | 7 + .../1-training/8-training_diagnostics.md | 143 ++++++++ .../3-framework_migration/6-reward_porting.md | 26 +- .../4-developer_guide/3-extending/0-index.md | 14 + .../3-extending/1-new_task.md | 9 +- .../3-extending/5-onboard_new_robot.md | 322 ++++++++++++++++++ .../3-extending/6-custom_reward.md | 121 +++++++ .../zh_CN/2-user_guide/1-training/0-index.md | 7 + .../1-training/8-training_diagnostics.md | 123 +++++++ .../3-framework_migration/6-reward_porting.md | 22 +- .../4-developer_guide/3-extending/0-index.md | 14 +- .../3-extending/1-new_task.md | 8 +- .../3-extending/5-onboard_new_robot.md | 309 +++++++++++++++++ .../3-extending/6-custom_reward.md | 115 +++++++ 14 files changed, 1220 insertions(+), 20 deletions(-) create mode 100644 docs/sphinx/source/en/2-user_guide/1-training/8-training_diagnostics.md create mode 100644 docs/sphinx/source/en/4-developer_guide/3-extending/5-onboard_new_robot.md create mode 100644 docs/sphinx/source/en/4-developer_guide/3-extending/6-custom_reward.md create mode 100644 docs/sphinx/source/zh_CN/2-user_guide/1-training/8-training_diagnostics.md create mode 100644 docs/sphinx/source/zh_CN/4-developer_guide/3-extending/5-onboard_new_robot.md create mode 100644 docs/sphinx/source/zh_CN/4-developer_guide/3-extending/6-custom_reward.md diff --git a/docs/sphinx/source/en/2-user_guide/1-training/0-index.md b/docs/sphinx/source/en/2-user_guide/1-training/0-index.md index 7fd29e864..7284ebb46 100644 --- a/docs/sphinx/source/en/2-user_guide/1-training/0-index.md +++ b/docs/sphinx/source/en/2-user_guide/1-training/0-index.md @@ -50,6 +50,12 @@ Run UniLab inside the checked-in Linux NVIDIA image workflow. Reproduce and operate the scoped single-GPU G1 FlashSAC/MJWarp long soak. ::: +:::{grid-item-card} Training diagnostics +:link: 8-training_diagnostics +:link-type: doc +NaN guard dumps, `info["log"]` keys, and common failure-mode playbooks. +::: + :::: ## When to Drop to `scripts/train_*.py` @@ -78,4 +84,5 @@ Day-to-day runs should use the unified CLI. Reach for the low-level 5-resume_and_checkpoints 6-docker 7-tensor_runtime_production +8-training_diagnostics ``` diff --git a/docs/sphinx/source/en/2-user_guide/1-training/8-training_diagnostics.md b/docs/sphinx/source/en/2-user_guide/1-training/8-training_diagnostics.md new file mode 100644 index 000000000..7115a0c8c --- /dev/null +++ b/docs/sphinx/source/en/2-user_guide/1-training/8-training_diagnostics.md @@ -0,0 +1,143 @@ +# Training Diagnostics + +This page collects the diagnosis paths for NaN/Inf failures, reward problems, +stalled learning, backend mismatches, and performance questions. Every key and +command below is wired in the repository; steps that need a simulator install, +a GPU, or a display are marked. + +## NaN Diagnostic Chain + +The NaN guard watches observations, rewards, and controls for non-finite rows +on-device and writes one state dump at an explicit host boundary. The chain: + +1. Configure it under `training.nan_guard` — shown with the PPO defaults from + `src/unilab/conf/ppo/config.yaml` (SAC/FlashSAC owners carry the same + block): + + ```yaml + training: + nan_guard: + enabled: true + buffer_size: 100 + max_envs_to_dump: 5 + output_dir: null # null -> /tmp/unilab/nan_dumps + defer_validation: true + check_interval: 32 + capture_interval: 32 + ``` + + Fields are defined by `NanGuardCfg` in + `src/unilab/training/tensor_diagnostics.py`. `capture_interval` controls how + often a physics-state snapshot is retained (0 disables snapshots); + `check_interval` is the poll period in deferred mode. +2. The training entrypoint attaches the guard to the env + (`apply_env_nan_guard` in `src/unilab/training/run.py`), and + `TorchEnv.step` calls `NanGuard.check(...)` each step. On detection, + `NanGuard.dump(...)` writes `nan_dump__step.npz` (state ring + buffer plus `meta_*` fields such as `detection_step` and `nan_env_ids`), + copies the model file next to it, and updates the `nan_dump_latest.npz` + symlink. +3. Replay the dump in a MuJoCo viewer (needs the `mujoco` extra and a + display): + + ```bash + uv run unilab-viz-nan /tmp/unilab/nan_dumps/nan_dump_latest.npz --env-index 0 + ``` + + `unilab-viz-nan` is the console entry for `src/unilab/utils/nan_viz.py` + (`pyproject.toml`); it prints the detection step and NaN env ids, then + replays the buffered states. Backends without physics-state playback ship + no states in the dump — the viewer prints the metadata only. + +Details and the viewer walkthrough: {doc}`../7-tooling/3-nan_visualizer`. +Round-trip coverage: `tests/training/test_tensor_diagnostics.py`. + +## Reading `info["log"]` + +Manager-Based environments place structured metrics in `info["log"]`; the +training logger forwards slash-prefixed entries to TensorBoard / W&B. The full +semantics table is in {doc}`3-logging`; the key families and their producers: + +| Key | Producer | Meaning | +| --- | --- | --- | +| `reward/` | `RewardManager.step_reward_extras()` (`src/unilab/managers/reward_manager.py`) | Per-term mean of `raw_value * weight` across envs, before `dt` scaling | +| `Episode_Termination/` | `TerminationManager.reset()` (`src/unilab/managers/termination_manager.py`) | Count of reset env rows for which that termination term fired | +| `Episode_Metrics/` | `MetricsManager` (`src/unilab/managers/metrics_manager.py`) | Completed-episode metric aggregates | + +Use `Train/mean_reward` and `Train/mean_episode_length` for overall progress +and `reward/` to attribute it; see {doc}`3-logging` for the distinction. + +## Common Failure Modes + +### Reward explosion or non-finite values + +- Symptom: training aborts with `RewardManager term '' returned NaN/Inf + for environments ...` (`src/unilab/managers/reward_manager.py`), or the NaN + guard writes a dump. +- Check: the per-term `reward/` curves — the diverging term usually + separates first. Compare its raw magnitude against its `weight` in the owner + YAML's `reward:` block. +- Fix: retune the weight from the CLI, e.g. + `uv run train --algo ppo --task --sim mujoco + reward..weight=-0.01` (override mechanics: the "Reward term weights" + section of {doc}`../2-algorithms/1-ppo`). Keep the smoothness/regularization + terms (`action_rate`-style penalties) enabled; dropping them commonly + destabilizes the action stream. + +### Policy does not learn + +- Symptom: `Train/mean_reward` stays flat while iterations advance. +- Check the hyperparameters against the per-algorithm reference tables + ({doc}`../2-algorithms/1-ppo` for PPO): `algo.algorithm.learning_rate`, + `algo.algorithm.entropy_coef`, `algo.algorithm.lam`. Check + `algo.num_envs` — it is a per-rank count, and very small counts starve the + gradient estimate. Check observation conditioning: + `algo.actor.obs_normalization` (the Go2 owner enables it) and whether any + observation term emits values with unexpected scale. +- Check the reward mix: if a tracking term (`reward/tracking_*`) stays near + zero while penalties dominate the `reward/` curves, the weights are + misbalanced, not the optimizer. + +### Two backends behave differently + +- Symptom: a checkpoint trained on backend A plays poorly or refuses to load + on backend B. +- Check: the sim2sim contract. Training snapshots the policy-defining fields + into `run_config.json`; play entrypoints validate the target config against + that snapshot before env construction (`resolve_sim2sim_config` in + `src/unilab/utils/sim2sim.py`). DENYLIST fields (observations, actions, + network shapes, `ctrl_dt`, ...) must match — under the default + `training.sim2sim_strict: true` a mismatch raises + `CrossBackendIncompatibleError`; WARNING_LIST fields only print a warning. + Audit the shipped pairs with `uv run scripts/audit_sim2sim_contracts.py`. +- Check capability differences per backend: + {doc}`../../3-deployment/2-sim_to_sim/6-capability_gaps`; exact support + grades: {doc}`../../5-reference/5-support_matrix`. + +### NaN appears mid-run + +- Follow the NaN chain above: the dump's `meta_detection_step` and + `meta_nan_env_ids` narrow the window, and the replayed states show whether + the pose diverged (asset/keyframe problem) or the controls blew up + (reward/action scaling problem). +- Cross-check the `reward/` curves just before the detection step to + find the first diverging term. + +### Performance and trace analysis + +- Off-policy configs (SAC/FlashSAC) expose `training.trace_enabled`, + `training.trace_output_dir`, `training.trace_thread_time`, and + `training.trace_cuda_events` (`src/unilab/conf/sac/config.yaml`), plus + `training.nvtx_profile_ranges` for NVTX annotations on CUDA. Traces are + Perfetto JSON; scalar timing keys are documented in {doc}`3-logging`. +- Analyze a captured trace host-side (no GPU needed): + + ```bash + uv run scripts/analyze_offpolicy_trace.py --training-e2e \ + --event collector/env_step --gap-event learner/training_e2e + ``` + + The script reports mean/median/p90/p95/p99 durations per event name; + `--steps-per-cycle` and `--drop-first` adjust the throughput summary. +- For the production long-soak setup and its failure classification fields, + see {doc}`7-tensor_runtime_production`. diff --git a/docs/sphinx/source/en/3-deployment/3-framework_migration/6-reward_porting.md b/docs/sphinx/source/en/3-deployment/3-framework_migration/6-reward_porting.md index cae37b4db..25468e3ce 100644 --- a/docs/sphinx/source/en/3-deployment/3-framework_migration/6-reward_porting.md +++ b/docs/sphinx/source/en/3-deployment/3-framework_migration/6-reward_porting.md @@ -2,7 +2,9 @@ Map Legged Gym's `_reward_*` methods to `reward` terms in the owner YAML. Manager-Based reward terms receive the env, read batched state through the -entity facade and managers, and return NumPy arrays of shape `(num_envs,)`. +entity facade and managers, and return float32 `torch.Tensor`s of shape +`(num_envs,)` on the env device. Since 1.3.4 the Manager runtime is +Torch-only: the reward manager rejects non-Tensor returns with a type error. ## Tracking, smoothness, and joint limits @@ -15,10 +17,11 @@ evaluation. ```yaml reward: tracking_lin_vel: - func: unilab.tasks.locomotion.common.manager_terms.track_lin_vel_xy_exp + func: unilab.tasks.locomotion.common.sensor_reward_terms.track_lin_vel weight: 1.0 params: - std: 0.5 + sensor_name: local_linvel + tracking_sigma: 0.5 command_name: twist action_rate: func: unilab.envs.mdp.action_rate_l2 @@ -31,10 +34,16 @@ reward: weight: -1.0 ``` -`track_lin_vel_xy_exp` uses body-frame xy velocity error and computes -`exp(-error_squared / std**2)`. If the source denominator is `tracking_sigma`, -use `std = sqrt(tracking_sigma)` and preserve coordinate frames and command -dimensions. +`track_lin_vel` is a class term +(`src/unilab/tasks/locomotion/common/sensor_reward_terms.py`) that binds one +named 3-D velocity sensor at construction and computes +`exp(-||cmd_xy - v_xy||^2 / tracking_sigma)` against the `twist` command. +`tracking_sigma` is the direct denominator (default 0.25; the Go2 PPO owner +sets 0.5), and the velocity frame is whichever frame the bound sensor reports — +the Go2 owners bind `local_linvel`, a velocimeter on the IMU site. Both the +sensor and the command fail closed at construction when missing. When porting, +keep the sensor frame and the command dimensions consistent with the source +implementation. `action_rate_l2` reads current and previous actions from the action manager. `joint_pos_limits` reads soft joint limits through the entity facade. @@ -87,6 +96,9 @@ whether timeouts should contribute to the source task's penalty separately. ## See also - {doc}`5-task_config_translation` +- {doc}`../../4-developer_guide/3-extending/6-custom_reward` — walkthrough for + defining and verifying a new reward term end to end - `src/unilab/envs/mdp/rewards.py` - `src/unilab/tasks/locomotion/common/manager_terms.py` +- `src/unilab/tasks/locomotion/common/sensor_reward_terms.py` - `src/unilab/tasks/locomotion/common/gait_terms.py` diff --git a/docs/sphinx/source/en/4-developer_guide/3-extending/0-index.md b/docs/sphinx/source/en/4-developer_guide/3-extending/0-index.md index cb37ae4e4..85bcdbf0f 100644 --- a/docs/sphinx/source/en/4-developer_guide/3-extending/0-index.md +++ b/docs/sphinx/source/en/4-developer_guide/3-extending/0-index.md @@ -30,6 +30,18 @@ Add configs, runner code, and script-level assembly. Extend terrain generation on cold paths. ::: +:::{grid-item-card} Onboard a new robot +:link: 5-onboard_new_robot +:link-type: doc +End-to-end walkthrough from URDF/MJCF assets to a first training run. +::: + +:::{grid-item-card} Custom reward term +:link: 6-custom_reward +:link-type: doc +Walkthrough: define, wire, and verify a reward term in the training log. +::: + :::: ```{toctree} @@ -39,4 +51,6 @@ Extend terrain generation on cold paths. 2-new_backend 3-new_algorithm 4-new_terrain +5-onboard_new_robot +6-custom_reward ``` diff --git a/docs/sphinx/source/en/4-developer_guide/3-extending/1-new_task.md b/docs/sphinx/source/en/4-developer_guide/3-extending/1-new_task.md index fa5740cfb..06ee6a4ae 100644 --- a/docs/sphinx/source/en/4-developer_guide/3-extending/1-new_task.md +++ b/docs/sphinx/source/en/4-developer_guide/3-extending/1-new_task.md @@ -41,5 +41,10 @@ Start from the contracts: {doc}`../2-contracts/1-env_contract`, - Registry API: `src/unilab/base/registry.py` - Env state contract: `src/unilab/base/torch_env.py` - Scene config: `src/unilab/base/scene.py` -- Existing task examples: `src/unilab/tasks/locomotion/go2/joystick.py`, - `src/unilab/tasks/motion_tracking/g1/manager_terms.py` +- Task declaration is owner YAML plus registry bootstrap — no per-task Python + env class: `src/unilab/tasks/locomotion/go2/__init__.py` registers + `Go2JoystickFlat` with `ManagerBasedRlEnvCfg` and `make_manager_based_rl_env` + per backend, and the owner YAMLs live under + `src/unilab/conf/ppo/task/go2_joystick_flat/`. Task-owned term library + example: `src/unilab/tasks/motion_tracking/g1/manager_terms.py`. End-to-end + walkthrough: {doc}`5-onboard_new_robot`. diff --git a/docs/sphinx/source/en/4-developer_guide/3-extending/5-onboard_new_robot.md b/docs/sphinx/source/en/4-developer_guide/3-extending/5-onboard_new_robot.md new file mode 100644 index 000000000..af3f2d618 --- /dev/null +++ b/docs/sphinx/source/en/4-developer_guide/3-extending/5-onboard_new_robot.md @@ -0,0 +1,322 @@ +# Walkthrough: Onboard a New Robot + +This walkthrough goes from a URDF/MJCF asset to a first PPO training run — +`uv run train --algo ppo --task --sim ` — one verified step at a +time. It composes existing owner layers instead of adding new code paths: assets +({doc}`../../2-user_guide/7-tooling/5-robot_import`), a task/scene keyframe, +owner YAMLs ({doc}`../../2-user_guide/1-training/2-hydra_config`), and a +registry entry (checklist view: {doc}`1-new_task`). + +Steps that need a simulator install, a display, or a GPU are marked. The +compose-only checks (`--cfg job`) still pass through the CLI's backend +preflight, so `--sim mujoco` requires the `mujoco` extra +(`uv sync --extra mujoco` in a source checkout). + +## 0. Pick Names + +Three names, used consistently everywhere: + +- `` — asset directory name, e.g. `go2` under + `src/unilab/assets/robots/go2/`. +- `` — snake_case CLI and owner-YAML directory name, e.g. + `go2_joystick_flat`. The CLI rejects names with slashes or dots + (`_check_task_name` in `src/unilab/cli.py`). +- `` — PascalCase registry key, e.g. `Go2JoystickFlat`. + +## 1. Prepare the Assets + +Follow {doc}`../../2-user_guide/7-tooling/5-robot_import`. For a URDF source: + +```bash +uv run scripts/tools/import_robot.py [robot_name] +``` + +The script converts URDF through `urdf-to-mjcf`, organizes the meshes, and +opens a `mujoco.viewer` for keyframe tuning (implementation: +`scripts/tools/import_robot.py`). If the source is already MJCF, lay out the +same directory contract by hand: + +- `src/unilab/assets/robots//assets/` — only `.stl` / `.obj` meshes. +- `src/unilab/assets/robots//.xml` — robot-only MJCF: bodies, + joints, actuators, sensors, mesh references. +- `src/unilab/assets/robots//scene_flat.xml` (the name is yours) — the + task/scene fragment: an `` of the robot XML, the ground, task + sensors, and the keyframe. Reference shape: + `src/unilab/assets/robots/go2/scene_flat.xml`. + +Prefer primitive collision geometry over visual meshes; high-poly collision +meshes hurt both simulation speed and contact stability. + +Verify: + +```bash +ls src/unilab/assets/robots// +# Expected: assets/ .xml scene_flat.xml +``` + +## 2. Tune the Keyframe + +The default pose is a `` in the task/scene fragment — never in +`.xml`. This is a hard repository rule: the robot XML stays +task-agnostic so several tasks can share one robot. Reference: the `home` +keyframe of `src/unilab/assets/robots/go2/scene_flat.xml`: + +```xml + + + +``` + +When the import tool's viewer closes, the tuned keyframe is written into the +scene fragment automatically. When checking it manually, confirm at least: + +- Standing pose: the floating-base height puts the feet in normal ground + contact — not floating, not penetrating. Press `c` in the viewer to show + contact points. +- Joint positions stay inside their joint ranges and near a natural stance; + `qpos` is ordered as the free joint (3 position + 4 quaternion) followed by + the actuated joints in XML order. +- Center of mass: the base sits above the support polygon, so the robot does + not tip at spawn. + +Verify: + +```bash +grep -n '/.xml +# Expected: 0 — keyframes stay out of the robot XML. +``` + +## 3. Write the Owner YAMLs + +PPO resolves each task×backend pair to two files (composition rules: +{doc}`../../2-user_guide/1-training/2-hydra_config`): + +- `src/unilab/conf/ppo/task//base.yaml` — the shared task declaration. +- `src/unilab/conf/ppo/task//mujoco.yaml` — the backend owner: inherits + the base and binds the registry name and the backend identity. + +A minimal base owner, adapted from the canonical Go2 owner +`src/unilab/conf/ppo/task/go2_joystick_flat/base.yaml`: + +```yaml +# @package _global_ +env: + scene: + model_file: src/unilab/assets/robots//scene_flat.xml + default_keyframe_name: home # the keyframe from step 2 + entities: + robot: + root_body_name: base + body_names: [base] + joint_names: [_hip_joint, ...] # XML joint names, in action order + actuator_names: [_hip, ...] # XML actuator names + sim_dt: 0.01 + ctrl_dt: 0.02 # must be an integer multiple of sim_dt + max_episode_seconds: 20.0 + observations: + policy: + terms: + base_ang_vel: + func: unilab.envs.mdp.builtin_sensor + params: {sensor_name: gyro} + projected_gravity: + func: unilab.envs.mdp.projected_gravity_from_sensor + params: {sensor_name: upvector} + joint_pos: + func: unilab.envs.mdp.joint_pos_rel + joint_vel: + func: unilab.envs.mdp.joint_vel_rel + actions: + func: unilab.envs.mdp.last_action + command: + func: unilab.envs.mdp.generated_commands + params: {command_name: twist} + actions: + joint_pos: + _target_: unilab.envs.mdp.JointPositionActionCfg + entity_name: robot + actuator_names: [".*"] + scale: 0.25 + use_default_offset: true + commands: + twist: + _target_: unilab.envs.mdp.UniformVelocityCommandCfg + entity_name: robot + resampling_time_range: [20.0, 20.0] + heading_command: false + ranges: + lin_vel_x: [-0.6, 1.0] + lin_vel_y: [-0.4, 0.4] + ang_vel_z: [-0.8, 0.8] + events: + reset_scene_to_default: + func: unilab.envs.mdp.reset_scene_to_default + mode: reset + reset_root_state_uniform: + func: unilab.envs.mdp.reset_root_state_uniform + mode: reset + params: + pose_range: + x: [-0.5, 0.5] + y: [-0.5, 0.5] + yaw: [-3.141592653589793, 3.141592653589793] + velocity_range: + x: [-0.5, 0.5] + y: [-0.5, 0.5] + terminations: + time_out: + func: unilab.envs.mdp.time_out + time_out: true + bad_orientation: + func: unilab.envs.mdp.bad_orientation + params: {limit_angle: 1.0471975511965976} + policy_observation_group: policy + +reward: + tracking_lin_vel: + func: unilab.envs.mdp.track_linear_velocity + weight: 1.0 + params: {std: 0.5, command_name: twist} + tracking_ang_vel: + func: unilab.envs.mdp.track_angular_velocity + weight: 0.2 + params: {std: 0.5, command_name: twist} + action_rate: + func: unilab.envs.mdp.action_rate_l2 + weight: -0.005 +``` + +Notes on the minimal set: + +- The sensor names (`gyro`, `upvector`) are the Go2 robot XML's names + (`src/unilab/assets/robots/go2/go2.xml`). Sensor-binding terms fail closed at + construction when the name is missing, so declare equivalent sensors in your + robot XML or drop the terms that bind them. +- The framework tracking terms (`unilab.envs.mdp.track_linear_velocity` / + `track_angular_velocity`) read root state instead of named sensors, so they + work on any robot whose command is named `twist`; parameters and semantics + are in {doc}`../../5-reference/6-manager_terms/1-rewards`. +- Add a `critic` observation group plus `env.critic_observation_group: critic` + when the algorithm should consume privileged observations (the Go2 owner + does both). + +The backend owner (`src/unilab/conf/ppo/task//mujoco.yaml`), following +`src/unilab/conf/ppo/task/go2_joystick_flat/mujoco.yaml`: + +```yaml +# @package _global_ +defaults: + - /task//base + - _self_ + +training: + task_name: # registry key from step 4 + sim_backend: mujoco +algo: + num_envs: 1024 + max_iterations: 151 +``` + +Verify — config composition only (still passes the CLI's backend preflight): + +```bash +uv run train --algo ppo --task --sim mujoco --cfg job \ + | grep -E 'task_name|default_keyframe_name|model_file' +# Expected: the composed values from your two YAMLs. +``` + +## 4. Register the Task + +The registry maps `` to a config factory plus one env factory per +backend. In-repo tasks register in a package under `src/unilab/tasks/`; the Go2 +package is the minimal template +(`src/unilab/tasks/locomotion/go2/__init__.py`): + +```python +"""Hydra-owned Manager-Based production registrations.""" + +from unilab.base import registry +from unilab.envs import ManagerBasedRlEnvCfg, make_manager_based_rl_env + +registry.register_env_config("", ManagerBasedRlEnvCfg) +registry.register_env("", make_manager_based_rl_env, sim_backend="mujoco") +``` + +Then make `ensure_registries()` import the module by adding it to the +`__unilab_registry_modules__` tuple in `src/unilab/tasks/__init__.py` (or in +your family package's `__init__.py`). Third-party packages outside this repo +register through the `unilab.tasks` entry-point group instead — see +{doc}`../1-architecture/5-registry` and `src/unilab/base/registry.py`. + +Verify — host-only, no backend needed: + +```bash +uv run python -c " +from unilab.base import registry +registry.ensure_registries() +print(registry.list_registered_envs()['']) +" +# Expected: {'config_factory': 'ManagerBasedRlEnvCfg', 'available_backends': ['mujoco']} +``` + +## 5. Pick the Backend + +Start with `mujoco` (the default backend). Each additional backend needs: + +1. a `register_env(..., sim_backend="")` line in the task package; +2. a matching owner YAML `src/unilab/conf/ppo/task//.yaml` with + `training.sim_backend: `; +3. evidence that the backend serves the terms your task binds — check the + generated {doc}`../../5-reference/5-support_matrix` and, for DR terms, + {doc}`../../2-user_guide/5-domain_randomization/2-field_reference`. + Unsupported capabilities fail closed at construction; set a term the + backend cannot serve to `null` in that backend's owner (pattern: + `pd_gains: null` in `src/unilab/conf/ppo/task/go2_joystick_flat/motrix.yaml`). + +Verify per added backend: repeat the `--cfg job` compose check with +`--sim `. + +## 6. First Training Run + +Needs the backend's Python extra; a GPU for any meaningful run. Start small: + +```bash +uv run train --algo ppo --task --sim mujoco \ + algo.num_envs=64 algo.max_iterations=5 training.no_play=true +``` + +First-run checkpoints: + +1. Composition and env construction fail fast and name the offending term or + field — for example, an observation term bound to a missing sensor raises + at manager construction. Fix the owner YAML and rerun. +2. The run directory appears under `logs/rsl_rl_ppo//` with + `run_config.json` (details: {doc}`../../2-user_guide/1-training/3-logging`). +3. Per-term reward keys `reward/` and termination counters + `Episode_Termination/` show up in TensorBoard + (`uv run tensorboard --logdir logs/rsl_rl_ppo/`). If any reward or + observation goes NaN/Inf, training raises with the term name and the NaN + guard writes a state dump — see + {doc}`../../2-user_guide/1-training/8-training_diagnostics`. + +Then scale back up (the owner values for `algo.num_envs` / +`algo.max_iterations`) and evaluate the latest run: + +```bash +uv run eval --algo ppo --task --sim mujoco --load-run -1 +``` + +## Where to Go Next + +- Add or tune reward terms: {doc}`6-custom_reward`. +- Built-in term tables: {doc}`../../5-reference/6-manager_terms/0-index`. +- Checklist view of the same contract: {doc}`1-new_task`. +- Interactive visual smoke check of the spawned scene (needs a display): + `uv run scripts/visualize_task_env.py --task --backend mujoco --num_envs 4`. diff --git a/docs/sphinx/source/en/4-developer_guide/3-extending/6-custom_reward.md b/docs/sphinx/source/en/4-developer_guide/3-extending/6-custom_reward.md new file mode 100644 index 000000000..1fe9e578c --- /dev/null +++ b/docs/sphinx/source/en/4-developer_guide/3-extending/6-custom_reward.md @@ -0,0 +1,121 @@ +# Walkthrough: Add a Custom Reward Term + +This walkthrough adds a reward term from zero to the point where +`reward/` shows up in the training log. The runtime is Torch-only (since +1.3.4; see the changelog and ADR-0011) — terms return `torch.Tensor`, and there +is no NumPy path. + +## 1. Define the Term + +Task-owned terms live in the task package, e.g. +`src/unilab/tasks/locomotion/common/sensor_reward_terms.py`; framework-level +terms live in `src/unilab/envs/mdp/rewards.py` and are re-exported at +`unilab.envs.mdp.`. Put a new task-specific term in the owning task +package. + +A term is a plain function taking the env first, then its `params`: + +```python +from __future__ import annotations + +from typing import TYPE_CHECKING + +import torch + +if TYPE_CHECKING: + from unilab.managers.types import ManagerBasedRlEnv + + +def action_magnitude_l1(env: ManagerBasedRlEnv) -> torch.Tensor: + """Sum of absolute raw policy actions, one value per environment.""" + return torch.sum(torch.abs(env.action_manager.action), dim=1) +``` + +Return contract, enforced by `RewardManager._compute_term` +(`src/unilab/managers/reward_manager.py`): a `torch.Tensor` of shape +`(num_envs,)`, dtype `float32`, on the env device. Wrong type, dtype, device, +or shape raises at the first `compute()` call; a non-finite value raises with +the term name. The manager invokes the term as `func(env, **params)`, so every +key under `params:` in the YAML must be an accepted parameter of the function. + +For stateful or sensor-bound terms use the class form instead: the class is +instantiated once with `(cfg, env)` at manager construction and then called per +step as `func(env, **params)`; an optional `reset(env_ids)` clears per-episode +state (`ManagerTermBaseCfg` in `src/unilab/managers/manager_base.py`). Binding a +named XML sensor at construction is the `SensorTermBase` pattern in +`src/unilab/tasks/locomotion/common/manager_terms.py`. + +## 2. Wire It into the Owner YAML + +Reward terms live under the top-level `reward:` mapping of the task owner YAML +(`src/unilab/conf//task//base.yaml` or a backend sibling), one +block per term with `func`, `weight`, and optional `params` — real structure +from `src/unilab/conf/ppo/task/go2_joystick_flat/base.yaml`: + +```yaml +reward: + action_magnitude: + func: unilab.tasks.locomotion.common.manager_terms.action_magnitude_l1 + weight: -0.001 +``` + +`func` is an importable `module.path.name`. The step reward contributed by the +term is `raw_value * weight` (scaled by `dt` inside the env; the log shows the +pre-dt weighted rate). Use a negative weight for penalties — the framework +penalty terms return non-negative costs, so do not negate them a second time. + +Verify composition (still passes the CLI's backend preflight; `--sim mujoco` +needs the `mujoco` extra): + +```bash +uv run train --algo ppo --task --sim mujoco --cfg job \ + | grep -A3 'action_magnitude' +# Expected: the func/weight lines from your owner YAML. +``` + +## 3. See It in the Training Log + +Run a short training session (needs the backend's extra; a GPU for any +meaningful run): + +```bash +uv run train --algo ppo --task --sim mujoco \ + algo.num_envs=64 algo.max_iterations=5 training.no_play=true +``` + +Manager-Based environments publish per-term reward diagnostics as +`reward/` keys in `info["log"]` — produced by +`RewardManager.step_reward_extras()` +(`src/unilab/managers/reward_manager.py`, mean across environments of +`raw_value * weight`, before `dt` scaling) — and the training logger forwards +them to TensorBoard / W&B ({doc}`../../2-user_guide/1-training/3-logging`): + +```bash +uv run tensorboard --logdir logs/rsl_rl_ppo/ +# Expected: a reward/action_magnitude tag alongside the other reward/ tags. +``` + +If the term misbehaves, the failure is explicit: construction errors name the +term, a wrong return type/shape raises from `RewardManager`, and a NaN/Inf +return raises `RewardManager term '' returned NaN/Inf` (plus a NaN-guard +state dump when enabled). Diagnosis workflow: +{doc}`../../2-user_guide/1-training/8-training_diagnostics`. + +## 4. Tune the Weight + +Retune a declared term from the command line — no YAML edit needed: + +```bash +uv run train --algo ppo --task --sim mujoco \ + reward.action_magnitude.weight=-0.01 +``` + +This works because `reward` is a root-level Hydra section of the composed +config (see the "Reward term weights" section of +{doc}`../../2-user_guide/2-algorithms/1-ppo`). Add new terms in the owner YAML +itself so the task stays reproducible; the CLI override is for sweeps. + +When porting terms from another framework, check the built-in tables first — +the term may already exist: {doc}`../../5-reference/6-manager_terms/1-rewards`. +The Legged Gym mapping guide is +{doc}`../../3-deployment/3-framework_migration/6-reward_porting`. diff --git a/docs/sphinx/source/zh_CN/2-user_guide/1-training/0-index.md b/docs/sphinx/source/zh_CN/2-user_guide/1-training/0-index.md index c6c2f2078..76c6b4e2e 100644 --- a/docs/sphinx/source/zh_CN/2-user_guide/1-training/0-index.md +++ b/docs/sphinx/source/zh_CN/2-user_guide/1-training/0-index.md @@ -49,6 +49,12 @@ TensorBoard、W&B、运行元数据以及 trace 选项。 复现并运维 scoped 单 GPU G1 FlashSAC/MJWarp 长 soak。 ::: +:::{grid-item-card} 训练诊断 +:link: 8-training_diagnostics +:link-type: doc +NaN guard dump、`info["log"]` 键与常见失败模式排查手册。 +::: + :::: ## 什么时候下沉到 `scripts/train_*.py` @@ -77,4 +83,5 @@ TensorBoard、W&B、运行元数据以及 trace 选项。 5-resume_and_checkpoints 6-docker 7-tensor_runtime_production +8-training_diagnostics ``` diff --git a/docs/sphinx/source/zh_CN/2-user_guide/1-training/8-training_diagnostics.md b/docs/sphinx/source/zh_CN/2-user_guide/1-training/8-training_diagnostics.md new file mode 100644 index 000000000..3b403a80a --- /dev/null +++ b/docs/sphinx/source/zh_CN/2-user_guide/1-training/8-training_diagnostics.md @@ -0,0 +1,123 @@ +# 训练诊断 + +本页汇总 NaN/Inf 失败、reward 异常、训练不收敛、后端行为不一致与性能问题的 +排查路径。文中所有键与命令都能在仓库中核实；需要安装仿真器、GPU 或显示器的 +步骤均已标注。 + +## NaN 诊断链 + +NaN guard 在设备侧检查观测、reward 与控制量中的非有限行，并在显式的主机 +边界写出一份状态 dump。链路如下： + +1. 在 `training.nan_guard` 下配置——以下为 `src/unilab/conf/ppo/config.yaml` + 中 PPO 的默认值（SAC/FlashSAC owner 带有相同的块）： + + ```yaml + training: + nan_guard: + enabled: true + buffer_size: 100 + max_envs_to_dump: 5 + output_dir: null # null -> /tmp/unilab/nan_dumps + defer_validation: true + check_interval: 32 + capture_interval: 32 + ``` + + 字段由 `src/unilab/training/tensor_diagnostics.py` 的 `NanGuardCfg` 定义。 + `capture_interval` 控制物理状态快照的保留频率（0 表示不快照）； + `check_interval` 是 deferred 模式下的轮询周期。 +2. 训练入口把 guard 挂到 env 上（`src/unilab/training/run.py` 的 + `apply_env_nan_guard`），`TorchEnv.step` 每步调用 `NanGuard.check(...)`。 + 检出时 `NanGuard.dump(...)` 写出 `nan_dump_<时间戳>_step.npz` + （状态环形缓冲加 `meta_*` 字段，如 `detection_step` 与 `nan_env_ids`）， + 把模型文件复制到旁边，并更新 `nan_dump_latest.npz` 软链。 +3. 在 MuJoCo viewer 中回放 dump（需要 `mujoco` extra 与显示器）： + + ```bash + uv run unilab-viz-nan /tmp/unilab/nan_dumps/nan_dump_latest.npz --env-index 0 + ``` + + `unilab-viz-nan` 是 `src/unilab/utils/nan_viz.py` 注册的 console 入口 + （`pyproject.toml`）；它先打印检出步数与 NaN env id，再回放缓冲状态。 + 不支持物理状态回放的后端 dump 中没有状态——viewer 只打印元数据。 + +细节与 viewer 操作见 {doc}`../7-tooling/3-nan_visualizer`。往返加载覆盖： +`tests/training/test_tensor_diagnostics.py`。 + +## 读懂 `info["log"]` + +Manager-Based 环境把结构化指标放进 `info["log"]`；训练 logger 把带斜杠前缀的 +条目转发到 TensorBoard / W&B。完整语义表见 {doc}`3-logging`；键族与生产者： + +| 键 | 生产者 | 含义 | +| --- | --- | --- | +| `reward/` | `RewardManager.step_reward_extras()`（`src/unilab/managers/reward_manager.py`） | 各 term 的 `raw_value * weight` 跨环境均值，未乘 `dt` | +| `Episode_Termination/` | `TerminationManager.reset()`（`src/unilab/managers/termination_manager.py`） | 因该终止 term 而 reset 的环境行数 | +| `Episode_Metrics/` | `MetricsManager`（`src/unilab/managers/metrics_manager.py`） | 已完成 episode 的指标聚合 | + +整体进展看 `Train/mean_reward` 与 `Train/mean_episode_length`，归因看 +`reward/`；两者区别见 {doc}`3-logging`。 + +## 常见失败模式 + +### Reward 爆炸或出现非有限值 + +- 症状：训练以 `RewardManager term '' returned NaN/Inf for + environments ...` 中止（`src/unilab/managers/reward_manager.py`），或 NaN + guard 写出 dump。 +- 查：逐项的 `reward/` 曲线——发散的 term 通常最先分离。对照 owner + YAML `reward:` 块中它的原始量级与 `weight`。 +- 改：用 CLI 调权，例如 `uv run train --algo ppo --task --sim mujoco + reward..weight=-0.01`（override 机制见 + {doc}`../2-algorithms/1-ppo` 的 “Reward term weights” 小节）。保留 + 平滑/正则项（`action_rate` 类惩罚）；删掉它们常常直接导致动作流不稳定。 + +### 策略不收敛 + +- 症状：`Train/mean_reward` 持平但迭代在推进。 +- 对照算法参考表查超参（PPO 见 {doc}`../2-algorithms/1-ppo`）： + `algo.algorithm.learning_rate`、`algo.algorithm.entropy_coef`、 + `algo.algorithm.lam`。检查 `algo.num_envs`——它是按 rank 计的数量，过小会 + 让梯度估计失真。检查观测条件：`algo.actor.obs_normalization`（Go2 owner + 开启了它），以及是否有观测 term 输出了异常量纲的值。 +- 检查 reward 配比：如果跟踪项（`reward/tracking_*`）一直接近零而惩罚项主导 + `reward/` 曲线，问题是权重失衡而不是优化器。 + +### 两个后端行为不一致 + +- 症状：在后端 A 训练的 checkpoint 在后端 B 上表现变差或拒绝加载。 +- 查：sim2sim 契约。训练把决定策略的字段快照进 `run_config.json`；play 入口 + 在构造 env 之前用该快照校验目标配置（`src/unilab/utils/sim2sim.py` 的 + `resolve_sim2sim_config`）。DENYLIST 字段（observations、actions、网络 + 形状、`ctrl_dt` 等）必须一致——默认 `training.sim2sim_strict: true` 下不一致 + 会抛 `CrossBackendIncompatibleError`；WARNING_LIST 字段只打印警告。审计已发布 + 的组合用 `uv run scripts/audit_sim2sim_contracts.py`。 +- 查后端能力差异：{doc}`../../3-deployment/2-sim_to_sim/6-capability_gaps`； + 精确支持等级：{doc}`../../5-reference/5-support_matrix`。 + +### 训练中途出现 NaN + +- 走上面的 NaN 链：dump 中的 `meta_detection_step` 与 `meta_nan_env_ids` + 缩小时间窗；回放状态可以看出是姿态发散（资产/keyframe 问题）还是控制量爆炸 + （reward/动作尺度问题）。 +- 对照检出步之前的 `reward/` 曲线，找最先发散的 term。 + +### 性能与 trace 分析 + +- Off-policy 配置（SAC/FlashSAC）暴露 `training.trace_enabled`、 + `training.trace_output_dir`、`training.trace_thread_time` 与 + `training.trace_cuda_events`（`src/unilab/conf/sac/config.yaml`），另有 + CUDA 上 NVTX 标注用的 `training.nvtx_profile_ranges`。trace 是 Perfetto + JSON；标量计时键见 {doc}`3-logging`。 +- 在主机侧分析已捕获的 trace（不需要 GPU）： + + ```bash + uv run scripts/analyze_offpolicy_trace.py --training-e2e \ + --event collector/env_step --gap-event learner/training_e2e + ``` + + 脚本按事件名报告 mean/median/p90/p95/p99 时长；`--steps-per-cycle` 与 + `--drop-first` 调整吞吐量汇总。 +- 生产环境长 soak 的设置与失败分类字段见 + {doc}`7-tensor_runtime_production`。 diff --git a/docs/sphinx/source/zh_CN/3-deployment/3-framework_migration/6-reward_porting.md b/docs/sphinx/source/zh_CN/3-deployment/3-framework_migration/6-reward_porting.md index b9ca425ee..affc3573d 100644 --- a/docs/sphinx/source/zh_CN/3-deployment/3-framework_migration/6-reward_porting.md +++ b/docs/sphinx/source/zh_CN/3-deployment/3-framework_migration/6-reward_porting.md @@ -2,7 +2,8 @@ 把 Legged Gym 的 `_reward_*` 方法映射到 owner YAML 的 `reward` term。 Manager-Based reward 接收 env，通过 entity facade 和各 manager 读取批量状态， -返回形状为 `(num_envs,)` 的 NumPy 数组。 +返回 env 设备上形状为 `(num_envs,)` 的 float32 `torch.Tensor`。1.3.4 起 +Manager 运行时是 Torch-only：reward manager 会对非 Tensor 的返回值抛类型错误。 ## 跟踪、平滑与关节限制 @@ -13,10 +14,11 @@ Manager-Based reward 接收 env，通过 entity facade 和各 manager 读取批 ```yaml reward: tracking_lin_vel: - func: unilab.tasks.locomotion.common.manager_terms.track_lin_vel_xy_exp + func: unilab.tasks.locomotion.common.sensor_reward_terms.track_lin_vel weight: 1.0 params: - std: 0.5 + sensor_name: local_linvel + tracking_sigma: 0.5 command_name: twist action_rate: func: unilab.envs.mdp.action_rate_l2 @@ -29,9 +31,14 @@ reward: weight: -1.0 ``` -`track_lin_vel_xy_exp` 使用底盘坐标系的 xy 速度误差，公式为 -`exp(-error_squared / std**2)`。若源实现分母为 `tracking_sigma`， -应令 `std = sqrt(tracking_sigma)`，并保持坐标系与 command 维度一致。 +`track_lin_vel` 是类形式的 term +（`src/unilab/tasks/locomotion/common/sensor_reward_terms.py`），在构造期绑定一个 +具名 3-D 速度传感器，并相对 `twist` command 计算 +`exp(-||cmd_xy - v_xy||^2 / tracking_sigma)`。`tracking_sigma` 直接作为分母 +（默认 0.25；Go2 PPO owner 设为 0.5）；速度坐标系取决于所绑定传感器的输出—— +Go2 owner 绑定的是 IMU site 上的 velocimeter `local_linvel`。传感器或 command +缺失时都会在构造期 fail closed。移植时保持传感器坐标系与 command 维度与源实现 +一致。 `action_rate_l2` 从 action manager 读取当前与前一步动作。 `joint_pos_limits` 从 entity facade 读取 soft joint limits。 @@ -81,6 +88,9 @@ command gating，再通过固定轨迹比较逐项输出。 ## 参考 - {doc}`5-task_config_translation` +- {doc}`../../4-developer_guide/3-extending/6-custom_reward` —— 端到端定义并验证 + 新 reward term 的演练 - `src/unilab/envs/mdp/rewards.py` - `src/unilab/tasks/locomotion/common/manager_terms.py` +- `src/unilab/tasks/locomotion/common/sensor_reward_terms.py` - `src/unilab/tasks/locomotion/common/gait_terms.py` diff --git a/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/0-index.md b/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/0-index.md index 270476c64..a24196239 100644 --- a/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/0-index.md +++ b/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/0-index.md @@ -24,10 +24,16 @@ 添加配置、runner 代码与脚本层组装。 ::: -:::{grid-item-card} 新地形 -:link: 4-new_terrain +:::{grid-item-card} 接入新机器人 +:link: 5-onboard_new_robot :link-type: doc -在冷路径上扩展地形生成。 +端到端演练：从 URDF/MJCF 资产到首次训练跑通。 +::: + +:::{grid-item-card} 自定义 reward term +:link: 6-custom_reward +:link-type: doc +演练：定义、接入并在训练日志中验证一个 reward term。 ::: :::: @@ -39,4 +45,6 @@ 2-new_backend 3-new_algorithm 4-new_terrain +5-onboard_new_robot +6-custom_reward ``` diff --git a/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/1-new_task.md b/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/1-new_task.md index 79e48268b..d0eb2ce17 100644 --- a/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/1-new_task.md +++ b/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/1-new_task.md @@ -41,5 +41,9 @@ - Registry API：`src/unilab/base/registry.py` - Env 状态契约：`src/unilab/base/torch_env.py` - 场景配置：`src/unilab/base/scene.py` -- 现有任务示例：`src/unilab/tasks/locomotion/go2/joystick.py`、 - `src/unilab/tasks/motion_tracking/g1/manager_terms.py` +- 任务声明是 owner YAML 加 registry 引导，没有按任务划分的 Python env 类： + `src/unilab/tasks/locomotion/go2/__init__.py` 用 `ManagerBasedRlEnvCfg` 与 + `make_manager_based_rl_env` 按后端注册 `Go2JoystickFlat`，owner YAML 位于 + `src/unilab/conf/ppo/task/go2_joystick_flat/`。任务自有 term 库示例： + `src/unilab/tasks/motion_tracking/g1/manager_terms.py`。端到端演练见 + {doc}`5-onboard_new_robot`。 diff --git a/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/5-onboard_new_robot.md b/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/5-onboard_new_robot.md new file mode 100644 index 000000000..7672326e4 --- /dev/null +++ b/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/5-onboard_new_robot.md @@ -0,0 +1,309 @@ +# 演练：接入新机器人 + +本演练从 URDF/MJCF 资产出发，一步步走到首次 PPO 训练跑通—— +`uv run train --algo ppo --task --sim `——每步都带验证。 +它只组合现有的 owner 层，不新增代码路径：资产 +（{doc}`../../2-user_guide/7-tooling/5-robot_import`）、任务/场景 keyframe、 +owner YAML（{doc}`../../2-user_guide/1-training/2-hydra_config`）以及 registry +条目（清单视角见 {doc}`1-new_task`）。 + +需要安装仿真器、显示器或 GPU 的步骤都已标注。纯组合检查（`--cfg job`）仍会 +经过 CLI 的后端预检，因此 `--sim mujoco` 需要 `mujoco` extra +（源码检出中执行 `uv sync --extra mujoco`）。 + +## 0. 选定名称 + +三个名称在所有地方保持一致： + +- `` —— 资产目录名，例如 `src/unilab/assets/robots/go2/` 下的 `go2`。 +- `` —— snake_case 的 CLI 与 owner YAML 目录名，例如 + `go2_joystick_flat`。CLI 会拒绝带斜杠或点的名称 + （`src/unilab/cli.py` 中的 `_check_task_name`）。 +- `` —— PascalCase 的 registry 键，例如 `Go2JoystickFlat`。 + +## 1. 准备资产 + +按 {doc}`../../2-user_guide/7-tooling/5-robot_import` 操作。URDF 源： + +```bash +uv run scripts/tools/import_robot.py [robot_name] +``` + +脚本通过 `urdf-to-mjcf` 转换 URDF、整理 mesh，并打开 `mujoco.viewer` 供 +keyframe 调整（实现：`scripts/tools/import_robot.py`）。如果源已经是 MJCF， +手工按同一目录契约摆放： + +- `src/unilab/assets/robots//assets/` —— 仅 `.stl` / `.obj` mesh。 +- `src/unilab/assets/robots//.xml` —— 仅含机器人的 MJCF： + body、joint、actuator、sensor 与 mesh 引用。 +- `src/unilab/assets/robots//scene_flat.xml`（名字自定）—— 任务/场景 + fragment：`` 机器人 XML、地面、任务传感器与 keyframe。参考形态： + `src/unilab/assets/robots/go2/scene_flat.xml`。 + +碰撞几何优先用简单 primitive，不要直接用高面数视觉 mesh；后者同时伤害仿真 +速度与接触稳定性。 + +验证： + +```bash +ls src/unilab/assets/robots// +# 预期：assets/ .xml scene_flat.xml +``` + +## 2. 调整 keyframe + +默认位姿是任务/场景 fragment 里的 ``——绝不放进 `.xml`。 +这是仓库硬约束：机器人 XML 保持任务无关，多个任务才能共享同一机器人。 +参考 `src/unilab/assets/robots/go2/scene_flat.xml` 的 `home` keyframe： + +```xml + + + +``` + +导入工具的 viewer 关闭时会把调好的 keyframe 自动写回场景 fragment。人工 +核对时至少确认： + +- 站立姿态：浮动基座高度让足底正常接触地面——不悬空、不穿透。在 viewer 中 + 按 `c` 显示接触点。 +- 关节角度位于关节行程内、接近自然站姿；`qpos` 的排列是自由关节 + （3 个位置 + 4 元数姿态）后接 XML 顺序的驱动关节。 +- 重心：基座位于支撑多边形上方，机器人出生时不会倾倒。 + +验证： + +```bash +grep -n '/.xml +# 预期：0 —— keyframe 不进入机器人 XML。 +``` + +## 3. 编写 owner YAML + +PPO 把每个 task×backend 组合解析为两个文件（组合规则见 +{doc}`../../2-user_guide/1-training/2-hydra_config`）： + +- `src/unilab/conf/ppo/task//base.yaml` —— 共享的任务声明。 +- `src/unilab/conf/ppo/task//mujoco.yaml` —— 后端 owner：继承 base， + 绑定 registry 名与后端身份。 + +最小 base owner，改写自 Go2 的规范 owner +`src/unilab/conf/ppo/task/go2_joystick_flat/base.yaml`： + +```yaml +# @package _global_ +env: + scene: + model_file: src/unilab/assets/robots//scene_flat.xml + default_keyframe_name: home # 第 2 步的 keyframe + entities: + robot: + root_body_name: base + body_names: [base] + joint_names: [_hip_joint, ...] # XML 关节名，按动作顺序 + actuator_names: [_hip, ...] # XML 执行器名 + sim_dt: 0.01 + ctrl_dt: 0.02 # 必须是 sim_dt 的整数倍 + max_episode_seconds: 20.0 + observations: + policy: + terms: + base_ang_vel: + func: unilab.envs.mdp.builtin_sensor + params: {sensor_name: gyro} + projected_gravity: + func: unilab.envs.mdp.projected_gravity_from_sensor + params: {sensor_name: upvector} + joint_pos: + func: unilab.envs.mdp.joint_pos_rel + joint_vel: + func: unilab.envs.mdp.joint_vel_rel + actions: + func: unilab.envs.mdp.last_action + command: + func: unilab.envs.mdp.generated_commands + params: {command_name: twist} + actions: + joint_pos: + _target_: unilab.envs.mdp.JointPositionActionCfg + entity_name: robot + actuator_names: [".*"] + scale: 0.25 + use_default_offset: true + commands: + twist: + _target_: unilab.envs.mdp.UniformVelocityCommandCfg + entity_name: robot + resampling_time_range: [20.0, 20.0] + heading_command: false + ranges: + lin_vel_x: [-0.6, 1.0] + lin_vel_y: [-0.4, 0.4] + ang_vel_z: [-0.8, 0.8] + events: + reset_scene_to_default: + func: unilab.envs.mdp.reset_scene_to_default + mode: reset + reset_root_state_uniform: + func: unilab.envs.mdp.reset_root_state_uniform + mode: reset + params: + pose_range: + x: [-0.5, 0.5] + y: [-0.5, 0.5] + yaw: [-3.141592653589793, 3.141592653589793] + velocity_range: + x: [-0.5, 0.5] + y: [-0.5, 0.5] + terminations: + time_out: + func: unilab.envs.mdp.time_out + time_out: true + bad_orientation: + func: unilab.envs.mdp.bad_orientation + params: {limit_angle: 1.0471975511965976} + policy_observation_group: policy + +reward: + tracking_lin_vel: + func: unilab.envs.mdp.track_linear_velocity + weight: 1.0 + params: {std: 0.5, command_name: twist} + tracking_ang_vel: + func: unilab.envs.mdp.track_angular_velocity + weight: 0.2 + params: {std: 0.5, command_name: twist} + action_rate: + func: unilab.envs.mdp.action_rate_l2 + weight: -0.005 +``` + +关于最小集的说明： + +- 传感器名（`gyro`、`upvector`）是 Go2 机器人 XML 里的名字 + （`src/unilab/assets/robots/go2/go2.xml`）。绑定传感器的 term 在构造期对 + 缺失的名字 fail closed，因此请在你的机器人 XML 里声明等价传感器，或删掉 + 绑定它们的 term。 +- 框架级跟踪 term（`unilab.envs.mdp.track_linear_velocity` / + `track_angular_velocity`）读 root state 而不是具名传感器，只要 command 名为 + `twist` 就能用于任意机器人；参数与语义见 + {doc}`../../5-reference/6-manager_terms/1-rewards`。 +- 当算法需要消费特权观测时，再加 `critic` 观测组和 + `env.critic_observation_group: critic`（Go2 owner 两者都有）。 + +后端 owner（`src/unilab/conf/ppo/task//mujoco.yaml`），参照 +`src/unilab/conf/ppo/task/go2_joystick_flat/mujoco.yaml`： + +```yaml +# @package _global_ +defaults: + - /task//base + - _self_ + +training: + task_name: # 第 4 步的 registry 键 + sim_backend: mujoco +algo: + num_envs: 1024 + max_iterations: 151 +``` + +验证——仅做配置组合（仍会经过 CLI 的后端预检）： + +```bash +uv run train --algo ppo --task --sim mujoco --cfg job \ + | grep -E 'task_name|default_keyframe_name|model_file' +# 预期：输出你两个 YAML 组合后的值。 +``` + +## 4. 注册任务 + +registry 把 `` 映射到一个 config 工厂和每个后端各一个 env +工厂。仓库内任务注册在 `src/unilab/tasks/` 下的包里；Go2 包就是最小模板 +（`src/unilab/tasks/locomotion/go2/__init__.py`）： + +```python +"""Hydra-owned Manager-Based production registrations.""" + +from unilab.base import registry +from unilab.envs import ManagerBasedRlEnvCfg, make_manager_based_rl_env + +registry.register_env_config("", ManagerBasedRlEnvCfg) +registry.register_env("", make_manager_based_rl_env, sim_backend="mujoco") +``` + +然后把该模块加入 `src/unilab/tasks/__init__.py`（或你的家族包 +`__init__.py`）的 `__unilab_registry_modules__` 元组，让 +`ensure_registries()` 导入它。仓库外的第三方任务包改用 `unilab.tasks` +entry-point 组注册——见 {doc}`../1-architecture/5-registry` 与 +`src/unilab/base/registry.py`。 + +验证——纯主机侧，不需要后端： + +```bash +uv run python -c " +from unilab.base import registry +registry.ensure_registries() +print(registry.list_registered_envs()['']) +" +# 预期：{'config_factory': 'ManagerBasedRlEnvCfg', 'available_backends': ['mujoco']} +``` + +## 5. 选择后端 + +从 `mujoco`（默认后端）开始。每增加一个后端需要： + +1. 任务包里加一行 `register_env(..., sim_backend="")`； +2. 对应的 owner YAML `src/unilab/conf/ppo/task//.yaml`，其中 + `training.sim_backend: `； +3. 该后端能提供你的任务所绑定 term 的证据——查生成的 + {doc}`../../5-reference/5-support_matrix`，DR term 另查 + {doc}`../../2-user_guide/5-domain_randomization/2-field_reference`。 + 不支持的能力在构造期 fail closed；后端无法服务的 term 在该后端 owner 里 + 置为 `null`（模式：`src/unilab/conf/ppo/task/go2_joystick_flat/motrix.yaml` + 中的 `pd_gains: null`）。 + +每加一个后端验证一次：用 `--sim ` 重跑 `--cfg job` 组合检查。 + +## 6. 首次训练 + +需要后端的 Python extra；有意义的运行需要 GPU。先小规模： + +```bash +uv run train --algo ppo --task --sim mujoco \ + algo.num_envs=64 algo.max_iterations=5 training.no_play=true +``` + +首跑检查点： + +1. 组合与 env 构造会快速失败并指出有问题的 term 或字段——例如观测 term + 绑定了不存在的传感器时会在 manager 构造期抛错。修 owner YAML 后重跑。 +2. 运行目录出现在 `logs/rsl_rl_ppo//`，其中包含 `run_config.json` + （细节见 {doc}`../../2-user_guide/1-training/3-logging`）。 +3. TensorBoard 中出现逐项 reward 键 `reward/` 与终止计数器 + `Episode_Termination/` + （`uv run tensorboard --logdir logs/rsl_rl_ppo/`）。任何 reward 或 + 观测出现 NaN/Inf 时，训练会带着 term 名抛错，并且 NaN guard 会写出状态 + dump——见 {doc}`../../2-user_guide/1-training/8-training_diagnostics`。 + +然后恢复到 owner 的规模（`algo.num_envs` / `algo.max_iterations`），并评估 +最近一次运行： + +```bash +uv run eval --algo ppo --task --sim mujoco --load-run -1 +``` + +## 下一步 + +- 添加或调权 reward term：{doc}`6-custom_reward`。 +- 内置 term 速查表：{doc}`../../5-reference/6-manager_terms/0-index`。 +- 同一契约的清单视角：{doc}`1-new_task`。 +- 对出生场景做交互式目视冒烟检查（需要显示器）： + `uv run scripts/visualize_task_env.py --task --backend mujoco --num_envs 4`。 diff --git a/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/6-custom_reward.md b/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/6-custom_reward.md new file mode 100644 index 000000000..f23829d82 --- /dev/null +++ b/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/6-custom_reward.md @@ -0,0 +1,115 @@ +# 演练：添加自定义 Reward Term + +本演练从零开始添加一个 reward term，直到 `reward/` 出现在训练日志中。 +运行时是 Torch-only（1.3.4 起；见 changelog 与 ADR-0011）——term 返回 +`torch.Tensor`，没有 NumPy 路径。 + +## 1. 定义 term + +任务自有的 term 放在任务包里，例如 +`src/unilab/tasks/locomotion/common/sensor_reward_terms.py`；框架级 term 位于 +`src/unilab/envs/mdp/rewards.py`，并统一再导出为 `unilab.envs.mdp.`。 +新的任务专属 term 应放在所属的任务包中。 + +term 是一个普通函数：第一个参数是 env，其余来自 `params`： + +```python +from __future__ import annotations + +from typing import TYPE_CHECKING + +import torch + +if TYPE_CHECKING: + from unilab.managers.types import ManagerBasedRlEnv + + +def action_magnitude_l1(env: ManagerBasedRlEnv) -> torch.Tensor: + """Sum of absolute raw policy actions, one value per environment.""" + return torch.sum(torch.abs(env.action_manager.action), dim=1) +``` + +返回契约由 `RewardManager._compute_term` 强制执行 +（`src/unilab/managers/reward_manager.py`）：形状为 `(num_envs,)` 的 +`torch.Tensor`，dtype 为 `float32`，位于 env 设备上。类型、dtype、设备或 +形状错误会在第一次 `compute()` 调用时抛错；返回非有限值会带 term 名抛错。 +manager 以 `func(env, **params)` 调用 term，因此 YAML 中 `params:` 下的每个 +键都必须是该函数接受的参数。 + +有状态或需要绑定传感器的 term 改用类形式：类在 manager 构造时以 +`(cfg, env)` 实例化一次，之后每步以 `func(env, **params)` 调用；可选的 +`reset(env_ids)` 清理按 episode 保存的状态（ +`src/unilab/managers/manager_base.py` 中的 `ManagerTermBaseCfg`）。在构造期 +绑定具名 XML 传感器是 +`src/unilab/tasks/locomotion/common/manager_terms.py` 中 `SensorTermBase` +的模式。 + +## 2. 接入 owner YAML + +reward term 位于任务 owner YAML 顶层的 `reward:` 映射下 +（`src/unilab/conf//task//base.yaml` 或其 backend 兄弟文件）， +每个 term 一段，含 `func`、`weight` 与可选 `params`——真实结构取自 +`src/unilab/conf/ppo/task/go2_joystick_flat/base.yaml`： + +```yaml +reward: + action_magnitude: + func: unilab.tasks.locomotion.common.manager_terms.action_magnitude_l1 + weight: -0.001 +``` + +`func` 是可导入的 `module.path.name`。该 term 贡献的单步 reward 为 +`raw_value * weight`（env 内部再乘 `dt`；日志显示的是未乘 dt 的加权速率）。 +惩罚项用负权重——框架的惩罚 term 返回非负代价，不要再取负一次。 + +验证组合结果（仍会经过 CLI 的后端预检；`--sim mujoco` 需要 `mujoco` +extra）： + +```bash +uv run train --algo ppo --task --sim mujoco --cfg job \ + | grep -A3 'action_magnitude' +# 预期：输出你 owner YAML 中的 func/weight 行。 +``` + +## 3. 在训练日志中确认 + +跑一次短训练（需要后端的 extra；有意义的运行需要 GPU）： + +```bash +uv run train --algo ppo --task --sim mujoco \ + algo.num_envs=64 algo.max_iterations=5 training.no_play=true +``` + +Manager-Based 环境把逐项 reward 诊断以 `reward/` 键发布到 +`info["log"]`——由 `RewardManager.step_reward_extras()` 产生 +（`src/unilab/managers/reward_manager.py`，跨环境对 `raw_value * weight` +取均值，未乘 `dt`）——训练 logger 再把它们转发到 TensorBoard / W&B +（{doc}`../../2-user_guide/1-training/3-logging`）： + +```bash +uv run tensorboard --logdir logs/rsl_rl_ppo/ +# 预期：与其他 reward/ 标签并列出现 reward/action_magnitude。 +``` + +term 出错时失败是显式的：构造错误会指出 term 名；返回类型/形状错误由 +`RewardManager` 抛出；返回 NaN/Inf 会抛出 +`RewardManager term '' returned NaN/Inf`（启用 NaN guard 时还会写出 +状态 dump）。诊断流程见 +{doc}`../../2-user_guide/1-training/8-training_diagnostics`。 + +## 4. 调整权重 + +已声明的 term 可以在命令行直接调权，无需改 YAML： + +```bash +uv run train --algo ppo --task --sim mujoco \ + reward.action_magnitude.weight=-0.01 +``` + +这可行是因为 `reward` 是组合配置顶层的 Hydra section（见 +{doc}`../../2-user_guide/2-algorithms/1-ppo` 的 “Reward term weights” 小节）。 +新增 term 请写进 owner YAML 以保证任务可复现；CLI override 适合扫参。 + +从其他框架移植 term 时先查内置表——该 term 可能已经存在： +{doc}`../../5-reference/6-manager_terms/1-rewards`。Legged Gym 的映射指南见 +{doc}`../../3-deployment/3-framework_migration/6-reward_porting`。 From 8ecd239d8dd2f03491e10a066198750748952326 Mon Sep 17 00:00:00 2001 From: LeeLeno <1846301824@qq.com> Date: Fri, 6 Nov 2026 00:34:23 +0800 Subject: [PATCH 3/8] docs+src: docstring baseline, interrogate gate, changelog discipline (#706) - add Google-style docstrings across high-traffic public interfaces: envs/mdp observations/actions/curriculums/ray_observation, managers types + manager_base, base torch_env + reset_state, structured_configs dataclasses (interrogate 43.6% -> 50.8%) - interrogate dev dependency with fail-under=40 gate; enforced in CI (ruff-lint job) and available as make check-docstrings; rocm pyproject/lock kept in sync - CONTRIBUTING.md: changelog discipline (Unreleased entries for user-visible PRs, bilingual Added/Changed/Fixed, issue links); AGENTS.md: docstring PR requirement + changelog pointer - changelog.md: add Unreleased section covering this PR --- .github/workflows/ci.yml | 2 + AGENTS.md | 5 + CONTRIBUTING.md | 14 + Makefile | 4 + docs/sphinx/source/changelog.md | 61 +++++ pyproject.rocm.toml | 8 +- pyproject.toml | 8 +- src/unilab/base/reset_state.py | 42 +++ src/unilab/base/torch_env.py | 68 +++++ src/unilab/envs/mdp/actions/actions.py | 32 +++ src/unilab/envs/mdp/curriculums.py | 20 ++ src/unilab/envs/mdp/observations.py | 104 ++++++++ src/unilab/envs/mdp/ray_observation.py | 21 ++ src/unilab/managers/manager_base.py | 13 + src/unilab/managers/types.py | 348 +++++++++++++++++++------ src/unilab/structured_configs.py | 186 +++++++++++++ uv.lock | 28 ++ uv.rocm.lock | 37 +++ 18 files changed, 914 insertions(+), 87 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8abab28d8..097cdf9cb 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -39,6 +39,8 @@ jobs: run: uv sync --only-group dev - name: Check run: uv run --no-sync ruff check --output-format=github . + - name: Check docstring coverage + run: uv run --no-sync interrogate src/unilab ruff-format: if: github.event_name == 'workflow_dispatch' || github.event_name == 'pull_request' diff --git a/AGENTS.md b/AGENTS.md index 414b9b923..d0da404ca 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -140,6 +140,11 @@ policy. In brief: define a reviewable outcome, choose the PR base before coding, link the driving issue, validate the final head, and wait for current-head remote CI only when the PR base is `main`. +New public interfaces must ship with docstrings; CI enforces the interrogate +floor in `pyproject.toml`. PRs with user-visible changes must add an entry to +the `Unreleased` section of `docs/sphinx/source/changelog.md` as described in +`CONTRIBUTING.md`. + When a change crosses runtime, backend, config, registry, or other public contracts, link the relevant ADR. Add an ADR only for a new structural decision. Treat new public contracts, lifecycle/protocol changes, routine CI, support diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 51dd179e8..da62457b4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -55,6 +55,20 @@ Use Conventional Commits: - `test:` test-related change - `chore:` build or tooling +## Changelog Discipline + +Every PR with user-visible changes must add an entry to the `## Unreleased` +section at the top of +[docs/sphinx/source/changelog.md](docs/sphinx/source/changelog.md), grouped +under the bilingual subsections `### Added / 新增`, `### Changed / 变更`, or +`### Fixed / 修复` (create the subsection when it is missing). Entries follow +the released-version style below: English prose first, then the Chinese +translation, with the driving GitHub issue or PR linked (for example +`([#706](https://github.com/Motphys/UniLab/issues/706))`). Purely internal +changes (refactors, test-only, or CI plumbing) may skip the changelog. Release +maintainers rename `## Unreleased` to the version number and PyPI upload date +at release time. + ## Pull Request Workflow 1. Choose and record the intended PR base before development. diff --git a/Makefile b/Makefile index 0e570ae20..6ea7497be 100644 --- a/Makefile +++ b/Makefile @@ -39,6 +39,10 @@ check: format type check-tests check-tests: uv run ruff check tests --select F401,F821,F811,F841 --output-format concise +.PHONY: check-docstrings +check-docstrings: + uv run interrogate src/unilab + .PHONY: test test: uv run pytest -m "not slow" diff --git a/docs/sphinx/source/changelog.md b/docs/sphinx/source/changelog.md index 7023580b4..57970d7c8 100644 --- a/docs/sphinx/source/changelog.md +++ b/docs/sphinx/source/changelog.md @@ -13,6 +13,67 @@ UniLab 遵循[语义化版本](https://semver.org/)。本共享页面以中英 PyPI 版本变更与未发布变更；发布日期采用 PyPI 上传日期。完整提交历史请参阅 [UniLab 仓库](https://github.com/unilabsim/UniLab)。 +## Unreleased + +### Added / 新增 + +- The documentation site is now versioned with sphinx-multiversion: the `main` + track is published with a version switcher in the navigation bar + ([#2584](https://github.com/Motphys/UniLab/pull/2584), + [#706](https://github.com/Motphys/UniLab/issues/706)). + 文档站现通过 sphinx-multiversion 版本化：发布 `main` 轨道，并在导航栏提供 + 版本切换器（[#2584](https://github.com/Motphys/UniLab/pull/2584)， + [#706](https://github.com/Motphys/UniLab/issues/706)）。 +- New manager terms reference pages tabulate the built-in reward, observation, + and termination terms + ([#2584](https://github.com/Motphys/UniLab/pull/2584), + [#706](https://github.com/Motphys/UniLab/issues/706)). + 新增 manager terms 参考页，以表格列出内置 reward、observation 与 + termination 项（[#2584](https://github.com/Motphys/UniLab/pull/2584)， + [#706](https://github.com/Motphys/UniLab/issues/706)）。 +- New domain randomization (DR) field configuration reference page documents + the supported reset randomization fields + ([#2584](https://github.com/Motphys/UniLab/pull/2584), + [#706](https://github.com/Motphys/UniLab/issues/706)). + 新增域随机化（DR）字段配置参考页，记录支持的 reset 随机化字段 + （[#2584](https://github.com/Motphys/UniLab/pull/2584)， + [#706](https://github.com/Motphys/UniLab/issues/706)）。 +- The four algorithm pages (PPO, APPO, SAC, FlashSAC) each gained a + configuration reference section + ([#2584](https://github.com/Motphys/UniLab/pull/2584), + [#706](https://github.com/Motphys/UniLab/issues/706)). + 四个算法页（PPO、APPO、SAC、FlashSAC）各新增配置参考小节 + （[#2584](https://github.com/Motphys/UniLab/pull/2584)， + [#706](https://github.com/Motphys/UniLab/issues/706)）。 +- New walkthroughs cover onboarding a new robot and writing a custom reward + term ([#2584](https://github.com/Motphys/UniLab/pull/2584), + [#706](https://github.com/Motphys/UniLab/issues/706)). + 新增"接入新机器人"与"自定义 reward"两篇 walkthrough + （[#2584](https://github.com/Motphys/UniLab/pull/2584)， + [#706](https://github.com/Motphys/UniLab/issues/706)）。 +- New training diagnostics page + ([#2584](https://github.com/Motphys/UniLab/pull/2584), + [#706](https://github.com/Motphys/UniLab/issues/706)). + 新增训练诊断页（[#2584](https://github.com/Motphys/UniLab/pull/2584)， + [#706](https://github.com/Motphys/UniLab/issues/706)）。 + +### Changed / 变更 + +- High-traffic public interfaces — the Manager protocols in + `unilab.managers.types`, `TorchEnv`, the reset-state transaction, the MDP + observation/action/curriculum/ray terms, and the PPO/APPO/SAC/FlashSAC + config dataclasses — now carry docstrings, and CI enforces an interrogate + docstring-coverage floor (`fail-under = 40` in `pyproject.toml`) + ([#2584](https://github.com/Motphys/UniLab/pull/2584), + [#706](https://github.com/Motphys/UniLab/issues/706)). + 高频公开接口——`unilab.managers.types` 中的 Manager protocol、 + `TorchEnv`、reset-state transaction、MDP + observation/action/curriculum/ray 项，以及 PPO/APPO/SAC/FlashSAC 配置 + dataclass——现已补齐 docstring；CI 通过 interrogate 强制执行 docstring + 覆盖率门槛（`pyproject.toml` 中 `fail-under = 40`） + （[#2584](https://github.com/Motphys/UniLab/pull/2584)， + [#706](https://github.com/Motphys/UniLab/issues/706)）。 + ## 1.5.8 (2026-11-05) ### Changed / 变更 diff --git a/pyproject.rocm.toml b/pyproject.rocm.toml index 319e4bd68..ba6c02014 100644 --- a/pyproject.rocm.toml +++ b/pyproject.rocm.toml @@ -81,7 +81,7 @@ unirl = [ [dependency-groups] # The RL stack (APPO/off-policy/multi-GPU) is merged into unilab itself # (src/unilab/rl/), so the dev environment no longer installs unilab-rl. -dev = ["pytest", "pytest-cov", "ruff", "mypy", "pyright>=1.1.408"] +dev = ["pytest", "pytest-cov", "ruff", "mypy", "pyright>=1.1.408", "interrogate"] [[tool.uv.index]] name = "pytorch-rocm72" @@ -129,6 +129,12 @@ ignore = [ "N812", ] +[tool.interrogate] +# Docstring coverage floor for src/unilab (issue #706). CI enforces this gate; +# raise the floor as coverage improves instead of lowering it to pass. +fail-under = 40 +exclude = ["tests", "docs", "build", "dist"] + [tool.mypy] python_version = "3.10" warn_return_any = true diff --git a/pyproject.toml b/pyproject.toml index cb0ab8cf4..f8bdb54e3 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -127,7 +127,7 @@ unirl = [ [dependency-groups] # The RL stack (APPO/off-policy/multi-GPU) is merged into unilab itself # (src/unilab/rl/), so the dev environment no longer installs unilab-rl. -dev = ["pytest", "pytest-cov", "ruff", "mypy", "pyright>=1.1.408"] +dev = ["pytest", "pytest-cov", "ruff", "mypy", "pyright>=1.1.408", "interrogate"] [[tool.uv.index]] name = "r2-cu130" @@ -168,6 +168,12 @@ ignore = [ "N812", ] +[tool.interrogate] +# Docstring coverage floor for src/unilab (issue #706). CI enforces this gate; +# raise the floor as coverage improves instead of lowering it to pass. +fail-under = 40 +exclude = ["tests", "docs", "build", "dist"] + [tool.mypy] python_version = "3.10" warn_return_any = true diff --git a/src/unilab/base/reset_state.py b/src/unilab/base/reset_state.py index 2bc9ee678..69938885e 100644 --- a/src/unilab/base/reset_state.py +++ b/src/unilab/base/reset_state.py @@ -73,6 +73,7 @@ def _randomization_term_tail(field: str) -> tuple[int, ...]: + """Return the per-column value tail of a canonical DR field name.""" try: return _RANDOMIZATION_TERM_TAILS[field] except KeyError as exc: @@ -80,6 +81,7 @@ def _randomization_term_tail(field: str) -> tuple[int, ...]: def _readonly_array(values: np.ndarray) -> np.ndarray: + """Return a read-only copy of ``values`` when it is not already immutable.""" result = np.asarray(values) if result.flags.writeable: result = result.copy() @@ -97,6 +99,19 @@ def __init__( default_qpos: np.ndarray | None = None, scene_layout: CompiledSceneLayout | None = None, ) -> None: + """Initialize the transaction over a backend's full environment batch. + + Args: + backend: Physics backend owning the packed reset composer. + default_qpos: Optional preselected default qpos; when ``None`` the + backend default is materialized lazily. + scene_layout: Optional compiled scene layout enabling mapped + physical-entity patches; only scalar (hinge/slide) joints are + supported. + + Raises: + NotImplementedError: If ``scene_layout`` declares non-scalar joints. + """ self._backend = backend self._num_envs = backend.num_envs self._selected_default_qpos = default_qpos @@ -402,6 +417,7 @@ def read_entity_root_pose(self, entity: str, env_ids: torch.Tensor) -> np.ndarra return self._backend.get_entity_state(entity)["root_pose"][host_ids].copy() def _commit_entities(self) -> None: + """Commit staged physical-entity patches through ``reset_entities``.""" assert self.scene_layout is not None if self._entity_rows is None: return @@ -1351,6 +1367,7 @@ def write_root_state_tensor( def _require_tensor_mask_write( self, reset_mask: torch.Tensor, *, capability: str ) -> torch.Tensor: + """Require an active mask transaction covering ``reset_mask`` and return it.""" if not self._tensor_active or self._tensor_active_mask is None: raise RuntimeError( f"ManagerBased tensor mask {capability} requires an active mask reset" @@ -1380,13 +1397,16 @@ def _tensor_mask_scope(self, reset_mask: torch.Tensor, *, capability: str) -> to def _tensor_mask_has_outside_rows( self, reset_mask: torch.Tensor, active_mask: torch.Tensor ) -> bool: + """Whether ``reset_mask`` selects rows outside ``active_mask``.""" return bool((reset_mask & ~active_mask).any()) def _mask_is_empty(self, reset_mask: torch.Tensor) -> bool: + """Whether ``reset_mask`` selects no rows.""" self._mask_count.copy_(reset_mask.sum()) return bool(self._mask_count.item() == 0) def _commit_has_rows(self, commit_mask: torch.Tensor) -> bool: + """Whether ``commit_mask`` selects at least one row.""" return not self._mask_is_empty(commit_mask) def _record_tensor_write_mask(self, reset_mask: torch.Tensor) -> None: @@ -1816,6 +1836,7 @@ def _packed_values(self, values: np.ndarray | torch.Tensor) -> torch.Tensor: return torch.as_tensor(np.ascontiguousarray(values, dtype=np.float32), device=device) def _materialize_default_state(self, term_name: str) -> None: + """Cache immutable default qpos/qvel from the backend on the cold path.""" if self._default_qpos is not None: return qpos = self._selected_default_qpos @@ -1884,6 +1905,7 @@ def _encode_tensor_root_velocity(self, root_state: torch.Tensor) -> torch.Tensor return velocity.contiguous() def _validate_tensor_root_quaternions(self, values: torch.Tensor, *, term_name: str) -> None: + """Require unit-length wxyz quaternions in a staged tensor payload.""" norms = torch.linalg.vector_norm(values, dim=-1) if not bool(torch.isclose(norms, torch.ones_like(norms), rtol=1e-5, atol=1e-6).all()): raise ValueError( @@ -1891,6 +1913,7 @@ def _validate_tensor_root_quaternions(self, values: torch.Tensor, *, term_name: ) def _materialize_default_actuator_gains(self, term_name: str) -> None: + """Cache immutable default actuator kp/kd tables from the backend.""" if self._default_kp is not None: return try: @@ -1934,6 +1957,7 @@ def _materialize_randomization_default( expected_tail: tuple[int, ...], term_name: str, ) -> np.ndarray: + """Bind and cache the immutable backend default table for one DR field.""" cached = self._randomization_defaults.get(field) if cached is not None: return cached @@ -1963,6 +1987,7 @@ def _fetch_reset_term_default( expected_tail: tuple[int, ...], term_name: str, ) -> np.ndarray: + """Fetch and validate one DR field's canonical or per-env default table.""" try: value = self._backend.get_reset_term_default(field) except (AttributeError, NotImplementedError) as exc: @@ -2008,6 +2033,7 @@ def _fetch_reset_term_default( return default def _require_randomization_default(self, field: str, term_name: str) -> np.ndarray: + """Return a DR field's cached default, requiring a prior bind call.""" try: return self._randomization_defaults[field] except KeyError as exc: @@ -2022,6 +2048,7 @@ def _default_is_per_env( *, field: str, ) -> bool: + """Whether a cached default table is stored per environment row.""" canonical_ndim = 1 + len(_randomization_term_tail(field)) if default.ndim == canonical_ndim: return False @@ -2038,6 +2065,7 @@ def _randomization_default_width( *, field: str, ) -> int: + """Return the model-column width of a cached default table.""" axis = 1 if self._default_is_per_env(default, field=field) else 0 return int(default.shape[axis]) @@ -2048,6 +2076,7 @@ def _select_randomization_default_columns( *, field: str, ) -> np.ndarray: + """Select the bound columns from a canonical or per-env default table.""" if self._default_is_per_env(default, field=field): return default[:, columns] return default[columns] @@ -2101,6 +2130,7 @@ def _readonly_binding( columns: np.ndarray, selected_default: np.ndarray, ) -> tuple[np.ndarray, np.ndarray]: + """Return validated columns and selected defaults as read-only arrays.""" return _readonly_array(columns), _readonly_array(selected_default) def _write_selected_randomization( @@ -2113,6 +2143,7 @@ def _write_selected_randomization( value_tail: tuple[int, ...], term_name: str, ) -> None: + """Stage selected columns of one DR field for the given rows.""" ids = self._prepare_state_write( env_ids, capability=field, @@ -2146,6 +2177,7 @@ def _prepare_state_write( capability: str, term_name: str, ) -> torch.Tensor: + """Validate ``env_ids`` against the active transaction and return them.""" self._require_active() if self._tensor_active: return self._prepare_tensor_randomization_write( @@ -2198,6 +2230,7 @@ def _validate_root_layout( *, term_name: str, ) -> tuple[np.ndarray, np.ndarray]: + """Validate a root-state layout against the default qpos/qvel widths.""" if not isinstance(layout, RootStateLayout): raise TypeError( f"EventManager term '{term_name}' root-state layout must be " @@ -2222,6 +2255,7 @@ def _validate_root_layout( def _tensor_root_columns_for_layout( self, layout: RootStateLayout, *, term_name: str ) -> tuple[torch.Tensor, torch.Tensor]: + """Return cached device column tensors for a root-state layout.""" qpos_columns, qvel_columns = self._validate_root_layout(layout, term_name=term_name) key = (tuple(int(i) for i in qpos_columns), tuple(int(i) for i in qvel_columns)) cached = self._tensor_root_columns @@ -2241,6 +2275,7 @@ def _tensor_joint_columns_for_indices( *, term_name: str, ) -> tuple[torch.Tensor, torch.Tensor]: + """Return cached device column tensors for joint qpos/qvel indices.""" assert self._default_qpos is not None and self._default_qvel is not None qpos_columns = self._validate_columns( np.asarray(qpos_indices, dtype=np.intp), @@ -2269,6 +2304,7 @@ def _tensor_joint_columns_for_indices( return qpos_tensor, qvel_tensor def _validate_quaternions(self, values: np.ndarray | torch.Tensor, *, term_name: str) -> None: + """Require unit-length wxyz quaternions in a staged payload.""" if isinstance(values, torch.Tensor): self._validate_tensor_root_quaternions(values, term_name=term_name) return @@ -2286,6 +2322,7 @@ def _validate_state_vector( capability: str, term_name: str, ) -> np.ndarray: + """Validate a 1-D finite floating state vector and return it read-only.""" if not isinstance(value, np.ndarray): raise TypeError( f"EventManager term '{term_name}' capability '{capability}' on backend " @@ -2363,6 +2400,7 @@ def _validate_columns( capability: str, term_name: str, ) -> np.ndarray: + """Validate 1-D integer column indices against the target width.""" if not isinstance(values, np.ndarray): raise TypeError( f"EventManager term '{term_name}' {capability} must be np.ndarray, " @@ -2397,6 +2435,7 @@ def _validate_tensor_values( capability: str, term_name: str, ) -> torch.Tensor: + """Validate shape, dtype, and finiteness of a staged tensor payload.""" if not isinstance(values, torch.Tensor): raise TypeError( f"EventManager term '{term_name}' {capability} must be torch.Tensor, " @@ -2468,16 +2507,19 @@ def _capability_error( capability: str, exc: BaseException, ) -> NotImplementedError: + """Build the fail-closed error for an unsupported reset-state capability.""" return NotImplementedError( f"EventManager term '{term_name}' reset-state capability '{capability}' is " f"unavailable on backend '{self._backend.backend_type}': {exc}" ) def _require_active(self) -> None: + """Require an open host or tensor reset transaction.""" if not (self._active or self._tensor_active): raise RuntimeError("ManagerBased reset-state mutation requires an active reset event") def _finish(self) -> None: + """Discard the packed composer staging and reset the transaction state.""" if self._packed_reset_transaction is not None: self._packed_reset_transaction.discard() self._active = False diff --git a/src/unilab/base/torch_env.py b/src/unilab/base/torch_env.py index 6c7017e3d..30b313fd5 100644 --- a/src/unilab/base/torch_env.py +++ b/src/unilab/base/torch_env.py @@ -60,10 +60,12 @@ class TorchEnvState: final_observation: dict[str, torch.Tensor] | None = None def replace(self, **updates: Any) -> "TorchEnvState": + """Return a copy of this transition with ``updates`` applied.""" return dataclasses.replace(self, **updates) def _cpu_time() -> float: + """Return the process CPU time (user + system) in seconds.""" process = os.times() return process.user + process.system @@ -92,6 +94,19 @@ def __init__( *, device: torch.device | str = "cpu", ): + """Initialize the backend-agnostic tensor lifecycle. + + Args: + cfg: Environment configuration. + backend: Physics backend bound to this environment. + num_envs: Number of vectorized environment rows. + device: Torch device owning the environment tensors; a bare + ``cuda`` device is resolved to the current device index. + + Raises: + ValueError: If ``finite_validation_interval`` is not a positive + integer. + """ apply_cpu_confinement(cfg.cpu_ids) self._cfg = cfg self._backend = backend @@ -128,9 +143,11 @@ def __init__( @property def device(self) -> torch.device: + """Torch device owning the environment tensors.""" return self._device def _bind_tensor_runtime(self) -> None: + """Validate and record the backend's tensor lifecycle capabilities once.""" if self._tensor_runtime_bound: return capabilities = self._backend.get_tensor_capabilities() @@ -155,6 +172,7 @@ def _bind_tensor_runtime(self) -> None: @property def cfg(self) -> EnvCfg: + """Environment configuration.""" return self._cfg @property @@ -177,24 +195,33 @@ def produces_completion_receipts(self) -> bool: @property def num_envs(self) -> int: + """Number of vectorized environment rows.""" return self._num_envs @property def state(self) -> Optional[TorchEnvState]: + """Latest environment transition, or ``None`` before :meth:`init_state`.""" return self._state @property def obs_groups_spec(self) -> dict[str, int]: + """Observation group names and per-group dimensions.""" raise NotImplementedError( f"{type(self).__name__} must declare observation group dimensions" ) @property def observation_space(self) -> gym.Space: + """Flat Box space spanning the concatenated observation groups.""" total = sum(self.obs_groups_spec.values()) return gym.spaces.Box(-np.inf, np.inf, shape=(total,), dtype=np.float32) def init_state(self) -> TorchEnvState: + """Allocate the zeroed initial state and run the bootstrap reset. + + Returns: + The initial ``TorchEnvState`` after the bootstrap reset. + """ self._bind_tensor_runtime() action_shape = self.action_space.shape if action_shape is not None: @@ -243,6 +270,19 @@ def reset( raise NotImplementedError(f"{type(self).__name__} must define a tensor reset lifecycle") def step(self, actions: torch.Tensor) -> TorchEnvState: + """Advance the environment by one policy step. + + Validates the action contract, applies action terms, steps the + backend, updates the tensor state, and runs the selected-row + autoreset. Per-phase timings are published under ``info["timing"]``. + + Args: + actions: Float32 action tensor of shape + ``(num_envs, *action_space.shape)`` on the env device. + + Returns: + The updated ``TorchEnvState``. + """ started = time.perf_counter() cpu_started = _cpu_time() phase = time.perf_counter() @@ -404,6 +444,7 @@ def apply_substep_action(self) -> torch.Tensor: ) def _step_physics(self, ctrl: torch.Tensor) -> dict[str, Any] | None: + """Step the backend ``sim_substeps`` times, merging timing and step receipts.""" if not self.requires_substep_state_feedback: return cast( dict[str, Any] | None, @@ -434,12 +475,14 @@ def _step_physics(self, ctrl: torch.Tensor) -> dict[str, Any] | None: return {"timing": backend_timing, "completion": backend_result.get("completion")} def _validate_action(self, actions: torch.Tensor) -> None: + """Validate the action contract and its finite verdict.""" self._validate_action_contract(actions) self._record_action_finite(actions) if not self._defer_finite_validation and not bool(self._action_finite_valid): raise ValueError("TorchEnv action contains NaN or Inf") def _validate_action_contract(self, actions: torch.Tensor) -> None: + """Check type, shape, dtype, contiguity, and device of an action tensor.""" if not isinstance(actions, torch.Tensor): raise TypeError(f"TorchEnv action must be a torch.Tensor, got {type(actions).__name__}") action_shape = self.action_space.shape @@ -478,6 +521,7 @@ def _validate_control_with_nan_guard(self, ctrl: torch.Tensor, *, defer: bool) - raise def _validate_control(self, ctrl: torch.Tensor) -> None: + """Validate the final control tensor after action-term adjustment.""" if not isinstance(ctrl, torch.Tensor): raise TypeError(f"TorchEnv control must be a torch.Tensor, got {type(ctrl).__name__}") if ctrl.dtype != torch.float32: @@ -507,6 +551,7 @@ def _validate_control(self, ctrl: torch.Tensor) -> None: raise ValueError("TorchEnv control contains NaN or Inf") def _validate_state(self, state: TorchEnvState) -> None: + """Validate the transition contract, aggregating finite verdicts on device.""" expected_keys = set(self.obs_groups_spec) if not isinstance(state.obs, dict) or set(state.obs) != expected_keys: raise ValueError("TorchEnvState.obs keys do not match obs_groups_spec") @@ -550,6 +595,7 @@ def _validate_state(self, state: TorchEnvState) -> None: raise ValueError(f"TorchEnvState {label} contains NaN or Inf") def _validate_finite_float(self, value: torch.Tensor, label: str) -> None: + """Require a floating-point, finite tensor.""" if not value.is_floating_point(): raise TypeError(f"TorchEnvState {label} must be floating-point") if not bool(torch.isfinite(value).all()): @@ -580,10 +626,12 @@ def _publish_control_finite_verdict(self) -> torch.Tensor | bool | None: return verdict def _require_floating_float(self, value: torch.Tensor) -> None: + """Require a floating-point tensor.""" if not value.is_floating_point(): raise TypeError("TorchEnvState floating outputs must be floating-point") def _validate_vector(self, value: Any, label: str, dtype: torch.dtype) -> None: + """Validate one ``(num_envs,)`` state vector's type, shape, dtype, and device.""" if not isinstance(value, torch.Tensor): raise TypeError(f"TorchEnvState {label} must be a torch.Tensor") if value.shape != (self._num_envs,): @@ -596,6 +644,7 @@ def _validate_vector(self, value: Any, label: str, dtype: torch.dtype) -> None: def _validate_final_observation( self, final_observation: dict[str, torch.Tensor] | None ) -> None: + """Validate the terminal observation carrier against ``obs``.""" if final_observation is None: return assert self._state is not None @@ -625,6 +674,7 @@ def _validate_final_observation( self._validate_finite_float(value, f"final_observation[{name!r}]") def _compute_truncated(self, state: TorchEnvState) -> torch.Tensor: + """Return per-row truncation flags from the episode step limit.""" self._truncated_scratch.fill_(False) max_steps = self._cfg.max_episode_steps if max_steps is not None: @@ -701,6 +751,7 @@ def _reset_done_envs(self) -> None: timing.update(detail_timing) def _validate_reset_observation_mask(self, obs: Mapping[str, torch.Tensor]) -> None: + """Validate full-width reset observations against the observation contract.""" if not isinstance(obs, dict) or set(obs) != set(self.obs_groups_spec): raise ValueError("TorchEnv mask reset observations do not match obs_groups_spec") for name, dim in self.obs_groups_spec.items(): @@ -735,6 +786,7 @@ def _validate_reset_observation_mask(self, obs: Mapping[str, torch.Tensor]) -> N def _scatter_reset_info_mask( self, reset_info: Mapping[str, Any], reset_mask: torch.Tensor ) -> None: + """Scatter per-row reset info tensors into the state info by mask.""" assert self._state is not None for key, value in reset_info.items(): current = self._state.info.get(key) @@ -747,6 +799,7 @@ def _scatter_reset_info_mask( current.copy_(torch.where(selected_reset_mask, value.to(device=self.device), current)) def _ensure_final_observation_scratch(self) -> dict[str, torch.Tensor]: + """Return the reusable terminal-observation scratch buffers.""" assert self._state is not None obs = self._state.obs scratch = self._final_observation_scratch @@ -765,6 +818,7 @@ def _ensure_final_observation_scratch(self) -> dict[str, torch.Tensor]: return scratch def _clear_reset_done_detail_timing(self, timing: dict[str, Any]) -> None: + """Zero the reset detail timing keys in ``timing``.""" for key in RESET_DONE_DETAIL_TIMING_KEYS: timing[key] = 0.0 @@ -785,6 +839,7 @@ def _collect_step_core_drain_ms(self, state: TorchEnvState) -> float: return 0.0 def _clear_step_final_observation(self) -> None: + """Clear the terminal observation carrier for the new step.""" assert self._state is not None self._state.final_observation = None @@ -887,6 +942,7 @@ def run_playback( @property def play_capabilities(self) -> EnvPlayCapabilities: + """Playback capabilities declared by the bound backend.""" capabilities = self._backend.get_play_capabilities() return EnvPlayCapabilities( supports_native_interactive_renderer=capabilities.supports_native_interactive_renderer, @@ -958,6 +1014,7 @@ def get_scene_visual_model_file(self) -> str | None: return cast(str | None, self._backend.get_scene_visual_model_file()) def set_autoreset(self, enabled: bool) -> None: + """Enable or disable the selected-row autoreset after each step.""" self._autoreset = bool(enabled) @property @@ -1112,9 +1169,19 @@ def get_backend_runtime_diagnostics(self) -> dict[str, Any]: return payload def export_training_state(self) -> dict[str, Any]: + """Export the training-resume state. + + Returns: + A versioned mapping carrying ``step_counter``. + """ return {"version": 1, "step_counter": self.step_counter} def import_training_state(self, state: Mapping[str, Any]) -> None: + """Restore a training-resume state produced by :meth:`export_training_state`. + + Raises: + ValueError: If the mapping keys, version, or counter are invalid. + """ if not isinstance(state, Mapping) or set(state) != {"version", "step_counter"}: raise ValueError("TorchEnv training state requires version and step_counter only") if type(state["version"]) is not int or state["version"] != 1: @@ -1125,6 +1192,7 @@ def import_training_state(self, state: Mapping[str, Any]) -> None: self.step_counter = counter def close(self) -> None: + """Flush deferred finite validation and release backend scene assets.""" try: # A short final cycle still owns device verdicts. Preserve an # existing training error when close runs from a finally block. diff --git a/src/unilab/envs/mdp/actions/actions.py b/src/unilab/envs/mdp/actions/actions.py index 2bef1c4c3..399f16a26 100644 --- a/src/unilab/envs/mdp/actions/actions.py +++ b/src/unilab/envs/mdp/actions/actions.py @@ -24,6 +24,7 @@ def _real(value: Any, *, label: str) -> float: + """Validate that ``value`` is a finite real number and return it as float.""" if isinstance(value, bool) or not isinstance(value, Real): raise TypeError(f"{label} must be a real number, got {type(value).__name__}") result = float(value) @@ -82,6 +83,7 @@ class BaseAction(ActionTerm): _entity: Entity def __init__(self, cfg: BaseActionCfg, env: ManagerBasedRlEnv): + """Resolve target joints and allocate the raw/processed action carriers.""" super().__init__(cfg=cfg, env=env) target_ids, target_names = self._entity.find_joints_by_actuator_names(cfg.actuator_names) self._target_ids = tuple(target_ids) @@ -104,6 +106,7 @@ def __init__(self, cfg: BaseActionCfg, env: ManagerBasedRlEnv): def _resolve_affine( self, value: float | dict[str, float], *, default: float, label: str ) -> float | torch.Tensor: + """Resolve a scalar or regex-keyed affine parameter.""" if isinstance(value, dict): indices, resolved = _resolve_named_values( value, self._target_names, label=f"{type(self).__name__} {label}" @@ -122,6 +125,7 @@ def _resolve_affine( return _real(value, label=f"{type(self).__name__} {label}") def _resolve_clip(self, value: dict[str, tuple] | None) -> torch.Tensor | None: + """Resolve regex-keyed ``(min, max)`` clip bounds into a tensor.""" if value is None: return None result = torch.empty((*self._affine_shape, 2), dtype=torch.float32, device=self._device) @@ -148,44 +152,54 @@ def _resolve_clip(self, value: dict[str, tuple] | None) -> torch.Tensor | None: return result def _cold_affine(self, value: float | torch.Tensor) -> float | torch.Tensor: + """Copy a resolved affine parameter onto the target device.""" if isinstance(value, torch.Tensor): return value.to(dtype=torch.float32, device=self._device, copy=True) return value def _cold_clip(self, value: torch.Tensor | None) -> torch.Tensor | None: + """Copy the resolved clip tensor onto the target device.""" if value is None: return None return value.to(dtype=torch.float32, device=self._device, copy=True) @property def scale(self) -> float | torch.Tensor: + """Action scaling factor.""" return self._scale @property def offset(self) -> float | torch.Tensor: + """Action offset.""" return self._offset @property def raw_action(self) -> torch.Tensor: + """Latest raw action tensor.""" return self._raw_actions @property def processed_action(self) -> torch.Tensor: + """Latest affine-transformed action tensor.""" return self._processed_actions @property def action_dim(self) -> int: + """Number of controlled joints.""" return self._action_dim @property def target_ids(self) -> tuple[int, ...]: + """Controlled joint indices.""" return self._target_ids @property def target_names(self) -> list[str]: + """Controlled joint names.""" return list(self._target_names) def process_actions(self, actions: torch.Tensor) -> None: + """Validate, store, and affine-transform one batch of raw actions.""" if not isinstance(actions, torch.Tensor): raise TypeError( f"{type(self).__name__} expected torch.Tensor, got {type(actions).__name__}" @@ -211,6 +225,7 @@ def validate_actions(self) -> None: self.publish_validation_verdict() def _apply_affine(self, source: torch.Tensor, destination: torch.Tensor) -> None: + """Apply scale/offset/clip from ``source`` into ``destination`` in place.""" destination.copy_(source) destination.mul_(self._scale).add_(self._offset) if self._clip is not None: @@ -243,6 +258,7 @@ class JointPositionActionCfg(BaseActionCfg): """Maximum position-target delay in physical substeps (inclusive).""" def __post_init__(self) -> None: + """Validate the delay-lag fields.""" for name, value in ( ("delay_min_lag", self.delay_min_lag), ("delay_max_lag", self.delay_max_lag), @@ -255,6 +271,7 @@ def __post_init__(self) -> None: raise ValueError("JointPositionActionCfg delay_max_lag must be >= delay_min_lag") def build(self, env: ManagerBasedRlEnv) -> JointPositionAction: + """Instantiate the action term for ``env``.""" return JointPositionAction(self, env) @@ -264,6 +281,7 @@ class JointPositionAction(BaseAction): cfg: JointPositionActionCfg def __init__(self, cfg: JointPositionActionCfg, env: ManagerBasedRlEnv): + """Bind the default-position offset and the optional delay buffer.""" if not isinstance(cfg.use_default_offset, bool): raise TypeError("JointPositionActionCfg use_default_offset must be bool") super().__init__(cfg=cfg, env=env) @@ -290,24 +308,29 @@ def __init__(self, cfg: JointPositionActionCfg, env: ManagerBasedRlEnv): @property def requires_substep_state_feedback(self) -> bool: + """Whether the term recomputes controls between physics substeps.""" return self._requires_substep_state_feedback @requires_substep_state_feedback.setter def requires_substep_state_feedback(self, value: bool) -> None: + """Set the substep state-feedback requirement.""" self._requires_substep_state_feedback = value def reset(self, env_ids: torch.Tensor | None = None) -> None: + """Zero raw actions and reset the delay buffer for the selected envs.""" super().reset(env_ids) if self._delay_buffer is not None: self._delay_buffer.reset(env_ids) def _delay_target(self, target: torch.Tensor) -> torch.Tensor: + """Push the target through the delay buffer when configured.""" if self._delay_buffer is None: return target self._delay_buffer.append(target) return self._delay_buffer.compute() def apply_actions(self) -> None: + """Publish joint position targets through the entity control boundary.""" control = self._entity.data.control_buffer if isinstance(control, torch.Tensor): # Stay on the control plane for tensor-native joint transmissions. @@ -349,6 +372,7 @@ class RelativeJointPositionActionCfg(BaseActionCfg): """ def __post_init__(self) -> None: + """Reject a non-zero offset, which is meaningless for relative control.""" if isinstance(self.offset, dict): resolved = [ _real(value, label="RelativeJointPositionActionCfg offset") @@ -362,6 +386,7 @@ def __post_init__(self) -> None: raise ValueError("RelativeJointPositionActionCfg does not support a non-zero offset") def build(self, env: ManagerBasedRlEnv) -> RelativeJointPositionAction: + """Instantiate the action term for ``env``.""" return RelativeJointPositionAction(self, env) @@ -369,12 +394,14 @@ class RelativeJointPositionAction(BaseAction): """Control joints via position targets relative to current positions.""" def _joint_state(self): + """Return the packed or entity joint state view.""" read_plan = getattr(self._env.scene, "read_plan", None) if read_plan is not None: return read_plan.joint_tensor_view(self._entity) return self._entity.joint_tensor_view(self._device) def apply_actions(self) -> None: + """Publish position targets relative to the current joint positions.""" joint_state = self._joint_state() target = joint_state.joint_pos.index_select(1, self._target_index) + ( self._processed_actions @@ -391,6 +418,7 @@ class JointVelocityActionCfg(BaseActionCfg): use_default_offset: bool = True def build(self, env: ManagerBasedRlEnv) -> JointVelocityAction: + """Instantiate the action term for ``env``.""" return JointVelocityAction(self, env) @@ -398,6 +426,7 @@ class JointVelocityAction(BaseAction): """Control joints via velocity targets.""" def __init__(self, cfg: JointVelocityActionCfg, env: ManagerBasedRlEnv): + """Bind the default velocity offset when configured.""" super().__init__(cfg=cfg, env=env) if not isinstance(cfg.use_default_offset, bool): raise TypeError("JointVelocityActionCfg use_default_offset must be bool") @@ -407,6 +436,7 @@ def __init__(self, cfg: JointVelocityActionCfg, env: ManagerBasedRlEnv): ) def apply_actions(self) -> None: + """Publish joint velocity targets through the entity boundary.""" self._entity.set_joint_velocity_target( self._entity_values(self._processed_actions), joint_ids=self._target_ids, @@ -418,6 +448,7 @@ class JointEffortActionCfg(BaseActionCfg): """Configuration for joint effort (torque) control.""" def build(self, env: ManagerBasedRlEnv) -> JointEffortAction: + """Instantiate the action term for ``env``.""" return JointEffortAction(self, env) @@ -425,6 +456,7 @@ class JointEffortAction(BaseAction): """Control joints via effort targets.""" def apply_actions(self) -> None: + """Publish joint effort targets through the entity boundary.""" self._entity.set_joint_effort_target( self._entity_values(self._processed_actions), joint_ids=self._target_ids, diff --git a/src/unilab/envs/mdp/curriculums.py b/src/unilab/envs/mdp/curriculums.py index 127dcea07..65f6f36ed 100644 --- a/src/unilab/envs/mdp/curriculums.py +++ b/src/unilab/envs/mdp/curriculums.py @@ -26,24 +26,34 @@ class _RewardCurriculumStageOptional(TypedDict, total=False): + """Optional fields of a reward curriculum stage.""" + weight: float params: dict[str, Any] class RewardCurriculumStage(_RewardCurriculumStageOptional): + """Stage for ``reward_curriculum``; requires a ``step`` threshold.""" + step: int class _TerminationCurriculumStageOptional(TypedDict, total=False): + """Optional fields of a termination curriculum stage.""" + params: dict[str, Any] time_out: bool class TerminationCurriculumStage(_TerminationCurriculumStageOptional): + """Stage for ``termination_curriculum``; requires a ``step`` threshold.""" + step: int class _CommandCurriculumStageOptional(TypedDict, total=False): + """Optional fields of a command curriculum stage.""" + params: dict[str, Any] @@ -59,6 +69,8 @@ class CommandCurriculumStage(_CommandCurriculumStageOptional): class _EventCurriculumStageOptional(TypedDict, total=False): + """Optional fields of an event curriculum stage.""" + params: dict[str, Any] @@ -202,6 +214,7 @@ class reward_curriculum: """ def __init__(self, cfg: CurriculumTermCfg, env: ManagerBasedRlEnv): + """Resolve the target reward term config and validate the stages.""" reward_name: str = cfg.params["reward_name"] stages: list[RewardCurriculumStage] = cfg.params["stages"] self._term_cfg = env.reward_manager.get_term_cfg(reward_name) @@ -215,6 +228,7 @@ def __call__( reward_name: str, stages: list[RewardCurriculumStage], ) -> dict[str, Any]: + """Apply all reached stages and return the logging snapshot.""" del env_ids, reward_name, stages return _apply_stages(self._term_cfg, env.common_step_counter, self._stages) @@ -239,6 +253,7 @@ class termination_curriculum: """ def __init__(self, cfg: CurriculumTermCfg, env: ManagerBasedRlEnv): + """Resolve the target termination term config and validate the stages.""" termination_name: str = cfg.params["termination_name"] stages: list[TerminationCurriculumStage] = cfg.params["stages"] self._term_cfg = env.termination_manager.get_term_cfg(termination_name) @@ -252,6 +267,7 @@ def __call__( termination_name: str, stages: list[TerminationCurriculumStage], ) -> dict[str, Any]: + """Apply all reached stages and return the logging snapshot.""" del env_ids, termination_name, stages return _apply_stages(self._term_cfg, env.common_step_counter, self._stages) @@ -278,6 +294,7 @@ class command_curriculum: """ def __init__(self, cfg: CurriculumTermCfg, env: ManagerBasedRlEnv): + """Resolve the target command term config and validate the stages.""" command_name: str = cfg.params["command_name"] stages: list[CommandCurriculumStage] = cfg.params["stages"] self._term_cfg = env.command_manager.get_term_cfg(command_name) @@ -293,6 +310,7 @@ def __call__( command_name: str, stages: list[CommandCurriculumStage], ) -> dict[str, Any]: + """Apply all reached stages and return the logging snapshot.""" del env_ids, command_name, stages return _apply_stages(self._term_cfg, env.common_step_counter, self._stages) @@ -317,6 +335,7 @@ class event_curriculum: """ def __init__(self, cfg: CurriculumTermCfg, env: ManagerBasedRlEnv): + """Resolve the target event term config and validate the stages.""" event_name: str = cfg.params["event_name"] stages: list[EventCurriculumStage] = cfg.params["stages"] self._term_cfg = env.event_manager.get_term_cfg(event_name) @@ -332,5 +351,6 @@ def __call__( event_name: str, stages: list[EventCurriculumStage], ) -> dict[str, Any]: + """Apply all reached stages and return the logging snapshot.""" del env_ids, event_name, stages return _apply_stages(self._term_cfg, env.common_step_counter, self._stages) diff --git a/src/unilab/envs/mdp/observations.py b/src/unilab/envs/mdp/observations.py index 4dfc881c5..f18ea4022 100644 --- a/src/unilab/envs/mdp/observations.py +++ b/src/unilab/envs/mdp/observations.py @@ -37,6 +37,7 @@ def quat_apply_inverse(quat_wxyz: torch.Tensor, vector: torch.Tensor) -> torch.T def require_read_plan(env: ManagerBasedRlEnv, term_name: str) -> Any: + """Return the scene's packed Torch read plan or fail closed for ``term_name``.""" read_plan = getattr(env.scene, "read_plan", None) if read_plan is None: raise NotImplementedError( @@ -46,6 +47,7 @@ def require_read_plan(env: ManagerBasedRlEnv, term_name: str) -> Any: def _root_body_name(asset: "Entity", term_name: str) -> str: + """Return the entity's declared root body name or fail closed for ``term_name``.""" root_body_name = getattr(asset, "root_body_name", None) if not isinstance(root_body_name, str) or not root_body_name: raise NotImplementedError( @@ -60,12 +62,19 @@ def root_body_view( asset_cfg: SceneEntityCfg, term_name: str, ) -> tuple[Entity, Any]: + """Return the entity and its packed root-body tensor view for ``term_name``.""" asset = cast("Entity", env.scene[asset_cfg.name]) read_plan = require_read_plan(env, term_name) return asset, read_plan.body_tensor_view(asset, (_root_body_name(asset, term_name),)) def joint_view(env: ManagerBasedRlEnv, asset_cfg: SceneEntityCfg, term_name: str) -> Any: + """Return the packed joint tensor view for the configured entity. + + Before the packed read plan is compiled, returns a zero-valued probe view + with the declared joint width so shape probing works during manager + construction. + """ asset = env.scene[asset_cfg.name] read_plan = getattr(env.scene, "read_plan", None) if read_plan is None: @@ -88,6 +97,7 @@ def joint_view(env: ManagerBasedRlEnv, asset_cfg: SceneEntityCfg, term_name: str def world_to_body_vector(quat_wxyz: torch.Tensor, vector_w: torch.Tensor) -> torch.Tensor: + """Rotate a world-frame vector into the body frame given by ``quat_wxyz``.""" return quat_apply_inverse(quat_wxyz, vector_w) @@ -98,6 +108,7 @@ class _NamedSensorObservation(ManagerTermBase): _tensor_sensor_names: tuple[str, ...] = () def __init__(self, cfg: ManagerTermBaseCfg, env: ManagerBasedRlEnv): + """Resolve the ``sensor_name``/``entity_name`` params and bind the sensor view.""" super().__init__(env) sensor_name = cfg.params.get("sensor_name") if not isinstance(sensor_name, str) or not sensor_name: @@ -117,12 +128,14 @@ def __init__(self, cfg: ManagerTermBaseCfg, env: ManagerBasedRlEnv): ) from exc def _resolve_entity_name(self, cfg: ManagerTermBaseCfg) -> str: + """Return the configured ``entity_name`` param, defaulting to ``"robot"``.""" entity_name = cfg.params.get("entity_name") if not isinstance(entity_name, str) or not entity_name: return "robot" return entity_name def _validate_call_name(self, sensor_name: str) -> None: + """Reject a call-time sensor name that differs from the bound one.""" if sensor_name != self._sensor_name: raise ValueError( f"Observation term '{self._term_name}' was bound to sensor " @@ -130,6 +143,7 @@ def _validate_call_name(self, sensor_name: str) -> None: ) def _read(self) -> Any: + """Read the cached sensor view with the term name attached to failures.""" try: return self._view.read() except (KeyError, TypeError, ValueError, NotImplementedError) as exc: @@ -139,6 +153,7 @@ def _read(self) -> Any: ) from exc def _read_tensor(self, env: ManagerBasedRlEnv) -> torch.Tensor: + """Read the sensor through the packed Torch read plan as a finite tensor.""" read_plan = getattr(env.scene, "read_plan", None) if read_plan is None: # Manager construction probes immutable observation shape before @@ -173,6 +188,7 @@ def _read_tensor(self, env: ManagerBasedRlEnv) -> torch.Tensor: @property def tensor_sensor_names(self) -> tuple[str, ...]: + """Sensor names this term requires in the packed Torch read plan.""" return self._tensor_sensor_names def reset(self, env_ids: torch.Tensor | None = None) -> None: @@ -191,6 +207,7 @@ def __call__( sensor_name: str, entity_name: str | None = None, ) -> torch.Tensor: + """Return the bound sensor's latest finite tensor reading.""" del entity_name self._validate_call_name(sensor_name) return self._read_tensor(env) @@ -202,6 +219,7 @@ class projected_gravity_from_sensor(_NamedSensorObservation): _term_name = "projected_gravity_from_sensor" def __init__(self, cfg: ManagerTermBaseCfg, env: ManagerBasedRlEnv): + """Bind the sensor view and require a 3-D up-vector output.""" super().__init__(cfg, env) if self._view.dimensions != (3,): raise ValueError( @@ -216,6 +234,7 @@ def __call__( sensor_name: str, entity_name: str | None = None, ) -> torch.Tensor: + """Return the negated 3-D up-vector reading (projected gravity).""" del entity_name self._validate_call_name(sensor_name) return -self._read_tensor(env) @@ -225,6 +244,16 @@ def base_lin_vel( env: ManagerBasedRlEnv, asset_cfg: SceneEntityCfg = _DEFAULT_ASSET_CFG, ) -> torch.Tensor: + """Root body linear velocity expressed in the root body frame. + + Args: + env: Manager-based RL environment providing the packed read plan. + asset_cfg: Scene entity selection; defaults to the ``"robot"`` entity. + + Returns: + A ``(num_envs, 3)`` tensor with the world-frame linear velocity + rotated into the root body frame. + """ _, view = root_body_view(env, asset_cfg, "base_lin_vel") return world_to_body_vector(view.quat_w[:, 0, :], view.lin_vel_w[:, 0, :]) @@ -233,6 +262,16 @@ def base_ang_vel( env: ManagerBasedRlEnv, asset_cfg: SceneEntityCfg = _DEFAULT_ASSET_CFG, ) -> torch.Tensor: + """Root body angular velocity expressed in the root body frame. + + Args: + env: Manager-based RL environment providing the packed read plan. + asset_cfg: Scene entity selection; defaults to the ``"robot"`` entity. + + Returns: + A ``(num_envs, 3)`` tensor with the world-frame angular velocity + rotated into the root body frame. + """ _, view = root_body_view(env, asset_cfg, "base_ang_vel") return world_to_body_vector(view.quat_w[:, 0, :], view.ang_vel_w[:, 0, :]) @@ -241,6 +280,16 @@ def projected_gravity( env: ManagerBasedRlEnv, asset_cfg: SceneEntityCfg = _DEFAULT_ASSET_CFG, ) -> torch.Tensor: + """Gravity vector projected into the root body frame. + + Args: + env: Manager-based RL environment providing the packed read plan. + asset_cfg: Scene entity selection; defaults to the ``"robot"`` entity. + + Returns: + A ``(num_envs, 3)`` tensor with the world gravity vector expressed in + the root body frame. + """ asset, view = root_body_view(env, asset_cfg, "projected_gravity") gravity = asset.data.gravity_vec_w(getattr(env, "device", torch.device("cpu")))[0] return quat_apply_inverse(view.quat_w[:, 0, :], gravity.expand((view.quat_w.shape[0], 3))) @@ -296,6 +345,7 @@ class _ImuMisalignedObservation(ManagerTermBase): _term_name = "imu_misaligned" def __init__(self, cfg: ManagerTermBaseCfg, env: ManagerBasedRlEnv): + """Validate ``max_angle_deg`` and bind the per-env misalignment quaternion.""" super().__init__(env) max_angle_deg = cfg.params.get("max_angle_deg", 1.0) if isinstance(max_angle_deg, bool) or not isinstance(max_angle_deg, (int, float)): @@ -313,6 +363,7 @@ def __init__(self, cfg: ManagerTermBaseCfg, env: ManagerBasedRlEnv): self._quat = _imu_misalignment_quat(env, math.radians(max_angle_deg)) def _validate_max_angle(self, max_angle_deg: float) -> None: + """Reject a call-time ``max_angle_deg`` that differs from the bound one.""" if float(max_angle_deg) != self._max_angle_deg: raise ValueError( f"Observation term '{self._term_name}' was bound to max_angle_deg " @@ -320,6 +371,7 @@ def _validate_max_angle(self, max_angle_deg: float) -> None: ) def _rotate(self, values: Any) -> torch.Tensor: + """Rotate vectors by this term's cached per-env misalignment quaternion.""" device = getattr(self._env, "device", torch.device("cpu")) vectors = torch.as_tensor(values, dtype=torch.float32, device=device) quat = self._quat.to(device=device) @@ -339,6 +391,7 @@ def __call__( max_angle_deg: float = 1.0, asset_cfg: SceneEntityCfg = _DEFAULT_ASSET_CFG, ) -> torch.Tensor: + """Return the base angular velocity rotated by the bound IMU misalignment.""" self._validate_max_angle(max_angle_deg) _, view = root_body_view(env, asset_cfg, self._term_name) return self._rotate(world_to_body_vector(view.quat_w[:, 0, :], view.ang_vel_w[:, 0, :])) @@ -355,6 +408,7 @@ def __call__( max_angle_deg: float = 1.0, asset_cfg: SceneEntityCfg = _DEFAULT_ASSET_CFG, ) -> torch.Tensor: + """Return projected gravity rotated by the bound IMU misalignment.""" self._validate_max_angle(max_angle_deg) asset, view = root_body_view(env, asset_cfg, self._term_name) gravity = asset.data.gravity_vec_w(getattr(env, "device", torch.device("cpu")))[0] @@ -369,6 +423,21 @@ def joint_pos_rel( biased: bool = False, asset_cfg: SceneEntityCfg = _DEFAULT_ASSET_CFG, ) -> torch.Tensor: + """Joint positions relative to the default joint positions. + + Args: + env: Manager-based RL environment. + biased: When True, add the per-joint encoder bias to the relative + positions. + asset_cfg: Scene entity selection; defaults to the ``"robot"`` entity. + + Returns: + A ``(num_envs, len(asset_cfg.joint_ids))`` tensor of relative joint + positions. + + Raises: + TypeError: If ``biased`` is not a ``bool``. + """ if not isinstance(biased, bool): raise TypeError(f"joint_pos_rel biased must be bool, got {type(biased).__name__}") joint_ids = asset_cfg.joint_ids @@ -387,6 +456,16 @@ def joint_vel_rel( env: ManagerBasedRlEnv, asset_cfg: SceneEntityCfg = _DEFAULT_ASSET_CFG, ) -> torch.Tensor: + """Joint velocities relative to the default joint velocities. + + Args: + env: Manager-based RL environment. + asset_cfg: Scene entity selection; defaults to the ``"robot"`` entity. + + Returns: + A ``(num_envs, len(asset_cfg.joint_ids))`` tensor of relative joint + velocities. + """ joint_ids = asset_cfg.joint_ids device = getattr(env, "device", torch.device("cpu")) asset = env.scene[asset_cfg.name] @@ -395,6 +474,19 @@ def joint_vel_rel( def last_action(env: ManagerBasedRlEnv, action_name: str | None = None) -> torch.Tensor: + """Latest raw action tensor published by the action manager. + + Args: + env: Manager-based RL environment. + action_name: When given, return only this action term's raw action + instead of the manager's combined action buffer. + + Returns: + A ``(num_envs, action_dim)`` tensor with the most recent raw actions. + + Raises: + KeyError: If ``action_name`` does not name an action term. + """ if action_name is None: return env.action_manager.action try: @@ -405,6 +497,18 @@ def last_action(env: ManagerBasedRlEnv, action_name: str | None = None) -> torch def generated_commands(env: ManagerBasedRlEnv, command_name: str) -> torch.Tensor: + """Latest command tensor generated by the named command term. + + Args: + env: Manager-based RL environment. + command_name: Name of the command term to read. + + Returns: + The command term's current command tensor. + + Raises: + KeyError: If ``command_name`` does not name an active command term. + """ try: command = env.command_manager.get_command(command_name) except KeyError as exc: diff --git a/src/unilab/envs/mdp/ray_observation.py b/src/unilab/envs/mdp/ray_observation.py index 5f2caf4a3..a4d7295ae 100644 --- a/src/unilab/envs/mdp/ray_observation.py +++ b/src/unilab/envs/mdp/ray_observation.py @@ -46,6 +46,7 @@ class RayRingPatternCfg: pitch: float = 0.0 def __post_init__(self) -> None: + """Validate the ray count and arc angles.""" if isinstance(self.num_rays, bool) or not isinstance(self.num_rays, int): raise TypeError("RayRingPatternCfg num_rays must be an integer") if self.num_rays <= 0: @@ -58,12 +59,15 @@ def __post_init__(self) -> None: @property def ray_count(self) -> int: + """Number of rays in the pattern.""" return self.num_rays def origins(self, device: torch.device) -> torch.Tensor: + """Return zero origins shared by all rays, ``(ray_count, 3)``.""" return torch.zeros((self.ray_count, 3), dtype=torch.float32, device=device) def directions(self, device: torch.device) -> torch.Tensor: + """Return the yaw-arc ray directions at the shared pitch, ``(ray_count, 3)``.""" angles = torch.linspace( self.start, self.stop, self.num_rays, dtype=torch.float32, device=device ) @@ -88,6 +92,7 @@ class RayGridPatternCfg: direction: Sequence[float] = (0.0, 0.0, -1.0) def __post_init__(self) -> None: + """Validate and normalize size, resolution, offset, and direction.""" size = _real_sequence(self.size, "RayGridPatternCfg size", 2) if any(value <= 0.0 for value in size): raise ValueError("RayGridPatternCfg size components must be positive") @@ -126,6 +131,7 @@ def __post_init__(self) -> None: @property def _counts(self) -> tuple[int, int]: + """Per-axis ray counts derived from size and resolution.""" resolution = self.resolution if not isinstance(resolution, Sequence): resolution = (resolution, resolution) @@ -136,10 +142,12 @@ def _counts(self) -> tuple[int, int]: @property def ray_count(self) -> int: + """Total number of rays in the grid.""" count_x, count_y = self._counts return count_x * count_y def origins(self, device: torch.device) -> torch.Tensor: + """Return the local grid ray origins, ``(ray_count, 3)``.""" count_x, count_y = self._counts x = torch.linspace(-self.size[0] / 2.0, self.size[0] / 2.0, count_x, device=device) y = torch.linspace(-self.size[1] / 2.0, self.size[1] / 2.0, count_y, device=device) @@ -150,6 +158,7 @@ def origins(self, device: torch.device) -> torch.Tensor: return origins.contiguous() def directions(self, device: torch.device) -> torch.Tensor: + """Return the shared normalized direction per ray, ``(ray_count, 3)``.""" direction = torch.tensor(self.direction, dtype=torch.float32, device=device) direction = direction / torch.linalg.vector_norm(direction) return direction.expand(self.ray_count, 3).clone().contiguous() @@ -166,6 +175,7 @@ class RayCameraPatternCfg: up: Sequence[float] = (0.0, 0.0, 1.0) def __post_init__(self) -> None: + """Validate the camera geometry and frame vectors.""" if isinstance(self.width, bool) or not isinstance(self.width, int): raise TypeError("RayCameraPatternCfg width must be an integer") if isinstance(self.height, bool) or not isinstance(self.height, int): @@ -191,12 +201,15 @@ def __post_init__(self) -> None: @property def ray_count(self) -> int: + """Total number of camera rays (``width * height``).""" return self.width * self.height def origins(self, device: torch.device) -> torch.Tensor: + """Return zero origins shared by all rays, ``(ray_count, 3)``.""" return torch.zeros((self.ray_count, 3), dtype=torch.float32, device=device) def directions(self, device: torch.device) -> torch.Tensor: + """Return the pinhole ray directions in the local frame, ``(ray_count, 3)``.""" forward = torch.tensor(self.forward, dtype=torch.float32, device=device) up = torch.tensor(self.up, dtype=torch.float32, device=device) right = torch.linalg.cross(forward, up) @@ -247,6 +260,7 @@ class RayPatternCfg: caster_options: dict[str, Any] = field(default_factory=dict) def __post_init__(self) -> None: + """Validate the pattern declaration and normalize its fields.""" if not isinstance(self.entity_name, str) or not self.entity_name: raise TypeError("RayPatternCfg entity_name must be a non-empty string") if not isinstance( @@ -331,6 +345,7 @@ class RayScan(ManagerTermBase): """Bind one tensor caster and publish a ray observation carrier.""" def __init__(self, cfg: ManagerTermBaseCfg, env: ManagerBasedRlEnv): + """Declare the ray pattern and allocate the deferred binding state.""" super().__init__(env) pattern = cfg.params.get("ray_pattern") if not isinstance(pattern, RayPatternCfg): @@ -523,6 +538,12 @@ def __call__( env: ManagerBasedRlEnv, ray_pattern: RayPatternCfg | None = None, ) -> torch.Tensor: + """Trace the pattern and return the ray observation tensor. + + Refreshes the full-scene pose from the packed read plan, transforms the + local pattern by the attachment frame, and returns per-ray distances + concatenated with optional normals and hit points. + """ if ray_pattern is not None and ray_pattern is not self.ray_pattern: raise ValueError( "RayScan was bound to a different ray_pattern; manager term parameters " diff --git a/src/unilab/managers/manager_base.py b/src/unilab/managers/manager_base.py index c4d0f5d65..50d61aa0c 100644 --- a/src/unilab/managers/manager_base.py +++ b/src/unilab/managers/manager_base.py @@ -1,6 +1,8 @@ # Derived from mujocolab/mjlab v1.6.0 (0fb8a681), src/mjlab/managers/manager_base.py. # Copyright 2025, The mjlab Developers. # Modified by UniLab for NumPy and UniLab contracts; licensed under Apache-2.0. +"""Base classes for manager terms and managers in the Manager-Based runtime.""" + from __future__ import annotations import abc @@ -73,17 +75,22 @@ def __call__(self, env, **kwargs) -> torch.Tensor: class ManagerTermBase: + """Base class for class-based manager terms instantiated with ``(cfg, env)``.""" + def __init__(self, env: ManagerBasedRlEnv): + """Retain the owning environment for later calls.""" self._env = env # Properties. @property def num_envs(self) -> int: + """Number of vectorized environment rows.""" return self._env.num_envs @property def name(self) -> str: + """Term name, defaulting to the class name.""" return self.__class__.__name__ # Methods. @@ -97,6 +104,7 @@ class ManagerBase(abc.ABC): """Base class for all managers.""" def __init__(self, env: ManagerBasedRlEnv): + """Retain the owning environment and prepare the configured terms.""" self._env = env self._prepare_terms() @@ -105,11 +113,13 @@ def __init__(self, env: ManagerBasedRlEnv): @property def num_envs(self) -> int: + """Number of vectorized environment rows.""" return self._env.num_envs @property @abc.abstractmethod def active_terms(self) -> list[str] | dict[Any, list[str]]: + """Names of the active terms, grouped per category when applicable.""" raise NotImplementedError # Methods. @@ -125,13 +135,16 @@ def reset(self, env_ids: torch.Tensor | slice | None = None) -> dict[str, Any]: return {} def get_active_iterable_terms(self, env_idx: int) -> Sequence[tuple[str, Sequence[float]]]: + """Return ``(term_name, values)`` pairs iterated by the given environment row.""" raise NotImplementedError @abc.abstractmethod def _prepare_terms(self) -> None: + """Resolve and instantiate the configured terms.""" raise NotImplementedError def _resolve_common_term_cfg(self, term_name: str, term_cfg: ManagerTermBaseCfg) -> None: + """Resolve ``SceneEntityCfg`` params and instantiate class-based term callables.""" for param_name, value in term_cfg.params.items(): if isinstance(value, SceneEntityCfg): try: diff --git a/src/unilab/managers/types.py b/src/unilab/managers/types.py index a867dcb24..2799fc58e 100644 --- a/src/unilab/managers/types.py +++ b/src/unilab/managers/types.py @@ -24,117 +24,185 @@ class ManagerEntity(Protocol): """Cold-path entity metadata required by :class:`SceneEntityCfg`.""" @property - def joint_names(self) -> Sequence[str]: ... + def joint_names(self) -> Sequence[str]: + """Declared joint names in model order.""" + ... @property - def body_names(self) -> Sequence[str]: ... + def body_names(self) -> Sequence[str]: + """Declared body names in model order.""" + ... @property - def geom_names(self) -> Sequence[str]: ... + def geom_names(self) -> Sequence[str]: + """Declared geom names in model order.""" + ... @property - def site_names(self) -> Sequence[str]: ... + def site_names(self) -> Sequence[str]: + """Declared site names in model order.""" + ... @property - def actuator_names(self) -> Sequence[str]: ... + def actuator_names(self) -> Sequence[str]: + """Declared actuator names in model order.""" + ... @property - def tendon_names(self) -> Sequence[str]: ... + def tendon_names(self) -> Sequence[str]: + """Declared tendon names in model order.""" + ... @property - def camera_names(self) -> Sequence[str]: ... + def camera_names(self) -> Sequence[str]: + """Declared camera names in model order.""" + ... @property - def light_names(self) -> Sequence[str]: ... + def light_names(self) -> Sequence[str]: + """Declared light names in model order.""" + ... @property - def material_names(self) -> Sequence[str]: ... + def material_names(self) -> Sequence[str]: + """Declared material names in model order.""" + ... @property - def texture_names(self) -> Sequence[str]: ... + def texture_names(self) -> Sequence[str]: + """Declared texture names in model order.""" + ... @property - def pair_names(self) -> Sequence[str]: ... + def pair_names(self) -> Sequence[str]: + """Declared body-pair names in model order.""" + ... @property - def data(self) -> Any: ... + def data(self) -> Any: + """Entity data facade consumed by manager terms.""" + ... @property - def num_joints(self) -> int: ... + def num_joints(self) -> int: + """Number of declared joints.""" + ... @property - def num_bodies(self) -> int: ... + def num_bodies(self) -> int: + """Number of declared bodies.""" + ... @property - def num_geoms(self) -> int: ... + def num_geoms(self) -> int: + """Number of declared geoms.""" + ... @property - def num_sites(self) -> int: ... + def num_sites(self) -> int: + """Number of declared sites.""" + ... @property - def num_actuators(self) -> int: ... + def num_actuators(self) -> int: + """Number of declared actuators.""" + ... @property - def num_tendons(self) -> int: ... + def num_tendons(self) -> int: + """Number of declared tendons.""" + ... @property - def num_cameras(self) -> int: ... + def num_cameras(self) -> int: + """Number of declared cameras.""" + ... @property - def num_lights(self) -> int: ... + def num_lights(self) -> int: + """Number of declared lights.""" + ... @property - def num_materials(self) -> int: ... + def num_materials(self) -> int: + """Number of declared materials.""" + ... @property - def num_textures(self) -> int: ... + def num_textures(self) -> int: + """Number of declared textures.""" + ... @property - def num_pairs(self) -> int: ... + def num_pairs(self) -> int: + """Number of declared body pairs.""" + ... def find_joints( self, keys: str | Sequence[str], preserve_order: bool = False - ) -> tuple[list[int], list[str]]: ... + ) -> tuple[list[int], list[str]]: + """Resolve name or regex keys to ``(ids, names)``; ``preserve_order`` keeps key order.""" + ... def find_bodies( self, keys: str | Sequence[str], preserve_order: bool = False - ) -> tuple[list[int], list[str]]: ... + ) -> tuple[list[int], list[str]]: + """Resolve name or regex keys to ``(ids, names)``; ``preserve_order`` keeps key order.""" + ... def find_geoms( self, keys: str | Sequence[str], preserve_order: bool = False - ) -> tuple[list[int], list[str]]: ... + ) -> tuple[list[int], list[str]]: + """Resolve name or regex keys to ``(ids, names)``; ``preserve_order`` keeps key order.""" + ... def find_sites( self, keys: str | Sequence[str], preserve_order: bool = False - ) -> tuple[list[int], list[str]]: ... + ) -> tuple[list[int], list[str]]: + """Resolve name or regex keys to ``(ids, names)``; ``preserve_order`` keeps key order.""" + ... def find_actuators( self, keys: str | Sequence[str], preserve_order: bool = False - ) -> tuple[list[int], list[str]]: ... + ) -> tuple[list[int], list[str]]: + """Resolve name or regex keys to ``(ids, names)``; ``preserve_order`` keeps key order.""" + ... def find_tendons( self, keys: str | Sequence[str], preserve_order: bool = False - ) -> tuple[list[int], list[str]]: ... + ) -> tuple[list[int], list[str]]: + """Resolve name or regex keys to ``(ids, names)``; ``preserve_order`` keeps key order.""" + ... def find_cameras( self, keys: str | Sequence[str], preserve_order: bool = False - ) -> tuple[list[int], list[str]]: ... + ) -> tuple[list[int], list[str]]: + """Resolve name or regex keys to ``(ids, names)``; ``preserve_order`` keeps key order.""" + ... def find_lights( self, keys: str | Sequence[str], preserve_order: bool = False - ) -> tuple[list[int], list[str]]: ... + ) -> tuple[list[int], list[str]]: + """Resolve name or regex keys to ``(ids, names)``; ``preserve_order`` keeps key order.""" + ... def find_materials( self, keys: str | Sequence[str], preserve_order: bool = False - ) -> tuple[list[int], list[str]]: ... + ) -> tuple[list[int], list[str]]: + """Resolve name or regex keys to ``(ids, names)``; ``preserve_order`` keeps key order.""" + ... def find_textures( self, keys: str | Sequence[str], preserve_order: bool = False - ) -> tuple[list[int], list[str]]: ... + ) -> tuple[list[int], list[str]]: + """Resolve name or regex keys to ``(ids, names)``; ``preserve_order`` keeps key order.""" + ... def find_pairs( self, keys: str | Sequence[str], preserve_order: bool = False - ) -> tuple[list[int], list[str]]: ... + ) -> tuple[list[int], list[str]]: + """Resolve name or regex keys to ``(ids, names)``; ``preserve_order`` keeps key order.""" + ... def write_joint_state_to_sim( self, @@ -142,13 +210,17 @@ def write_joint_state_to_sim( velocity: torch.Tensor, joint_ids: torch.Tensor | Sequence[int] | slice | None = None, env_ids: torch.Tensor | slice | None = None, - ) -> None: ... + ) -> None: + """Write joint position/velocity columns for the selected rows into the sim.""" + ... def write_root_state_tensor_to_sim( self, root_state: torch.Tensor, reset_mask: torch.Tensor, - ) -> None: ... + ) -> None: + """Write a 13-D root state for the rows selected by ``reset_mask``.""" + ... def write_motion_state_tensor_to_sim( self, @@ -156,102 +228,168 @@ def write_motion_state_tensor_to_sim( position: torch.Tensor, velocity: torch.Tensor, reset_mask: torch.Tensor, - ) -> None: ... + ) -> None: + """Write root and joint motion state for the rows selected by ``reset_mask``.""" + ... class ManagerSensorView(Protocol): """Backend-owned named-sensor view retained by a manager term.""" @property - def backend_type(self) -> str: ... + def backend_type(self) -> str: + """Backend type identifier of the owning view.""" + ... @property - def names(self) -> tuple[str, ...]: ... + def names(self) -> tuple[str, ...]: + """Bound sensor names.""" + ... @property - def dimensions(self) -> tuple[int, ...]: ... + def dimensions(self) -> tuple[int, ...]: + """Per-reading dimensions of the bound sensor.""" + ... - def read(self) -> Any: ... + def read(self) -> Any: + """Return the latest sensor reading.""" + ... class ManagerScene(Protocol): """Minimal name-addressable scene surface consumed by managers.""" @property - def entities(self) -> Mapping[str, ManagerEntity]: ... + def entities(self) -> Mapping[str, ManagerEntity]: + """Name-addressable scene entities.""" + ... @property - def terrain(self) -> Terrain: ... + def terrain(self) -> Terrain: + """Scene terrain provider.""" + ... - def compile_tensor_reads(self, device: str | torch.device, specs: Sequence[Any]) -> Any: ... + def compile_tensor_reads(self, device: str | torch.device, specs: Sequence[Any]) -> Any: + """Compile the packed Torch read plan for ``specs`` on ``device``.""" + ... - def env_origins(self, device: str | torch.device) -> torch.Tensor: ... + def env_origins(self, device: str | torch.device) -> torch.Tensor: + """Return per-environment origin positions on ``device``.""" + ... - def __getitem__(self, name: str) -> ManagerEntity: ... + def __getitem__(self, name: str) -> ManagerEntity: + """Return the entity registered under ``name``.""" + ... - def bind_sensor_data(self, names: Sequence[str]) -> ManagerSensorView: ... + def bind_sensor_data(self, names: Sequence[str]) -> ManagerSensorView: + """Bind a named-sensor view retained by a manager term.""" + ... - def reset_to_default(self, env_ids: torch.Tensor, *, term_name: str) -> None: ... + def reset_to_default(self, env_ids: torch.Tensor, *, term_name: str) -> None: + """Restore default state for the rows selected by ``env_ids``.""" + ... def bind_gravity_write_tensor( self, *, term_name: str, device: str | torch.device - ) -> torch.Tensor: ... + ) -> torch.Tensor: + """Bind the per-environment gravity write table on ``device``.""" + ... def write_gravity_to_sim( self, values: torch.Tensor, env_ids: torch.Tensor, *, term_name: str - ) -> None: ... + ) -> None: + """Write per-row gravity vectors into the sim.""" + ... @property - def read_plan(self) -> Any: ... + def read_plan(self) -> Any: + """Compiled packed Torch read plan, when available.""" + ... class ManagerActionTerm(Protocol): + """Action-term surface consumed by observation terms.""" + @property - def raw_action(self) -> torch.Tensor: ... + def raw_action(self) -> torch.Tensor: + """Latest raw action tensor received by the term.""" + ... - def process_actions(self, actions: torch.Tensor) -> None: ... + def process_actions(self, actions: torch.Tensor) -> None: + """Process one batch of raw policy actions.""" + ... class ManagerActionManager(Protocol): + """Action-manager surface consumed by manager terms.""" + @property - def action(self) -> torch.Tensor: ... + def action(self) -> torch.Tensor: + """Combined latest action tensor across terms.""" + ... @property - def prev_action(self) -> torch.Tensor: ... + def prev_action(self) -> torch.Tensor: + """Action tensor from the previous step.""" + ... @property - def prev_prev_action(self) -> torch.Tensor: ... + def prev_prev_action(self) -> torch.Tensor: + """Action tensor from two steps ago.""" + ... - def get_term(self, name: str) -> ManagerActionTerm: ... + def get_term(self, name: str) -> ManagerActionTerm: + """Return the action term registered under ``name``.""" + ... - def clear_action_state(self, env_ids: torch.Tensor | None = None) -> None: ... + def clear_action_state(self, env_ids: torch.Tensor | None = None) -> None: + """Clear buffered action state for the selected rows.""" + ... class ManagerCommandManager(Protocol): - def get_command(self, name: str) -> torch.Tensor | None: ... + """Command-manager surface consumed by observation terms.""" - def get_term(self, name: str) -> Any: ... + def get_command(self, name: str) -> torch.Tensor | None: + """Return the named term's current command tensor, or ``None``.""" + ... - def get_term_cfg(self, name: str) -> Any: ... + def get_term(self, name: str) -> Any: + """Return the command term registered under ``name``.""" + ... + + def get_term_cfg(self, name: str) -> Any: + """Return the live config of the command term registered under ``name``.""" + ... class ManagerEventManager(Protocol): """Cold-path event term config surface consumed by curriculum terms.""" - def get_term_cfg(self, term_name: str) -> Any: ... + def get_term_cfg(self, term_name: str) -> Any: + """Return the live config of the event term registered under ``term_name``.""" + ... class ManagerTerminationManager(Protocol): + """Termination-manager surface consumed by manager terms.""" + @property - def terminated(self) -> torch.Tensor: ... + def terminated(self) -> torch.Tensor: + """Per-row termination flags for the current step.""" + ... - def get_term_cfg(self, term_name: str) -> Any: ... + def get_term_cfg(self, term_name: str) -> Any: + """Return the live config of the termination term registered under ``term_name``.""" + ... class ManagerRewardManager(Protocol): """Cold-path reward term config surface consumed by curriculum terms.""" - def get_term_cfg(self, term_name: str) -> Any: ... + def get_term_cfg(self, term_name: str) -> Any: + """Return the live config of the reward term registered under ``term_name``.""" + ... class ManagerBasedRlEnv(Protocol): @@ -262,19 +400,29 @@ class ManagerBasedRlEnv(Protocol): """ @property - def num_envs(self) -> int: ... + def num_envs(self) -> int: + """Number of vectorized environment rows.""" + ... @property - def device(self) -> torch.device: ... + def device(self) -> torch.device: + """Torch device owning the environment tensors.""" + ... @property - def rng(self) -> ManagerRng: ... + def rng(self) -> ManagerRng: + """Manager-owned Torch random generator.""" + ... @property - def cfg(self) -> Any: ... + def cfg(self) -> Any: + """Resolved environment configuration.""" + ... @property - def backend(self) -> SimBackend: ... + def backend(self) -> SimBackend: + """Physics backend bound to this environment.""" + ... tensor_reset_pose_bounds: torch.Tensor | None tensor_reset_velocity_bounds: torch.Tensor | None @@ -284,49 +432,79 @@ def backend(self) -> SimBackend: ... event_term_mask_buffer: ResetMaskBuffer @property - def physics_dt(self) -> float: ... + def physics_dt(self) -> float: + """Physics integration timestep in seconds.""" + ... @property - def step_dt(self) -> float: ... + def step_dt(self) -> float: + """Policy step interval in seconds.""" + ... @property - def scene(self) -> ManagerScene: ... + def scene(self) -> ManagerScene: + """Name-addressable scene surface.""" + ... @property - def action_manager(self) -> ManagerActionManager: ... + def action_manager(self) -> ManagerActionManager: + """Action manager instance.""" + ... @property - def command_manager(self) -> ManagerCommandManager: ... + def command_manager(self) -> ManagerCommandManager: + """Command manager instance.""" + ... @property - def observation_manager(self) -> Any: ... + def observation_manager(self) -> Any: + """Observation manager instance.""" + ... @property - def event_manager(self) -> ManagerEventManager: ... + def event_manager(self) -> ManagerEventManager: + """Event manager instance.""" + ... @property - def termination_manager(self) -> ManagerTerminationManager: ... + def termination_manager(self) -> ManagerTerminationManager: + """Termination manager instance.""" + ... @property - def reward_manager(self) -> ManagerRewardManager: ... + def reward_manager(self) -> ManagerRewardManager: + """Reward manager instance.""" + ... @property - def metrics_manager(self) -> Any: ... + def metrics_manager(self) -> Any: + """Metrics manager instance.""" + ... @property - def episode_length_buf(self) -> torch.Tensor: ... + def episode_length_buf(self) -> torch.Tensor: + """Per-row episode length counters.""" + ... @property - def reset_buf(self) -> torch.Tensor: ... + def reset_buf(self) -> torch.Tensor: + """Per-row reset flags.""" + ... @property - def common_step_counter(self) -> int: ... + def common_step_counter(self) -> int: + """Total number of environment steps taken.""" + ... @property - def max_episode_length(self) -> int: ... + def max_episode_length(self) -> int: + """Maximum episode length in steps.""" + ... @property - def max_episode_length_s(self) -> float: ... + def max_episode_length_s(self) -> float: + """Maximum episode length in seconds.""" + ... # Concrete task terms may still type their own richer env subclass. The # standalone manager core deliberately depends only on the properties above. diff --git a/src/unilab/structured_configs.py b/src/unilab/structured_configs.py index b5bcb5ab9..9f5b69bf1 100644 --- a/src/unilab/structured_configs.py +++ b/src/unilab/structured_configs.py @@ -12,7 +12,10 @@ class BaseConfig: + """Base class for typed configs with a plain-dict export helper.""" + def to_dict(self) -> dict[str, Any]: + """Return the config as a plain nested dict.""" return cast(dict[str, Any], dataclasses.asdict(cast(Any, self))) @@ -21,40 +24,73 @@ def to_dict(self) -> dict[str, Any]: @dataclass class SACAlgoParams: + """SAC entropy-temperature and optimization parameters.""" + + # Learning rate for the entropy temperature (alpha). alpha_lr: float = 3e-4 + # Initial entropy temperature value. alpha_init: float = 0.01 + # Target entropy is -action_dim * ratio; 0 keeps a zero target. target_entropy_ratio: float = 0.0 + # Gradient clipping norm; 0 disables clipping. max_grad_norm: float = 0.0 + # AMP dtype selection ("auto" resolves per device). amp_dtype: str = "auto" + # Enable torch.compile for the update path. use_compile: bool = True @dataclass class SACConfig(BaseConfig): + """Top-level SAC training configuration.""" + + # Algorithm registry key. algo: str = "sac" + # Name used in logs and run directories. algo_log_name: str = "sac" + # Optional runtime implementation override. runtime_impl: Optional[str] = None + # Optional runtime resolver override. runtime_resolver: Optional[str] = None + # Global random seed. seed: int = 1 + # Number of parallel environments. num_envs: int = 4096 # Learner batch size for one SAC update. batch_size: int = 8192 + # Replay buffer length in steps per environment. replay_buffer_n: int = 512 + # Gradient updates per environment step. updates_per_step: int = 4 + # Collection steps before learning starts. learning_starts: int = 1 + # Environment steps between policy updates. policy_frequency: int = 4 + # Maximum number of training iterations. max_iterations: int = 500 + # Iterations between checkpoint saves. save_interval: int = 500 + # Discount factor. gamma: float = 0.97 + # Target network soft-update coefficient. tau: float = 0.125 + # Actor learning rate. actor_lr: float = 3e-4 + # Critic learning rate. critic_lr: float = 3e-4 + # Actor hidden layer width. actor_hidden_dim: int = 512 + # Critic hidden layer width. critic_hidden_dim: int = 768 + # Number of atoms in the distributional critic. num_atoms: int = 101 + # Enable observation normalization. obs_normalization: bool = False + # Enable layer normalization in the networks. use_layer_norm: bool = True + # Extra actor overrides forwarded to the actor factory. actor: dict[str, Any] = field(default_factory=dict) + # Entropy-temperature and optimization parameters. algo_params: SACAlgoParams = field(default_factory=SACAlgoParams) @@ -63,52 +99,99 @@ class SACConfig(BaseConfig): @dataclass class FlashSACAlgoParams: + """FlashSAC learner parameters.""" + + # Normalize rewards before computing returns. normalize_reward: bool = True + # Clamp of the normalized return target magnitude. normalized_g_max: float = 5.0 + # Number of residual blocks in the actor. actor_num_blocks: int = 2 + # Number of residual blocks in the critic. critic_num_blocks: int = 2 + # Behavior-cloning loss weight on the actor; 0 disables it. actor_bc_alpha: float = 0.0 + # Mean of the actor noise zeta schedule. actor_noise_zeta_mu: float = 2.0 + # Maximum of the actor noise zeta schedule. actor_noise_zeta_max: int = 16 + # Lower bound of the distributional critic value support. critic_min_v: float = -5.0 + # Upper bound of the distributional critic value support. critic_max_v: float = 5.0 + # Initial entropy temperature value. temp_initial_value: float = 0.01 + # Target standard deviation for the temperature adaptation. temp_target_sigma: float = 0.15 + # Explicit target entropy; None derives it from the action dimension. temp_target_entropy: float | None = None + # Initial learning rate of the schedule. learning_rate_init: float = 3e-4 + # Peak learning rate of the schedule. learning_rate_peak: float = 3e-4 + # Final learning rate of the schedule. learning_rate_end: float = 1.5e-4 + # Learning-rate warmup steps. learning_rate_warmup_steps: int = 0 + # Learning-rate decay steps. learning_rate_decay_steps: int = 500000 + # N-step return horizon. n_step: int = 1 + # AMP dtype selection ("auto" resolves per device). amp_dtype: str = "auto" + # Enable torch.compile for the update path. use_compile: bool = True + # Compile the full objective functions, not only the networks. compile_full_objectives: bool = True + # Capture the whole training cycle in a CUDA graph. use_whole_cycle_cuda_graph: bool = True @dataclass class FlashSACConfig(BaseConfig): + """Top-level FlashSAC training configuration.""" + + # Algorithm registry key. algo: str = "flashsac" + # Name used in logs and run directories. algo_log_name: str = "flash_sac" + # Global random seed. seed: int = 1 + # Number of parallel environments. num_envs: int = 1024 + # Learner batch size for one update. batch_size: int = 2048 + # Replay buffer length in steps per environment. replay_buffer_n: int = 512 + # Gradient updates per environment step. updates_per_step: int = 2 + # Collection steps before learning starts. learning_starts: int = 98 + # Environment steps between policy updates. policy_frequency: int = 2 + # Maximum number of training iterations. max_iterations: int = 5000 + # Iterations between checkpoint saves. save_interval: int = 1000 + # Discount factor. gamma: float = 0.97 + # Target network soft-update coefficient. tau: float = 0.01 + # Actor learning rate. actor_lr: float = 3e-4 + # Critic learning rate. critic_lr: float = 3e-4 + # Actor hidden layer width. actor_hidden_dim: int = 128 + # Critic hidden layer width. critic_hidden_dim: int = 256 + # Number of atoms in the distributional critic. num_atoms: int = 101 + # Enable observation normalization. obs_normalization: bool = False + # Enable layer normalization in the networks. use_layer_norm: bool = False + # FlashSAC learner parameters. algo_params: FlashSACAlgoParams = field(default_factory=FlashSACAlgoParams) @@ -117,62 +200,113 @@ class FlashSACConfig(BaseConfig): @dataclass class APPOAlgorithmConfig: + """APPO PPO-style objective, schedule, and target-update parameters.""" + + # Learning epochs per collected batch. num_learning_epochs: int = 5 + # Mini-batches per learning epoch. num_mini_batches: int = 4 + # PPO surrogate clipping range. clip_param: float = 0.2 + # Discount factor. gamma: float = 0.99 + # GAE lambda. lam: float = 0.95 + # Value loss coefficient. value_loss_coef: float = 1.0 + # Entropy bonus coefficient. entropy_coef: float = 0.01 + # Optimizer learning rate. learning_rate: float = 1e-3 + # Gradient clipping norm. max_grad_norm: float = 1.0 + # Clip the value loss around the old value estimate. use_clipped_value_loss: bool = True + # Learning-rate schedule ("adaptive" tracks KL). schedule: str = "adaptive" + # Desired KL divergence for the adaptive schedule. desired_kl: float = 0.01 + # KL threshold factor for the adaptive schedule; must exceed 1.0. adaptive_kl_factor: float = 1.2 + # Learning-rate adjustment factor for the adaptive schedule. adaptive_lr_factor: float = 1.1 + # Optimizer name. optimizer: str = "adam" + # Target network soft-update coefficient. tau: float = 1.0 + # Updates between target network refreshes. target_update_freq: int = 1 + # V-trace importance-weight clip for rho. vtrace_clip_rho: float = 1.0 + # V-trace importance-weight clip for c. vtrace_clip_c: float = 1.0 + # Enable torch.compile for the update path. enable_compile: bool = True @dataclass class APPODistributionConfig: + """Action distribution configuration for APPO.""" + + # Distribution class path resolved by the actor factory. class_name: str = "rsl_rl.modules.distribution.GaussianDistribution" + # Initial distribution standard deviation. init_std: float = 1.0 + # Standard-deviation parameterization ("scalar" shares one value). std_type: str = "scalar" @dataclass class APPOActorConfig: + """APPO actor network configuration.""" + + # Actor model class path. class_name: str = "rsl_rl.models.MLPModel" + # Hidden layer widths. hidden_dims: list = field(default_factory=lambda: [512, 256, 128]) + # Activation function name. activation: str = "elu" + # Action distribution configuration. distribution_cfg: APPODistributionConfig = field(default_factory=APPODistributionConfig) @dataclass class APPOCriticConfig: + """APPO critic network configuration.""" + + # Critic model class path. class_name: str = "rsl_rl.models.MLPModel" + # Hidden layer widths. hidden_dims: list = field(default_factory=lambda: [512, 256, 128]) + # Activation function name. activation: str = "elu" @dataclass class APPOConfig(BaseConfig): + """Top-level APPO training configuration.""" + + # Algorithm registry key. algo: str = "appo" + # Name used in logs and run directories. algo_log_name: str = "appo" + # Global random seed. seed: int = 1 + # Number of parallel environments. num_envs: int = 2048 + # Rollout steps collected per environment per iteration. steps_per_env: int = 24 + # Maximum number of training iterations. max_iterations: int = 150 + # Iterations between checkpoint saves. save_interval: int = 50 + # Observation group assignment per network input. obs_groups: dict = field(default_factory=lambda: {"actor": {"policy": 0}}) + # Actor network configuration. actor: APPOActorConfig = field(default_factory=APPOActorConfig) + # Critic network configuration. critic: APPOCriticConfig = field(default_factory=APPOCriticConfig) + # Objective, schedule, and target-update parameters. algorithm: APPOAlgorithmConfig = field(default_factory=APPOAlgorithmConfig) @@ -181,61 +315,113 @@ class APPOConfig(BaseConfig): @dataclass class PPODistributionConfig: + """Action distribution configuration for PPO (rsl-rl).""" + + # Distribution class path resolved by the actor factory. class_name: str = "rsl_rl.modules.distribution.GaussianDistribution" + # Initial distribution standard deviation. init_std: float = 1.0 + # Standard-deviation parameterization ("scalar" shares one value). std_type: str = "scalar" @dataclass class PPOActorConfig: + """PPO actor network configuration.""" + + # Actor model class path. class_name: str = "rsl_rl.models.MLPModel" + # Hidden layer widths. hidden_dims: list = field(default_factory=lambda: [512, 256, 128]) + # Activation function name. activation: str = "elu" + # Enable observation normalization in the actor. obs_normalization: bool = False + # Action distribution configuration. distribution_cfg: PPODistributionConfig = field(default_factory=PPODistributionConfig) @dataclass class PPOCriticConfig: + """PPO critic network configuration.""" + + # Critic model class path. class_name: str = "rsl_rl.models.MLPModel" + # Hidden layer widths. hidden_dims: list = field(default_factory=lambda: [512, 256, 128]) + # Activation function name. activation: str = "elu" + # Enable observation normalization in the critic. obs_normalization: bool = False @dataclass class PPOAlgorithmConfig: + """PPO objective and schedule parameters (rsl-rl).""" + + # Algorithm class path resolved by rsl-rl. class_name: str = "rsl_rl.algorithms:PPO" + # Value loss coefficient. value_loss_coef: float = 1.0 + # Clip the value loss around the old value estimate. use_clipped_value_loss: bool = True + # PPO surrogate clipping range. clip_param: float = 0.2 + # Entropy bonus coefficient. entropy_coef: float = 0.01 + # Learning epochs per collected batch. num_learning_epochs: int = 5 + # Mini-batches per learning epoch. num_mini_batches: int = 4 + # Optimizer learning rate. learning_rate: float = 1e-3 + # Learning-rate schedule ("adaptive" tracks KL). schedule: str = "adaptive" + # Discount factor. gamma: float = 0.99 + # GAE lambda. lam: float = 0.95 + # Desired KL divergence for the adaptive schedule. desired_kl: float = 0.01 + # Gradient clipping norm. max_grad_norm: float = 1.0 @dataclass class PPOConfig(BaseConfig): + """Top-level PPO training configuration (direct rsl-rl integration).""" + + # Algorithm registry key. algo: str = "ppo" + # Name used in logs and run directories. algo_log_name: str = "rsl_rl_ppo" + # Global random seed. seed: int = 1 + # Number of parallel environments. num_envs: int = 4096 + # Rollout steps collected per environment per iteration. num_steps_per_env: int = 24 + # Maximum number of training iterations. max_iterations: int = 101 + # Iterations between checkpoint saves. save_interval: int = 100 + # Observation group assignment per network input. obs_groups: dict = field(default_factory=lambda: {"actor": ["policy"]}) + # Experiment name used for run directories. experiment_name: str = "test" + # Optional run name suffix. run_name: str = "" + # Resume training from a checkpoint. resume: bool = False + # Run directory to load from; -1 selects the most recent. load_run: str = "-1" + # Checkpoint iteration to load; -1 selects the latest. checkpoint: int = -1 + # Explicit checkpoint path overriding run/checkpoint selection. resume_path: Optional[str] = None + # Actor network configuration. actor: PPOActorConfig = field(default_factory=PPOActorConfig) + # Critic network configuration. critic: PPOCriticConfig = field(default_factory=PPOCriticConfig) + # Objective and schedule parameters. algorithm: PPOAlgorithmConfig = field(default_factory=PPOAlgorithmConfig) diff --git a/uv.lock b/uv.lock index 14b24cc4e..30c7cc096 100644 --- a/uv.lock +++ b/uv.lock @@ -1492,6 +1492,23 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" }, ] +[[package]] +name = "interrogate" +version = "1.7.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "attrs" }, + { name = "click" }, + { name = "colorama" }, + { name = "py" }, + { name = "tabulate" }, + { name = "tomli", marker = "python_full_version < '3.11'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/8b/22/74f7fcc96280eea46cf2bcbfa1354ac31de0e60a4be6f7966f12cef20893/interrogate-1.7.0.tar.gz", hash = "sha256:a320d6ec644dfd887cc58247a345054fc4d9f981100c45184470068f4b3719b0", size = 159636, upload-time = "2024-04-07T22:30:46.217Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/12/c9/6869a1dcf4aaf309b9543ec070be3ec3adebee7c9bec9af8c230494134b9/interrogate-1.7.0-py3-none-any.whl", hash = "sha256:b13ff4dd8403369670e2efe684066de9fcb868ad9d7f2b4095d8112142dc9d12", size = 46982, upload-time = "2024-04-07T22:30:44.277Z" }, +] + [[package]] name = "ipython" version = "8.39.0" @@ -3564,6 +3581,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/8e/37/efad0257dc6e593a18957422533ff0f87ede7c9c6ea010a2177d738fb82f/pure_eval-0.2.3-py3-none-any.whl", hash = "sha256:1db8e35b67b3d218d818ae653e27f06c3aa420901fa7b081ca98cbedc874e0d0", size = 11842, upload-time = "2024-07-21T12:58:20.04Z" }, ] +[[package]] +name = "py" +version = "1.11.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/98/ff/fec109ceb715d2a6b4c4a85a61af3b40c723a961e8828319fbcb15b868dc/py-1.11.0.tar.gz", hash = "sha256:51c75c4126074b472f746a24399ad32f6053d1b34b68d2fa41e558e6f4a98719", size = 207796, upload-time = "2021-11-04T17:17:01.377Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f6/f0/10642828a8dfb741e5f3fbaac830550a518a775c7fff6f04a007259b0548/py-1.11.0-py2.py3-none-any.whl", hash = "sha256:607c53218732647dff4acdfcd50cb62615cedf612e72d1724fb1a0cc6405b378", size = 98708, upload-time = "2021-11-04T17:17:00.152Z" }, +] + [[package]] name = "py-cpuinfo" version = "9.0.0" @@ -5225,6 +5251,7 @@ unirl = [ [package.dev-dependencies] dev = [ + { name = "interrogate" }, { name = "mypy" }, { name = "pyright" }, { name = "pytest" }, @@ -5283,6 +5310,7 @@ provides-extras = ["drake", "mujoco", "mjwarp", "newton", "motrix", "genesis", " [package.metadata.requires-dev] dev = [ + { name = "interrogate" }, { name = "mypy" }, { name = "pyright", specifier = ">=1.1.408" }, { name = "pytest" }, diff --git a/uv.rocm.lock b/uv.rocm.lock index 990d8f6d9..0275d8d3a 100644 --- a/uv.rocm.lock +++ b/uv.rocm.lock @@ -1368,6 +1368,23 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" }, ] +[[package]] +name = "interrogate" +version = "1.7.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "attrs" }, + { name = "click" }, + { name = "colorama" }, + { name = "py" }, + { name = "tabulate" }, + { name = "tomli", marker = "python_full_version < '3.11'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/8b/22/74f7fcc96280eea46cf2bcbfa1354ac31de0e60a4be6f7966f12cef20893/interrogate-1.7.0.tar.gz", hash = "sha256:a320d6ec644dfd887cc58247a345054fc4d9f981100c45184470068f4b3719b0", size = 159636, upload-time = "2024-04-07T22:30:46.217Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/12/c9/6869a1dcf4aaf309b9543ec070be3ec3adebee7c9bec9af8c230494134b9/interrogate-1.7.0-py3-none-any.whl", hash = "sha256:b13ff4dd8403369670e2efe684066de9fcb868ad9d7f2b4095d8112142dc9d12", size = 46982, upload-time = "2024-04-07T22:30:44.277Z" }, +] + [[package]] name = "ipython" version = "8.39.0" @@ -3271,6 +3288,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/8e/37/efad0257dc6e593a18957422533ff0f87ede7c9c6ea010a2177d738fb82f/pure_eval-0.2.3-py3-none-any.whl", hash = "sha256:1db8e35b67b3d218d818ae653e27f06c3aa420901fa7b081ca98cbedc874e0d0", size = 11842, upload-time = "2024-07-21T12:58:20.04Z" }, ] +[[package]] +name = "py" +version = "1.11.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/98/ff/fec109ceb715d2a6b4c4a85a61af3b40c723a961e8828319fbcb15b868dc/py-1.11.0.tar.gz", hash = "sha256:51c75c4126074b472f746a24399ad32f6053d1b34b68d2fa41e558e6f4a98719", size = 207796, upload-time = "2021-11-04T17:17:01.377Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f6/f0/10642828a8dfb741e5f3fbaac830550a518a775c7fff6f04a007259b0548/py-1.11.0-py2.py3-none-any.whl", hash = "sha256:607c53218732647dff4acdfcd50cb62615cedf612e72d1724fb1a0cc6405b378", size = 98708, upload-time = "2021-11-04T17:17:00.152Z" }, +] + [[package]] name = "py-cpuinfo" version = "9.0.0" @@ -4278,6 +4304,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/a2/09/77d55d46fd61b4a135c444fc97158ef34a095e5681d0a6c10b75bf356191/sympy-1.14.0-py3-none-any.whl", hash = "sha256:e091cc3e99d2141a0ba2847328f5479b05d94a6635cb96148ccb3f34671bd8f5", size = 6299353, upload-time = "2025-04-27T18:04:59.103Z" }, ] +[[package]] +name = "tabulate" +version = "0.10.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/46/58/8c37dea7bbf769b20d58e7ace7e5edfe65b849442b00ffcdd56be88697c6/tabulate-0.10.0.tar.gz", hash = "sha256:e2cfde8f79420f6deeffdeda9aaec3b6bc5abce947655d17ac662b126e48a60d", size = 91754, upload-time = "2026-03-04T18:55:34.402Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl", hash = "sha256:f0b0622e567335c8fabaaa659f1b33bcb6ddfe2e496071b743aa113f8774f2d3", size = 39814, upload-time = "2026-03-04T18:55:31.284Z" }, +] + [[package]] name = "tenacity" version = "9.1.4" @@ -4740,6 +4775,7 @@ unirl = [ [package.dev-dependencies] dev = [ + { name = "interrogate" }, { name = "mypy" }, { name = "pyright" }, { name = "pytest" }, @@ -4788,6 +4824,7 @@ provides-extras = ["drake", "mujoco", "motrix", "genesis", "unirl"] [package.metadata.requires-dev] dev = [ + { name = "interrogate" }, { name = "mypy" }, { name = "pyright", specifier = ">=1.1.408" }, { name = "pytest" }, From bad6b7a672017d5e2d47617e68c36f5f673c39c9 Mon Sep 17 00:00:00 2001 From: LeeLeno <1846301824@qq.com> Date: Fri, 6 Nov 2026 13:14:26 +0800 Subject: [PATCH 4/8] src: drop docstrings from private helpers (#706) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Keep the P2 docstring baseline on public API surface (public functions, classes, dunder/protocol methods — these feed the api_reference autodoc pages). Remove the one-liners added to private helpers (_validate_*, _materialize_*, etc.), which never render in public docs. interrogate still passes: 49.1% (gate 40%). --- src/unilab/base/reset_state.py | 29 -------------------------- src/unilab/base/torch_env.py | 17 --------------- src/unilab/envs/mdp/actions/actions.py | 8 ------- src/unilab/envs/mdp/observations.py | 7 ------- src/unilab/envs/mdp/ray_observation.py | 1 - src/unilab/managers/manager_base.py | 2 -- 6 files changed, 64 deletions(-) diff --git a/src/unilab/base/reset_state.py b/src/unilab/base/reset_state.py index 69938885e..a4db35ab0 100644 --- a/src/unilab/base/reset_state.py +++ b/src/unilab/base/reset_state.py @@ -73,7 +73,6 @@ def _randomization_term_tail(field: str) -> tuple[int, ...]: - """Return the per-column value tail of a canonical DR field name.""" try: return _RANDOMIZATION_TERM_TAILS[field] except KeyError as exc: @@ -81,7 +80,6 @@ def _randomization_term_tail(field: str) -> tuple[int, ...]: def _readonly_array(values: np.ndarray) -> np.ndarray: - """Return a read-only copy of ``values`` when it is not already immutable.""" result = np.asarray(values) if result.flags.writeable: result = result.copy() @@ -417,7 +415,6 @@ def read_entity_root_pose(self, entity: str, env_ids: torch.Tensor) -> np.ndarra return self._backend.get_entity_state(entity)["root_pose"][host_ids].copy() def _commit_entities(self) -> None: - """Commit staged physical-entity patches through ``reset_entities``.""" assert self.scene_layout is not None if self._entity_rows is None: return @@ -1367,7 +1364,6 @@ def write_root_state_tensor( def _require_tensor_mask_write( self, reset_mask: torch.Tensor, *, capability: str ) -> torch.Tensor: - """Require an active mask transaction covering ``reset_mask`` and return it.""" if not self._tensor_active or self._tensor_active_mask is None: raise RuntimeError( f"ManagerBased tensor mask {capability} requires an active mask reset" @@ -1397,16 +1393,13 @@ def _tensor_mask_scope(self, reset_mask: torch.Tensor, *, capability: str) -> to def _tensor_mask_has_outside_rows( self, reset_mask: torch.Tensor, active_mask: torch.Tensor ) -> bool: - """Whether ``reset_mask`` selects rows outside ``active_mask``.""" return bool((reset_mask & ~active_mask).any()) def _mask_is_empty(self, reset_mask: torch.Tensor) -> bool: - """Whether ``reset_mask`` selects no rows.""" self._mask_count.copy_(reset_mask.sum()) return bool(self._mask_count.item() == 0) def _commit_has_rows(self, commit_mask: torch.Tensor) -> bool: - """Whether ``commit_mask`` selects at least one row.""" return not self._mask_is_empty(commit_mask) def _record_tensor_write_mask(self, reset_mask: torch.Tensor) -> None: @@ -1836,7 +1829,6 @@ def _packed_values(self, values: np.ndarray | torch.Tensor) -> torch.Tensor: return torch.as_tensor(np.ascontiguousarray(values, dtype=np.float32), device=device) def _materialize_default_state(self, term_name: str) -> None: - """Cache immutable default qpos/qvel from the backend on the cold path.""" if self._default_qpos is not None: return qpos = self._selected_default_qpos @@ -1905,7 +1897,6 @@ def _encode_tensor_root_velocity(self, root_state: torch.Tensor) -> torch.Tensor return velocity.contiguous() def _validate_tensor_root_quaternions(self, values: torch.Tensor, *, term_name: str) -> None: - """Require unit-length wxyz quaternions in a staged tensor payload.""" norms = torch.linalg.vector_norm(values, dim=-1) if not bool(torch.isclose(norms, torch.ones_like(norms), rtol=1e-5, atol=1e-6).all()): raise ValueError( @@ -1913,7 +1904,6 @@ def _validate_tensor_root_quaternions(self, values: torch.Tensor, *, term_name: ) def _materialize_default_actuator_gains(self, term_name: str) -> None: - """Cache immutable default actuator kp/kd tables from the backend.""" if self._default_kp is not None: return try: @@ -1957,7 +1947,6 @@ def _materialize_randomization_default( expected_tail: tuple[int, ...], term_name: str, ) -> np.ndarray: - """Bind and cache the immutable backend default table for one DR field.""" cached = self._randomization_defaults.get(field) if cached is not None: return cached @@ -1987,7 +1976,6 @@ def _fetch_reset_term_default( expected_tail: tuple[int, ...], term_name: str, ) -> np.ndarray: - """Fetch and validate one DR field's canonical or per-env default table.""" try: value = self._backend.get_reset_term_default(field) except (AttributeError, NotImplementedError) as exc: @@ -2033,7 +2021,6 @@ def _fetch_reset_term_default( return default def _require_randomization_default(self, field: str, term_name: str) -> np.ndarray: - """Return a DR field's cached default, requiring a prior bind call.""" try: return self._randomization_defaults[field] except KeyError as exc: @@ -2048,7 +2035,6 @@ def _default_is_per_env( *, field: str, ) -> bool: - """Whether a cached default table is stored per environment row.""" canonical_ndim = 1 + len(_randomization_term_tail(field)) if default.ndim == canonical_ndim: return False @@ -2065,7 +2051,6 @@ def _randomization_default_width( *, field: str, ) -> int: - """Return the model-column width of a cached default table.""" axis = 1 if self._default_is_per_env(default, field=field) else 0 return int(default.shape[axis]) @@ -2076,7 +2061,6 @@ def _select_randomization_default_columns( *, field: str, ) -> np.ndarray: - """Select the bound columns from a canonical or per-env default table.""" if self._default_is_per_env(default, field=field): return default[:, columns] return default[columns] @@ -2130,7 +2114,6 @@ def _readonly_binding( columns: np.ndarray, selected_default: np.ndarray, ) -> tuple[np.ndarray, np.ndarray]: - """Return validated columns and selected defaults as read-only arrays.""" return _readonly_array(columns), _readonly_array(selected_default) def _write_selected_randomization( @@ -2143,7 +2126,6 @@ def _write_selected_randomization( value_tail: tuple[int, ...], term_name: str, ) -> None: - """Stage selected columns of one DR field for the given rows.""" ids = self._prepare_state_write( env_ids, capability=field, @@ -2177,7 +2159,6 @@ def _prepare_state_write( capability: str, term_name: str, ) -> torch.Tensor: - """Validate ``env_ids`` against the active transaction and return them.""" self._require_active() if self._tensor_active: return self._prepare_tensor_randomization_write( @@ -2230,7 +2211,6 @@ def _validate_root_layout( *, term_name: str, ) -> tuple[np.ndarray, np.ndarray]: - """Validate a root-state layout against the default qpos/qvel widths.""" if not isinstance(layout, RootStateLayout): raise TypeError( f"EventManager term '{term_name}' root-state layout must be " @@ -2255,7 +2235,6 @@ def _validate_root_layout( def _tensor_root_columns_for_layout( self, layout: RootStateLayout, *, term_name: str ) -> tuple[torch.Tensor, torch.Tensor]: - """Return cached device column tensors for a root-state layout.""" qpos_columns, qvel_columns = self._validate_root_layout(layout, term_name=term_name) key = (tuple(int(i) for i in qpos_columns), tuple(int(i) for i in qvel_columns)) cached = self._tensor_root_columns @@ -2275,7 +2254,6 @@ def _tensor_joint_columns_for_indices( *, term_name: str, ) -> tuple[torch.Tensor, torch.Tensor]: - """Return cached device column tensors for joint qpos/qvel indices.""" assert self._default_qpos is not None and self._default_qvel is not None qpos_columns = self._validate_columns( np.asarray(qpos_indices, dtype=np.intp), @@ -2304,7 +2282,6 @@ def _tensor_joint_columns_for_indices( return qpos_tensor, qvel_tensor def _validate_quaternions(self, values: np.ndarray | torch.Tensor, *, term_name: str) -> None: - """Require unit-length wxyz quaternions in a staged payload.""" if isinstance(values, torch.Tensor): self._validate_tensor_root_quaternions(values, term_name=term_name) return @@ -2322,7 +2299,6 @@ def _validate_state_vector( capability: str, term_name: str, ) -> np.ndarray: - """Validate a 1-D finite floating state vector and return it read-only.""" if not isinstance(value, np.ndarray): raise TypeError( f"EventManager term '{term_name}' capability '{capability}' on backend " @@ -2400,7 +2376,6 @@ def _validate_columns( capability: str, term_name: str, ) -> np.ndarray: - """Validate 1-D integer column indices against the target width.""" if not isinstance(values, np.ndarray): raise TypeError( f"EventManager term '{term_name}' {capability} must be np.ndarray, " @@ -2435,7 +2410,6 @@ def _validate_tensor_values( capability: str, term_name: str, ) -> torch.Tensor: - """Validate shape, dtype, and finiteness of a staged tensor payload.""" if not isinstance(values, torch.Tensor): raise TypeError( f"EventManager term '{term_name}' {capability} must be torch.Tensor, " @@ -2507,19 +2481,16 @@ def _capability_error( capability: str, exc: BaseException, ) -> NotImplementedError: - """Build the fail-closed error for an unsupported reset-state capability.""" return NotImplementedError( f"EventManager term '{term_name}' reset-state capability '{capability}' is " f"unavailable on backend '{self._backend.backend_type}': {exc}" ) def _require_active(self) -> None: - """Require an open host or tensor reset transaction.""" if not (self._active or self._tensor_active): raise RuntimeError("ManagerBased reset-state mutation requires an active reset event") def _finish(self) -> None: - """Discard the packed composer staging and reset the transaction state.""" if self._packed_reset_transaction is not None: self._packed_reset_transaction.discard() self._active = False diff --git a/src/unilab/base/torch_env.py b/src/unilab/base/torch_env.py index 30b313fd5..f83f4e119 100644 --- a/src/unilab/base/torch_env.py +++ b/src/unilab/base/torch_env.py @@ -65,7 +65,6 @@ def replace(self, **updates: Any) -> "TorchEnvState": def _cpu_time() -> float: - """Return the process CPU time (user + system) in seconds.""" process = os.times() return process.user + process.system @@ -147,7 +146,6 @@ def device(self) -> torch.device: return self._device def _bind_tensor_runtime(self) -> None: - """Validate and record the backend's tensor lifecycle capabilities once.""" if self._tensor_runtime_bound: return capabilities = self._backend.get_tensor_capabilities() @@ -444,7 +442,6 @@ def apply_substep_action(self) -> torch.Tensor: ) def _step_physics(self, ctrl: torch.Tensor) -> dict[str, Any] | None: - """Step the backend ``sim_substeps`` times, merging timing and step receipts.""" if not self.requires_substep_state_feedback: return cast( dict[str, Any] | None, @@ -475,14 +472,12 @@ def _step_physics(self, ctrl: torch.Tensor) -> dict[str, Any] | None: return {"timing": backend_timing, "completion": backend_result.get("completion")} def _validate_action(self, actions: torch.Tensor) -> None: - """Validate the action contract and its finite verdict.""" self._validate_action_contract(actions) self._record_action_finite(actions) if not self._defer_finite_validation and not bool(self._action_finite_valid): raise ValueError("TorchEnv action contains NaN or Inf") def _validate_action_contract(self, actions: torch.Tensor) -> None: - """Check type, shape, dtype, contiguity, and device of an action tensor.""" if not isinstance(actions, torch.Tensor): raise TypeError(f"TorchEnv action must be a torch.Tensor, got {type(actions).__name__}") action_shape = self.action_space.shape @@ -521,7 +516,6 @@ def _validate_control_with_nan_guard(self, ctrl: torch.Tensor, *, defer: bool) - raise def _validate_control(self, ctrl: torch.Tensor) -> None: - """Validate the final control tensor after action-term adjustment.""" if not isinstance(ctrl, torch.Tensor): raise TypeError(f"TorchEnv control must be a torch.Tensor, got {type(ctrl).__name__}") if ctrl.dtype != torch.float32: @@ -551,7 +545,6 @@ def _validate_control(self, ctrl: torch.Tensor) -> None: raise ValueError("TorchEnv control contains NaN or Inf") def _validate_state(self, state: TorchEnvState) -> None: - """Validate the transition contract, aggregating finite verdicts on device.""" expected_keys = set(self.obs_groups_spec) if not isinstance(state.obs, dict) or set(state.obs) != expected_keys: raise ValueError("TorchEnvState.obs keys do not match obs_groups_spec") @@ -595,7 +588,6 @@ def _validate_state(self, state: TorchEnvState) -> None: raise ValueError(f"TorchEnvState {label} contains NaN or Inf") def _validate_finite_float(self, value: torch.Tensor, label: str) -> None: - """Require a floating-point, finite tensor.""" if not value.is_floating_point(): raise TypeError(f"TorchEnvState {label} must be floating-point") if not bool(torch.isfinite(value).all()): @@ -626,12 +618,10 @@ def _publish_control_finite_verdict(self) -> torch.Tensor | bool | None: return verdict def _require_floating_float(self, value: torch.Tensor) -> None: - """Require a floating-point tensor.""" if not value.is_floating_point(): raise TypeError("TorchEnvState floating outputs must be floating-point") def _validate_vector(self, value: Any, label: str, dtype: torch.dtype) -> None: - """Validate one ``(num_envs,)`` state vector's type, shape, dtype, and device.""" if not isinstance(value, torch.Tensor): raise TypeError(f"TorchEnvState {label} must be a torch.Tensor") if value.shape != (self._num_envs,): @@ -644,7 +634,6 @@ def _validate_vector(self, value: Any, label: str, dtype: torch.dtype) -> None: def _validate_final_observation( self, final_observation: dict[str, torch.Tensor] | None ) -> None: - """Validate the terminal observation carrier against ``obs``.""" if final_observation is None: return assert self._state is not None @@ -674,7 +663,6 @@ def _validate_final_observation( self._validate_finite_float(value, f"final_observation[{name!r}]") def _compute_truncated(self, state: TorchEnvState) -> torch.Tensor: - """Return per-row truncation flags from the episode step limit.""" self._truncated_scratch.fill_(False) max_steps = self._cfg.max_episode_steps if max_steps is not None: @@ -751,7 +739,6 @@ def _reset_done_envs(self) -> None: timing.update(detail_timing) def _validate_reset_observation_mask(self, obs: Mapping[str, torch.Tensor]) -> None: - """Validate full-width reset observations against the observation contract.""" if not isinstance(obs, dict) or set(obs) != set(self.obs_groups_spec): raise ValueError("TorchEnv mask reset observations do not match obs_groups_spec") for name, dim in self.obs_groups_spec.items(): @@ -786,7 +773,6 @@ def _validate_reset_observation_mask(self, obs: Mapping[str, torch.Tensor]) -> N def _scatter_reset_info_mask( self, reset_info: Mapping[str, Any], reset_mask: torch.Tensor ) -> None: - """Scatter per-row reset info tensors into the state info by mask.""" assert self._state is not None for key, value in reset_info.items(): current = self._state.info.get(key) @@ -799,7 +785,6 @@ def _scatter_reset_info_mask( current.copy_(torch.where(selected_reset_mask, value.to(device=self.device), current)) def _ensure_final_observation_scratch(self) -> dict[str, torch.Tensor]: - """Return the reusable terminal-observation scratch buffers.""" assert self._state is not None obs = self._state.obs scratch = self._final_observation_scratch @@ -818,7 +803,6 @@ def _ensure_final_observation_scratch(self) -> dict[str, torch.Tensor]: return scratch def _clear_reset_done_detail_timing(self, timing: dict[str, Any]) -> None: - """Zero the reset detail timing keys in ``timing``.""" for key in RESET_DONE_DETAIL_TIMING_KEYS: timing[key] = 0.0 @@ -839,7 +823,6 @@ def _collect_step_core_drain_ms(self, state: TorchEnvState) -> float: return 0.0 def _clear_step_final_observation(self) -> None: - """Clear the terminal observation carrier for the new step.""" assert self._state is not None self._state.final_observation = None diff --git a/src/unilab/envs/mdp/actions/actions.py b/src/unilab/envs/mdp/actions/actions.py index 399f16a26..8492a5199 100644 --- a/src/unilab/envs/mdp/actions/actions.py +++ b/src/unilab/envs/mdp/actions/actions.py @@ -24,7 +24,6 @@ def _real(value: Any, *, label: str) -> float: - """Validate that ``value`` is a finite real number and return it as float.""" if isinstance(value, bool) or not isinstance(value, Real): raise TypeError(f"{label} must be a real number, got {type(value).__name__}") result = float(value) @@ -106,7 +105,6 @@ def __init__(self, cfg: BaseActionCfg, env: ManagerBasedRlEnv): def _resolve_affine( self, value: float | dict[str, float], *, default: float, label: str ) -> float | torch.Tensor: - """Resolve a scalar or regex-keyed affine parameter.""" if isinstance(value, dict): indices, resolved = _resolve_named_values( value, self._target_names, label=f"{type(self).__name__} {label}" @@ -125,7 +123,6 @@ def _resolve_affine( return _real(value, label=f"{type(self).__name__} {label}") def _resolve_clip(self, value: dict[str, tuple] | None) -> torch.Tensor | None: - """Resolve regex-keyed ``(min, max)`` clip bounds into a tensor.""" if value is None: return None result = torch.empty((*self._affine_shape, 2), dtype=torch.float32, device=self._device) @@ -152,13 +149,11 @@ def _resolve_clip(self, value: dict[str, tuple] | None) -> torch.Tensor | None: return result def _cold_affine(self, value: float | torch.Tensor) -> float | torch.Tensor: - """Copy a resolved affine parameter onto the target device.""" if isinstance(value, torch.Tensor): return value.to(dtype=torch.float32, device=self._device, copy=True) return value def _cold_clip(self, value: torch.Tensor | None) -> torch.Tensor | None: - """Copy the resolved clip tensor onto the target device.""" if value is None: return None return value.to(dtype=torch.float32, device=self._device, copy=True) @@ -225,7 +220,6 @@ def validate_actions(self) -> None: self.publish_validation_verdict() def _apply_affine(self, source: torch.Tensor, destination: torch.Tensor) -> None: - """Apply scale/offset/clip from ``source`` into ``destination`` in place.""" destination.copy_(source) destination.mul_(self._scale).add_(self._offset) if self._clip is not None: @@ -323,7 +317,6 @@ def reset(self, env_ids: torch.Tensor | None = None) -> None: self._delay_buffer.reset(env_ids) def _delay_target(self, target: torch.Tensor) -> torch.Tensor: - """Push the target through the delay buffer when configured.""" if self._delay_buffer is None: return target self._delay_buffer.append(target) @@ -394,7 +387,6 @@ class RelativeJointPositionAction(BaseAction): """Control joints via position targets relative to current positions.""" def _joint_state(self): - """Return the packed or entity joint state view.""" read_plan = getattr(self._env.scene, "read_plan", None) if read_plan is not None: return read_plan.joint_tensor_view(self._entity) diff --git a/src/unilab/envs/mdp/observations.py b/src/unilab/envs/mdp/observations.py index f18ea4022..9e4d6cf04 100644 --- a/src/unilab/envs/mdp/observations.py +++ b/src/unilab/envs/mdp/observations.py @@ -47,7 +47,6 @@ def require_read_plan(env: ManagerBasedRlEnv, term_name: str) -> Any: def _root_body_name(asset: "Entity", term_name: str) -> str: - """Return the entity's declared root body name or fail closed for ``term_name``.""" root_body_name = getattr(asset, "root_body_name", None) if not isinstance(root_body_name, str) or not root_body_name: raise NotImplementedError( @@ -128,14 +127,12 @@ def __init__(self, cfg: ManagerTermBaseCfg, env: ManagerBasedRlEnv): ) from exc def _resolve_entity_name(self, cfg: ManagerTermBaseCfg) -> str: - """Return the configured ``entity_name`` param, defaulting to ``"robot"``.""" entity_name = cfg.params.get("entity_name") if not isinstance(entity_name, str) or not entity_name: return "robot" return entity_name def _validate_call_name(self, sensor_name: str) -> None: - """Reject a call-time sensor name that differs from the bound one.""" if sensor_name != self._sensor_name: raise ValueError( f"Observation term '{self._term_name}' was bound to sensor " @@ -143,7 +140,6 @@ def _validate_call_name(self, sensor_name: str) -> None: ) def _read(self) -> Any: - """Read the cached sensor view with the term name attached to failures.""" try: return self._view.read() except (KeyError, TypeError, ValueError, NotImplementedError) as exc: @@ -153,7 +149,6 @@ def _read(self) -> Any: ) from exc def _read_tensor(self, env: ManagerBasedRlEnv) -> torch.Tensor: - """Read the sensor through the packed Torch read plan as a finite tensor.""" read_plan = getattr(env.scene, "read_plan", None) if read_plan is None: # Manager construction probes immutable observation shape before @@ -363,7 +358,6 @@ def __init__(self, cfg: ManagerTermBaseCfg, env: ManagerBasedRlEnv): self._quat = _imu_misalignment_quat(env, math.radians(max_angle_deg)) def _validate_max_angle(self, max_angle_deg: float) -> None: - """Reject a call-time ``max_angle_deg`` that differs from the bound one.""" if float(max_angle_deg) != self._max_angle_deg: raise ValueError( f"Observation term '{self._term_name}' was bound to max_angle_deg " @@ -371,7 +365,6 @@ def _validate_max_angle(self, max_angle_deg: float) -> None: ) def _rotate(self, values: Any) -> torch.Tensor: - """Rotate vectors by this term's cached per-env misalignment quaternion.""" device = getattr(self._env, "device", torch.device("cpu")) vectors = torch.as_tensor(values, dtype=torch.float32, device=device) quat = self._quat.to(device=device) diff --git a/src/unilab/envs/mdp/ray_observation.py b/src/unilab/envs/mdp/ray_observation.py index a4d7295ae..9a8364fee 100644 --- a/src/unilab/envs/mdp/ray_observation.py +++ b/src/unilab/envs/mdp/ray_observation.py @@ -131,7 +131,6 @@ def __post_init__(self) -> None: @property def _counts(self) -> tuple[int, int]: - """Per-axis ray counts derived from size and resolution.""" resolution = self.resolution if not isinstance(resolution, Sequence): resolution = (resolution, resolution) diff --git a/src/unilab/managers/manager_base.py b/src/unilab/managers/manager_base.py index 50d61aa0c..84b98040c 100644 --- a/src/unilab/managers/manager_base.py +++ b/src/unilab/managers/manager_base.py @@ -140,11 +140,9 @@ def get_active_iterable_terms(self, env_idx: int) -> Sequence[tuple[str, Sequenc @abc.abstractmethod def _prepare_terms(self) -> None: - """Resolve and instantiate the configured terms.""" raise NotImplementedError def _resolve_common_term_cfg(self, term_name: str, term_cfg: ManagerTermBaseCfg) -> None: - """Resolve ``SceneEntityCfg`` params and instantiate class-based term callables.""" for param_name, value in term_cfg.params.items(): if isinstance(value, SceneEntityCfg): try: From ca51347e2b830e561abd80423d09acc754d7bd91 Mon Sep 17 00:00:00 2001 From: LeeLeno <1846301824@qq.com> Date: Fri, 6 Nov 2026 14:11:16 +0800 Subject: [PATCH 5/8] docs: task-oriented entry points on landing and guide indexes (#706) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The P0/P1 pages were only reachable by walking the section tree. Add an "I want to…" card grid to the top of both landing pages (onboard a robot, custom reward, term tables, DR fields, hyperparameters, diagnostics) and surface the two walkthroughs on the developer-guide index, so new users reach the answer pages in one click. --- docs/sphinx/source/en/0-index.md | 49 +++++++++++++++++++ .../source/en/4-developer_guide/0-index.md | 12 +++++ docs/sphinx/source/zh_CN/0-index.md | 46 +++++++++++++++++ .../source/zh_CN/4-developer_guide/0-index.md | 12 +++++ 4 files changed, 119 insertions(+) diff --git a/docs/sphinx/source/en/0-index.md b/docs/sphinx/source/en/0-index.md index 887729a6b..be4dc1207 100644 --- a/docs/sphinx/source/en/0-index.md +++ b/docs/sphinx/source/en/0-index.md @@ -39,6 +39,55 @@ User guide :::: +## I want to… + +::::{grid} 1 1 2 3 +:gutter: 3 + +:::{grid-item-card} Onboard my own robot +:link: 4-developer_guide/3-extending/5-onboard_new_robot +:link-type: doc +End-to-end walkthrough from URDF/MJCF to a first training run: asset import, +keyframe, minimal owner YAML, with a verification command per step. +::: + +:::{grid-item-card} Add a custom reward +:link: 4-developer_guide/3-extending/6-custom_reward +:link-type: doc +From function definition to the term showing up in training logs: owner YAML +injection, `info["log"]` verification, weight tuning. +::: + +:::{grid-item-card} Look up built-in rewards / observations / terminations +:link: 5-reference/6-manager_terms/0-index +:link-type: doc +Reference tables for every built-in term: parameters, types, defaults, and +semantics, grouped by framework-level and task-layer modules. +::: + +:::{grid-item-card} Configure domain randomization +:link: 2-user_guide/5-domain_randomization/2-field_reference +:link-type: doc +Every DR field with value guidance, lifecycle timing, and minimal YAML +examples, including cross-backend setups. +::: + +:::{grid-item-card} Tune training hyperparameters +:link: 2-user_guide/2-algorithms/0-index +:link-type: doc +Per-algorithm configuration tables for PPO / APPO / SAC / FlashSAC: learning +rate, GAE lam, entropy, reward weights. +::: + +:::{grid-item-card} Debug training problems +:link: 2-user_guide/1-training/8-training_diagnostics +:link-type: doc +Triage paths and tooling for NaNs, reward anomalies, and cross-backend +behavior mismatches. +::: + +:::: + ## Why UniLab ::::{grid} 1 1 3 3 diff --git a/docs/sphinx/source/en/4-developer_guide/0-index.md b/docs/sphinx/source/en/4-developer_guide/0-index.md index 8e8273c27..6a2e845a8 100644 --- a/docs/sphinx/source/en/4-developer_guide/0-index.md +++ b/docs/sphinx/source/en/4-developer_guide/0-index.md @@ -61,6 +61,18 @@ Contract ownership, schema policy, benchmark acceptance, and release transition. Add env config, registration, owner YAMLs, and tests. ::: +:::{grid-item-card} Onboard a new robot (walkthrough) +:link: 3-extending/5-onboard_new_robot +:link-type: doc +End-to-end from URDF/MJCF to a first training run, with verification per step. +::: + +:::{grid-item-card} Custom reward (walkthrough) +:link: 3-extending/6-custom_reward +:link-type: doc +From function definition to the term appearing in training logs. +::: + :::{grid-item-card} New backend :link: 3-extending/2-new_backend :link-type: doc diff --git a/docs/sphinx/source/zh_CN/0-index.md b/docs/sphinx/source/zh_CN/0-index.md index 254ba0709..f811214ff 100644 --- a/docs/sphinx/source/zh_CN/0-index.md +++ b/docs/sphinx/source/zh_CN/0-index.md @@ -38,6 +38,52 @@ UniLab 将任务语义变成可复用配置：组装 manager term、选择物理 :::: +## 我要…… + +::::{grid} 1 1 2 3 +:gutter: 3 + +:::{grid-item-card} 接入我自己的机器人 +:link: 4-developer_guide/3-extending/5-onboard_new_robot +:link-type: doc +从 URDF/MJCF 到首训跑通的端到端演练：资产导入、keyframe、最小 owner +YAML，每步带验证命令。 +::: + +:::{grid-item-card} 添加自定义 reward +:link: 4-developer_guide/3-extending/6-custom_reward +:link-type: doc +从函数定义到 reward term 出现在训练日志：owner YAML 注入、`info["log"]` +验证、权重调整。 +::: + +:::{grid-item-card} 查内置 reward / observation / termination +:link: 5-reference/6-manager_terms/0-index +:link-type: doc +全部内置 term 的参考表：参数、类型、默认值、语义，按框架级与任务层分组。 +::: + +:::{grid-item-card} 配置域随机化（DR） +:link: 2-user_guide/5-domain_randomization/2-field_reference +:link-type: doc +全部 DR 字段的取值线索、生效时机与最小 YAML 示例，含跨 backend 写法。 +::: + +:::{grid-item-card} 调训练超参数 +:link: 2-user_guide/2-algorithms/0-index +:link-type: doc +PPO / APPO / SAC / FlashSAC 各算法页的配置参考表：learning rate、GAE +lam、entropy、reward 权重。 +::: + +:::{grid-item-card} 训练出问题了 +:link: 2-user_guide/1-training/8-training_diagnostics +:link-type: doc +NaN 定位、reward 异常、两 backend 行为不一致的排查路径与工具链。 +::: + +:::: + ## 为什么选择 UniLab ::::{grid} 1 1 3 3 diff --git a/docs/sphinx/source/zh_CN/4-developer_guide/0-index.md b/docs/sphinx/source/zh_CN/4-developer_guide/0-index.md index 5282a7519..5cd4d4724 100644 --- a/docs/sphinx/source/zh_CN/4-developer_guide/0-index.md +++ b/docs/sphinx/source/zh_CN/4-developer_guide/0-index.md @@ -61,6 +61,18 @@ Manager-Based construction、reset、interval 与后端能力边界。 添加 env config、注册、owner YAML 与测试。 ::: +:::{grid-item-card} 接入新机器人（演练） +:link: 3-extending/5-onboard_new_robot +:link-type: doc +从 URDF/MJCF 到首训跑通的端到端流程，每步带验证命令。 +::: + +:::{grid-item-card} 自定义 reward（演练） +:link: 3-extending/6-custom_reward +:link-type: doc +从函数定义到 reward term 出现在训练日志。 +::: + :::{grid-item-card} 新后端 :link: 3-extending/2-new_backend :link-type: doc From 7e9790d44b4ee8184a0c2e350ae253f8441f3977 Mon Sep 17 00:00:00 2001 From: LeeLeno <1846301824@qq.com> Date: Fri, 6 Nov 2026 19:27:57 +0800 Subject: [PATCH 6/8] Revert "docs: task-oriented entry points on landing and guide indexes (#706)" This reverts commit ca51347e2b830e561abd80423d09acc754d7bd91. --- docs/sphinx/source/en/0-index.md | 49 ------------------- .../source/en/4-developer_guide/0-index.md | 12 ----- docs/sphinx/source/zh_CN/0-index.md | 46 ----------------- .../source/zh_CN/4-developer_guide/0-index.md | 12 ----- 4 files changed, 119 deletions(-) diff --git a/docs/sphinx/source/en/0-index.md b/docs/sphinx/source/en/0-index.md index be4dc1207..887729a6b 100644 --- a/docs/sphinx/source/en/0-index.md +++ b/docs/sphinx/source/en/0-index.md @@ -39,55 +39,6 @@ User guide :::: -## I want to… - -::::{grid} 1 1 2 3 -:gutter: 3 - -:::{grid-item-card} Onboard my own robot -:link: 4-developer_guide/3-extending/5-onboard_new_robot -:link-type: doc -End-to-end walkthrough from URDF/MJCF to a first training run: asset import, -keyframe, minimal owner YAML, with a verification command per step. -::: - -:::{grid-item-card} Add a custom reward -:link: 4-developer_guide/3-extending/6-custom_reward -:link-type: doc -From function definition to the term showing up in training logs: owner YAML -injection, `info["log"]` verification, weight tuning. -::: - -:::{grid-item-card} Look up built-in rewards / observations / terminations -:link: 5-reference/6-manager_terms/0-index -:link-type: doc -Reference tables for every built-in term: parameters, types, defaults, and -semantics, grouped by framework-level and task-layer modules. -::: - -:::{grid-item-card} Configure domain randomization -:link: 2-user_guide/5-domain_randomization/2-field_reference -:link-type: doc -Every DR field with value guidance, lifecycle timing, and minimal YAML -examples, including cross-backend setups. -::: - -:::{grid-item-card} Tune training hyperparameters -:link: 2-user_guide/2-algorithms/0-index -:link-type: doc -Per-algorithm configuration tables for PPO / APPO / SAC / FlashSAC: learning -rate, GAE lam, entropy, reward weights. -::: - -:::{grid-item-card} Debug training problems -:link: 2-user_guide/1-training/8-training_diagnostics -:link-type: doc -Triage paths and tooling for NaNs, reward anomalies, and cross-backend -behavior mismatches. -::: - -:::: - ## Why UniLab ::::{grid} 1 1 3 3 diff --git a/docs/sphinx/source/en/4-developer_guide/0-index.md b/docs/sphinx/source/en/4-developer_guide/0-index.md index 6a2e845a8..8e8273c27 100644 --- a/docs/sphinx/source/en/4-developer_guide/0-index.md +++ b/docs/sphinx/source/en/4-developer_guide/0-index.md @@ -61,18 +61,6 @@ Contract ownership, schema policy, benchmark acceptance, and release transition. Add env config, registration, owner YAMLs, and tests. ::: -:::{grid-item-card} Onboard a new robot (walkthrough) -:link: 3-extending/5-onboard_new_robot -:link-type: doc -End-to-end from URDF/MJCF to a first training run, with verification per step. -::: - -:::{grid-item-card} Custom reward (walkthrough) -:link: 3-extending/6-custom_reward -:link-type: doc -From function definition to the term appearing in training logs. -::: - :::{grid-item-card} New backend :link: 3-extending/2-new_backend :link-type: doc diff --git a/docs/sphinx/source/zh_CN/0-index.md b/docs/sphinx/source/zh_CN/0-index.md index f811214ff..254ba0709 100644 --- a/docs/sphinx/source/zh_CN/0-index.md +++ b/docs/sphinx/source/zh_CN/0-index.md @@ -38,52 +38,6 @@ UniLab 将任务语义变成可复用配置：组装 manager term、选择物理 :::: -## 我要…… - -::::{grid} 1 1 2 3 -:gutter: 3 - -:::{grid-item-card} 接入我自己的机器人 -:link: 4-developer_guide/3-extending/5-onboard_new_robot -:link-type: doc -从 URDF/MJCF 到首训跑通的端到端演练：资产导入、keyframe、最小 owner -YAML，每步带验证命令。 -::: - -:::{grid-item-card} 添加自定义 reward -:link: 4-developer_guide/3-extending/6-custom_reward -:link-type: doc -从函数定义到 reward term 出现在训练日志：owner YAML 注入、`info["log"]` -验证、权重调整。 -::: - -:::{grid-item-card} 查内置 reward / observation / termination -:link: 5-reference/6-manager_terms/0-index -:link-type: doc -全部内置 term 的参考表：参数、类型、默认值、语义，按框架级与任务层分组。 -::: - -:::{grid-item-card} 配置域随机化（DR） -:link: 2-user_guide/5-domain_randomization/2-field_reference -:link-type: doc -全部 DR 字段的取值线索、生效时机与最小 YAML 示例，含跨 backend 写法。 -::: - -:::{grid-item-card} 调训练超参数 -:link: 2-user_guide/2-algorithms/0-index -:link-type: doc -PPO / APPO / SAC / FlashSAC 各算法页的配置参考表：learning rate、GAE -lam、entropy、reward 权重。 -::: - -:::{grid-item-card} 训练出问题了 -:link: 2-user_guide/1-training/8-training_diagnostics -:link-type: doc -NaN 定位、reward 异常、两 backend 行为不一致的排查路径与工具链。 -::: - -:::: - ## 为什么选择 UniLab ::::{grid} 1 1 3 3 diff --git a/docs/sphinx/source/zh_CN/4-developer_guide/0-index.md b/docs/sphinx/source/zh_CN/4-developer_guide/0-index.md index 5cd4d4724..5282a7519 100644 --- a/docs/sphinx/source/zh_CN/4-developer_guide/0-index.md +++ b/docs/sphinx/source/zh_CN/4-developer_guide/0-index.md @@ -61,18 +61,6 @@ Manager-Based construction、reset、interval 与后端能力边界。 添加 env config、注册、owner YAML 与测试。 ::: -:::{grid-item-card} 接入新机器人（演练） -:link: 3-extending/5-onboard_new_robot -:link-type: doc -从 URDF/MJCF 到首训跑通的端到端流程，每步带验证命令。 -::: - -:::{grid-item-card} 自定义 reward（演练） -:link: 3-extending/6-custom_reward -:link-type: doc -从函数定义到 reward term 出现在训练日志。 -::: - :::{grid-item-card} 新后端 :link: 3-extending/2-new_backend :link-type: doc From 6d5a485673def9d49433d8c9253c12e1870d3fe4 Mon Sep 17 00:00:00 2001 From: LeeLeno <1846301824@qq.com> Date: Fri, 6 Nov 2026 20:03:52 +0800 Subject: [PATCH 7/8] docs: add top-level 6-how_to section and landing task entries (#706) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Move three task walkthroughs into a new top-level How-To section (git mv, en + zh_CN mirrors): 4-developer_guide/3-extending/5-onboard_new_robot -> 6-how_to/1-onboard_new_robot 4-developer_guide/3-extending/6-custom_reward -> 6-how_to/2-custom_reward 2-user_guide/1-training/8-training_diagnostics -> 6-how_to/3-training_diagnostics - Add 6-how_to/0-index (en + zh_CN) with grid cards and reference quick links; register the section in both language root toctrees. - Fix all relative {doc} links for the new depth and repoint inbound links (reward_porting, new_task, extending/training section indexes). - Add 'I want to...' / '我要……' task-entry grid on both landing pages. - Document the new section in docs/sphinx/AGENTS.md and docs/README.md. --- docs/README.md | 1 + docs/sphinx/AGENTS.md | 5 +++ docs/sphinx/source/en/0-index.md | 45 +++++++++++++++++++ .../en/2-user_guide/1-training/0-index.md | 7 --- .../3-framework_migration/6-reward_porting.md | 2 +- .../4-developer_guide/3-extending/0-index.md | 14 ------ .../3-extending/1-new_task.md | 2 +- docs/sphinx/source/en/6-how_to/0-index.md | 44 ++++++++++++++++++ .../1-onboard_new_robot.md} | 28 ++++++------ .../2-custom_reward.md} | 10 ++--- .../3-training_diagnostics.md} | 18 ++++---- docs/sphinx/source/zh_CN/0-index.md | 44 ++++++++++++++++++ .../zh_CN/2-user_guide/1-training/0-index.md | 7 --- .../3-framework_migration/6-reward_porting.md | 2 +- .../4-developer_guide/3-extending/0-index.md | 14 ------ .../3-extending/1-new_task.md | 2 +- docs/sphinx/source/zh_CN/6-how_to/0-index.md | 44 ++++++++++++++++++ .../1-onboard_new_robot.md} | 28 ++++++------ .../2-custom_reward.md} | 10 ++--- .../3-training_diagnostics.md} | 18 ++++---- 20 files changed, 243 insertions(+), 102 deletions(-) create mode 100644 docs/sphinx/source/en/6-how_to/0-index.md rename docs/sphinx/source/en/{4-developer_guide/3-extending/5-onboard_new_robot.md => 6-how_to/1-onboard_new_robot.md} (91%) rename docs/sphinx/source/en/{4-developer_guide/3-extending/6-custom_reward.md => 6-how_to/2-custom_reward.md} (92%) rename docs/sphinx/source/en/{2-user_guide/1-training/8-training_diagnostics.md => 6-how_to/3-training_diagnostics.md} (88%) create mode 100644 docs/sphinx/source/zh_CN/6-how_to/0-index.md rename docs/sphinx/source/zh_CN/{4-developer_guide/3-extending/5-onboard_new_robot.md => 6-how_to/1-onboard_new_robot.md} (91%) rename docs/sphinx/source/zh_CN/{4-developer_guide/3-extending/6-custom_reward.md => 6-how_to/2-custom_reward.md} (92%) rename docs/sphinx/source/zh_CN/{2-user_guide/1-training/8-training_diagnostics.md => 6-how_to/3-training_diagnostics.md} (87%) diff --git a/docs/README.md b/docs/README.md index 8503e940c..28b01e168 100644 --- a/docs/README.md +++ b/docs/README.md @@ -21,6 +21,7 @@ UniLab 的全部文档源码统一在 [`docs/sphinx/`](sphinx/) 下,通过 Sphin | 中文用户指南 | [`docs/sphinx/source/zh_CN/2-user_guide/`](sphinx/source/zh_CN/2-user_guide/) | | 英文 Developer 指南 | [`docs/sphinx/source/en/4-developer_guide/`](sphinx/source/en/4-developer_guide/) | | 中文 Developer 指南 | [`docs/sphinx/source/zh_CN/4-developer_guide/`](sphinx/source/zh_CN/4-developer_guide/) | +| How-to 指南(端到端任务演练,中英镜像) | [`docs/sphinx/source/en/6-how_to/`](sphinx/source/en/6-how_to/) | | 英文 Deployment(sim-to-real / sim-to-sim) | [`docs/sphinx/source/en/3-deployment/`](sphinx/source/en/3-deployment/) | | 中文 Agent 速查 | [`docs/sphinx/source/zh_CN/4-developer_guide/6-agent_quick_reference.md`](sphinx/source/zh_CN/4-developer_guide/6-agent_quick_reference.md) | | ADR(共享,中文为主) | [`docs/sphinx/source/adr/`](sphinx/source/adr/) | diff --git a/docs/sphinx/AGENTS.md b/docs/sphinx/AGENTS.md index aa6224c6a..bcced05b4 100644 --- a/docs/sphinx/AGENTS.md +++ b/docs/sphinx/AGENTS.md @@ -49,6 +49,11 @@ language. The language switcher uses an explicit path map in `conf.py` page, add a corresponding entry to that map** so the switcher lands somewhere sensible (or omit the entry to fall back to the zh_CN index). +Top-level numbered sections run `1-getting_started` through `6-how_to/`; +`6-how_to/` is the home of user-facing, end-to-end task walkthroughs whose +every step carries a verification command (the en/zh_CN trees mirror it 1:1, +so its pages need no `_LANGUAGE_PATH_FORWARD` entries). + Do not add per-page language button blocks or hand-written cross-language navigation. The sidebar language switcher handles language changes globally. diff --git a/docs/sphinx/source/en/0-index.md b/docs/sphinx/source/en/0-index.md index 887729a6b..537468113 100644 --- a/docs/sphinx/source/en/0-index.md +++ b/docs/sphinx/source/en/0-index.md @@ -39,6 +39,50 @@ User guide :::: +## I want to… + +::::{grid} 1 1 2 3 +:gutter: 3 + +:::{grid-item-card} Onboard my own robot +:link: 6-how_to/1-onboard_new_robot +:link-type: doc +End-to-end walkthrough from URDF/MJCF to a first training run, with a +verification command at every step. +::: + +:::{grid-item-card} Add a custom reward +:link: 6-how_to/2-custom_reward +:link-type: doc +From a function definition to the reward term showing up in the training log. +::: + +:::{grid-item-card} Look up built-in rewards / observations / terminations +:link: 5-reference/6-manager_terms/0-index +:link-type: doc +Reference tables for all built-in terms: parameters, types, defaults, semantics. +::: + +:::{grid-item-card} Configure domain randomization (DR) +:link: 2-user_guide/5-domain_randomization/2-field_reference +:link-type: doc +Value guidance, effective timing, and minimal YAML examples for every DR field. +::: + +:::{grid-item-card} Tune training hyperparameters +:link: 2-user_guide/2-algorithms/0-index +:link-type: doc +Config reference tables on each algorithm page: PPO / APPO / SAC / FlashSAC. +::: + +:::{grid-item-card} Troubleshoot training +:link: 6-how_to/3-training_diagnostics +:link-type: doc +Diagnosis paths for NaN localization, reward anomalies, and backend mismatches. +::: + +:::: + ## Why UniLab ::::{grid} 1 1 3 3 @@ -169,4 +213,5 @@ why_unilab 3-deployment/0-index 4-developer_guide/0-index 5-reference/0-index +6-how_to/0-index ``` diff --git a/docs/sphinx/source/en/2-user_guide/1-training/0-index.md b/docs/sphinx/source/en/2-user_guide/1-training/0-index.md index 7284ebb46..7fd29e864 100644 --- a/docs/sphinx/source/en/2-user_guide/1-training/0-index.md +++ b/docs/sphinx/source/en/2-user_guide/1-training/0-index.md @@ -50,12 +50,6 @@ Run UniLab inside the checked-in Linux NVIDIA image workflow. Reproduce and operate the scoped single-GPU G1 FlashSAC/MJWarp long soak. ::: -:::{grid-item-card} Training diagnostics -:link: 8-training_diagnostics -:link-type: doc -NaN guard dumps, `info["log"]` keys, and common failure-mode playbooks. -::: - :::: ## When to Drop to `scripts/train_*.py` @@ -84,5 +78,4 @@ Day-to-day runs should use the unified CLI. Reach for the low-level 5-resume_and_checkpoints 6-docker 7-tensor_runtime_production -8-training_diagnostics ``` diff --git a/docs/sphinx/source/en/3-deployment/3-framework_migration/6-reward_porting.md b/docs/sphinx/source/en/3-deployment/3-framework_migration/6-reward_porting.md index 25468e3ce..55445927d 100644 --- a/docs/sphinx/source/en/3-deployment/3-framework_migration/6-reward_porting.md +++ b/docs/sphinx/source/en/3-deployment/3-framework_migration/6-reward_porting.md @@ -96,7 +96,7 @@ whether timeouts should contribute to the source task's penalty separately. ## See also - {doc}`5-task_config_translation` -- {doc}`../../4-developer_guide/3-extending/6-custom_reward` — walkthrough for +- {doc}`../../6-how_to/2-custom_reward` — walkthrough for defining and verifying a new reward term end to end - `src/unilab/envs/mdp/rewards.py` - `src/unilab/tasks/locomotion/common/manager_terms.py` diff --git a/docs/sphinx/source/en/4-developer_guide/3-extending/0-index.md b/docs/sphinx/source/en/4-developer_guide/3-extending/0-index.md index 85bcdbf0f..cb37ae4e4 100644 --- a/docs/sphinx/source/en/4-developer_guide/3-extending/0-index.md +++ b/docs/sphinx/source/en/4-developer_guide/3-extending/0-index.md @@ -30,18 +30,6 @@ Add configs, runner code, and script-level assembly. Extend terrain generation on cold paths. ::: -:::{grid-item-card} Onboard a new robot -:link: 5-onboard_new_robot -:link-type: doc -End-to-end walkthrough from URDF/MJCF assets to a first training run. -::: - -:::{grid-item-card} Custom reward term -:link: 6-custom_reward -:link-type: doc -Walkthrough: define, wire, and verify a reward term in the training log. -::: - :::: ```{toctree} @@ -51,6 +39,4 @@ Walkthrough: define, wire, and verify a reward term in the training log. 2-new_backend 3-new_algorithm 4-new_terrain -5-onboard_new_robot -6-custom_reward ``` diff --git a/docs/sphinx/source/en/4-developer_guide/3-extending/1-new_task.md b/docs/sphinx/source/en/4-developer_guide/3-extending/1-new_task.md index 06ee6a4ae..636e3833c 100644 --- a/docs/sphinx/source/en/4-developer_guide/3-extending/1-new_task.md +++ b/docs/sphinx/source/en/4-developer_guide/3-extending/1-new_task.md @@ -47,4 +47,4 @@ Start from the contracts: {doc}`../2-contracts/1-env_contract`, per backend, and the owner YAMLs live under `src/unilab/conf/ppo/task/go2_joystick_flat/`. Task-owned term library example: `src/unilab/tasks/motion_tracking/g1/manager_terms.py`. End-to-end - walkthrough: {doc}`5-onboard_new_robot`. + walkthrough: {doc}`../../6-how_to/1-onboard_new_robot`. diff --git a/docs/sphinx/source/en/6-how_to/0-index.md b/docs/sphinx/source/en/6-how_to/0-index.md new file mode 100644 index 000000000..e4def32a6 --- /dev/null +++ b/docs/sphinx/source/en/6-how_to/0-index.md @@ -0,0 +1,44 @@ +# How-To Guides + +End-to-end walkthroughs for common tasks — each step carries the command that +verifies it, so you can confirm the state of your checkout as you go. + +::::{grid} 1 1 2 3 +:gutter: 3 + +:::{grid-item-card} Onboard a new robot +:link: 1-onboard_new_robot +:link-type: doc +From URDF/MJCF assets to a first PPO training run, one verified step at a time. +::: + +:::{grid-item-card} Add a custom reward term +:link: 2-custom_reward +:link-type: doc +From a function definition to the `reward/` key in the training log. +::: + +:::{grid-item-card} Diagnose training problems +:link: 3-training_diagnostics +:link-type: doc +NaN guard dumps, reward anomalies, and backend-mismatch playbooks. +::: + +:::: + +## Reference Quick Links + +- Built-in reward / observation / termination terms: + {doc}`../5-reference/6-manager_terms/0-index` +- Domain randomization fields: + {doc}`../2-user_guide/5-domain_randomization/2-field_reference` +- Algorithm pages with per-algorithm config reference tables: + {doc}`../2-user_guide/2-algorithms/0-index` + +```{toctree} +:hidden: + +1-onboard_new_robot +2-custom_reward +3-training_diagnostics +``` diff --git a/docs/sphinx/source/en/4-developer_guide/3-extending/5-onboard_new_robot.md b/docs/sphinx/source/en/6-how_to/1-onboard_new_robot.md similarity index 91% rename from docs/sphinx/source/en/4-developer_guide/3-extending/5-onboard_new_robot.md rename to docs/sphinx/source/en/6-how_to/1-onboard_new_robot.md index af3f2d618..20a3ce8de 100644 --- a/docs/sphinx/source/en/4-developer_guide/3-extending/5-onboard_new_robot.md +++ b/docs/sphinx/source/en/6-how_to/1-onboard_new_robot.md @@ -3,9 +3,9 @@ This walkthrough goes from a URDF/MJCF asset to a first PPO training run — `uv run train --algo ppo --task --sim ` — one verified step at a time. It composes existing owner layers instead of adding new code paths: assets -({doc}`../../2-user_guide/7-tooling/5-robot_import`), a task/scene keyframe, -owner YAMLs ({doc}`../../2-user_guide/1-training/2-hydra_config`), and a -registry entry (checklist view: {doc}`1-new_task`). +({doc}`../2-user_guide/7-tooling/5-robot_import`), a task/scene keyframe, +owner YAMLs ({doc}`../2-user_guide/1-training/2-hydra_config`), and a +registry entry (checklist view: {doc}`../4-developer_guide/3-extending/1-new_task`). Steps that need a simulator install, a display, or a GPU are marked. The compose-only checks (`--cfg job`) still pass through the CLI's backend @@ -25,7 +25,7 @@ Three names, used consistently everywhere: ## 1. Prepare the Assets -Follow {doc}`../../2-user_guide/7-tooling/5-robot_import`. For a URDF source: +Follow {doc}`../2-user_guide/7-tooling/5-robot_import`. For a URDF source: ```bash uv run scripts/tools/import_robot.py [robot_name] @@ -96,7 +96,7 @@ grep -c '/.xml ## 3. Write the Owner YAMLs PPO resolves each task×backend pair to two files (composition rules: -{doc}`../../2-user_guide/1-training/2-hydra_config`): +{doc}`../2-user_guide/1-training/2-hydra_config`): - `src/unilab/conf/ppo/task//base.yaml` — the shared task declaration. - `src/unilab/conf/ppo/task//mujoco.yaml` — the backend owner: inherits @@ -202,7 +202,7 @@ Notes on the minimal set: - The framework tracking terms (`unilab.envs.mdp.track_linear_velocity` / `track_angular_velocity`) read root state instead of named sensors, so they work on any robot whose command is named `twist`; parameters and semantics - are in {doc}`../../5-reference/6-manager_terms/1-rewards`. + are in {doc}`../5-reference/6-manager_terms/1-rewards`. - Add a `critic` observation group plus `env.critic_observation_group: critic` when the algorithm should consume privileged observations (the Go2 owner does both). @@ -253,7 +253,7 @@ Then make `ensure_registries()` import the module by adding it to the `__unilab_registry_modules__` tuple in `src/unilab/tasks/__init__.py` (or in your family package's `__init__.py`). Third-party packages outside this repo register through the `unilab.tasks` entry-point group instead — see -{doc}`../1-architecture/5-registry` and `src/unilab/base/registry.py`. +{doc}`../4-developer_guide/1-architecture/5-registry` and `src/unilab/base/registry.py`. Verify — host-only, no backend needed: @@ -274,8 +274,8 @@ Start with `mujoco` (the default backend). Each additional backend needs: 2. a matching owner YAML `src/unilab/conf/ppo/task//.yaml` with `training.sim_backend: `; 3. evidence that the backend serves the terms your task binds — check the - generated {doc}`../../5-reference/5-support_matrix` and, for DR terms, - {doc}`../../2-user_guide/5-domain_randomization/2-field_reference`. + generated {doc}`../5-reference/5-support_matrix` and, for DR terms, + {doc}`../2-user_guide/5-domain_randomization/2-field_reference`. Unsupported capabilities fail closed at construction; set a term the backend cannot serve to `null` in that backend's owner (pattern: `pd_gains: null` in `src/unilab/conf/ppo/task/go2_joystick_flat/motrix.yaml`). @@ -298,13 +298,13 @@ First-run checkpoints: field — for example, an observation term bound to a missing sensor raises at manager construction. Fix the owner YAML and rerun. 2. The run directory appears under `logs/rsl_rl_ppo//` with - `run_config.json` (details: {doc}`../../2-user_guide/1-training/3-logging`). + `run_config.json` (details: {doc}`../2-user_guide/1-training/3-logging`). 3. Per-term reward keys `reward/` and termination counters `Episode_Termination/` show up in TensorBoard (`uv run tensorboard --logdir logs/rsl_rl_ppo/`). If any reward or observation goes NaN/Inf, training raises with the term name and the NaN guard writes a state dump — see - {doc}`../../2-user_guide/1-training/8-training_diagnostics`. + {doc}`3-training_diagnostics`. Then scale back up (the owner values for `algo.num_envs` / `algo.max_iterations`) and evaluate the latest run: @@ -315,8 +315,8 @@ uv run eval --algo ppo --task --sim mujoco --load-run -1 ## Where to Go Next -- Add or tune reward terms: {doc}`6-custom_reward`. -- Built-in term tables: {doc}`../../5-reference/6-manager_terms/0-index`. -- Checklist view of the same contract: {doc}`1-new_task`. +- Add or tune reward terms: {doc}`2-custom_reward`. +- Built-in term tables: {doc}`../5-reference/6-manager_terms/0-index`. +- Checklist view of the same contract: {doc}`../4-developer_guide/3-extending/1-new_task`. - Interactive visual smoke check of the spawned scene (needs a display): `uv run scripts/visualize_task_env.py --task --backend mujoco --num_envs 4`. diff --git a/docs/sphinx/source/en/4-developer_guide/3-extending/6-custom_reward.md b/docs/sphinx/source/en/6-how_to/2-custom_reward.md similarity index 92% rename from docs/sphinx/source/en/4-developer_guide/3-extending/6-custom_reward.md rename to docs/sphinx/source/en/6-how_to/2-custom_reward.md index 1fe9e578c..f949f0a05 100644 --- a/docs/sphinx/source/en/4-developer_guide/3-extending/6-custom_reward.md +++ b/docs/sphinx/source/en/6-how_to/2-custom_reward.md @@ -88,7 +88,7 @@ Manager-Based environments publish per-term reward diagnostics as `RewardManager.step_reward_extras()` (`src/unilab/managers/reward_manager.py`, mean across environments of `raw_value * weight`, before `dt` scaling) — and the training logger forwards -them to TensorBoard / W&B ({doc}`../../2-user_guide/1-training/3-logging`): +them to TensorBoard / W&B ({doc}`../2-user_guide/1-training/3-logging`): ```bash uv run tensorboard --logdir logs/rsl_rl_ppo/ @@ -99,7 +99,7 @@ If the term misbehaves, the failure is explicit: construction errors name the term, a wrong return type/shape raises from `RewardManager`, and a NaN/Inf return raises `RewardManager term '' returned NaN/Inf` (plus a NaN-guard state dump when enabled). Diagnosis workflow: -{doc}`../../2-user_guide/1-training/8-training_diagnostics`. +{doc}`3-training_diagnostics`. ## 4. Tune the Weight @@ -112,10 +112,10 @@ uv run train --algo ppo --task --sim mujoco \ This works because `reward` is a root-level Hydra section of the composed config (see the "Reward term weights" section of -{doc}`../../2-user_guide/2-algorithms/1-ppo`). Add new terms in the owner YAML +{doc}`../2-user_guide/2-algorithms/1-ppo`). Add new terms in the owner YAML itself so the task stays reproducible; the CLI override is for sweeps. When porting terms from another framework, check the built-in tables first — -the term may already exist: {doc}`../../5-reference/6-manager_terms/1-rewards`. +the term may already exist: {doc}`../5-reference/6-manager_terms/1-rewards`. The Legged Gym mapping guide is -{doc}`../../3-deployment/3-framework_migration/6-reward_porting`. +{doc}`../3-deployment/3-framework_migration/6-reward_porting`. diff --git a/docs/sphinx/source/en/2-user_guide/1-training/8-training_diagnostics.md b/docs/sphinx/source/en/6-how_to/3-training_diagnostics.md similarity index 88% rename from docs/sphinx/source/en/2-user_guide/1-training/8-training_diagnostics.md rename to docs/sphinx/source/en/6-how_to/3-training_diagnostics.md index 7115a0c8c..1ed9b303e 100644 --- a/docs/sphinx/source/en/2-user_guide/1-training/8-training_diagnostics.md +++ b/docs/sphinx/source/en/6-how_to/3-training_diagnostics.md @@ -49,14 +49,14 @@ on-device and writes one state dump at an explicit host boundary. The chain: replays the buffered states. Backends without physics-state playback ship no states in the dump — the viewer prints the metadata only. -Details and the viewer walkthrough: {doc}`../7-tooling/3-nan_visualizer`. +Details and the viewer walkthrough: {doc}`../2-user_guide/7-tooling/3-nan_visualizer`. Round-trip coverage: `tests/training/test_tensor_diagnostics.py`. ## Reading `info["log"]` Manager-Based environments place structured metrics in `info["log"]`; the training logger forwards slash-prefixed entries to TensorBoard / W&B. The full -semantics table is in {doc}`3-logging`; the key families and their producers: +semantics table is in {doc}`../2-user_guide/1-training/3-logging`; the key families and their producers: | Key | Producer | Meaning | | --- | --- | --- | @@ -65,7 +65,7 @@ semantics table is in {doc}`3-logging`; the key families and their producers: | `Episode_Metrics/` | `MetricsManager` (`src/unilab/managers/metrics_manager.py`) | Completed-episode metric aggregates | Use `Train/mean_reward` and `Train/mean_episode_length` for overall progress -and `reward/` to attribute it; see {doc}`3-logging` for the distinction. +and `reward/` to attribute it; see {doc}`../2-user_guide/1-training/3-logging` for the distinction. ## Common Failure Modes @@ -80,7 +80,7 @@ and `reward/` to attribute it; see {doc}`3-logging` for the distinction. - Fix: retune the weight from the CLI, e.g. `uv run train --algo ppo --task --sim mujoco reward..weight=-0.01` (override mechanics: the "Reward term weights" - section of {doc}`../2-algorithms/1-ppo`). Keep the smoothness/regularization + section of {doc}`../2-user_guide/2-algorithms/1-ppo`). Keep the smoothness/regularization terms (`action_rate`-style penalties) enabled; dropping them commonly destabilizes the action stream. @@ -88,7 +88,7 @@ and `reward/` to attribute it; see {doc}`3-logging` for the distinction. - Symptom: `Train/mean_reward` stays flat while iterations advance. - Check the hyperparameters against the per-algorithm reference tables - ({doc}`../2-algorithms/1-ppo` for PPO): `algo.algorithm.learning_rate`, + ({doc}`../2-user_guide/2-algorithms/1-ppo` for PPO): `algo.algorithm.learning_rate`, `algo.algorithm.entropy_coef`, `algo.algorithm.lam`. Check `algo.num_envs` — it is a per-rank count, and very small counts starve the gradient estimate. Check observation conditioning: @@ -111,8 +111,8 @@ and `reward/` to attribute it; see {doc}`3-logging` for the distinction. `CrossBackendIncompatibleError`; WARNING_LIST fields only print a warning. Audit the shipped pairs with `uv run scripts/audit_sim2sim_contracts.py`. - Check capability differences per backend: - {doc}`../../3-deployment/2-sim_to_sim/6-capability_gaps`; exact support - grades: {doc}`../../5-reference/5-support_matrix`. + {doc}`../3-deployment/2-sim_to_sim/6-capability_gaps`; exact support + grades: {doc}`../5-reference/5-support_matrix`. ### NaN appears mid-run @@ -129,7 +129,7 @@ and `reward/` to attribute it; see {doc}`3-logging` for the distinction. `training.trace_output_dir`, `training.trace_thread_time`, and `training.trace_cuda_events` (`src/unilab/conf/sac/config.yaml`), plus `training.nvtx_profile_ranges` for NVTX annotations on CUDA. Traces are - Perfetto JSON; scalar timing keys are documented in {doc}`3-logging`. + Perfetto JSON; scalar timing keys are documented in {doc}`../2-user_guide/1-training/3-logging`. - Analyze a captured trace host-side (no GPU needed): ```bash @@ -140,4 +140,4 @@ and `reward/` to attribute it; see {doc}`3-logging` for the distinction. The script reports mean/median/p90/p95/p99 durations per event name; `--steps-per-cycle` and `--drop-first` adjust the throughput summary. - For the production long-soak setup and its failure classification fields, - see {doc}`7-tensor_runtime_production`. + see {doc}`../2-user_guide/1-training/7-tensor_runtime_production`. diff --git a/docs/sphinx/source/zh_CN/0-index.md b/docs/sphinx/source/zh_CN/0-index.md index 254ba0709..5e856d1f4 100644 --- a/docs/sphinx/source/zh_CN/0-index.md +++ b/docs/sphinx/source/zh_CN/0-index.md @@ -38,6 +38,49 @@ UniLab 将任务语义变成可复用配置：组装 manager term、选择物理 :::: +## 我要…… + +::::{grid} 1 1 2 3 +:gutter: 3 + +:::{grid-item-card} 接入我自己的机器人 +:link: 6-how_to/1-onboard_new_robot +:link-type: doc +从 URDF/MJCF 到首训跑通的端到端演练，每步带验证命令。 +::: + +:::{grid-item-card} 添加自定义 reward +:link: 6-how_to/2-custom_reward +:link-type: doc +从函数定义到 reward term 出现在训练日志。 +::: + +:::{grid-item-card} 查内置 reward / observation / termination +:link: 5-reference/6-manager_terms/0-index +:link-type: doc +全部内置 term 的参考表：参数、类型、默认值、语义。 +::: + +:::{grid-item-card} 配置域随机化（DR） +:link: 2-user_guide/5-domain_randomization/2-field_reference +:link-type: doc +全部 DR 字段的取值线索、生效时机与最小 YAML 示例。 +::: + +:::{grid-item-card} 调训练超参数 +:link: 2-user_guide/2-algorithms/0-index +:link-type: doc +PPO / APPO / SAC / FlashSAC 各算法页的配置参考表。 +::: + +:::{grid-item-card} 训练出问题了 +:link: 6-how_to/3-training_diagnostics +:link-type: doc +NaN 定位、reward 异常、backend 不一致的排查路径。 +::: + +:::: + ## 为什么选择 UniLab ::::{grid} 1 1 3 3 @@ -161,4 +204,5 @@ why_unilab 3-deployment/0-index 4-developer_guide/0-index 5-reference/0-index +6-how_to/0-index ``` diff --git a/docs/sphinx/source/zh_CN/2-user_guide/1-training/0-index.md b/docs/sphinx/source/zh_CN/2-user_guide/1-training/0-index.md index 76c6b4e2e..c6c2f2078 100644 --- a/docs/sphinx/source/zh_CN/2-user_guide/1-training/0-index.md +++ b/docs/sphinx/source/zh_CN/2-user_guide/1-training/0-index.md @@ -49,12 +49,6 @@ TensorBoard、W&B、运行元数据以及 trace 选项。 复现并运维 scoped 单 GPU G1 FlashSAC/MJWarp 长 soak。 ::: -:::{grid-item-card} 训练诊断 -:link: 8-training_diagnostics -:link-type: doc -NaN guard dump、`info["log"]` 键与常见失败模式排查手册。 -::: - :::: ## 什么时候下沉到 `scripts/train_*.py` @@ -83,5 +77,4 @@ NaN guard dump、`info["log"]` 键与常见失败模式排查手册。 5-resume_and_checkpoints 6-docker 7-tensor_runtime_production -8-training_diagnostics ``` diff --git a/docs/sphinx/source/zh_CN/3-deployment/3-framework_migration/6-reward_porting.md b/docs/sphinx/source/zh_CN/3-deployment/3-framework_migration/6-reward_porting.md index affc3573d..58355c865 100644 --- a/docs/sphinx/source/zh_CN/3-deployment/3-framework_migration/6-reward_porting.md +++ b/docs/sphinx/source/zh_CN/3-deployment/3-framework_migration/6-reward_porting.md @@ -88,7 +88,7 @@ command gating，再通过固定轨迹比较逐项输出。 ## 参考 - {doc}`5-task_config_translation` -- {doc}`../../4-developer_guide/3-extending/6-custom_reward` —— 端到端定义并验证 +- {doc}`../../6-how_to/2-custom_reward` —— 端到端定义并验证 新 reward term 的演练 - `src/unilab/envs/mdp/rewards.py` - `src/unilab/tasks/locomotion/common/manager_terms.py` diff --git a/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/0-index.md b/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/0-index.md index a24196239..838e9213b 100644 --- a/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/0-index.md +++ b/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/0-index.md @@ -24,18 +24,6 @@ 添加配置、runner 代码与脚本层组装。 ::: -:::{grid-item-card} 接入新机器人 -:link: 5-onboard_new_robot -:link-type: doc -端到端演练：从 URDF/MJCF 资产到首次训练跑通。 -::: - -:::{grid-item-card} 自定义 reward term -:link: 6-custom_reward -:link-type: doc -演练：定义、接入并在训练日志中验证一个 reward term。 -::: - :::: ```{toctree} @@ -45,6 +33,4 @@ 2-new_backend 3-new_algorithm 4-new_terrain -5-onboard_new_robot -6-custom_reward ``` diff --git a/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/1-new_task.md b/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/1-new_task.md index d0eb2ce17..20e9c80a3 100644 --- a/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/1-new_task.md +++ b/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/1-new_task.md @@ -46,4 +46,4 @@ `make_manager_based_rl_env` 按后端注册 `Go2JoystickFlat`，owner YAML 位于 `src/unilab/conf/ppo/task/go2_joystick_flat/`。任务自有 term 库示例： `src/unilab/tasks/motion_tracking/g1/manager_terms.py`。端到端演练见 - {doc}`5-onboard_new_robot`。 + {doc}`../../6-how_to/1-onboard_new_robot`。 diff --git a/docs/sphinx/source/zh_CN/6-how_to/0-index.md b/docs/sphinx/source/zh_CN/6-how_to/0-index.md new file mode 100644 index 000000000..dfd10ea85 --- /dev/null +++ b/docs/sphinx/source/zh_CN/6-how_to/0-index.md @@ -0,0 +1,44 @@ +# How-To 指南 + +常见任务的端到端操作指南——每一步都附带验证命令，可以边做边确认本地 +检出的状态。 + +::::{grid} 1 1 2 3 +:gutter: 3 + +:::{grid-item-card} 接入新机器人 +:link: 1-onboard_new_robot +:link-type: doc +从 URDF/MJCF 资产到首次 PPO 训练跑通，每步都带验证。 +::: + +:::{grid-item-card} 添加自定义 reward term +:link: 2-custom_reward +:link-type: doc +从函数定义到训练日志中出现 `reward/` 键。 +::: + +:::{grid-item-card} 诊断训练问题 +:link: 3-training_diagnostics +:link-type: doc +NaN guard dump、reward 异常与后端不一致的排查手册。 +::: + +:::: + +## 参考速查 + +- 内置 reward / observation / termination term： + {doc}`../5-reference/6-manager_terms/0-index` +- 域随机化（DR）字段： + {doc}`../2-user_guide/5-domain_randomization/2-field_reference` +- 各算法页含配置参考表： + {doc}`../2-user_guide/2-algorithms/0-index` + +```{toctree} +:hidden: + +1-onboard_new_robot +2-custom_reward +3-training_diagnostics +``` diff --git a/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/5-onboard_new_robot.md b/docs/sphinx/source/zh_CN/6-how_to/1-onboard_new_robot.md similarity index 91% rename from docs/sphinx/source/zh_CN/4-developer_guide/3-extending/5-onboard_new_robot.md rename to docs/sphinx/source/zh_CN/6-how_to/1-onboard_new_robot.md index 7672326e4..9358b4f5f 100644 --- a/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/5-onboard_new_robot.md +++ b/docs/sphinx/source/zh_CN/6-how_to/1-onboard_new_robot.md @@ -3,9 +3,9 @@ 本演练从 URDF/MJCF 资产出发，一步步走到首次 PPO 训练跑通—— `uv run train --algo ppo --task --sim `——每步都带验证。 它只组合现有的 owner 层，不新增代码路径：资产 -（{doc}`../../2-user_guide/7-tooling/5-robot_import`）、任务/场景 keyframe、 -owner YAML（{doc}`../../2-user_guide/1-training/2-hydra_config`）以及 registry -条目（清单视角见 {doc}`1-new_task`）。 +（{doc}`../2-user_guide/7-tooling/5-robot_import`）、任务/场景 keyframe、 +owner YAML（{doc}`../2-user_guide/1-training/2-hydra_config`）以及 registry +条目（清单视角见 {doc}`../4-developer_guide/3-extending/1-new_task`）。 需要安装仿真器、显示器或 GPU 的步骤都已标注。纯组合检查（`--cfg job`）仍会 经过 CLI 的后端预检，因此 `--sim mujoco` 需要 `mujoco` extra @@ -23,7 +23,7 @@ owner YAML（{doc}`../../2-user_guide/1-training/2-hydra_config`）以及 regist ## 1. 准备资产 -按 {doc}`../../2-user_guide/7-tooling/5-robot_import` 操作。URDF 源： +按 {doc}`../2-user_guide/7-tooling/5-robot_import` 操作。URDF 源： ```bash uv run scripts/tools/import_robot.py [robot_name] @@ -88,7 +88,7 @@ grep -c '/.xml ## 3. 编写 owner YAML PPO 把每个 task×backend 组合解析为两个文件（组合规则见 -{doc}`../../2-user_guide/1-training/2-hydra_config`）： +{doc}`../2-user_guide/1-training/2-hydra_config`）： - `src/unilab/conf/ppo/task//base.yaml` —— 共享的任务声明。 - `src/unilab/conf/ppo/task//mujoco.yaml` —— 后端 owner：继承 base， @@ -194,7 +194,7 @@ reward: - 框架级跟踪 term（`unilab.envs.mdp.track_linear_velocity` / `track_angular_velocity`）读 root state 而不是具名传感器，只要 command 名为 `twist` 就能用于任意机器人；参数与语义见 - {doc}`../../5-reference/6-manager_terms/1-rewards`。 + {doc}`../5-reference/6-manager_terms/1-rewards`。 - 当算法需要消费特权观测时，再加 `critic` 观测组和 `env.critic_observation_group: critic`（Go2 owner 两者都有）。 @@ -242,7 +242,7 @@ registry.register_env("", make_manager_based_rl_env, sim_backend=" 然后把该模块加入 `src/unilab/tasks/__init__.py`（或你的家族包 `__init__.py`）的 `__unilab_registry_modules__` 元组，让 `ensure_registries()` 导入它。仓库外的第三方任务包改用 `unilab.tasks` -entry-point 组注册——见 {doc}`../1-architecture/5-registry` 与 +entry-point 组注册——见 {doc}`../4-developer_guide/1-architecture/5-registry` 与 `src/unilab/base/registry.py`。 验证——纯主机侧，不需要后端： @@ -264,8 +264,8 @@ print(registry.list_registered_envs()['']) 2. 对应的 owner YAML `src/unilab/conf/ppo/task//.yaml`，其中 `training.sim_backend: `； 3. 该后端能提供你的任务所绑定 term 的证据——查生成的 - {doc}`../../5-reference/5-support_matrix`，DR term 另查 - {doc}`../../2-user_guide/5-domain_randomization/2-field_reference`。 + {doc}`../5-reference/5-support_matrix`，DR term 另查 + {doc}`../2-user_guide/5-domain_randomization/2-field_reference`。 不支持的能力在构造期 fail closed；后端无法服务的 term 在该后端 owner 里 置为 `null`（模式：`src/unilab/conf/ppo/task/go2_joystick_flat/motrix.yaml` 中的 `pd_gains: null`）。 @@ -286,12 +286,12 @@ uv run train --algo ppo --task --sim mujoco \ 1. 组合与 env 构造会快速失败并指出有问题的 term 或字段——例如观测 term 绑定了不存在的传感器时会在 manager 构造期抛错。修 owner YAML 后重跑。 2. 运行目录出现在 `logs/rsl_rl_ppo//`，其中包含 `run_config.json` - （细节见 {doc}`../../2-user_guide/1-training/3-logging`）。 + （细节见 {doc}`../2-user_guide/1-training/3-logging`）。 3. TensorBoard 中出现逐项 reward 键 `reward/` 与终止计数器 `Episode_Termination/` （`uv run tensorboard --logdir logs/rsl_rl_ppo/`）。任何 reward 或 观测出现 NaN/Inf 时，训练会带着 term 名抛错，并且 NaN guard 会写出状态 - dump——见 {doc}`../../2-user_guide/1-training/8-training_diagnostics`。 + dump——见 {doc}`3-training_diagnostics`。 然后恢复到 owner 的规模（`algo.num_envs` / `algo.max_iterations`），并评估 最近一次运行： @@ -302,8 +302,8 @@ uv run eval --algo ppo --task --sim mujoco --load-run -1 ## 下一步 -- 添加或调权 reward term：{doc}`6-custom_reward`。 -- 内置 term 速查表：{doc}`../../5-reference/6-manager_terms/0-index`。 -- 同一契约的清单视角：{doc}`1-new_task`。 +- 添加或调权 reward term：{doc}`2-custom_reward`。 +- 内置 term 速查表：{doc}`../5-reference/6-manager_terms/0-index`。 +- 同一契约的清单视角：{doc}`../4-developer_guide/3-extending/1-new_task`。 - 对出生场景做交互式目视冒烟检查（需要显示器）： `uv run scripts/visualize_task_env.py --task --backend mujoco --num_envs 4`。 diff --git a/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/6-custom_reward.md b/docs/sphinx/source/zh_CN/6-how_to/2-custom_reward.md similarity index 92% rename from docs/sphinx/source/zh_CN/4-developer_guide/3-extending/6-custom_reward.md rename to docs/sphinx/source/zh_CN/6-how_to/2-custom_reward.md index f23829d82..fffcc0bf2 100644 --- a/docs/sphinx/source/zh_CN/4-developer_guide/3-extending/6-custom_reward.md +++ b/docs/sphinx/source/zh_CN/6-how_to/2-custom_reward.md @@ -84,7 +84,7 @@ Manager-Based 环境把逐项 reward 诊断以 `reward/` 键发布到 `info["log"]`——由 `RewardManager.step_reward_extras()` 产生 （`src/unilab/managers/reward_manager.py`，跨环境对 `raw_value * weight` 取均值，未乘 `dt`）——训练 logger 再把它们转发到 TensorBoard / W&B -（{doc}`../../2-user_guide/1-training/3-logging`）： +（{doc}`../2-user_guide/1-training/3-logging`）： ```bash uv run tensorboard --logdir logs/rsl_rl_ppo/ @@ -95,7 +95,7 @@ term 出错时失败是显式的：构造错误会指出 term 名；返回类型 `RewardManager` 抛出；返回 NaN/Inf 会抛出 `RewardManager term '' returned NaN/Inf`（启用 NaN guard 时还会写出 状态 dump）。诊断流程见 -{doc}`../../2-user_guide/1-training/8-training_diagnostics`。 +{doc}`3-training_diagnostics`。 ## 4. 调整权重 @@ -107,9 +107,9 @@ uv run train --algo ppo --task --sim mujoco \ ``` 这可行是因为 `reward` 是组合配置顶层的 Hydra section（见 -{doc}`../../2-user_guide/2-algorithms/1-ppo` 的 “Reward term weights” 小节）。 +{doc}`../2-user_guide/2-algorithms/1-ppo` 的 “Reward term weights” 小节）。 新增 term 请写进 owner YAML 以保证任务可复现；CLI override 适合扫参。 从其他框架移植 term 时先查内置表——该 term 可能已经存在： -{doc}`../../5-reference/6-manager_terms/1-rewards`。Legged Gym 的映射指南见 -{doc}`../../3-deployment/3-framework_migration/6-reward_porting`。 +{doc}`../5-reference/6-manager_terms/1-rewards`。Legged Gym 的映射指南见 +{doc}`../3-deployment/3-framework_migration/6-reward_porting`。 diff --git a/docs/sphinx/source/zh_CN/2-user_guide/1-training/8-training_diagnostics.md b/docs/sphinx/source/zh_CN/6-how_to/3-training_diagnostics.md similarity index 87% rename from docs/sphinx/source/zh_CN/2-user_guide/1-training/8-training_diagnostics.md rename to docs/sphinx/source/zh_CN/6-how_to/3-training_diagnostics.md index 3b403a80a..3a0a6c5c6 100644 --- a/docs/sphinx/source/zh_CN/2-user_guide/1-training/8-training_diagnostics.md +++ b/docs/sphinx/source/zh_CN/6-how_to/3-training_diagnostics.md @@ -42,13 +42,13 @@ NaN guard 在设备侧检查观测、reward 与控制量中的非有限行，并 （`pyproject.toml`）；它先打印检出步数与 NaN env id，再回放缓冲状态。 不支持物理状态回放的后端 dump 中没有状态——viewer 只打印元数据。 -细节与 viewer 操作见 {doc}`../7-tooling/3-nan_visualizer`。往返加载覆盖： +细节与 viewer 操作见 {doc}`../2-user_guide/7-tooling/3-nan_visualizer`。往返加载覆盖： `tests/training/test_tensor_diagnostics.py`。 ## 读懂 `info["log"]` Manager-Based 环境把结构化指标放进 `info["log"]`；训练 logger 把带斜杠前缀的 -条目转发到 TensorBoard / W&B。完整语义表见 {doc}`3-logging`；键族与生产者： +条目转发到 TensorBoard / W&B。完整语义表见 {doc}`../2-user_guide/1-training/3-logging`；键族与生产者： | 键 | 生产者 | 含义 | | --- | --- | --- | @@ -57,7 +57,7 @@ Manager-Based 环境把结构化指标放进 `info["log"]`；训练 logger 把 | `Episode_Metrics/` | `MetricsManager`（`src/unilab/managers/metrics_manager.py`） | 已完成 episode 的指标聚合 | 整体进展看 `Train/mean_reward` 与 `Train/mean_episode_length`，归因看 -`reward/`；两者区别见 {doc}`3-logging`。 +`reward/`；两者区别见 {doc}`../2-user_guide/1-training/3-logging`。 ## 常见失败模式 @@ -70,13 +70,13 @@ Manager-Based 环境把结构化指标放进 `info["log"]`；训练 logger 把 YAML `reward:` 块中它的原始量级与 `weight`。 - 改：用 CLI 调权，例如 `uv run train --algo ppo --task --sim mujoco reward..weight=-0.01`（override 机制见 - {doc}`../2-algorithms/1-ppo` 的 “Reward term weights” 小节）。保留 + {doc}`../2-user_guide/2-algorithms/1-ppo` 的 “Reward term weights” 小节）。保留 平滑/正则项（`action_rate` 类惩罚）；删掉它们常常直接导致动作流不稳定。 ### 策略不收敛 - 症状：`Train/mean_reward` 持平但迭代在推进。 -- 对照算法参考表查超参（PPO 见 {doc}`../2-algorithms/1-ppo`）： +- 对照算法参考表查超参（PPO 见 {doc}`../2-user_guide/2-algorithms/1-ppo`）： `algo.algorithm.learning_rate`、`algo.algorithm.entropy_coef`、 `algo.algorithm.lam`。检查 `algo.num_envs`——它是按 rank 计的数量，过小会 让梯度估计失真。检查观测条件：`algo.actor.obs_normalization`（Go2 owner @@ -93,8 +93,8 @@ Manager-Based 环境把结构化指标放进 `info["log"]`；训练 logger 把 形状、`ctrl_dt` 等）必须一致——默认 `training.sim2sim_strict: true` 下不一致 会抛 `CrossBackendIncompatibleError`；WARNING_LIST 字段只打印警告。审计已发布 的组合用 `uv run scripts/audit_sim2sim_contracts.py`。 -- 查后端能力差异：{doc}`../../3-deployment/2-sim_to_sim/6-capability_gaps`； - 精确支持等级：{doc}`../../5-reference/5-support_matrix`。 +- 查后端能力差异：{doc}`../3-deployment/2-sim_to_sim/6-capability_gaps`； + 精确支持等级：{doc}`../5-reference/5-support_matrix`。 ### 训练中途出现 NaN @@ -109,7 +109,7 @@ Manager-Based 环境把结构化指标放进 `info["log"]`；训练 logger 把 `training.trace_output_dir`、`training.trace_thread_time` 与 `training.trace_cuda_events`（`src/unilab/conf/sac/config.yaml`），另有 CUDA 上 NVTX 标注用的 `training.nvtx_profile_ranges`。trace 是 Perfetto - JSON；标量计时键见 {doc}`3-logging`。 + JSON；标量计时键见 {doc}`../2-user_guide/1-training/3-logging`。 - 在主机侧分析已捕获的 trace（不需要 GPU）： ```bash @@ -120,4 +120,4 @@ Manager-Based 环境把结构化指标放进 `info["log"]`；训练 logger 把 脚本按事件名报告 mean/median/p90/p95/p99 时长；`--steps-per-cycle` 与 `--drop-first` 调整吞吐量汇总。 - 生产环境长 soak 的设置与失败分类字段见 - {doc}`7-tensor_runtime_production`。 + {doc}`../2-user_guide/1-training/7-tensor_runtime_production`。 From 153abf26ff5625fc543c88544dd0f52ce4c75fdf Mon Sep 17 00:00:00 2001 From: LeeLeno <1846301824@qq.com> Date: Fri, 6 Nov 2026 20:05:27 +0800 Subject: [PATCH 8/8] docs: changelog entry for how-to section and landing entry cards (#706) --- docs/sphinx/source/changelog.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/sphinx/source/changelog.md b/docs/sphinx/source/changelog.md index 57970d7c8..2715347d3 100644 --- a/docs/sphinx/source/changelog.md +++ b/docs/sphinx/source/changelog.md @@ -56,6 +56,15 @@ PyPI 版本变更与未发布变更；发布日期采用 PyPI 上传日期。完 [#706](https://github.com/Motphys/UniLab/issues/706)). 新增训练诊断页（[#2584](https://github.com/Motphys/UniLab/pull/2584)， [#706](https://github.com/Motphys/UniLab/issues/706)）。 +- New top-level How-to Guides section groups the walkthroughs and the training + diagnostics page, and the landing pages gain task-oriented entry cards + linking to the reference tables + ([#2584](https://github.com/Motphys/UniLab/pull/2584), + [#706](https://github.com/Motphys/UniLab/issues/706)). + 新增顶级"How-to 指南"分区，集中收纳各 walkthrough 与训练诊断页；首页新增 + 任务导向入口卡片，直达各参考表 + （[#2584](https://github.com/Motphys/UniLab/pull/2584)， + [#706](https://github.com/Motphys/UniLab/issues/706)）。 ### Changed / 变更