diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 82cd4d4..e6dcfce 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,6 +1,6 @@ -# The kit's own CI: unit tests (Python + JS), both bundle builds, the example -# course built with the dev bundle plus the browser smoke test, and the -# reusable course-check workflow run against the example course. +# The kit's own CI, on pull requests: unit tests (Python + JS), both bundle +# builds, the example course built with the dev bundle plus the browser smoke +# test, and the reusable course-check workflow run against the example course. name: CI # Pull requests only. Branch protection requires PRs to be up to date with @@ -41,7 +41,11 @@ jobs: - name: Dev build + tests run: npm run build:dev && npm test + # The slowest job. A release PR only changes version strings and the + # changelog, so it is skipped there (a skipped job counts as passing for + # required checks). example-course: + if: github.head_ref != 'release-please--branches--main--components--course-kit' needs: [python, js] runs-on: ubuntu-latest steps: diff --git a/AGENTS.md b/AGENTS.md index c0853c9..3eaaeb3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -155,11 +155,17 @@ downloads); CI retries failed tests once. ## CI -- `ci.yml`: pytest; JS release + dev builds with tests; example course build + - e2e; the reusable `course-check.yml` run against `example-course` with - `kit-source: local`. +- `ci.yml` runs on pull requests only (branch protection requires PRs to be up + to date with `main`): pytest; JS release + dev builds with tests; example + course build + e2e (skipped on the release-please PR, which only changes + version strings); the reusable `course-check.yml` run against + `tests/example-course` with `kit-source: local`. - `release.yml`: release-please PR -> tag `vX.Y.Z` -> npm (provenance) and - PyPI (trusted publishing) -> move `vX.Y` and `vX`. + PyPI (trusted publishing) -> move `vX.Y` and `vX`. release-please uses the + `RELEASE_PLEASE_TOKEN` secret (a PAT or app token with Contents, Pull + requests and Workflows access), because it edits `KIT_VERSION` in + `.github/workflows/` and the default token may not; the token also makes CI + run on the release PR. Setup: README, "Releasing". - `course-publish.yml` job order: check -> notebooks -> publish-notebooks -> render-scrolly -> prepare-site -> myst build -> Pages deploy. One run per repository at a time (notebook pushes never race). No CNAME step: with the diff --git a/CHANGELOG.md b/CHANGELOG.md index 9beb705..c213f22 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ Maintained by release-please from conventional commit messages. * don't force provenance in package.json (CI passes --provenance) ([5199e87](https://github.com/neuromatch/course-kit/commit/5199e875c476013ebcf573ce721e4460382fce07)) * reorganize repo, add release manifest and package metadata ([b1a96c7](https://github.com/neuromatch/course-kit/commit/b1a96c725a5c94ceefd08cf8d1ac61e180a3c381)) -## 0.1.0 (unreleased) +## 0.1.0 First version, extracted from `neuromatch/course-template`: diff --git a/README.md b/README.md index 3ce5451..c1bc0d7 100644 --- a/README.md +++ b/README.md @@ -105,9 +105,9 @@ See [AGENTS.md](AGENTS.md) for the layout, the Python/JS twin rules and the test commands. In short: ```bash -cd python && uv venv && uv pip install -e '.[test]' -r ../tests/example-course/requirements.txt && uv run pytest -q ../tests/python -cd js && npm ci && npm run build && npm test && npm run build:dev && npm test -tests/scripts/build-example.sh # build example-course with the dev bundle +(cd python && uv venv && uv pip install -e '.[test]' -r ../tests/example-course/requirements.txt && uv run pytest -q ../tests/python) +(cd js && npm ci && npm run build && npm test && npm run build:dev && npm test) +tests/scripts/build-example.sh # build tests/example-course with the dev bundle ``` To try a kit change in a course without releasing, point the course's @@ -120,16 +120,19 @@ Commits on `main` use conventional messages (`fix:`, `feat:`, `feat!:`). release-please keeps a release PR open; merging it tags `vX.Y.Z`, publishes npm and PyPI, and moves the `vX.Y` and `vX` tags. -One-time setup (after this repo exists as `neuromatch/course-kit`): - -1. **npm**: create the free npm organization `neuromatch`. Publish the first - version by hand (`cd js && npm run build && npm publish --access public`), - because npm trusted publishing can only be configured on an existing - package. Then add a trusted publisher: repository `neuromatch/course-kit`, - workflow `release.yml`. -2. **PyPI**: add a *pending publisher* for project `neuromatch-course-kit`: +The release flow is already set up. This is what it depends on, in case it +has to be rebuilt: + +1. **npm**: the `neuromatch` organization owns `@neuromatch/course-kit`. The + first version was published by hand + (`cd js && npm run build && npm publish --otp=`), because npm trusted + publishing can only be configured on an existing package. The package has a + trusted publisher: repository `neuromatch/course-kit`, workflow + `release.yml`, no environment, direct publishing allowed (the workflow runs + a plain `npm publish --provenance`), dist-tag management off. +2. **PyPI**: the project `neuromatch-course-kit` has a trusted publisher: repository `neuromatch/course-kit`, workflow `release.yml`, environment - `release`. The first release creates the project. + `release`. 3. Create the `release` environment in the repo settings. 4. Add a second owner/maintainer on npm and PyPI. 5. Create a fine-grained personal access token (or GitHub App) for this repo diff --git a/js/README.md b/js/README.md new file mode 100644 index 0000000..28ae868 --- /dev/null +++ b/js/README.md @@ -0,0 +1,92 @@ +# @neuromatch/course-kit + +The [MyST](https://mystmd.org) plugin for Neuromatch courses: one bundled +file plus the browser assets it uses. It is one half of the +[Neuromatch course kit](https://github.com/neuromatch/course-kit); the other +half is the [`nma-kit`](https://pypi.org/project/neuromatch-course-kit/) +command line, which checks content and generates notebooks. Both ship under +the same version. + +You normally don't install this package. A course's `myst.yml` loads it from +jsDelivr: + +```yaml +version: 1 +extend: https://cdn.jsdelivr.net/npm/@neuromatch/course-kit@0.1/dist/nma.yml +project: + github: https://github.com// +``` + +`dist/nma.yml` adds the plugin and turns on `project.jupyter.lite`. In CI, +`nma-kit prepare-site` rewrites `@0.1` to the exact version of the CLI, so the +plugin, its assets and the CLI always match. See the +[kit README](https://github.com/neuromatch/course-kit#using-the-kit-in-a-course) +for the full course setup. + +## What it adds to MyST + +| Directive / feature | What it does on the site | +|---|---| +| `{nma-video}`, `{nma-slides}` | Embedded YouTube, Bilibili or OSF videos and slide decks | +| `{nma-solution}`, `{nma-explanation}` | A "Click for solution" link; the solution text itself is never on the page | +| `{nma-feedback}` | A feedback widget that posts to datatops from the browser | +| `layout: scrolly`, `{step}`, `{stage}` | Scrollytelling pages with live Python (Pyodide) | +| Editable cells | Code cells become editable and runnable in the browser on pages with a `kernelspec`; `runtime: colab` pages get a Colab banner instead | + +Examples: + +```markdown +:::{nma-video} Video 1: Linear dynamical systems +:youtube: vPYQPI4nKT8 +:bilibili: BV1up4y1S7wj +::: + +:::{nma-feedback} Linear_Dynamical_Systems_Video +::: +``` + +Directives must be at the top level of a page. Their options are checked by +`nma-kit check`. + +Settings live in the `nma:` block of `myst.yml` (all optional): + +```yaml +nma: + paths: + tutorials: tutorials + notebooks: + branch: notebooks-branch + feedback: { url: https://..., project: ..., user_key: ... } + scrolly: + pyodide_url: https://cdn.jsdelivr.net/pyodide/v314.0.7/full/ +``` + +One rule belongs in the course's own CSS, because a plugin can't add +site-wide CSS: `.myst-jp-nb-toolbar { display: none !important; }`. + +## What's in the package + +``` +dist/nma.mjs the plugin (all features, one file) +dist/nma.yml the config courses `extend:` +dist/assets// browser widgets and CSS, one folder per feature +dist/build-info.json {mode, version} +``` + +Asset URLs inside the bundle point at this exact version on jsDelivr. MyST +downloads them at build time and serves them from the course's own site, so +readers' browsers don't contact the CDN for them. The one exception is +the Python runtime: editable cells and scrolly pages load Pyodide from +`cdn.jsdelivr.net`, and micropip installs packages from PyPI. + +## Versions + +The kit is on 0.x. A breaking change bumps the minor version, so `@0.1` only +receives compatible fixes. Don't use `@latest` in `myst.yml`: jsDelivr caches +alias URLs for up to 7 days and `@latest` crosses major versions. + +## Development + +Source, tests and release process: . + +BSD-3-Clause. diff --git a/python/README.md b/python/README.md index 2982730..0f7af05 100644 --- a/python/README.md +++ b/python/README.md @@ -1,5 +1,64 @@ # neuromatch-course-kit -The `nma-kit` command line tool for Neuromatch MyST courses: notebook -generation, content checks and site preparation. See the -[course kit repository](https://github.com/neuromatch/course-kit) for docs. +The `nma-kit` command line for Neuromatch courses written in +[MyST](https://mystmd.org) Markdown. It is one half of the +[Neuromatch course kit](https://github.com/neuromatch/course-kit); the other +half is the MyST plugin on npm, +[`@neuromatch/course-kit`](https://www.npmjs.com/package/@neuromatch/course-kit). +Both ship under the same version. + +Courses usually don't install this by hand: the reusable GitHub workflows in +the kit repository do. To use it locally: + +```bash +uv venv && uv pip install -r requirements.txt "neuromatch-course-kit==0.1.*" +uv run nma-kit check +uv run myst start +``` + +Requires Python 3.10 or newer. Run every command from the course root (the +folder with `myst.yml`). + +## Commands + +| Command | What it does | +|---|---| +| `nma-kit check` | Validate the content rules and the `nma:` block of `myst.yml` | +| `nma-kit notebooks [--dry-run]` | Generate student and instructor notebooks and the solution files into `notebooks/` | +| `nma-kit render-scrolly` | Run each scrolly page's instructor notebook and save outputs to `_build/scrolly/` | +| `nma-kit prepare-site [--dry-run] [--kit-version V]` | Add badges and fallback cells to the pages and pin the kit URL in `myst.yml`. **CI only: it edits files in place, never commit the result** | +| `nma-kit publish-notebooks [--remote R] [--message M]` | Push `notebooks/` to the notebooks branch (created if missing) | + +Exit codes: `0` ok, `1` content problems or a failed publish, `2` usage or +configuration errors. + +## Settings + +Read from `myst.yml`: `project.github` gives the repository, and the optional +top-level `nma:` block holds the rest. + +```yaml +nma: + paths: + tutorials: tutorials # where the .md pages live + notebooks: + branch: notebooks-branch # where generated notebooks are published + feedback: # required if any page uses {nma-feedback} + url: https://... + project: ... + user_key: ... +``` + +Values in `nma:` must be plain strings in one level of sections. + +## Versions + +The kit is on 0.x. A breaking change bumps the minor version, so +`==0.1.*` only receives compatible fixes. The CLI and the npm plugin always +share a version number, and `prepare-site` pins the plugin URL to the CLI's +own version. + +Source, docs and release process: +. + +BSD-3-Clause. diff --git a/python/src/nma_kit/config.py b/python/src/nma_kit/config.py index 195924c..e1e3e18 100644 --- a/python/src/nma_kit/config.py +++ b/python/src/nma_kit/config.py @@ -5,7 +5,9 @@ with string values. js/src/shared/config.mjs reads the same block with a YAML parser too; both sides are tested against tests/fixtures/config/settings.json. -New settings (all optional, defaults are the template's conventions): +Settings read here (all optional; the defaults are the course template's +conventions). The feedback and scrolly sections are read by nma_kit.feedback +and the plugin (`nma.scrolly.pyodide_url`): nma: paths: diff --git a/python/src/nma_kit/solutions.py b/python/src/nma_kit/solutions.py index 5ea8e5b..ab829da 100644 --- a/python/src/nma_kit/solutions.py +++ b/python/src/nma_kit/solutions.py @@ -98,7 +98,7 @@ def close_block(block, end): stop = end if end != -1 else len(lines) body = lines[block.start + 1:stop] block.options, block.body_lines = _split_options(body) - if block.kind == "code-cell" and any(EXERCISE_RE.match(l) for l in block.body_lines): + if block.kind == "code-cell" and any(EXERCISE_RE.match(line) for line in block.body_lines): block.kind = "exercise" for i, line in enumerate(lines): @@ -184,7 +184,7 @@ def validate(text: str) -> list[str]: for b in blocks: where = f"line {b.start + 1}:" if b.kind in ("code-cell", "exercise"): - if any(TO_REMOVE_RE.match(l) for l in b.body_lines): + if any(TO_REMOVE_RE.match(line) for line in b.body_lines): errors.append( f"{where} code cell has a `# to_remove` marker. Move the solution into " f"an {{nma-solution}} or the explanation into an {{nma-explanation}} directive" @@ -267,7 +267,7 @@ def exercise_pairs(text: str) -> list[tuple[str, str]]: """(exercise source, solution code) for each exercise. Assumes validate() passed.""" blocks = [b for b in find_blocks(text) if not b.nested] pairs = [] - for ex, sol in zip(blocks, blocks[1:]): + for ex, sol in zip(blocks, blocks[1:], strict=False): if ex.kind == "exercise" and sol.kind == SOLUTION: exercise = "\n".join(ex.body_lines).strip("\n") pairs.append((exercise, solution_code(sol) or "")) @@ -328,7 +328,7 @@ def split_notebook(nb: dict, stem: str, rel_dir: str, repo: str, lines = text.splitlines(keepends=True) base = cell.get("id", "cell")[:50] cursor = 0 - for k, (block, item) in enumerate(zip(directive_blocks, items)): + for k, (block, item) in enumerate(zip(directive_blocks, items, strict=False)): prose = "".join(lines[cursor:block.start]).strip("\n") if prose.strip(): fragment = _markdown(prose, f"{base}-m{k}") diff --git a/tests/python/test_release.py b/tests/python/test_release.py index d7a2857..103119e 100644 --- a/tests/python/test_release.py +++ b/tests/python/test_release.py @@ -52,7 +52,7 @@ def test_release_please_bumps_every_file(): def test_generic_files_carry_the_marker(): for path in [*(WORKFLOWS / n for n in REUSABLE), KIT / "python" / "src" / "nma_kit" / "__init__.py"]: - line = next(l for l in path.read_text().splitlines() if re.search(r"\d+\.\d+\.\d+", l) and "VERSION" in l.upper()) + line = next(ln for ln in path.read_text().splitlines() if re.search(r"\d+\.\d+\.\d+", ln) and "VERSION" in ln.upper()) assert MARKER in line, path diff --git a/tests/scripts/compare_builds.py b/tests/scripts/compare_builds.py index e8c2f0b..efaaec1 100644 --- a/tests/scripts/compare_builds.py +++ b/tests/scripts/compare_builds.py @@ -63,7 +63,7 @@ def first_difference(a, b, path="$"): return found return None if isinstance(a, list): - for i, (x, y) in enumerate(zip(a, b)): + for i, (x, y) in enumerate(zip(a, b, strict=False)): found = first_difference(x, y, f"{path}[{i}]") if found: return found