fix(web): a release works on Cloudflare Pages — the terminal opens (node_modules is served), the sign-in page keeps its headers

docs/WEB.md

@@ -287,7 +287,8 @@ node web/serve.mjs --dist dist-web            # http://127.0.0.1:8800
 **Dev server** (`web/serve.mjs`): `--static <vscode-web dir> --extensions <dir> --extension-names
 levelcode-ai,levelcode-web --account <origin>`, plus `--product <json>` and `--config-defaults
 <json>` to override the product configuration. The extension is built with
-`node web/ai-extension/build.mjs --out <dir> --vscode <checkout>`.
+`node web/ai-extension/build.mjs --out <dir> --vscode <checkout>`. With `--dist <release> --pages` it answers
+as Cloudflare Pages does (below): the release-level end-to-end checks take `--pages` too.
 
 **Release layout.** `index.html` and the callback page are at the root, un-cached. Everything
 else is under `/_/<id>/` and cached for a year (`immutable`): `static/` (the client), `extensions/`,
@@ -323,6 +324,29 @@ Content-Security-Policy for the editor is the tightest the workbench allows:
 
 The callback page has `default-src 'none'` and only its own script and style.
 
+### On Cloudflare Pages
+
+Pages reads `_headers` as written, and three things it does differently from a plain file server are
+handled in the release itself. `web/serve.mjs --dist <release> --pages` answers the way Pages does
+(`web/lib/pages.mjs`), so the end-to-end checks run against a release as it will be served there:
+
+- **It is never sent a folder named `node_modules`.** `wrangler pages deploy` leaves out `**/node_modules`
+  (and `.git`, `.DS_Store`), with no option to keep them, and the client loads its terminal (xterm), its syntax
+  colouring (vscode-textmate, vscode-oniguruma) and more from `static/node_modules`. On the first deploy every
+  one of those came back as the page (Pages answers an address it has no file for with `index.html`), and
+  the terminal did not open. The build stages the same files again as `static/node-modules`, and the
+  release's `_redirects` serves them from there at the address the client asks for
+  (`/_/:id/static/node_modules/* /_/:id/static/node-modules/:splat 200`). Any other host serves
+  `node_modules` itself and ignores `_redirects`. The build refuses a `node_modules` folder anywhere else.
+- **It serves `x.html` at `/x`** (and redirects `/x.html` there, query kept), and matches `_headers` rules
+  against the address asked for: every page's rule is also written for its address without `.html`, so the
+  sign-in page keeps its own policy at `/callback`.
+- **A file it does not have is answered with the page, status 200.** A check that only looks at status codes
+  does not see it; `e2e.mjs` fails on any script, JSON or WebAssembly response that is HTML.
+
+The limits that matter: 25 MiB per file (the largest is a 24.6 MiB diagram bundle in the client) and 20,000
+files on the free plan (1,840 uploaded with the copy, measured; about twice that with the previous build kept online).
+
 ### On the account site
 
 Three settings on the Rails app (all default **off**; with none set nothing changes):
@@ -427,7 +451,7 @@ transaction, so both end with the same key.
 | The shell | `./scripts/test-extensions.sh` (the unit suites `web/test/unit/shell-*.test.js`) | The engine: a differential against GNU bash 5.1 (905 cases, a shrink-only table of the ones that differ), fuzzing, every patch of the vendored code with its anchors, the build; the runtime: the commit and its refusals, kill and deadline against a **real engine in a worker thread**, limits; the terminal: key bytes, the line editor against real xterm.js 6.1 in its headless build, width, the output filter, history. With the Code-OSS checkout present the real-xterm halves run; `LEVELCODE_REQUIRE_XTERM=1` and `LEVELCODE_REQUIRE_WORKER=1` make a skip a failure, which is what CI sets. |
 | Mutations | `node web/test/helpers/mutation/mutate.mjs` | 371 edits to the runtime, the commit and the sanitiser, each of which a suite must catch (a tool, not a suite). |
 | The shell, in a browser | `node web/test/e2e-shell.mjs --static … --extensions …` | Real headless Chrome and the real nested Worker: nothing of the shell is fetched until it is used; a terminal from the palette; typing, files, a working folder that persists, 127, Ctrl+C on `sleep 60`, a link to `path:line:col` that opens the file at that line, a pasted three-line buffer through a resize; hostile output that does not type into the prompt or rename the terminal; the agent offered `run_command` without background, a file written through the review (Keep/Undo), a `.vscode` write refused whole, a timeout that applies nothing, the mirror tab. |
-| End to end | `node web/test/e2e.mjs --dist dist-web --stub-port <port the build used>` | Real headless Chrome: boot from static files, the extension activates in the worker, sign in through the real callback page, agent tool calls (write / list / search / read / edit) against the scratch workspace, Quick Open, Keep and Undo, a reload that keeps the session and files. |
+| End to end | `node web/test/e2e.mjs --dist dist-web --stub-port <port the build used> [--pages]` | Real headless Chrome: boot from static files, the extension activates in the worker, sign in through the real callback page, agent tool calls (write / list / search / read / edit) against the scratch workspace, Quick Open, Keep and Undo, a reload that keeps the session and files, a terminal that opens, and no script, JSON or WebAssembly file answered with HTML. `--pages`: served as Cloudflare Pages serves it (also for the look and same-tab sign-in checks). |
 | The look | `node web/test/e2e-look.mjs --dist dist-web --stub-port <port>` | The skinned chat (greeting, starters, composer in the reading column, the rail at 1440 px and the dialog at 900 px), light and dark following the system with this build's own colours (also in the chat's webview), no Welcome page, no second chat, source control / run / extensions not offered, the layout switch both ways, and the docked layout docked again after a reload. Release builds only. |
 | Same-tab sign-in | `node web/test/e2e-signin-tab.mjs --dist dist-web --stub-port <port>` | No second window; verifier survives the reload; no repeat; no sign-in for a signed-in editor; a second tab uses the same session; a stranded result is delivered, an old one dropped. |
 | Bring your own key | `node web/test/e2e-byok.mjs --static … --extensions …` | The request goes from the tab to the provider with the user's key and nothing of LevelCode's. |

scripts/build-web.mjs

@@ -13,7 +13,7 @@
 //      workspace extension (web/workspace) and every declarative built-in (grammars, themes, icons);
 //   3. names the build after a hash of everything it staged, lays that out under the build-addressed
 //      prefix (/_/<id>/...), renders index.html with the configuration baked in, and writes the HTTP
-//      policy for the host (_headers, nginx).
+//      policy for the host (_headers, nginx) and, for Cloudflare Pages, a _redirects.
 //
 // It changes nothing outside --out, except that building the client from the checkout leaves gulp's own output,
 // vscode-web/, next to it (pass --static to use a client built earlier and skip that). It does not deploy.
@@ -26,6 +26,7 @@ import { webConfig, renderIndex } from '../web/lib/config.mjs';
 import { stageDeclarativeExtensions, copyExtension, withoutConfigurationDefaults } from '../web/lib/extensions.mjs';
 import { policy, toHeadersFile, toNginx } from '../web/lib/headers.mjs';
 import { hashTree } from '../web/lib/fingerprint.mjs';
+import { NODE_MODULES_COPY, toRedirectsFile } from '../web/lib/pages.mjs';
 
 const REPO = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..');
 const WEB = path.join(REPO, 'web');
@@ -46,6 +47,15 @@ const log = (m) => console.log('[build-web] ' + m);
 const die = (m) => { console.error('[build-web] ' + m); process.exit(1); };
 const sizeOf = (p) => { let n = 0; const walk = (d) => { for (const e of fs.readdirSync(d, { withFileTypes: true })) { const f = path.join(d, e.name); e.isDirectory() ? walk(f) : (n += fs.statSync(f).size); } }; walk(p); return n; };
 const mb = (n) => (n / 1048576).toFixed(1) + ' MB';
+/** Every folder under `dir` named `name`, not looking inside one that is. */
+const directoriesNamed = (dir, name, found = []) => {
+	for (const e of fs.readdirSync(dir, { withFileTypes: true })) {
+		if (!e.isDirectory()) { continue; }
+		const f = path.join(dir, e.name);
+		if (e.name === name) { found.push(f); } else { directoriesNamed(f, name, found); }
+	}
+	return found;
+};
 
 /* ----- 1. Code-OSS's web client ---------------------------------------------------------------- */
 let staticDir;
@@ -78,6 +88,12 @@ let root = staging;
 
 log('copying the web client');
 fs.cpSync(staticDir, path.join(root, 'static'), { recursive: true, filter: (src) => !/\.(js|css)\.map$/.test(src) });
+// The client loads its terminal (xterm), its syntax colouring (vscode-textmate, vscode-oniguruma) and more from
+// static/node_modules. Cloudflare Pages is never sent a folder by that name (wrangler leaves out **/node_modules): the
+// same files are staged again as static/node-modules, and the release's _redirects (written below) serves them from
+// there at the address the client asks for. Any other host serves node_modules itself and ignores _redirects.
+const clientModules = path.join(root, 'static', 'node_modules');
+if (fs.existsSync(clientModules)) { fs.cpSync(clientModules, path.join(root, 'static', NODE_MODULES_COPY), { recursive: true }); }
 const baked = new Set(fs.existsSync(path.join(staticDir, 'extensions')) ? fs.readdirSync(path.join(staticDir, 'extensions')) : []);
 
 const extOut = path.join(root, 'extensions');
@@ -98,8 +114,10 @@ fs.writeFileSync(path.join(scratchOut, 'package.nls.json'), '{}\n');
 const extensionsSource = path.join(vscodeDir, 'extensions');
 if (!fs.existsSync(extensionsSource)) { die('no extensions/ in ' + vscodeDir + ' to take the declarative built-ins from'); }
 const declarative = stageDeclarativeExtensions(extensionsSource, extOut, { exclude: [...baked, 'levelcode-ai', 'levelcode-web', 'levelcode-shell'] });
-// A checkout that has not had LevelCode's extensions copied in still builds; the themes are then absent.
-if (!declarative.includes('levelcode-themes')) {
+// A checkout that has not had LevelCode's extensions copied in still builds; the themes are then absent. A client built
+// from a checkout that has them has them baked in, and a second copy here would be skipped with an error for each of
+// its translations.
+if (!declarative.includes('levelcode-themes') && !baked.has('levelcode-themes')) {
 	const themes = path.join(REPO, 'extensions', 'levelcode-themes');
 	if (fs.existsSync(themes)) { copyExtension(themes, path.join(extOut, 'levelcode-themes')); declarative.push('levelcode-themes'); }
 }
@@ -116,6 +134,11 @@ log(`${extensions.length} extensions: levelcode-ai, levelcode-web, levelcode-web
 // main.js and boot.css are addressed by the build id like everything else; the page names them.
 for (const f of ['main.js', 'boot.css', 'chrome.css']) { fs.copyFileSync(path.join(WEB, f), path.join(root, f)); }
 
+// Nothing else may be in a folder named node_modules: Pages would not have it, and nothing would serve it from elsewhere.
+for (const dir of directoriesNamed(staging, 'node_modules')) {
+	if (dir !== clientModules) { die(path.relative(staging, dir) + ' is a node_modules folder, which Cloudflare Pages is never sent'); }
+}
+
 /* ----- 3. identity of this build: what was staged ------------------------------------------------- */
 const id = String(args.id || hashTree(createHash('sha256'), staging, { skip: ['.DS_Store'] }).digest('hex').slice(0, 12));
 const prefix = `/_/${id}`;
@@ -159,6 +182,9 @@ for (const f of ['callback.html', 'callback.js', 'callback.css', 'icon.svg', 'ic
 const toPattern = (o) => (o ? o.replace('{{uuid}}', '*') : '');
 const pol = policy({ account, webviewOrigin: toPattern(webviewOrigin), extHostOrigin: toPattern(extHostOrigin), base: '/_' });
 fs.writeFileSync(path.join(outDir, '_headers'), toHeadersFile(pol));
+if (fs.existsSync(path.join(root, 'static', NODE_MODULES_COPY))) {
+	fs.writeFileSync(path.join(outDir, '_redirects'), toRedirectsFile());
+}
 fs.mkdirSync(path.join(outDir, 'deploy'), { recursive: true });
 fs.writeFileSync(path.join(outDir, 'deploy', 'nginx.conf'), toNginx(pol));
 const info = {

web/lib/headers.mjs

@@ -70,7 +70,11 @@ export function policy(o) {
 
 /** Netlify / Cloudflare Pages `_headers` text. */
 export function toHeadersFile(p) {
-	return p.rules.map((r) => r.path + '\n' + Object.entries(r.headers).map(([k, v]) => `  ${k}: ${v}`).join('\n')).join('\n\n') + '\n';
+	// Cloudflare Pages serves a page `x.html` at `/x` (it redirects /x.html there), and matches rules against the address
+	// asked for: a rule for /callback.html alone would leave the sign-in page it serves without its headers. So each
+	// page's rule is also written for its address without `.html` (/index.html's is `/`, which has its own rule).
+	const rules = p.rules.flatMap((r) => (/^\/[^/]+\.html$/.test(r.path) && r.path !== '/index.html' ? [r, { ...r, path: r.path.slice(0, -'.html'.length) }] : [r]));
+	return rules.map((r) => r.path + '\n' + Object.entries(r.headers).map(([k, v]) => `  ${k}: ${v}`).join('\n')).join('\n\n') + '\n';
 }
 
 /** Parse a `_headers` file back into rules (the dev server serves a built release with it). */

web/lib/pages.mjs

@@ -0,0 +1,90 @@
+// How Cloudflare Pages answers a request for a release (scripts/build-web.mjs), for web/serve.mjs --pages. A release
+// that only ever ran under serve.mjs's plain file server went live on Pages without its terminal: Pages had never
+// been sent the client's node_modules, and answered every one of those scripts with the page. This is what Pages
+// does differently from a plain file server, so a release can be checked the way it will be served there.
+//
+//   1. What `wrangler pages deploy` does not upload is not there: any path with a segment named node_modules or .git,
+//      .DS_Store, and _worker.js, _redirects, _headers, _routes.json and functions/ at the root (its IGNORE_LIST, in
+//      workers-sdk packages/wrangler/src/pages/validate.ts). Symbolic links are skipped too.
+//   2. _redirects comes first, whatever the files: the first rule that matches applies. 200 serves another path of the
+//      site in place (a rewrite); 301, 302, 303, 307 and 308 redirect. `:name` stands for one segment, `*` for the rest.
+//   3. An .html address is redirected to the same address without it (/a.html -> /a, /x/index.html -> /x/), and
+//      /a serves a.html, /x/ serves x/index.html.
+//   4. An address that matches no file is answered with /index.html and 200 (what Pages does when there is no
+//      404.html): a missing script comes back as the page, which is the failure this is here to show.
+// Pure: no file system access. Callers say whether a path exists.
+
+/**
+ * The release's own answer to (1) for the client's node_modules (scripts/build-web.mjs): the same files are staged
+ * again under this name, and _redirects serves them from there at the address the client asks for.
+ */
+export const NODE_MODULES_COPY = 'node-modules';
+
+/** The release's _redirects. One rule for every build folder (:id), so a previous build kept online is served too. */
+export function toRedirectsFile() {
+	return `/_/:id/static/node_modules/* /_/:id/static/${NODE_MODULES_COPY}/:splat 200\n`;
+}
+
+/** Is this path (relative to the release, `/`-separated) one that `wrangler pages deploy` uploads? */
+export function uploaded(rel) {
+	const parts = rel.split('/').filter(Boolean);
+	if (parts.some((p) => p === 'node_modules' || p === '.git' || p === '.DS_Store')) { return false; }
+	if (parts.length === 1 && ['_worker.js', '_redirects', '_headers', '_routes.json'].includes(parts[0])) { return false; }
+	return parts[0] !== 'functions';
+}
+
+/**
+ * The rules of a _redirects file, in order: `from to [status]`, blank lines and `#` comments skipped.
+ * @param {string} text
+ * @returns {Array<{ from: string, to: string, status: number }>}
+ */
+export function parseRedirects(text) {
+	const rules = [];
+	for (const line of text.split('\n')) {
+		const t = line.trim();
+		if (!t || t.startsWith('#')) { continue; }
+		const [from, to, status] = t.split(/\s+/);
+		if (from && to) { rules.push({ from, to, status: status ? Number(status) : 302 }); }
+	}
+	return rules;
+}
+
+/**
+ * The first rule that matches `pathname`, with its destination filled in, or null.
+ * @param {Array<{ from: string, to: string, status: number }>} rules
+ * @param {string} pathname
+ * @returns {{ to: string, status: number } | null}
+ */
+export function matchRedirect(rules, pathname) {
+	for (const r of rules) {
+		const names = [];
+		const pattern = r.from.split('/').map((seg, i, all) => {
+			if (seg === '*' && i === all.length - 1) { names.push('splat'); return '(.*)'; }
+			if (seg.startsWith(':')) { names.push(seg.slice(1)); return '([^/]+)'; }
+			return seg.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
+		}).join('/');
+		const m = new RegExp('^' + pattern + '$').exec(pathname);
+		if (!m) { continue; }
+		const values = Object.fromEntries(names.map((n, i) => [n, m[i + 1]]));
+		const to = r.to.replace(/:([A-Za-z_][A-Za-z0-9_]*)/g, (all, n) => (n in values ? values[n] : all));
+		return { to, status: r.status };
+	}
+	return null;
+}
+
+/**
+ * What Pages does with an address before it looks for a file: redirect an .html one, or name the file an
+ * extensionless one stands for.
+ * @param {string} pathname
+ * @param {(rel: string) => boolean} isFile  is there an uploaded file at this release-relative path?
+ * @returns {{ redirect: string } | { file: string } | null}  null: no such page; serve the fallback
+ */
+export function resolvePage(pathname, isFile) {
+	if (pathname.endsWith('/index.html')) { return { redirect: pathname.slice(0, -'index.html'.length) }; }
+	if (pathname.endsWith('.html') && isFile(pathname.slice(1))) { return { redirect: pathname.slice(0, -'.html'.length) }; }
+	const rel = pathname.slice(1);
+	if (pathname.endsWith('/')) { return isFile(rel + 'index.html') ? { file: rel + 'index.html' } : null; }
+	if (isFile(rel)) { return { file: rel }; }
+	if (isFile(rel + '.html')) { return { file: rel + '.html' }; }
+	return null;
+}