Skip to content

Commit f02fa04

Browse files
committed
fix(web): a release works on Cloudflare Pages: the client's node_modules is served, the sign-in page keeps its headers
Three things a release did on Pages that it does not do on a plain file server (the previous commit shows them): - node_modules. The build stages static/node_modules a second time as static/node-modules, and writes a _redirects with one rule, /_/:id/static/node_modules/* -> /_/:id/static/node-modules/:splat (200, a rewrite), so Pages serves the client's files at the address it asks for, for this build and for a previous one kept online. Other hosts serve node_modules and ignore _redirects. The build refuses a node_modules folder anywhere else in a release. (+959 files, +35 MB: 1,840 files are uploaded, of Pages' 20,000.) - /callback. Pages serves callback.html at /callback and matches _headers by the address asked for: the sign-in page went without its default-src 'none' policy and no-store there. Every page's rule is now also written without `.html` (index.html's is `/`, which has its own). - levelcode-themes. A client built from a checkout that has LevelCode's extensions bakes it in; the build still staged a second copy (without a package.nls.json), which the workbench skipped with an error per translation lookup. It is staged only when the client does not have it. Checked with the client the live release was built from: e2e.mjs --pages 32/32 (the terminal opens; no script, JSON or WebAssembly answered with HTML), e2e-look.mjs --pages 26/26 (the web themes are in use: the baked theme's default is outranked by following the system), e2e-signin-tab.mjs --pages 20/20 (the sign-in page reached through Pages' .html redirect), e2e.mjs without --pages 32/32. Unit gate 127 files on Node 24 and Node 18. Not run on Cloudflare itself.
1 parent d4c6e5d commit f02fa04

4 files changed

Lines changed: 71 additions & 6 deletions

File tree

‎docs/WEB.md‎

Lines changed: 26 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -287,7 +287,8 @@ node web/serve.mjs --dist dist-web # http://127.0.0.1:8800
287287
**Dev server** (`web/serve.mjs`): `--static <vscode-web dir> --extensions <dir> --extension-names
288288
levelcode-ai,levelcode-web --account <origin>`, plus `--product <json>` and `--config-defaults
289289
<json>` to override the product configuration. The extension is built with
290-
`node web/ai-extension/build.mjs --out <dir> --vscode <checkout>`.
290+
`node web/ai-extension/build.mjs --out <dir> --vscode <checkout>`. With `--dist <release> --pages` it answers
291+
as Cloudflare Pages does (below): the release-level end-to-end checks take `--pages` too.
291292

292293
**Release layout.** `index.html` and the callback page are at the root, un-cached. Everything
293294
else is under `/_/<id>/` and cached for a year (`immutable`): `static/` (the client), `extensions/`,
@@ -323,6 +324,29 @@ Content-Security-Policy for the editor is the tightest the workbench allows:
323324

324325
The callback page has `default-src 'none'` and only its own script and style.
325326

327+
### On Cloudflare Pages
328+
329+
Pages reads `_headers` as written, and three things it does differently from a plain file server are
330+
handled in the release itself. `web/serve.mjs --dist <release> --pages` answers the way Pages does
331+
(`web/lib/pages.mjs`), so the end-to-end checks run against a release as it will be served there:
332+
333+
- **It is never sent a folder named `node_modules`.** `wrangler pages deploy` leaves out `**/node_modules`
334+
(and `.git`, `.DS_Store`), with no option to keep them, and the client loads its terminal (xterm), its syntax
335+
colouring (vscode-textmate, vscode-oniguruma) and more from `static/node_modules`. On the first deploy every
336+
one of those came back as the page (Pages answers an address it has no file for with `index.html`), and
337+
the terminal did not open. The build stages the same files again as `static/node-modules`, and the
338+
release's `_redirects` serves them from there at the address the client asks for
339+
(`/_/:id/static/node_modules/* /_/:id/static/node-modules/:splat 200`). Any other host serves
340+
`node_modules` itself and ignores `_redirects`. The build refuses a `node_modules` folder anywhere else.
341+
- **It serves `x.html` at `/x`** (and redirects `/x.html` there, query kept), and matches `_headers` rules
342+
against the address asked for: every page's rule is also written for its address without `.html`, so the
343+
sign-in page keeps its own policy at `/callback`.
344+
- **A file it does not have is answered with the page, status 200.** A check that only looks at status codes
345+
does not see it; `e2e.mjs` fails on any script, JSON or WebAssembly response that is HTML.
346+
347+
The limits that matter: 25 MiB per file (the largest is a 24.6 MiB diagram bundle in the client) and 20,000
348+
files on the free plan (1,840 uploaded with the copy, measured; about twice that with the previous build kept online).
349+
326350
### On the account site
327351

328352
Three settings on the Rails app (all default **off**; with none set nothing changes):
@@ -427,7 +451,7 @@ transaction, so both end with the same key.
427451
| The shell | `./scripts/test-extensions.sh` (the unit suites `web/test/unit/shell-*.test.js`) | The engine: a differential against GNU bash 5.1 (905 cases, a shrink-only table of the ones that differ), fuzzing, every patch of the vendored code with its anchors, the build; the runtime: the commit and its refusals, kill and deadline against a **real engine in a worker thread**, limits; the terminal: key bytes, the line editor against real xterm.js 6.1 in its headless build, width, the output filter, history. With the Code-OSS checkout present the real-xterm halves run; `LEVELCODE_REQUIRE_XTERM=1` and `LEVELCODE_REQUIRE_WORKER=1` make a skip a failure, which is what CI sets. |
428452
| Mutations | `node web/test/helpers/mutation/mutate.mjs` | 371 edits to the runtime, the commit and the sanitiser, each of which a suite must catch (a tool, not a suite). |
429453
| The shell, in a browser | `node web/test/e2e-shell.mjs --static … --extensions …` | Real headless Chrome and the real nested Worker: nothing of the shell is fetched until it is used; a terminal from the palette; typing, files, a working folder that persists, 127, Ctrl+C on `sleep 60`, a link to `path:line:col` that opens the file at that line, a pasted three-line buffer through a resize; hostile output that does not type into the prompt or rename the terminal; the agent offered `run_command` without background, a file written through the review (Keep/Undo), a `.vscode` write refused whole, a timeout that applies nothing, the mirror tab. |
430-
| End to end | `node web/test/e2e.mjs --dist dist-web --stub-port <port the build used>` | Real headless Chrome: boot from static files, the extension activates in the worker, sign in through the real callback page, agent tool calls (write / list / search / read / edit) against the scratch workspace, Quick Open, Keep and Undo, a reload that keeps the session and files. |
454+
| End to end | `node web/test/e2e.mjs --dist dist-web --stub-port <port the build used> [--pages]` | Real headless Chrome: boot from static files, the extension activates in the worker, sign in through the real callback page, agent tool calls (write / list / search / read / edit) against the scratch workspace, Quick Open, Keep and Undo, a reload that keeps the session and files, a terminal that opens, and no script, JSON or WebAssembly file answered with HTML. `--pages`: served as Cloudflare Pages serves it (also for the look and same-tab sign-in checks). |
431455
| The look | `node web/test/e2e-look.mjs --dist dist-web --stub-port <port>` | The skinned chat (greeting, starters, composer in the reading column, the rail at 1440 px and the dialog at 900 px), light and dark following the system with this build's own colours (also in the chat's webview), no Welcome page, no second chat, source control / run / extensions not offered, the layout switch both ways, and the docked layout docked again after a reload. Release builds only. |
432456
| Same-tab sign-in | `node web/test/e2e-signin-tab.mjs --dist dist-web --stub-port <port>` | No second window; verifier survives the reload; no repeat; no sign-in for a signed-in editor; a second tab uses the same session; a stranded result is delivered, an old one dropped. |
433457
| Bring your own key | `node web/test/e2e-byok.mjs --static … --extensions …` | The request goes from the tab to the provider with the user's key and nothing of LevelCode's. |

‎scripts/build-web.mjs‎

Lines changed: 29 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313
// workspace extension (web/workspace) and every declarative built-in (grammars, themes, icons);
1414
// 3. names the build after a hash of everything it staged, lays that out under the build-addressed
1515
// prefix (/_/<id>/...), renders index.html with the configuration baked in, and writes the HTTP
16-
// policy for the host (_headers, nginx).
16+
// policy for the host (_headers, nginx) and, for Cloudflare Pages, a _redirects.
1717
//
1818
// It changes nothing outside --out, except that building the client from the checkout leaves gulp's own output,
1919
// vscode-web/, next to it (pass --static to use a client built earlier and skip that). It does not deploy.
@@ -26,6 +26,7 @@ import { webConfig, renderIndex } from '../web/lib/config.mjs';
2626
import { stageDeclarativeExtensions, copyExtension, withoutConfigurationDefaults } from '../web/lib/extensions.mjs';
2727
import { policy, toHeadersFile, toNginx } from '../web/lib/headers.mjs';
2828
import { hashTree } from '../web/lib/fingerprint.mjs';
29+
import { NODE_MODULES_COPY, toRedirectsFile } from '../web/lib/pages.mjs';
2930

3031
const REPO = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..');
3132
const WEB = path.join(REPO, 'web');
@@ -46,6 +47,15 @@ const log = (m) => console.log('[build-web] ' + m);
4647
const die = (m) => { console.error('[build-web] ' + m); process.exit(1); };
4748
const sizeOf = (p) => { let n = 0; const walk = (d) => { for (const e of fs.readdirSync(d, { withFileTypes: true })) { const f = path.join(d, e.name); e.isDirectory() ? walk(f) : (n += fs.statSync(f).size); } }; walk(p); return n; };
4849
const mb = (n) => (n / 1048576).toFixed(1) + ' MB';
50+
/** Every folder under `dir` named `name`, not looking inside one that is. */
51+
const directoriesNamed = (dir, name, found = []) => {
52+
for (const e of fs.readdirSync(dir, { withFileTypes: true })) {
53+
if (!e.isDirectory()) { continue; }
54+
const f = path.join(dir, e.name);
55+
if (e.name === name) { found.push(f); } else { directoriesNamed(f, name, found); }
56+
}
57+
return found;
58+
};
4959

5060
/* ----- 1. Code-OSS's web client ---------------------------------------------------------------- */
5161
let staticDir;
@@ -78,6 +88,12 @@ let root = staging;
7888

7989
log('copying the web client');
8090
fs.cpSync(staticDir, path.join(root, 'static'), { recursive: true, filter: (src) => !/\.(js|css)\.map$/.test(src) });
91+
// The client loads its terminal (xterm), its syntax colouring (vscode-textmate, vscode-oniguruma) and more from
92+
// static/node_modules. Cloudflare Pages is never sent a folder by that name (wrangler leaves out **/node_modules): the
93+
// same files are staged again as static/node-modules, and the release's _redirects (written below) serves them from
94+
// there at the address the client asks for. Any other host serves node_modules itself and ignores _redirects.
95+
const clientModules = path.join(root, 'static', 'node_modules');
96+
if (fs.existsSync(clientModules)) { fs.cpSync(clientModules, path.join(root, 'static', NODE_MODULES_COPY), { recursive: true }); }
8197
const baked = new Set(fs.existsSync(path.join(staticDir, 'extensions')) ? fs.readdirSync(path.join(staticDir, 'extensions')) : []);
8298

8399
const extOut = path.join(root, 'extensions');
@@ -98,8 +114,10 @@ fs.writeFileSync(path.join(scratchOut, 'package.nls.json'), '{}\n');
98114
const extensionsSource = path.join(vscodeDir, 'extensions');
99115
if (!fs.existsSync(extensionsSource)) { die('no extensions/ in ' + vscodeDir + ' to take the declarative built-ins from'); }
100116
const declarative = stageDeclarativeExtensions(extensionsSource, extOut, { exclude: [...baked, 'levelcode-ai', 'levelcode-web', 'levelcode-shell'] });
101-
// A checkout that has not had LevelCode's extensions copied in still builds; the themes are then absent.
102-
if (!declarative.includes('levelcode-themes')) {
117+
// A checkout that has not had LevelCode's extensions copied in still builds; the themes are then absent. A client built
118+
// from a checkout that has them has them baked in, and a second copy here would be skipped with an error for each of
119+
// its translations.
120+
if (!declarative.includes('levelcode-themes') && !baked.has('levelcode-themes')) {
103121
const themes = path.join(REPO, 'extensions', 'levelcode-themes');
104122
if (fs.existsSync(themes)) { copyExtension(themes, path.join(extOut, 'levelcode-themes')); declarative.push('levelcode-themes'); }
105123
}
@@ -116,6 +134,11 @@ log(`${extensions.length} extensions: levelcode-ai, levelcode-web, levelcode-web
116134
// main.js and boot.css are addressed by the build id like everything else; the page names them.
117135
for (const f of ['main.js', 'boot.css', 'chrome.css']) { fs.copyFileSync(path.join(WEB, f), path.join(root, f)); }
118136

137+
// Nothing else may be in a folder named node_modules: Pages would not have it, and nothing would serve it from elsewhere.
138+
for (const dir of directoriesNamed(staging, 'node_modules')) {
139+
if (dir !== clientModules) { die(path.relative(staging, dir) + ' is a node_modules folder, which Cloudflare Pages is never sent'); }
140+
}
141+
119142
/* ----- 3. identity of this build: what was staged ------------------------------------------------- */
120143
const id = String(args.id || hashTree(createHash('sha256'), staging, { skip: ['.DS_Store'] }).digest('hex').slice(0, 12));
121144
const prefix = `/_/${id}`;
@@ -159,6 +182,9 @@ for (const f of ['callback.html', 'callback.js', 'callback.css', 'icon.svg', 'ic
159182
const toPattern = (o) => (o ? o.replace('{{uuid}}', '*') : '');
160183
const pol = policy({ account, webviewOrigin: toPattern(webviewOrigin), extHostOrigin: toPattern(extHostOrigin), base: '/_' });
161184
fs.writeFileSync(path.join(outDir, '_headers'), toHeadersFile(pol));
185+
if (fs.existsSync(path.join(root, 'static', NODE_MODULES_COPY))) {
186+
fs.writeFileSync(path.join(outDir, '_redirects'), toRedirectsFile());
187+
}
162188
fs.mkdirSync(path.join(outDir, 'deploy'), { recursive: true });
163189
fs.writeFileSync(path.join(outDir, 'deploy', 'nginx.conf'), toNginx(pol));
164190
const info = {

‎web/lib/headers.mjs‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,11 @@ export function policy(o) {
7070

7171
/** Netlify / Cloudflare Pages `_headers` text. */
7272
export function toHeadersFile(p) {
73-
return p.rules.map((r) => r.path + '\n' + Object.entries(r.headers).map(([k, v]) => ` ${k}: ${v}`).join('\n')).join('\n\n') + '\n';
73+
// Cloudflare Pages serves a page `x.html` at `/x` (it redirects /x.html there), and matches rules against the address
74+
// asked for: a rule for /callback.html alone would leave the sign-in page it serves without its headers. So each
75+
// page's rule is also written for its address without `.html` (/index.html's is `/`, which has its own rule).
76+
const rules = p.rules.flatMap((r) => (/^\/[^/]+\.html$/.test(r.path) && r.path !== '/index.html' ? [r, { ...r, path: r.path.slice(0, -'.html'.length) }] : [r]));
77+
return rules.map((r) => r.path + '\n' + Object.entries(r.headers).map(([k, v]) => ` ${k}: ${v}`).join('\n')).join('\n\n') + '\n';
7478
}
7579

7680
/** Parse a `_headers` file back into rules (the dev server serves a built release with it). */

‎web/test/unit/headers.test.js‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,5 +46,16 @@ const block = (conf, match) => {
4646
assert.strictEqual(headersFor(p, '/')['Content-Security-Policy'], headersFor(p, '/index.html')['Content-Security-Policy']);
4747
});
4848

49+
await test('Cloudflare Pages: the sign-in page keeps its headers at /callback, where Pages serves callback.html', async () => {
50+
const { parseHeadersFile } = await import(pathToFileURL(path.join(__dirname, '..', '..', 'lib', 'headers.mjs')).href);
51+
const read = parseHeadersFile(toHeadersFile(p));
52+
const at = (address) => (read.rules.find((r) => r.path === address) || { headers: {} }).headers;
53+
assert.deepStrictEqual(at('/callback'), at('/callback.html'), 'the same rule under both addresses');
54+
assert.match(at('/callback')['Content-Security-Policy'] || '', /^default-src 'none'/);
55+
assert.strictEqual(at('/callback')['Cache-Control'], 'no-store');
56+
assert.ok(!read.rules.some((r) => r.path === '/index'), '/index.html is served at /, which has its own rule');
57+
assert.ok(!/location = \/callback \{/.test(conf), 'nginx serves callback.html as named and needs no second address');
58+
});
59+
4960
console.log(`\n${n} tests passed`);
5061
})().catch((e) => { console.error(e); process.exit(1); });

0 commit comments

Comments
 (0)