Skip to content

Commit 08a2cc8

Browse files
authored
Merge pull request #111 from levelcodeai/fix/web-cloudflare-pages
fix(web): a release works on Cloudflare Pages — the terminal opens (node_modules is served), the sign-in page keeps its headers
2 parents 3091c99 + f02fa04 commit 08a2cc8

11 files changed

Lines changed: 271 additions & 13 deletions

File tree

‎docs/WEB.md‎

Lines changed: 26 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -287,7 +287,8 @@ node web/serve.mjs --dist dist-web # http://127.0.0.1:8800
287287
**Dev server** (`web/serve.mjs`): `--static <vscode-web dir> --extensions <dir> --extension-names
288288
levelcode-ai,levelcode-web --account <origin>`, plus `--product <json>` and `--config-defaults
289289
<json>` to override the product configuration. The extension is built with
290-
`node web/ai-extension/build.mjs --out <dir> --vscode <checkout>`.
290+
`node web/ai-extension/build.mjs --out <dir> --vscode <checkout>`. With `--dist <release> --pages` it answers
291+
as Cloudflare Pages does (below): the release-level end-to-end checks take `--pages` too.
291292

292293
**Release layout.** `index.html` and the callback page are at the root, un-cached. Everything
293294
else is under `/_/<id>/` and cached for a year (`immutable`): `static/` (the client), `extensions/`,
@@ -323,6 +324,29 @@ Content-Security-Policy for the editor is the tightest the workbench allows:
323324

324325
The callback page has `default-src 'none'` and only its own script and style.
325326

327+
### On Cloudflare Pages
328+
329+
Pages reads `_headers` as written, and three things it does differently from a plain file server are
330+
handled in the release itself. `web/serve.mjs --dist <release> --pages` answers the way Pages does
331+
(`web/lib/pages.mjs`), so the end-to-end checks run against a release as it will be served there:
332+
333+
- **It is never sent a folder named `node_modules`.** `wrangler pages deploy` leaves out `**/node_modules`
334+
(and `.git`, `.DS_Store`), with no option to keep them, and the client loads its terminal (xterm), its syntax
335+
colouring (vscode-textmate, vscode-oniguruma) and more from `static/node_modules`. On the first deploy every
336+
one of those came back as the page (Pages answers an address it has no file for with `index.html`), and
337+
the terminal did not open. The build stages the same files again as `static/node-modules`, and the
338+
release's `_redirects` serves them from there at the address the client asks for
339+
(`/_/:id/static/node_modules/* /_/:id/static/node-modules/:splat 200`). Any other host serves
340+
`node_modules` itself and ignores `_redirects`. The build refuses a `node_modules` folder anywhere else.
341+
- **It serves `x.html` at `/x`** (and redirects `/x.html` there, query kept), and matches `_headers` rules
342+
against the address asked for: every page's rule is also written for its address without `.html`, so the
343+
sign-in page keeps its own policy at `/callback`.
344+
- **A file it does not have is answered with the page, status 200.** A check that only looks at status codes
345+
does not see it; `e2e.mjs` fails on any script, JSON or WebAssembly response that is HTML.
346+
347+
The limits that matter: 25 MiB per file (the largest is a 24.6 MiB diagram bundle in the client) and 20,000
348+
files on the free plan (1,840 uploaded with the copy, measured; about twice that with the previous build kept online).
349+
326350
### On the account site
327351

328352
Three settings on the Rails app (all default **off**; with none set nothing changes):
@@ -427,7 +451,7 @@ transaction, so both end with the same key.
427451
| The shell | `./scripts/test-extensions.sh` (the unit suites `web/test/unit/shell-*.test.js`) | The engine: a differential against GNU bash 5.1 (905 cases, a shrink-only table of the ones that differ), fuzzing, every patch of the vendored code with its anchors, the build; the runtime: the commit and its refusals, kill and deadline against a **real engine in a worker thread**, limits; the terminal: key bytes, the line editor against real xterm.js 6.1 in its headless build, width, the output filter, history. With the Code-OSS checkout present the real-xterm halves run; `LEVELCODE_REQUIRE_XTERM=1` and `LEVELCODE_REQUIRE_WORKER=1` make a skip a failure, which is what CI sets. |
428452
| Mutations | `node web/test/helpers/mutation/mutate.mjs` | 371 edits to the runtime, the commit and the sanitiser, each of which a suite must catch (a tool, not a suite). |
429453
| The shell, in a browser | `node web/test/e2e-shell.mjs --static … --extensions …` | Real headless Chrome and the real nested Worker: nothing of the shell is fetched until it is used; a terminal from the palette; typing, files, a working folder that persists, 127, Ctrl+C on `sleep 60`, a link to `path:line:col` that opens the file at that line, a pasted three-line buffer through a resize; hostile output that does not type into the prompt or rename the terminal; the agent offered `run_command` without background, a file written through the review (Keep/Undo), a `.vscode` write refused whole, a timeout that applies nothing, the mirror tab. |
430-
| End to end | `node web/test/e2e.mjs --dist dist-web --stub-port <port the build used>` | Real headless Chrome: boot from static files, the extension activates in the worker, sign in through the real callback page, agent tool calls (write / list / search / read / edit) against the scratch workspace, Quick Open, Keep and Undo, a reload that keeps the session and files. |
454+
| End to end | `node web/test/e2e.mjs --dist dist-web --stub-port <port the build used> [--pages]` | Real headless Chrome: boot from static files, the extension activates in the worker, sign in through the real callback page, agent tool calls (write / list / search / read / edit) against the scratch workspace, Quick Open, Keep and Undo, a reload that keeps the session and files, a terminal that opens, and no script, JSON or WebAssembly file answered with HTML. `--pages`: served as Cloudflare Pages serves it (also for the look and same-tab sign-in checks). |
431455
| The look | `node web/test/e2e-look.mjs --dist dist-web --stub-port <port>` | The skinned chat (greeting, starters, composer in the reading column, the rail at 1440 px and the dialog at 900 px), light and dark following the system with this build's own colours (also in the chat's webview), no Welcome page, no second chat, source control / run / extensions not offered, the layout switch both ways, and the docked layout docked again after a reload. Release builds only. |
432456
| Same-tab sign-in | `node web/test/e2e-signin-tab.mjs --dist dist-web --stub-port <port>` | No second window; verifier survives the reload; no repeat; no sign-in for a signed-in editor; a second tab uses the same session; a stranded result is delivered, an old one dropped. |
433457
| Bring your own key | `node web/test/e2e-byok.mjs --static … --extensions …` | The request goes from the tab to the provider with the user's key and nothing of LevelCode's. |

‎scripts/build-web.mjs‎

Lines changed: 29 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313
// workspace extension (web/workspace) and every declarative built-in (grammars, themes, icons);
1414
// 3. names the build after a hash of everything it staged, lays that out under the build-addressed
1515
// prefix (/_/<id>/...), renders index.html with the configuration baked in, and writes the HTTP
16-
// policy for the host (_headers, nginx).
16+
// policy for the host (_headers, nginx) and, for Cloudflare Pages, a _redirects.
1717
//
1818
// It changes nothing outside --out, except that building the client from the checkout leaves gulp's own output,
1919
// vscode-web/, next to it (pass --static to use a client built earlier and skip that). It does not deploy.
@@ -26,6 +26,7 @@ import { webConfig, renderIndex } from '../web/lib/config.mjs';
2626
import { stageDeclarativeExtensions, copyExtension, withoutConfigurationDefaults } from '../web/lib/extensions.mjs';
2727
import { policy, toHeadersFile, toNginx } from '../web/lib/headers.mjs';
2828
import { hashTree } from '../web/lib/fingerprint.mjs';
29+
import { NODE_MODULES_COPY, toRedirectsFile } from '../web/lib/pages.mjs';
2930

3031
const REPO = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..');
3132
const WEB = path.join(REPO, 'web');
@@ -46,6 +47,15 @@ const log = (m) => console.log('[build-web] ' + m);
4647
const die = (m) => { console.error('[build-web] ' + m); process.exit(1); };
4748
const sizeOf = (p) => { let n = 0; const walk = (d) => { for (const e of fs.readdirSync(d, { withFileTypes: true })) { const f = path.join(d, e.name); e.isDirectory() ? walk(f) : (n += fs.statSync(f).size); } }; walk(p); return n; };
4849
const mb = (n) => (n / 1048576).toFixed(1) + ' MB';
50+
/** Every folder under `dir` named `name`, not looking inside one that is. */
51+
const directoriesNamed = (dir, name, found = []) => {
52+
for (const e of fs.readdirSync(dir, { withFileTypes: true })) {
53+
if (!e.isDirectory()) { continue; }
54+
const f = path.join(dir, e.name);
55+
if (e.name === name) { found.push(f); } else { directoriesNamed(f, name, found); }
56+
}
57+
return found;
58+
};
4959

5060
/* ----- 1. Code-OSS's web client ---------------------------------------------------------------- */
5161
let staticDir;
@@ -78,6 +88,12 @@ let root = staging;
7888

7989
log('copying the web client');
8090
fs.cpSync(staticDir, path.join(root, 'static'), { recursive: true, filter: (src) => !/\.(js|css)\.map$/.test(src) });
91+
// The client loads its terminal (xterm), its syntax colouring (vscode-textmate, vscode-oniguruma) and more from
92+
// static/node_modules. Cloudflare Pages is never sent a folder by that name (wrangler leaves out **/node_modules): the
93+
// same files are staged again as static/node-modules, and the release's _redirects (written below) serves them from
94+
// there at the address the client asks for. Any other host serves node_modules itself and ignores _redirects.
95+
const clientModules = path.join(root, 'static', 'node_modules');
96+
if (fs.existsSync(clientModules)) { fs.cpSync(clientModules, path.join(root, 'static', NODE_MODULES_COPY), { recursive: true }); }
8197
const baked = new Set(fs.existsSync(path.join(staticDir, 'extensions')) ? fs.readdirSync(path.join(staticDir, 'extensions')) : []);
8298

8399
const extOut = path.join(root, 'extensions');
@@ -98,8 +114,10 @@ fs.writeFileSync(path.join(scratchOut, 'package.nls.json'), '{}\n');
98114
const extensionsSource = path.join(vscodeDir, 'extensions');
99115
if (!fs.existsSync(extensionsSource)) { die('no extensions/ in ' + vscodeDir + ' to take the declarative built-ins from'); }
100116
const declarative = stageDeclarativeExtensions(extensionsSource, extOut, { exclude: [...baked, 'levelcode-ai', 'levelcode-web', 'levelcode-shell'] });
101-
// A checkout that has not had LevelCode's extensions copied in still builds; the themes are then absent.
102-
if (!declarative.includes('levelcode-themes')) {
117+
// A checkout that has not had LevelCode's extensions copied in still builds; the themes are then absent. A client built
118+
// from a checkout that has them has them baked in, and a second copy here would be skipped with an error for each of
119+
// its translations.
120+
if (!declarative.includes('levelcode-themes') && !baked.has('levelcode-themes')) {
103121
const themes = path.join(REPO, 'extensions', 'levelcode-themes');
104122
if (fs.existsSync(themes)) { copyExtension(themes, path.join(extOut, 'levelcode-themes')); declarative.push('levelcode-themes'); }
105123
}
@@ -116,6 +134,11 @@ log(`${extensions.length} extensions: levelcode-ai, levelcode-web, levelcode-web
116134
// main.js and boot.css are addressed by the build id like everything else; the page names them.
117135
for (const f of ['main.js', 'boot.css', 'chrome.css']) { fs.copyFileSync(path.join(WEB, f), path.join(root, f)); }
118136

137+
// Nothing else may be in a folder named node_modules: Pages would not have it, and nothing would serve it from elsewhere.
138+
for (const dir of directoriesNamed(staging, 'node_modules')) {
139+
if (dir !== clientModules) { die(path.relative(staging, dir) + ' is a node_modules folder, which Cloudflare Pages is never sent'); }
140+
}
141+
119142
/* ----- 3. identity of this build: what was staged ------------------------------------------------- */
120143
const id = String(args.id || hashTree(createHash('sha256'), staging, { skip: ['.DS_Store'] }).digest('hex').slice(0, 12));
121144
const prefix = `/_/${id}`;
@@ -159,6 +182,9 @@ for (const f of ['callback.html', 'callback.js', 'callback.css', 'icon.svg', 'ic
159182
const toPattern = (o) => (o ? o.replace('{{uuid}}', '*') : '');
160183
const pol = policy({ account, webviewOrigin: toPattern(webviewOrigin), extHostOrigin: toPattern(extHostOrigin), base: '/_' });
161184
fs.writeFileSync(path.join(outDir, '_headers'), toHeadersFile(pol));
185+
if (fs.existsSync(path.join(root, 'static', NODE_MODULES_COPY))) {
186+
fs.writeFileSync(path.join(outDir, '_redirects'), toRedirectsFile());
187+
}
162188
fs.mkdirSync(path.join(outDir, 'deploy'), { recursive: true });
163189
fs.writeFileSync(path.join(outDir, 'deploy', 'nginx.conf'), toNginx(pol));
164190
const info = {

‎web/lib/headers.mjs‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,11 @@ export function policy(o) {
7070

7171
/** Netlify / Cloudflare Pages `_headers` text. */
7272
export function toHeadersFile(p) {
73-
return p.rules.map((r) => r.path + '\n' + Object.entries(r.headers).map(([k, v]) => ` ${k}: ${v}`).join('\n')).join('\n\n') + '\n';
73+
// Cloudflare Pages serves a page `x.html` at `/x` (it redirects /x.html there), and matches rules against the address
74+
// asked for: a rule for /callback.html alone would leave the sign-in page it serves without its headers. So each
75+
// page's rule is also written for its address without `.html` (/index.html's is `/`, which has its own rule).
76+
const rules = p.rules.flatMap((r) => (/^\/[^/]+\.html$/.test(r.path) && r.path !== '/index.html' ? [r, { ...r, path: r.path.slice(0, -'.html'.length) }] : [r]));
77+
return rules.map((r) => r.path + '\n' + Object.entries(r.headers).map(([k, v]) => ` ${k}: ${v}`).join('\n')).join('\n\n') + '\n';
7478
}
7579

7680
/** Parse a `_headers` file back into rules (the dev server serves a built release with it). */

‎web/lib/pages.mjs‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
// How Cloudflare Pages answers a request for a release (scripts/build-web.mjs), for web/serve.mjs --pages. A release
2+
// that only ever ran under serve.mjs's plain file server went live on Pages without its terminal: Pages had never
3+
// been sent the client's node_modules, and answered every one of those scripts with the page. This is what Pages
4+
// does differently from a plain file server, so a release can be checked the way it will be served there.
5+
//
6+
// 1. What `wrangler pages deploy` does not upload is not there: any path with a segment named node_modules or .git,
7+
// .DS_Store, and _worker.js, _redirects, _headers, _routes.json and functions/ at the root (its IGNORE_LIST, in
8+
// workers-sdk packages/wrangler/src/pages/validate.ts). Symbolic links are skipped too.
9+
// 2. _redirects comes first, whatever the files: the first rule that matches applies. 200 serves another path of the
10+
// site in place (a rewrite); 301, 302, 303, 307 and 308 redirect. `:name` stands for one segment, `*` for the rest.
11+
// 3. An .html address is redirected to the same address without it (/a.html -> /a, /x/index.html -> /x/), and
12+
// /a serves a.html, /x/ serves x/index.html.
13+
// 4. An address that matches no file is answered with /index.html and 200 (what Pages does when there is no
14+
// 404.html): a missing script comes back as the page, which is the failure this is here to show.
15+
// Pure: no file system access. Callers say whether a path exists.
16+
17+
/**
18+
* The release's own answer to (1) for the client's node_modules (scripts/build-web.mjs): the same files are staged
19+
* again under this name, and _redirects serves them from there at the address the client asks for.
20+
*/
21+
export const NODE_MODULES_COPY = 'node-modules';
22+
23+
/** The release's _redirects. One rule for every build folder (:id), so a previous build kept online is served too. */
24+
export function toRedirectsFile() {
25+
return `/_/:id/static/node_modules/* /_/:id/static/${NODE_MODULES_COPY}/:splat 200\n`;
26+
}
27+
28+
/** Is this path (relative to the release, `/`-separated) one that `wrangler pages deploy` uploads? */
29+
export function uploaded(rel) {
30+
const parts = rel.split('/').filter(Boolean);
31+
if (parts.some((p) => p === 'node_modules' || p === '.git' || p === '.DS_Store')) { return false; }
32+
if (parts.length === 1 && ['_worker.js', '_redirects', '_headers', '_routes.json'].includes(parts[0])) { return false; }
33+
return parts[0] !== 'functions';
34+
}
35+
36+
/**
37+
* The rules of a _redirects file, in order: `from to [status]`, blank lines and `#` comments skipped.
38+
* @param {string} text
39+
* @returns {Array<{ from: string, to: string, status: number }>}
40+
*/
41+
export function parseRedirects(text) {
42+
const rules = [];
43+
for (const line of text.split('\n')) {
44+
const t = line.trim();
45+
if (!t || t.startsWith('#')) { continue; }
46+
const [from, to, status] = t.split(/\s+/);
47+
if (from && to) { rules.push({ from, to, status: status ? Number(status) : 302 }); }
48+
}
49+
return rules;
50+
}
51+
52+
/**
53+
* The first rule that matches `pathname`, with its destination filled in, or null.
54+
* @param {Array<{ from: string, to: string, status: number }>} rules
55+
* @param {string} pathname
56+
* @returns {{ to: string, status: number } | null}
57+
*/
58+
export function matchRedirect(rules, pathname) {
59+
for (const r of rules) {
60+
const names = [];
61+
const pattern = r.from.split('/').map((seg, i, all) => {
62+
if (seg === '*' && i === all.length - 1) { names.push('splat'); return '(.*)'; }
63+
if (seg.startsWith(':')) { names.push(seg.slice(1)); return '([^/]+)'; }
64+
return seg.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
65+
}).join('/');
66+
const m = new RegExp('^' + pattern + '$').exec(pathname);
67+
if (!m) { continue; }
68+
const values = Object.fromEntries(names.map((n, i) => [n, m[i + 1]]));
69+
const to = r.to.replace(/:([A-Za-z_][A-Za-z0-9_]*)/g, (all, n) => (n in values ? values[n] : all));
70+
return { to, status: r.status };
71+
}
72+
return null;
73+
}
74+
75+
/**
76+
* What Pages does with an address before it looks for a file: redirect an .html one, or name the file an
77+
* extensionless one stands for.
78+
* @param {string} pathname
79+
* @param {(rel: string) => boolean} isFile is there an uploaded file at this release-relative path?
80+
* @returns {{ redirect: string } | { file: string } | null} null: no such page; serve the fallback
81+
*/
82+
export function resolvePage(pathname, isFile) {
83+
if (pathname.endsWith('/index.html')) { return { redirect: pathname.slice(0, -'index.html'.length) }; }
84+
if (pathname.endsWith('.html') && isFile(pathname.slice(1))) { return { redirect: pathname.slice(0, -'.html'.length) }; }
85+
const rel = pathname.slice(1);
86+
if (pathname.endsWith('/')) { return isFile(rel + 'index.html') ? { file: rel + 'index.html' } : null; }
87+
if (isFile(rel)) { return { file: rel }; }
88+
if (isFile(rel + '.html')) { return { file: rel + '.html' }; }
89+
return null;
90+
}

0 commit comments

Comments
 (0)