diff --git a/docs/WEB.md b/docs/WEB.md index 5036ca1..3711f3d 100644 --- a/docs/WEB.md +++ b/docs/WEB.md @@ -287,7 +287,8 @@ node web/serve.mjs --dist dist-web # http://127.0.0.1:8800 **Dev server** (`web/serve.mjs`): `--static --extensions --extension-names levelcode-ai,levelcode-web --account `, plus `--product ` and `--config-defaults ` to override the product configuration. The extension is built with -`node web/ai-extension/build.mjs --out --vscode `. +`node web/ai-extension/build.mjs --out --vscode `. With `--dist --pages` it answers +as Cloudflare Pages does (below): the release-level end-to-end checks take `--pages` too. **Release layout.** `index.html` and the callback page are at the root, un-cached. Everything else is under `/_//` and cached for a year (`immutable`): `static/` (the client), `extensions/`, @@ -323,6 +324,29 @@ Content-Security-Policy for the editor is the tightest the workbench allows: The callback page has `default-src 'none'` and only its own script and style. +### On Cloudflare Pages + +Pages reads `_headers` as written, and three things it does differently from a plain file server are +handled in the release itself. `web/serve.mjs --dist --pages` answers the way Pages does +(`web/lib/pages.mjs`), so the end-to-end checks run against a release as it will be served there: + +- **It is never sent a folder named `node_modules`.** `wrangler pages deploy` leaves out `**/node_modules` + (and `.git`, `.DS_Store`), with no option to keep them, and the client loads its terminal (xterm), its syntax + colouring (vscode-textmate, vscode-oniguruma) and more from `static/node_modules`. On the first deploy every + one of those came back as the page (Pages answers an address it has no file for with `index.html`), and + the terminal did not open. The build stages the same files again as `static/node-modules`, and the + release's `_redirects` serves them from there at the address the client asks for + (`/_/:id/static/node_modules/* /_/:id/static/node-modules/:splat 200`). Any other host serves + `node_modules` itself and ignores `_redirects`. The build refuses a `node_modules` folder anywhere else. +- **It serves `x.html` at `/x`** (and redirects `/x.html` there, query kept), and matches `_headers` rules + against the address asked for: every page's rule is also written for its address without `.html`, so the + sign-in page keeps its own policy at `/callback`. +- **A file it does not have is answered with the page, status 200.** A check that only looks at status codes + does not see it; `e2e.mjs` fails on any script, JSON or WebAssembly response that is HTML. + +The limits that matter: 25 MiB per file (the largest is a 24.6 MiB diagram bundle in the client) and 20,000 +files on the free plan (1,840 uploaded with the copy, measured; about twice that with the previous build kept online). + ### On the account site Three settings on the Rails app (all default **off**; with none set nothing changes): @@ -427,7 +451,7 @@ transaction, so both end with the same key. | The shell | `./scripts/test-extensions.sh` (the unit suites `web/test/unit/shell-*.test.js`) | The engine: a differential against GNU bash 5.1 (905 cases, a shrink-only table of the ones that differ), fuzzing, every patch of the vendored code with its anchors, the build; the runtime: the commit and its refusals, kill and deadline against a **real engine in a worker thread**, limits; the terminal: key bytes, the line editor against real xterm.js 6.1 in its headless build, width, the output filter, history. With the Code-OSS checkout present the real-xterm halves run; `LEVELCODE_REQUIRE_XTERM=1` and `LEVELCODE_REQUIRE_WORKER=1` make a skip a failure, which is what CI sets. | | Mutations | `node web/test/helpers/mutation/mutate.mjs` | 371 edits to the runtime, the commit and the sanitiser, each of which a suite must catch (a tool, not a suite). | | The shell, in a browser | `node web/test/e2e-shell.mjs --static … --extensions …` | Real headless Chrome and the real nested Worker: nothing of the shell is fetched until it is used; a terminal from the palette; typing, files, a working folder that persists, 127, Ctrl+C on `sleep 60`, a link to `path:line:col` that opens the file at that line, a pasted three-line buffer through a resize; hostile output that does not type into the prompt or rename the terminal; the agent offered `run_command` without background, a file written through the review (Keep/Undo), a `.vscode` write refused whole, a timeout that applies nothing, the mirror tab. | -| End to end | `node web/test/e2e.mjs --dist dist-web --stub-port ` | Real headless Chrome: boot from static files, the extension activates in the worker, sign in through the real callback page, agent tool calls (write / list / search / read / edit) against the scratch workspace, Quick Open, Keep and Undo, a reload that keeps the session and files. | +| End to end | `node web/test/e2e.mjs --dist dist-web --stub-port [--pages]` | Real headless Chrome: boot from static files, the extension activates in the worker, sign in through the real callback page, agent tool calls (write / list / search / read / edit) against the scratch workspace, Quick Open, Keep and Undo, a reload that keeps the session and files, a terminal that opens, and no script, JSON or WebAssembly file answered with HTML. `--pages`: served as Cloudflare Pages serves it (also for the look and same-tab sign-in checks). | | The look | `node web/test/e2e-look.mjs --dist dist-web --stub-port ` | The skinned chat (greeting, starters, composer in the reading column, the rail at 1440 px and the dialog at 900 px), light and dark following the system with this build's own colours (also in the chat's webview), no Welcome page, no second chat, source control / run / extensions not offered, the layout switch both ways, and the docked layout docked again after a reload. Release builds only. | | Same-tab sign-in | `node web/test/e2e-signin-tab.mjs --dist dist-web --stub-port ` | No second window; verifier survives the reload; no repeat; no sign-in for a signed-in editor; a second tab uses the same session; a stranded result is delivered, an old one dropped. | | Bring your own key | `node web/test/e2e-byok.mjs --static … --extensions …` | The request goes from the tab to the provider with the user's key and nothing of LevelCode's. | diff --git a/scripts/build-web.mjs b/scripts/build-web.mjs index 45b6512..85da0f6 100644 --- a/scripts/build-web.mjs +++ b/scripts/build-web.mjs @@ -13,7 +13,7 @@ // workspace extension (web/workspace) and every declarative built-in (grammars, themes, icons); // 3. names the build after a hash of everything it staged, lays that out under the build-addressed // prefix (/_//...), renders index.html with the configuration baked in, and writes the HTTP -// policy for the host (_headers, nginx). +// policy for the host (_headers, nginx) and, for Cloudflare Pages, a _redirects. // // It changes nothing outside --out, except that building the client from the checkout leaves gulp's own output, // vscode-web/, next to it (pass --static to use a client built earlier and skip that). It does not deploy. @@ -26,6 +26,7 @@ import { webConfig, renderIndex } from '../web/lib/config.mjs'; import { stageDeclarativeExtensions, copyExtension, withoutConfigurationDefaults } from '../web/lib/extensions.mjs'; import { policy, toHeadersFile, toNginx } from '../web/lib/headers.mjs'; import { hashTree } from '../web/lib/fingerprint.mjs'; +import { NODE_MODULES_COPY, toRedirectsFile } from '../web/lib/pages.mjs'; const REPO = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..'); const WEB = path.join(REPO, 'web'); @@ -46,6 +47,15 @@ const log = (m) => console.log('[build-web] ' + m); const die = (m) => { console.error('[build-web] ' + m); process.exit(1); }; const sizeOf = (p) => { let n = 0; const walk = (d) => { for (const e of fs.readdirSync(d, { withFileTypes: true })) { const f = path.join(d, e.name); e.isDirectory() ? walk(f) : (n += fs.statSync(f).size); } }; walk(p); return n; }; const mb = (n) => (n / 1048576).toFixed(1) + ' MB'; +/** Every folder under `dir` named `name`, not looking inside one that is. */ +const directoriesNamed = (dir, name, found = []) => { + for (const e of fs.readdirSync(dir, { withFileTypes: true })) { + if (!e.isDirectory()) { continue; } + const f = path.join(dir, e.name); + if (e.name === name) { found.push(f); } else { directoriesNamed(f, name, found); } + } + return found; +}; /* ----- 1. Code-OSS's web client ---------------------------------------------------------------- */ let staticDir; @@ -78,6 +88,12 @@ let root = staging; log('copying the web client'); fs.cpSync(staticDir, path.join(root, 'static'), { recursive: true, filter: (src) => !/\.(js|css)\.map$/.test(src) }); +// The client loads its terminal (xterm), its syntax colouring (vscode-textmate, vscode-oniguruma) and more from +// static/node_modules. Cloudflare Pages is never sent a folder by that name (wrangler leaves out **/node_modules): the +// same files are staged again as static/node-modules, and the release's _redirects (written below) serves them from +// there at the address the client asks for. Any other host serves node_modules itself and ignores _redirects. +const clientModules = path.join(root, 'static', 'node_modules'); +if (fs.existsSync(clientModules)) { fs.cpSync(clientModules, path.join(root, 'static', NODE_MODULES_COPY), { recursive: true }); } const baked = new Set(fs.existsSync(path.join(staticDir, 'extensions')) ? fs.readdirSync(path.join(staticDir, 'extensions')) : []); const extOut = path.join(root, 'extensions'); @@ -98,8 +114,10 @@ fs.writeFileSync(path.join(scratchOut, 'package.nls.json'), '{}\n'); const extensionsSource = path.join(vscodeDir, 'extensions'); if (!fs.existsSync(extensionsSource)) { die('no extensions/ in ' + vscodeDir + ' to take the declarative built-ins from'); } const declarative = stageDeclarativeExtensions(extensionsSource, extOut, { exclude: [...baked, 'levelcode-ai', 'levelcode-web', 'levelcode-shell'] }); -// A checkout that has not had LevelCode's extensions copied in still builds; the themes are then absent. -if (!declarative.includes('levelcode-themes')) { +// A checkout that has not had LevelCode's extensions copied in still builds; the themes are then absent. A client built +// from a checkout that has them has them baked in, and a second copy here would be skipped with an error for each of +// its translations. +if (!declarative.includes('levelcode-themes') && !baked.has('levelcode-themes')) { const themes = path.join(REPO, 'extensions', 'levelcode-themes'); if (fs.existsSync(themes)) { copyExtension(themes, path.join(extOut, 'levelcode-themes')); declarative.push('levelcode-themes'); } } @@ -116,6 +134,11 @@ log(`${extensions.length} extensions: levelcode-ai, levelcode-web, levelcode-web // main.js and boot.css are addressed by the build id like everything else; the page names them. for (const f of ['main.js', 'boot.css', 'chrome.css']) { fs.copyFileSync(path.join(WEB, f), path.join(root, f)); } +// Nothing else may be in a folder named node_modules: Pages would not have it, and nothing would serve it from elsewhere. +for (const dir of directoriesNamed(staging, 'node_modules')) { + if (dir !== clientModules) { die(path.relative(staging, dir) + ' is a node_modules folder, which Cloudflare Pages is never sent'); } +} + /* ----- 3. identity of this build: what was staged ------------------------------------------------- */ const id = String(args.id || hashTree(createHash('sha256'), staging, { skip: ['.DS_Store'] }).digest('hex').slice(0, 12)); const prefix = `/_/${id}`; @@ -159,6 +182,9 @@ for (const f of ['callback.html', 'callback.js', 'callback.css', 'icon.svg', 'ic const toPattern = (o) => (o ? o.replace('{{uuid}}', '*') : ''); const pol = policy({ account, webviewOrigin: toPattern(webviewOrigin), extHostOrigin: toPattern(extHostOrigin), base: '/_' }); fs.writeFileSync(path.join(outDir, '_headers'), toHeadersFile(pol)); +if (fs.existsSync(path.join(root, 'static', NODE_MODULES_COPY))) { + fs.writeFileSync(path.join(outDir, '_redirects'), toRedirectsFile()); +} fs.mkdirSync(path.join(outDir, 'deploy'), { recursive: true }); fs.writeFileSync(path.join(outDir, 'deploy', 'nginx.conf'), toNginx(pol)); const info = { diff --git a/web/lib/headers.mjs b/web/lib/headers.mjs index a993434..211ba8a 100644 --- a/web/lib/headers.mjs +++ b/web/lib/headers.mjs @@ -70,7 +70,11 @@ export function policy(o) { /** Netlify / Cloudflare Pages `_headers` text. */ export function toHeadersFile(p) { - return p.rules.map((r) => r.path + '\n' + Object.entries(r.headers).map(([k, v]) => ` ${k}: ${v}`).join('\n')).join('\n\n') + '\n'; + // Cloudflare Pages serves a page `x.html` at `/x` (it redirects /x.html there), and matches rules against the address + // asked for: a rule for /callback.html alone would leave the sign-in page it serves without its headers. So each + // page's rule is also written for its address without `.html` (/index.html's is `/`, which has its own rule). + const rules = p.rules.flatMap((r) => (/^\/[^/]+\.html$/.test(r.path) && r.path !== '/index.html' ? [r, { ...r, path: r.path.slice(0, -'.html'.length) }] : [r])); + return rules.map((r) => r.path + '\n' + Object.entries(r.headers).map(([k, v]) => ` ${k}: ${v}`).join('\n')).join('\n\n') + '\n'; } /** Parse a `_headers` file back into rules (the dev server serves a built release with it). */ diff --git a/web/lib/pages.mjs b/web/lib/pages.mjs new file mode 100644 index 0000000..bb7a044 --- /dev/null +++ b/web/lib/pages.mjs @@ -0,0 +1,90 @@ +// How Cloudflare Pages answers a request for a release (scripts/build-web.mjs), for web/serve.mjs --pages. A release +// that only ever ran under serve.mjs's plain file server went live on Pages without its terminal: Pages had never +// been sent the client's node_modules, and answered every one of those scripts with the page. This is what Pages +// does differently from a plain file server, so a release can be checked the way it will be served there. +// +// 1. What `wrangler pages deploy` does not upload is not there: any path with a segment named node_modules or .git, +// .DS_Store, and _worker.js, _redirects, _headers, _routes.json and functions/ at the root (its IGNORE_LIST, in +// workers-sdk packages/wrangler/src/pages/validate.ts). Symbolic links are skipped too. +// 2. _redirects comes first, whatever the files: the first rule that matches applies. 200 serves another path of the +// site in place (a rewrite); 301, 302, 303, 307 and 308 redirect. `:name` stands for one segment, `*` for the rest. +// 3. An .html address is redirected to the same address without it (/a.html -> /a, /x/index.html -> /x/), and +// /a serves a.html, /x/ serves x/index.html. +// 4. An address that matches no file is answered with /index.html and 200 (what Pages does when there is no +// 404.html): a missing script comes back as the page, which is the failure this is here to show. +// Pure: no file system access. Callers say whether a path exists. + +/** + * The release's own answer to (1) for the client's node_modules (scripts/build-web.mjs): the same files are staged + * again under this name, and _redirects serves them from there at the address the client asks for. + */ +export const NODE_MODULES_COPY = 'node-modules'; + +/** The release's _redirects. One rule for every build folder (:id), so a previous build kept online is served too. */ +export function toRedirectsFile() { + return `/_/:id/static/node_modules/* /_/:id/static/${NODE_MODULES_COPY}/:splat 200\n`; +} + +/** Is this path (relative to the release, `/`-separated) one that `wrangler pages deploy` uploads? */ +export function uploaded(rel) { + const parts = rel.split('/').filter(Boolean); + if (parts.some((p) => p === 'node_modules' || p === '.git' || p === '.DS_Store')) { return false; } + if (parts.length === 1 && ['_worker.js', '_redirects', '_headers', '_routes.json'].includes(parts[0])) { return false; } + return parts[0] !== 'functions'; +} + +/** + * The rules of a _redirects file, in order: `from to [status]`, blank lines and `#` comments skipped. + * @param {string} text + * @returns {Array<{ from: string, to: string, status: number }>} + */ +export function parseRedirects(text) { + const rules = []; + for (const line of text.split('\n')) { + const t = line.trim(); + if (!t || t.startsWith('#')) { continue; } + const [from, to, status] = t.split(/\s+/); + if (from && to) { rules.push({ from, to, status: status ? Number(status) : 302 }); } + } + return rules; +} + +/** + * The first rule that matches `pathname`, with its destination filled in, or null. + * @param {Array<{ from: string, to: string, status: number }>} rules + * @param {string} pathname + * @returns {{ to: string, status: number } | null} + */ +export function matchRedirect(rules, pathname) { + for (const r of rules) { + const names = []; + const pattern = r.from.split('/').map((seg, i, all) => { + if (seg === '*' && i === all.length - 1) { names.push('splat'); return '(.*)'; } + if (seg.startsWith(':')) { names.push(seg.slice(1)); return '([^/]+)'; } + return seg.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); + }).join('/'); + const m = new RegExp('^' + pattern + '$').exec(pathname); + if (!m) { continue; } + const values = Object.fromEntries(names.map((n, i) => [n, m[i + 1]])); + const to = r.to.replace(/:([A-Za-z_][A-Za-z0-9_]*)/g, (all, n) => (n in values ? values[n] : all)); + return { to, status: r.status }; + } + return null; +} + +/** + * What Pages does with an address before it looks for a file: redirect an .html one, or name the file an + * extensionless one stands for. + * @param {string} pathname + * @param {(rel: string) => boolean} isFile is there an uploaded file at this release-relative path? + * @returns {{ redirect: string } | { file: string } | null} null: no such page; serve the fallback + */ +export function resolvePage(pathname, isFile) { + if (pathname.endsWith('/index.html')) { return { redirect: pathname.slice(0, -'index.html'.length) }; } + if (pathname.endsWith('.html') && isFile(pathname.slice(1))) { return { redirect: pathname.slice(0, -'.html'.length) }; } + const rel = pathname.slice(1); + if (pathname.endsWith('/')) { return isFile(rel + 'index.html') ? { file: rel + 'index.html' } : null; } + if (isFile(rel)) { return { file: rel }; } + if (isFile(rel + '.html')) { return { file: rel + '.html' }; } + return null; +} diff --git a/web/serve.mjs b/web/serve.mjs index 0700202..dbcf072 100644 --- a/web/serve.mjs +++ b/web/serve.mjs @@ -2,6 +2,7 @@ // Development / end-to-end server for LevelCode in the browser. Dependency-free. // // node web/serve.mjs --static --extensions [--port 8800] +// node web/serve.mjs --dist [--pages] // // It serves exactly what a static host serves in production (see scripts/build-web.mjs), with // the Code-OSS web build mounted at /static and the LevelCode extensions at /extensions. @@ -10,6 +11,7 @@ import fs from 'node:fs'; import path from 'node:path'; import { webConfig, renderIndex, WEB_DIR } from './lib/config.mjs'; import { parseHeadersFile, headersFor } from './lib/headers.mjs'; +import { uploaded, parseRedirects, matchRedirect, resolvePage } from './lib/pages.mjs'; const args = Object.fromEntries(process.argv.slice(2).reduce((a, v, i, all) => { if (v.startsWith('--')) { a.push([v.slice(2), all[i + 1] && !all[i + 1].startsWith('--') ? all[i + 1] : 'true']); } @@ -23,6 +25,16 @@ const STATIC_DIR = path.resolve(args.static || process.env.LEVELCODE_WEB_STATIC const EXT_DIR = path.resolve(args.extensions || process.env.LEVELCODE_WEB_EXTENSIONS || ''); if (!DIST && !args.static && !process.env.LEVELCODE_WEB_STATIC) { console.error('missing --static (or --dist )'); process.exit(2); } const RULES = DIST && fs.existsSync(path.join(DIST, '_headers')) ? parseHeadersFile(fs.readFileSync(path.join(DIST, '_headers'), 'utf8')) : { rules: [] }; +// --pages (with --dist): answer as Cloudflare Pages answers that release — only what wrangler uploads, its _redirects, +// its .html redirects, and the page for an address that matches nothing (web/lib/pages.mjs). +const PAGES = !!DIST && args.pages === 'true'; +const REDIRECTS = PAGES && fs.existsSync(path.join(DIST, '_redirects')) ? parseRedirects(fs.readFileSync(path.join(DIST, '_redirects'), 'utf8')) : []; +/** A file Pages would have: uploaded, a regular file, not a symbolic link. */ +function pagesFile(rel) { + if (!uploaded(rel)) { return false; } + const p = within(DIST, rel); + try { return !!p && fs.lstatSync(p).isFile(); } catch { return false; } +} const config = webConfig({ account: args.account || process.env.LEVELCODE_ACCOUNT_ORIGIN, @@ -51,7 +63,13 @@ const server = http.createServer((req, res) => { try { pathname = decodeURIComponent(url.pathname); } catch { res.writeHead(400).end('bad path'); return; } let file = null; - if (DIST) { + if (PAGES) { + const r = matchRedirect(REDIRECTS, pathname); + if (r && r.status !== 200) { res.writeHead(r.status, { location: r.to + url.search }).end(); return; } + const page = resolvePage(r ? r.to : pathname, pagesFile); + if (page && 'redirect' in page) { res.writeHead(308, { location: page.redirect + url.search }).end(); return; } + file = path.join(DIST, page ? page.file : 'index.html'); // no such file: the page, as Pages answers it + } else if (DIST) { file = within(DIST, pathname === '/' ? 'index.html' : pathname.slice(1)); if (file && fs.existsSync(file) && fs.statSync(file).isDirectory()) { file = path.join(file, 'index.html'); } } else { diff --git a/web/test/cdp.mjs b/web/test/cdp.mjs index 6792ffd..be302d0 100644 --- a/web/test/cdp.mjs +++ b/web/test/cdp.mjs @@ -62,7 +62,7 @@ export class Page { if (d.params.entry.level === 'error') { this.logs.push({ t: Date.now(), tag, type: 'log-error', text: `${d.params.entry.text} ${d.params.entry.url || ''}`.slice(0, 500) }); } break; case 'Network.responseReceived': - this.requests.push({ url: d.params.response.url, status: d.params.response.status, type: d.params.type }); + this.requests.push({ url: d.params.response.url, status: d.params.response.status, type: d.params.type, mimeType: d.params.response.mimeType }); if (d.params.response.status >= 400) { this.failed.push(`${d.params.response.status} ${d.params.response.url}`); } break; case 'Network.loadingFailed': diff --git a/web/test/e2e-look.mjs b/web/test/e2e-look.mjs index 75d8971..7950004 100644 --- a/web/test/e2e-look.mjs +++ b/web/test/e2e-look.mjs @@ -12,7 +12,7 @@ // The sign-in is the stand-in's (?signin=1), exactly as in e2e-signin-tab.mjs; the chat's empty state is what // a signed-in visitor sees. // -// node web/test/e2e-look.mjs --dist --stub-port [--shots ] +// node web/test/e2e-look.mjs --dist --stub-port [--shots ] [--pages] // // Release only: the look is a property of what the build ships (the theme package, chrome.css, the skinned chat // page), and the development server's pieces do not include them. @@ -39,7 +39,7 @@ const results = []; const check = (name, ok, detail = '') => { results.push(ok); console.log(`${ok ? 'PASS' : 'FAIL'} ${name}${detail ? ' — ' + detail : ''}`); return ok; }; const stub = await startStub({ editorOrigin, port: args['stub-port'] ? Number(args['stub-port']) : 0 }); -const server = spawn(process.execPath, [path.join(HERE, '..', 'serve.mjs'), '--port', String(PORT), '--dist', DIST], { stdio: 'inherit' }); +const server = spawn(process.execPath, [path.join(HERE, '..', 'serve.mjs'), '--port', String(PORT), '--dist', DIST, ...(args.pages ? ['--pages'] : [])], { stdio: 'inherit' }); let chrome; const finish = async (code) => { try { chrome && chrome.close(); } catch {} try { server.kill(); } catch {} try { await stub.close(); } catch {} process.exit(code); }; process.on('SIGINT', () => finish(130)); diff --git a/web/test/e2e-signin-tab.mjs b/web/test/e2e-signin-tab.mjs index 1801f7e..72e2123 100644 --- a/web/test/e2e-signin-tab.mjs +++ b/web/test/e2e-signin-tab.mjs @@ -17,7 +17,7 @@ // and an old one is not. // // node web/test/e2e-signin-tab.mjs --static --extensions [--product ] -// node web/test/e2e-signin-tab.mjs --dist --stub-port +// node web/test/e2e-signin-tab.mjs --dist --stub-port [--pages] import { spawn } from 'node:child_process'; import fs from 'node:fs'; import path from 'node:path'; @@ -43,7 +43,7 @@ const check = (name, ok, detail = '') => { results.push(ok); console.log(`${ok ? const stub = await startStub({ editorOrigin, port: args['stub-port'] ? Number(args['stub-port']) : 0 }); const server = spawn(process.execPath, [path.join(HERE, '..', 'serve.mjs'), '--port', String(PORT), ...(DIST - ? ['--dist', DIST] + ? ['--dist', DIST, ...(args.pages ? ['--pages'] : [])] : ['--static', STATIC, '--extensions', EXTS, '--extension-names', 'levelcode-ai,levelcode-web', '--account', stub.origin, ...(args.product ? ['--product', args.product] : [])])], { stdio: 'inherit' }); // With the extension host on an origin of its own (--product with webEndpointUrlTemplate, or a release built with diff --git a/web/test/e2e.mjs b/web/test/e2e.mjs index 3953534..088ac6d 100644 --- a/web/test/e2e.mjs +++ b/web/test/e2e.mjs @@ -8,6 +8,7 @@ // the real one must. It proves the editor's half of the contract; thin.ly's half is its own specs. // // node web/test/e2e.mjs --static --extensions [--shots ] +// node web/test/e2e.mjs --dist --stub-port [--pages] (--pages: served as Cloudflare Pages serves it) import { spawn } from 'node:child_process'; import fs from 'node:fs'; import path from 'node:path'; @@ -37,7 +38,7 @@ const shot = async (page, name) => { if (SHOTS) { await page.screenshot(path.joi // A release has its account origin baked into index.html, so the stand-in has to be where it was built to look. const stub = await startStub({ editorOrigin, port: args['stub-port'] ? Number(args['stub-port']) : 0 }); const server = spawn(process.execPath, [path.join(HERE, '..', 'serve.mjs'), '--port', String(EDITOR_PORT), ...(DIST - ? ['--dist', DIST] + ? ['--dist', DIST, ...(args.pages ? ['--pages'] : [])] : ['--static', STATIC, '--extensions', EXTS, '--extension-names', args['extension-names'] || 'levelcode-ai,levelcode-web', '--account', stub.origin, ...(args.product ? ['--product', args.product] : [])])], { stdio: 'inherit' }); let chrome; @@ -226,6 +227,18 @@ try { check('after a reload the file that was kept is still there', (await scratchFile('/scratch/notes.md')) === 'notes\n'); await shot(page, '4-reload'); + /* 4b. what the client loads from its own node_modules: the terminal's library (xterm), the syntax colouring + (vscode-textmate, vscode-oniguruma). A host that does not have a file answers with something else — Cloudflare + Pages, which is never sent a folder named node_modules, answers with the page — and the editor runs without it. */ + if (DIST) { + await page.palette('LevelCode Shell: New Terminal'); + const term = await page.waitFor(() => page.eval(`[...document.querySelectorAll('.xterm .xterm-screen')].some((e) => e.getBoundingClientRect().width > 0)`), { ms: 30000, label: 'a terminal' }).catch(() => false); + check('a terminal opens (LevelCode Shell: New Terminal): its library loaded', !!term); + } + const asPage = page.requests.filter((r) => /\.(m?js|json|wasm)(\?|$)/.test(r.url) && /text\/html/.test(r.mimeType || '')); + check('no script, JSON or WebAssembly file came back as an HTML page (a file the host does not have)', asPage.length === 0, + JSON.stringify(asPage.slice(0, 4).map((r) => r.url.replace(/^https?:\/\/[^/]+/, '')))); + /* 5. a folder from "the computer". The OS picker needs a person, so the page's File System Access entry point is replaced by one that hands back a real FileSystemDirectoryHandle (a folder of the origin-private file system); everything after the picker — the command, the workbench's file provider, the agent's reads and writes — is real. */ diff --git a/web/test/unit/headers.test.js b/web/test/unit/headers.test.js index c399cf2..349eaf9 100644 --- a/web/test/unit/headers.test.js +++ b/web/test/unit/headers.test.js @@ -46,5 +46,16 @@ const block = (conf, match) => { assert.strictEqual(headersFor(p, '/')['Content-Security-Policy'], headersFor(p, '/index.html')['Content-Security-Policy']); }); + await test('Cloudflare Pages: the sign-in page keeps its headers at /callback, where Pages serves callback.html', async () => { + const { parseHeadersFile } = await import(pathToFileURL(path.join(__dirname, '..', '..', 'lib', 'headers.mjs')).href); + const read = parseHeadersFile(toHeadersFile(p)); + const at = (address) => (read.rules.find((r) => r.path === address) || { headers: {} }).headers; + assert.deepStrictEqual(at('/callback'), at('/callback.html'), 'the same rule under both addresses'); + assert.match(at('/callback')['Content-Security-Policy'] || '', /^default-src 'none'/); + assert.strictEqual(at('/callback')['Cache-Control'], 'no-store'); + assert.ok(!read.rules.some((r) => r.path === '/index'), '/index.html is served at /, which has its own rule'); + assert.ok(!/location = \/callback \{/.test(conf), 'nginx serves callback.html as named and needs no second address'); + }); + console.log(`\n${n} tests passed`); })().catch((e) => { console.error(e); process.exit(1); }); diff --git a/web/test/unit/pages.test.js b/web/test/unit/pages.test.js new file mode 100644 index 0000000..74efc62 --- /dev/null +++ b/web/test/unit/pages.test.js @@ -0,0 +1,72 @@ +/*--------------------------------------------------------------------------------------------- + * web/lib/pages.mjs — run: node web/test/unit/pages.test.js + * + * How Cloudflare Pages answers for a release, as web/serve.mjs --pages answers: what wrangler uploads, the _redirects + * rules, the .html redirects and the page for anything else. And the release's own answer to the one that broke the + * terminal: every file of the client's node_modules, which Pages is never sent, is served from the copy it is sent. + *--------------------------------------------------------------------------------------------*/ +// @ts-check +'use strict'; + +const assert = require('assert'); +const path = require('path'); +const { pathToFileURL } = require('url'); + +let n = 0; +async function test(name, fn) { await fn(); n++; console.log(' ok - ' + name); } + +(async () => { + const { uploaded, parseRedirects, matchRedirect, resolvePage, toRedirectsFile, NODE_MODULES_COPY } = + await import(pathToFileURL(path.join(__dirname, '..', '..', 'lib', 'pages.mjs')).href); + + await test('what wrangler uploads: nothing in a folder named node_modules or .git, anywhere, and not its own files at the root', async () => { + for (const rel of ['_/abc/static/node_modules/@xterm/xterm/lib/xterm.js', 'node_modules/x.js', '_/abc/extensions/x/node_modules/y.js', + '_/abc/.git/HEAD', '_/abc/static/.DS_Store', '_headers', '_redirects', '_worker.js', '_routes.json', 'functions/api.js']) { + assert.strictEqual(uploaded(rel), false, rel); + } + for (const rel of ['index.html', 'callback.html', '_/abc/static/node-modules/@xterm/xterm/lib/xterm.js', '_/abc/static/out/nls.messages.js', + '_/abc/extensions/levelcode-ai/package.json', '_/abc/functions/x.js', '_/abc/static/node_modules_x/y.js']) { + assert.strictEqual(uploaded(rel), true, rel); + } + }); + + await test('_redirects: rules in order, comments and blank lines skipped, 302 when no status is given', async () => { + const rules = parseRedirects('# a comment\n\n/old /new 301\n/a/:x/* /b/:x/:splat 200\n/plain /there\n'); + assert.deepStrictEqual(rules, [{ from: '/old', to: '/new', status: 301 }, { from: '/a/:x/*', to: '/b/:x/:splat', status: 200 }, { from: '/plain', to: '/there', status: 302 }]); + assert.deepStrictEqual(matchRedirect(rules, '/a/one/two/three.js'), { to: '/b/one/two/three.js', status: 200 }); + assert.strictEqual(matchRedirect(rules, '/a/one'), null, 'a splat needs the slash before it'); + assert.deepStrictEqual(matchRedirect(parseRedirects('/x /first 200\n/x /second 200\n'), '/x'), { to: '/first', status: 200 }, 'the first match wins'); + }); + + await test('the release\'s rule serves every node_modules file from the copy that was uploaded, in this build and in a kept one', async () => { + const rules = parseRedirects(toRedirectsFile()); + for (const id of ['5bfb4df6caa1', 'ae34c91f6d37']) { + for (const file of ['@xterm/xterm/lib/xterm.js', 'vscode-textmate/release/main.js', 'vscode-oniguruma/release/onig.wasm', '@vscode/codicons/dist/codicon.ttf']) { + const asked = `/_/${id}/static/node_modules/${file}`; + const r = matchRedirect(rules, asked); + assert.ok(r && r.status === 200, asked + ' is rewritten, not redirected'); + assert.strictEqual(r.to, `/_/${id}/static/${NODE_MODULES_COPY}/${file}`); + assert.ok(uploaded(r.to.slice(1)), r.to + ' is a path Pages is sent'); + } + } + assert.strictEqual(matchRedirect(rules, '/_/abc/static/out/vs/workbench/workbench.web.main.internal.js'), null, 'nothing else is rewritten'); + assert.strictEqual(matchRedirect(rules, '/_/abc/def/static/node_modules/x.js'), null, 'a build id is one segment'); + }); + + await test('.html addresses: redirected without it; the address without it serves the page', async () => { + const files = new Set(['index.html', 'callback.html', 'callback.js', '_/abc/static/out/x/frame.html', '_/abc/static/out/x/index.html']); + const isFile = (rel) => files.has(rel); + assert.deepStrictEqual(resolvePage('/', isFile), { file: 'index.html' }); + assert.deepStrictEqual(resolvePage('/index.html', isFile), { redirect: '/' }); + assert.deepStrictEqual(resolvePage('/callback.html', isFile), { redirect: '/callback' }); + assert.deepStrictEqual(resolvePage('/callback', isFile), { file: 'callback.html' }); + assert.deepStrictEqual(resolvePage('/callback.js', isFile), { file: 'callback.js' }); + assert.deepStrictEqual(resolvePage('/_/abc/static/out/x/frame.html', isFile), { redirect: '/_/abc/static/out/x/frame' }); + assert.deepStrictEqual(resolvePage('/_/abc/static/out/x/frame', isFile), { file: '_/abc/static/out/x/frame.html' }); + assert.deepStrictEqual(resolvePage('/_/abc/static/out/x/index.html', isFile), { redirect: '/_/abc/static/out/x/' }); + assert.deepStrictEqual(resolvePage('/_/abc/static/out/x/', isFile), { file: '_/abc/static/out/x/index.html' }); + assert.strictEqual(resolvePage('/missing.js', isFile), null, 'no such file: the caller answers with the page'); + }); + + console.log(`\n${n} tests passed`); +})().catch((e) => { console.error(e); process.exit(1); });