[JeremyYao]

Closes #1462

Description

Adds Playwright end-to-end (E2E) tests. They start a real VS Code and use the extension the way a person would, including inside webviews like the Data Editor. The existing tests can't reach inside webviews, so that UI has only been checked by hand, using the checklist in src/tests/README.md.

The new tests are in their own e2e folder, and the existing tests aren't changed. There are 3 tests so far:

• Opening a file in the Data Editor and checking its bytes
• Changing a byte, saving, and checking the file on disk
• Debugging a schema until it pauses on entry

Why Playwright

#1462 asks for vscode-extension-tester, so this PR is a change of direction. Playwright was chosen for these reasons:

• It reaches inside webviews. Most of the manual checklist is Data Editor behaviour, like drag selection, tooltips and scrolling. Playwright can reach into VS Code's nested webview iframes directly. The earlier discussion noted that neither vscode-extension-tester nor WebdriverIO handles webviews well.
• It avoids problems we've already hit. In Exploratory: Create simple integration/end-to-end (e2e) tests using vscode-extension-tester #1290, vscode-extension-tester tests passed and failed inconsistently, hit StaleElementReferenceError, and extest was hard to configure. An earlier UI test effort also needed delays that varied between machines. Playwright waits for elements automatically, so these tests contain no fixed sleeps.
• CI failures are easier to debug. Every test keeps a trace with a timeline, screenshots and page snapshots, and CI uploads them when tests fail. That's how the Java bug below was diagnosed on Linux from a Windows machine.
• It fits this repo. The current vscode-extension-tester needs Node 22+, which should fail yarn install on our Node 20 CI jobs. It also tests only the latest 3 VS Code versions, and treats 1.90, our minimum, as best-effort. Playwright installs on Node 20 and isn't tied to VS Code releases.
• WebdriverIO has less to offer here. The example tried in the discussion failed with a __dirname error, its docs were less thorough, and it would add a second test runner and a third-party service while still using WebDriver.
• Others use this approach. VS Code's own smoke tests launch through Playwright's _electron API, and so does Confluent's VS Code extension.

Downsides

• Playwright's Electron support is officially experimental.
  [Task]: Set up vscode-extension-tester Correctly #1462 asks for vscode-extension-tester, so this PR is a change of direction. Playwright was chosen for these reasons:

• It reaches inside webviews. Most of the manual checklist is Data Editor behaviour, like drag selection, tooltips and scrolling. Playwright can reach into VS Code's nested webview iframes directly. The earlier discussion noted that neither vscode-extension-tester nor WebdriverIO handles webviews well.

• It avoids problems we've already hit. In Exploratory: Create simple integration/end-to-end (e2e) tests using vscode-extension-tester #1290, vscode-extension-tester tests passed and failed inconsistently, hit StaleElementReferenceEro configure. An earlier UI test effort also needed delays that varied between machines. Playwright waits for elements automatically, so these tests s approach.** VS Code's own smoke tests launch through Playwright's _electron` API, and so does Confluent's VS Code extension.

Downsides

• Playwright's Electron support is officially experimental.
• Selectors for VS Code's own UI are hand-written and can break between VS Code releases. They're all in e2e/pages/workbench.ts to limit that.
• vscode-extension-tester has stronger helpers for debug UI.
• vscode-extension-tester wasn't tried again for this PR. This comparison comes from Exploratory: Create simple integration/end-to-end (e2e) tests using vscode-extension-tester #1290, the discussion above, and each tool's current documentation.

#1462 acceptance criteria

Criterion	Status
VS Code Extension Tester is installed	Replaced by Playwright, for the reasons above
A test instance doesn't bleed into the user's VS Code	Met. Tests use a separately downloaded VS Code, temporary user data and extensions folders, and port 9100 for the Data Editor server.
Behavior is consistent	Mostly met. The tests have no fixed sleeps, but the debugger test relies on the 5-second wait under known issues. In oned because of it.

The debugger test found that a valid schema could fail with "Schema compilation failed. Fix the schema and try again." The real error was an UnsupportedClassVersionError, because the CLI was running on a Java version that was too old. There were two causes:

• Daffodil's launcher scripts ignore JAVA_HOME and run the first java on the PATH.
• Vite replaces process.env with {} when bundling, so on Linux and macOS the CLI ran with no PATH and picked up the system Java.

The CLI now runs with the real environment, with the chosen JDK first on the PATH.

Other changes

• Adds the test:e2e, test:e2e:ui, test:e2e:debug and test:e2e:report scripts
• Adds an e2e CI job on Windows, macOS and Linux, which uploads the HTML report when tests fail
• Adds a testId prop to Button, used by the Data Editor's save buttons
• Includes e2e in yarn lint, and excludes its output from the RAT check
• Removes vscode-extension-tester, which was never used
• Removes @types/glob, which described an older glob API and broke the build once vscode-extension-tester was removed

Known issues not fixed here

The debugger start uses a fixed 5-second wait. After starting the debugger, runScript waits 5 seconds and assumes it's ready. A slow start fails with connect ECONNREFUSED, and a fast start waits longer than it needs to. The wait dates back to at least May 2022 (c24daf3).

From src/utils.ts:

if (type.includes('daffodil')) {
  await delay(5000).then(() => {})
} else {
  const wait_port = require('wait-port')
  await wait_port({ host: '127.0.0.1', port: port, output: 'silent' })
}

The debugger always takes the first branch, because runDebugger passes the type 'daffodil' and no port. Switching it to wait_port is untested, and the port check might look like a
client connecting to the debugger.

process.env is empty in the rest of the bundle. getJavaHome can't see JAVA_HOME, and environment variables like OMEGA_EDIT_SERVER_LOG_LEVEL have no effect.

Daffodil jar downloads can be slow or stall. Each debugger test downloads the jars again, because every test gets a new user data folder. The extension sets no timeout on the download. It also only checks that the folder exists, so an interrupted download may leave a partial folder that later runs reuse.

Wiki

• I have determined that no documentation updates are needed for these changes

• I have added the following documentation for these changes

• Added an "End-to-End Tests" section to DEVELOPMENT.md

Review Instructions including Screenshots

You'll need Node 22 or newer, JDK 17 and sbt. Run yarn install first.

Each test opens its own VS Code window, so windows will open and close during a run. Tests use port 9100 for the Data Editor server, so they won'te open.

To explore the tests:

yarn test:e2e:ui             # interactive viewer: pick tests and inspect each step
yarn test:e2e:debug debugger # step through a test in the Playwright Inspector
yarn test:e2e:report         # report from the last run, with a trace for each test

Confirmation Testing

Already tested

• All 3 tests pass locally on Windows, on VS Code 1.144.0 and 1.90.0
• The e2e CI job passes on Windows, macOS and Linux with VS Code stable, and on Linux with VS Code 1.90.0
• A test that fails on purpose leaves a trace in the report
• The interactive viewer was driven with a script: all tests passed, and a passing test showed its actions and page snapshots
• The Inspector opens attached to VS Code. Stepping through a test wasn't tried.

Try it yourself

1. Run the tests on the oldest supported VS Code, which is the default. All 3 should pass.

   yarn test:e2e

2. Run them on the latest VS Code. All 3 should pass again.

   # bash
   DAFFODIL_TEST_VSCODE_VERSION=stable yarn test:e2e
   
   # PowerShell
   $env:DAFFODIL_TEST_VSCODE_VERSION = 'stable'; yarn test:e2e

3. Check that a test can really fail. In e2e/dataEditor.spec.ts, change the expected '54' to '55', then run:

   yarn test:e2e dataEditor
   yarn test:e2e:report

   shows the bytes of the opened file should fail, and its trace should show the first byte as 54. Undo the edit with git checkout -- e2e/dataEditor.spec.ts.

4. Check that the tests stay isolated. Open a Data Editor in your normal VS Code, run yarn test:e2e dataEditor, and check that your editor still works. git status should show no new files afterwards.

The debugger test can fail with connect ECONNREFUSED 127.0.0.1:4711, or stall while downloading the Daffodil jars (see known issues). This happened 4 times in one set of 8 local runs. CI retries failed tests but local runs don't, so re-run it if you see either.

Try the Java fix

This needs Linux, with JDK 17 or newer first on the PATH and an older Java at /usr/bin/java. The bug was found on the ubuntu-22.04 CI runners.

yarn test:e2e debugger                              # passes
git checkout main -- src/daffodilDebugger/utils.ts  # puts the old code back
yarn test:e2e debugger                              # fails
git checkout HEAD -- src/daffodilDebugger/utils.ts  # restores the fix

The second run should fail after about 2 minutes, and its trace should show "Schema compilation failed. Fix the schema and try again."

Regression Testing

Already tested

• No test files under src/tests were changed
• On the fork's CI run for this commit, 32 of the 33 Build, Test, and Package jobs passed
• The one failure (windows-2022, Node 20.19.4, Java 17, VS Code 1.90.0) is an existing flake: the Daffodil Debugger before all hook timed out, on retry. main failed the same way at 360290c (run 40520441064)
• yarn install --frozen-lockfile passes on Node 20.19.4, so the Node 20 CI jobs still install
• License checks and sbt ratCheck pass

Try it yourself

1. Confirm the existing tests are unchanged. This should print nothing:

   git diff --stat main...HEAD -- src/tests

2. Run the existing sbt, Vitest and Mocha tests. All should pass, apart from the flake above if it occurs.

   yarn test

3. Run the repo checks:

   yarn lint
   sbt ratCheck

   On a Windows checkout with core.autocrlf=true, use yarn prettier src e2e -c --end-of-line auto instead of yarn lint.

4. On Node 20, confirm that the dependency changes still install:

   yarn install --frozen-lockfile

5. Check the production code changes by hand. Press F5 to start the extension, as described under "Debugging the Extension" in DEVELOPMENT.md, t

   • Data Editor save buttons (Button, FileMetrics): run Daffodil Debug: Data Editor, open a copy of a file, and change a byte. Save… should reveal Save and Save As, and both should work.
   • Schema errors (validateSchemaWithDaffodilCLI): add a deliberate error to a copy of a schema and run Daffodil Debug: Debug File. It should still show "Schema compilation failed. Fix the schema and try again."

Made with assistance from Claude Code