Add manage folders E2E tests for mobile and desktop (BrowserStack) by rwood-moz · Pull Request #102 · thunderbird/stormbox

rwood-moz · 2026-08-11T20:56:39Z

What changed?

Add E2E tests for managing folders in WebMail, and update the UI smoke test also.

Why?

Expand the current E2E test suite that runs in BrowserStack on desktop and mobile browsers.

Limitations and Notes

More specifically:

Updated the smoke test to run regardless on if the inbox is empty or not at the test start (bug fix)
Updated the smoke test to verify the settings dialog appears
Updated the light/dark theme test (in the smoke test) to first turn off the 'follow system theme' option so that the switch theme/mode button appears
Updated the smoke test to exercise the Manage Folders dialog controls (basic check only)
Updated the smoke test for other recent app/locator changes
Consolidated the UI smoke test into one test (no need to have separate tests for desktop and mobile)
Added a new Manage Folders E2E test to actually add/modify/delete folders via the Manage Folders dialog
Moving messages in/out of folders will be done in the separate Message Processing test in Add message processing E2E tests to run in BrowserStack #68
Updated to the newest browserstack-node-sdk 1.67.0 I tested this out with these E2E tests in BrowserStack and it worked fine

I worked with Codex AI to make this patch. I reviewed all code changes and tested thoroughly.

Applicable Issues

Fixes #70.

QA Log

I ran the updated E2E tests (UI smoke test and the new folder management test):

On my local machine on Firefox desktop ✅
On my local machine on the Playwright android mobile viewport ✅
On Firefox desktop in BrowserStack ✅
On Chromium desktop in BrowserStack ✅
On Safari desktop in BrowserStack ✅
On Android Chrome on a Google Pixel 10 device in BrowserStack ✅

All tests pass.

rwood-moz · 2026-09-10T20:32:49Z

After the most recent Stormbox updates this new test no longer works. Moving back into WIP.

…e test

rwood-moz · 2026-09-11T19:14:59Z

After the most recent Stormbox updates this new test no longer works. Moving back into WIP.

Ok I've updated the new folder management test and the UI smoke test to work with the latest Webmail changes, and this patch is now ready for review.

Sancus · 2026-09-15T18:18:54Z

+  });
+
+  test('add, rename, move, search, and delete folders', async ({ page }, testInfo) => {
+    let foldersToCleanUp: Array<string> = [];


There is no cleanup path if anything fails. Every folder this test creates is tracked only in this local, and it is only deleted in the last step, so any failure before then leaves E2E-<ts>-* folders on the shared prod account. On CI (retries: 1) the retry runs in a fresh worker, so fNamePrefix (line 13) gets a new timestamp and the first attempt's folders are orphaned permanently. This runs nightly against prod.

Please add a test.afterEach (or try/finally) that opens Manage Folders, searches for the prefix, selects everything it finds and bulk-deletes, and ideally a pre-test sweep for stale E2E- folders from earlier runs, along the lines of sweepOrphanTestMessages in tests/e2e/helpers/.

Sancus · 2026-09-15T18:18:54Z

    });
    await this.assertAccountMenuItemsVisible();
    await expect(this.selectAllMessagesCheckbox).toBeVisible();
+    await expect(this.messageCount).toBeVisible();


This doesn't achieve the "works on an empty inbox" goal. SelectableListHeader.vue:104-109 renders no text in the count <span> when totalCount === 0, and the span has no padding, so it is zero-width and toBeVisible() (which requires a non-empty bounding box) fails.

On current main (already on stage) the count is also hidden whenever the list is narrower than 520 px (HEADER_COUNT_MIN_WIDTH in useMessageListHeader.ts), which includes the Pixel 7 viewport, so the mobile smoke test will start failing on prod as soon as the multi-column release ships.

Assert on the header (.msg-list__header) instead, or use toBeAttached() here.

Sancus · 2026-09-15T18:18:54Z

+      const starButton = page.getByRole('button', {
+        name: `Star folder ${fName}`,
+      });
+
+      // the folder is new so shouldn't be starred yet
+      expect(await starButton.getAttribute('aria-pressed')).toBe('false');
+
+      // star it and verify
+      console.log(`starring folder: ${fName}`);
+      await starButton.click();
+      expect(await starButton.getAttribute('aria-pressed')).toBe('true');


Two problems here:

expect(await starButton.getAttribute('aria-pressed')).toBe('true') right after click() is a non-retrying read. setFolderStarred goes through the DB worker asynchronously, so this can read false and flake. Use await expect(starButton).toHaveAttribute('aria-pressed', 'true') (and the same for the 'false' check).

The locator is name: 'Star folder X', but after starring the label becomes Unstar folder X (FolderManagerDialog.vue:1150-1152). It keeps matching only because non-exact role matching is a case-insensitive substring match. Use [data-folder-star="X"], which the app exposes for this.

Sancus · 2026-09-15T18:18:54Z

+      const firstFolder = page
+        .getByRole('heading', { name: 'Folders', exact: true })
+        .locator('xpath=..')
+        .locator('xpath=following-sibling::div[contains(@class, "folder-node")][1]');
+
+      await expect(firstFolder.locator('.folder-node__name')).toHaveText(fName);


starredUserFolders is sorted by name (FolderTree.vue:100-105), so "first .folder-node after the Folders heading" equals fName only if no other starred folder sorts before E2E-…. A starred folder left behind by a failed run, or one a human starred on this account with a name below E, makes this fail on every run until someone cleans up manually.

Assert membership in the favorites group instead, e.g.

await expect( page.locator('.folder-node[data-tour="folder-favorites"]').filter({ hasText: fName }), ).toBeVisible();

Sancus · 2026-09-15T18:18:54Z

+    this.manageFoldersText = this.manageFoldersDialog.getByText('Drag a folder to move it, or select several to delete them');
+    this.manageFoldersSearchInput = this.manageFoldersDialog.locator('.folder-subs__search-input');
+    this.manageFoldersCloseBtn = this.manageFoldersDialog.getByRole('button', { name: 'Close manage folders' });
+    this.manageFoldersExpandBtn = this.manageFoldersDialog.getByRole('button', { name: 'Expand default folders' });
+    this.manageFoldersAddTopLevelBtn = this.manageFoldersDialog.getByRole('button', { name: 'New folder', exact: true });
+    this.manageFoldersNewFolderDialog = page.getByRole('dialog', { name: 'New folder' });
+    this.manageFoldersNewFolderNameInput = this.manageFoldersNewFolderDialog.getByRole('textbox', { name: 'Name' });
+    this.manageFoldersNewFolderParentSelect = this.manageFoldersNewFolderDialog.locator('select[data-folder-create-parent]');
+    this.manageFoldersNewFolderCreateBtn = this.manageFoldersNewFolderDialog.getByRole('button', { name: 'Create' });
+    this.manageFoldersNewFolderCancelBtn = this.manageFoldersNewFolderDialog.getByRole('button', { name: 'Cancel' });
+    this.manageFoldersNewFolderNameExistsText = this.manageFoldersNewFolderDialog.getByText('A folder with that name already exists here.', { exact: true });
+    this.manageFoldersExpandInboxBtn = this.manageFoldersDialog.getByRole('button', { name: 'Expand inbox' });


Prefer the data-folder-* hooks the app ships for tests over visible copy. FolderManagerDialog.vue / FolderCreateDialog.vue expose data-folder-new, data-folder-create-name, data-folder-create-submit, data-folder-toggle="<name>", data-folder-edit="<name>", data-folder-rename-input, data-folder-save, data-folder-select="<name>", data-folder-star="<name>", data-account-toggle, and tests/e2e/folder-crud.spec.js already drives the dialog with them.

Concrete examples in this block: 'Expand inbox' only matches the real label Expand Inbox because non-exact role matching is case-insensitive ([data-folder-toggle="Inbox"] is unambiguous); the hint sentence and 'Create' are copy that can change without anyone thinking of this suite. Copy changes already broke this test once (your Sep 10 comment); hooks would have survived that.

Sancus · 2026-09-15T18:18:54Z

+      await expect(
+        page.locator('.folder-subs__name').getByText(fName, { exact: true })
+      ).toBeVisible()


The dialog virtualizes its rows (renderedItems / TanStack virtualizer in FolderManagerDialog.vue), so rows below the fold are not in the DOM at all and toBeVisible() fails for them. This works today because the list is short, but it will break as the account accumulates folders (see the cleanup comment). Filtering with the search box before asserting, as the delete step already does, is robust to that; the same applies to the other .folder-subs__name visibility checks in this file.

Sancus · 2026-09-15T18:18:54Z

+
+    // verify default folders (we already expanded the folders list in openManageFoldersDialog)
+    for (const folderName of FOLDER_NAMES_TO_EXERCISE) {
+      await expect(this.page.locator('.folder-subs__name', { hasText: folderName })).toBeVisible();


page.locator('.folder-subs__name', { hasText: folderName }) matches every row containing that text anywhere on the page, including a shared account's Inbox / Sent Items if the test account has shared folders, and toBeVisible() throws a strict-mode violation on multiple matches. Scope it to this.manageFoldersDialog and the own-account section (or at least use getByText(folderName, { exact: true }) and .first()).

Sancus · 2026-09-15T18:18:54Z

+  private async isInboxEmptyTextVisible(timeout: number) {
+    try {
+      await expect(this.inboxEmptyText).toBeVisible({ timeout });
+      return true;
+    } catch {
+      return false;
+    }
+  }


Dead code: isInboxEmptyTextVisible is never called, and inboxEmptyText (declared at line 53, assigned at line 129) is only referenced here. Please drop all three.

Sancus · 2026-09-15T18:18:54Z

+    if (duplicate) {
+      await expect(this.manageFoldersNewFolderNameExistsText).toBeVisible();
+      await this.manageFoldersNewFolderCancelBtn.click({ force: projectName.toLowerCase().includes('android')});
+    } else {


Dead path: every addFolder call in folder-management.spec.ts passes duplicate = false, so this branch and manageFoldersNewFolderNameExistsText are never exercised. Either add a step that creates a duplicate name (the app has a specific error for it, worth one assertion) or remove the flag and the locator.

rwood-moz force-pushed the e2e/folders branch 6 times, most recently from 5033e40 to f24af0c Compare August 19, 2026 14:16

rwood-moz force-pushed the e2e/folders branch 4 times, most recently from 5ad5cf0 to a97ca84 Compare August 26, 2026 20:46

rwood-moz force-pushed the e2e/folders branch from a97ca84 to 28246f1 Compare August 27, 2026 14:37

rwood-moz changed the title ~~WIP: Add manage folders E2E tests for mobile and desktop (BrowserStack)~~ Add manage folders E2E tests for mobile and desktop (BrowserStack) Aug 27, 2026

rwood-moz marked this pull request as ready for review August 27, 2026 14:52

rwood-moz requested a review from Sancus August 27, 2026 14:52

rwood-moz added 3 commits September 10, 2026 16:13

Update smoke test so it will work regardless on if inbox is empty or not

f7c7ce3

Update smoke test to exercise manage folders dialog

f61b98a

Consolidate ui-smoke test into one test for both desktop and mobile

57f2039

rwood-moz force-pushed the e2e/folders branch 2 times, most recently from 3e950e4 to 9e36820 Compare September 10, 2026 20:32

rwood-moz changed the title ~~Add manage folders E2E tests for mobile and desktop (BrowserStack)~~ WIP: Add manage folders E2E tests for mobile and desktop (BrowserStack) Sep 10, 2026

rwood-moz removed the request for review from Sancus September 10, 2026 20:33

rwood-moz marked this pull request as draft September 10, 2026 20:33

rwood-moz force-pushed the e2e/folders branch from 9e36820 to 1c262a7 Compare September 11, 2026 17:10

Add folder management test for desktop and mobile, and update UI smok…

45c8cad

…e test

rwood-moz force-pushed the e2e/folders branch from 1c262a7 to 45c8cad Compare September 11, 2026 18:34

rwood-moz changed the title ~~WIP: Add manage folders E2E tests for mobile and desktop (BrowserStack)~~ Add manage folders E2E tests for mobile and desktop (BrowserStack) Sep 11, 2026

rwood-moz marked this pull request as ready for review September 11, 2026 19:14

rwood-moz requested a review from Sancus September 11, 2026 19:15

rwood-moz requested review from Sancus and removed request for Sancus September 14, 2026 13:42

Sancus reviewed Sep 15, 2026

View reviewed changes

Update folder picker locator

d339cbb

Navigation Menu

Add manage folders E2E tests for mobile and desktop (BrowserStack) - #102

Add manage folders E2E tests for mobile and desktop (BrowserStack)#102
rwood-moz wants to merge 5 commits into
mainfrom
e2e/folders

rwood-moz commented Aug 11, 2026 •

edited

Loading

Uh oh!

rwood-moz commented Sep 10, 2026

Uh oh!

rwood-moz commented Sep 11, 2026

Uh oh!

Uh oh!

Sancus Sep 15, 2026

Uh oh!

Sancus Sep 15, 2026

Uh oh!

Sancus Sep 15, 2026

Uh oh!

Sancus Sep 15, 2026

Uh oh!

Sancus Sep 15, 2026

Uh oh!

Sancus Sep 15, 2026

Uh oh!

Sancus Sep 15, 2026

Uh oh!

Sancus Sep 15, 2026

Uh oh!

Sancus Sep 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

rwood-moz commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed?

Why?

Limitations and Notes

Applicable Issues

QA Log

Uh oh!

rwood-moz commented Sep 10, 2026

Uh oh!

rwood-moz commented Sep 11, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rwood-moz commented Aug 11, 2026 •

edited

Loading