Skip to content

Fix failed Android SauceLabs system tests - #7036

Open
valfirst with Copilot wants to merge 3 commits into
masterfrom
fix/android-saucelabs-failures
Open

valfirst with Copilot wants to merge 3 commits into
masterfrom
fix/android-saucelabs-failures

Conversation

Copilot AI commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Verify Android device uploads by comparing the pulled file bytes with the resource, avoiding unreliable Photos indexing.
  • Verify uploaded files are removed after deletion.
  • Use coordinate-based drags for carousel gesture checks, matching the successful pattern already used in the scenario.

Validation

  • Mobile-app module Checkstyle and unit tests pass.
  • SpotBugs could not run because only JDK 27 is installed; the analysis tool rejects Java 27 class files. PMD reports existing violations elsewhere in the module.

Summary by Gitar

  • Binary comparison:
    • Compares byte[] variables by content in VariableComparator and VariablesSteps
  • New step:
    • Added Then file ... does not exist on device backed by DeviceActions.fileExists
  • Documentation:
    • Documented the file-absence verification step in plugin-mobile-app.adoc

This will update automatically on new commits.

Copilot AI balanced review requested due to automatic review settings November 5, 2026 13:06
@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 2c2d6e82-0052-4467-b951-d03e335de041

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

gitar-bot[bot]

This comment was marked as resolved.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new file-verification steps lack user documentation, and tests do not cover the alternative missing-file error or the accessibility guard that determines whether read failures are propagated.

2 open issues
What changed in this PR

This PR aims to stabilize Android SauceLabs system tests by checking device files directly instead of relying on Photos indexing, and by adjusting carousel gestures.

Changes:

  • Add steps to compare device-file bytes with a resource and verify file absence.
  • Update Android upload and deletion scenarios to use these assertions.
  • Use coordinate-based carousel drags and add unit tests for file verification.
File Description
vividus-tests/​src/​main/​resources/​story/​system/​mobile_app/​MobileAppStepsTests.story Replaces Photos checks with file assertions and adjusts carousel drags.
vividus-plugin-mobile-app/​src/​test/​java/​org/​vividus/​mobileapp/​steps/​DeviceStepsTests.java Tests the new file-verification steps.
vividus-plugin-mobile-app/​src/​test/​java/​org/​vividus/​mobileapp/​action/​DeviceActionsTests.java Tests file existence checks and unexpected-error propagation.
vividus-plugin-mobile-app/​src/​main/​java/​org/​vividus/​mobileapp/​steps/​DeviceSteps.java Adds resource-content comparison and file-absence assertions.
vividus-plugin-mobile-app/​src/​main/​java/​org/​vividus/​mobileapp/​action/​DeviceActions.java Adds file existence detection using file retrieval and missing-file errors.

Review effort: Balanced

@codecov

codecov Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.22222% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 97.84%. Comparing base (bd13a26) to head (065f9c9).
⚠️ Report is 7 commits behind head on master.

Files with missing lines Patch % Lines
...ain/java/org/vividus/steps/VariableComparator.java 92.30% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##             master    #7036   +/-   ##
=========================================
  Coverage     97.83%   97.84%           
- Complexity     7434     7448   +14     
=========================================
  Files          1028     1028           
  Lines         21916    21955   +39     
  Branches       1451     1458    +7     
=========================================
+ Hits          21442    21481   +39     
  Misses          357      357           
  Partials        117      117           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Qodana for JVM

2 new problems were found

Inspection name Severity Problems
Simplifiable conditional expression 🔶 Warning 2

💡 Qodana analysis was run in the pull request mode: only the changed files were checked
☁️ View the detailed Qodana report

Contact Qodana team

Contact us at qodana-support@jetbrains.com

@valfirst
valfirst force-pushed the fix/android-saucelabs-failures branch from 13666e0 to 56238c9 Compare November 5, 2026 14:55

@gitar-bot gitar-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gitar has auto-approved this PR (configure)

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The mixed-type comparison test uses an unstubbed softAssert mock that always returns false, so an incorrect comparison would not fail the test.

1 open issue
2 resolved since last review

Review effort: Balanced

@gitar-bot gitar-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gitar has auto-approved this PR (configure)

@gitar-bot gitar-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gitar has auto-approved this PR (configure)

Copilot AI requested a review from valfirst November 6, 2026 09:34

@gitar-bot gitar-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gitar has auto-approved this PR (configure)

Copilot AI requested a review from valfirst November 6, 2026 13:04
@valfirst
valfirst force-pushed the fix/android-saucelabs-failures branch from a7fcc0e to eeaa081 Compare November 6, 2026 14:03
@valfirst
valfirst marked this pull request as ready for review November 6, 2026 14:03
@valfirst
valfirst requested a review from a team as a code owner November 6, 2026 14:03
@valfirst
valfirst requested review from ikalinin1 and a lite review from Copilot November 6, 2026 14:03

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A moderate active issue remains because NOT_EQUAL_TO bypasses byte-array content comparison and incorrectly relies on identity strings, requiring a fix to comparison and soft-assert behavior.

1 open issue
1 resolved since last review

Review effort: Lite

Comment on lines +34 to +37
if (variable1 instanceof byte[] bytes1 && variable2 instanceof byte[] bytes2)
{
return compareBinary(bytes1, bytes2);
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Binary byte-array comparisons now use content equality for NOT_EQUAL_TO as well, with the corresponding assertion outcome/message and regression tests. Commit: 065f9c931.

Co-authored-by: valfirst <5081226+valfirst@users.noreply.github.com>

@gitar-bot gitar-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gitar has auto-approved this PR (configure)

@gitar-bot

gitar-bot Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
CI failed: Android and iOS SauceLabs system test scenarios failed while creating Appium sessions because Sauce Labs reported infrastructure errors preparing or starting devices; the build failure is a downstream result and the PR changes do not appear to cause it (one finding, unrelated, infrastructure).

Overview

Two unique error templates were found across 28 analyzed logs for commit 065f9c9. Both point to Sauce Labs / device session-creation failures during the mobile system test run, and the analyzer judged them unrelated to the PR changes.

Failures

Sauce Labs session creation failed with infrastructure errors (confidence: high)

  • Type: infrastructure
  • Affected jobs: 128393132859
  • Related to change: no
  • Root cause: ProxyMobileStepsTests scenarios could not start Appium sessions. Sauce Labs returned HTTP 500 with explicit messages such as "Infrastructure Error -- The Sauce VM failed to prepare for this test" and "The Sauce VMs failed to start the browser or device". On the iOS side, WebDriverAgent reported "Device setup is not yet complete". The log also shows repeated Connection refused to localhost:80, but the analyzer did not establish that as the cause of the Sauce session failures.
  • Suggested fix: Retry the job and check Sauce Labs service/device availability and tunnel status. If failures persist, investigate the localhost:80 connection refusals and confirm the proxy or upstream service expected by these tests is running.

Summary

  • Change-related failures: None. The analyzer reported no change-related failures; the PR's mobile-app file-check, binary-comparison, and carousel-gesture changes do not explain the Sauce VM startup errors.
  • Infrastructure/flaky failures: Sauce Labs device/VM startup failures causing session creation to fail for the Android (ProxyMobileStepsTests) and iOS (HealthCheck) system test scenarios. The debugStories Gradle task failed as a downstream result.
  • Recommended action: Retry the workflow and verify Sauce Labs and tunnel availability. If the failures reproduce, investigate the localhost:80 connection refusals separately; no code change in this PR is indicated by the logs.
Code Review ✅ Approved 1 closed / 1 findings

🟡 Medium risk · Binary comparisons now use byte-content equality, and device-file checks infer absence from driver errors; misclassified errors could make system tests report files missing.

Adds byte-level comparison for byte[] variables, a new Then file ... does not exist on device step backed by DeviceActions.fileExists, and coordinate-based carousel gestures for Android SauceLabs tests. Resolved the byte-array comparison issue that dumped multi-MB decimal output into the report. No open issues found.

✅ 1 closed
✅ Performance: Byte-array comparison puts multi-MB decimal dumps into report

📄 vividus/src/main/java/org/vividus/steps/VariableComparator.java:34-37 📄 vividus-tests/src/main/resources/story/system/mobile_app/MobileAppStepsTests.story:246-249
VariableComparator now handles byte[] by turning both arrays into strings with Arrays.toString and passing them to compareValues. In VariablesSteps, compareValues builds the assertion description Checking if "%s" is %s "%s" from both full values, and on failure it publishes a diff of them. The new story step compares file_for_upload_step.png, which is 445,508 bytes. As a decimal list that is about 2 MB per side, so each run writes roughly 4 MB of description into the soft-assert record, the logs and the Allure report. A failing run also has to build and attach a diff of two 2 MB single-line strings. That is unreadable and makes the report very large.

Fix: compare the arrays directly with Arrays.equals (or MessageDigest.isEqual) and record a short description, for example the array lengths or a SHA-256 of each side, instead of the full contents.

Verification

Rules No rules evaluated

Functional validation Not enabled

Set up

Auto-approval Approved

For more verification details, comment gitar display:verbose.

Tip

Comment Gitar fix CI or enable auto-apply: gitar auto-apply:on

Options

Auto-apply is off → Gitar will not commit updates to this branch.
Display: compact → Showing compact verification results.

Comment with these commands to change the behavior for this request:

Auto-apply Compact
gitar auto-apply:on         
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source

Copilot AI requested a review from valfirst November 7, 2026 11:42
protected boolean compareBinary(byte[] bytes1, ComparisonRule condition, byte[] bytes2)
{
boolean equal = Arrays.equals(bytes1, bytes2);
return condition == ComparisonRule.EQUAL_TO ? equal : !equal;
protected boolean compareBinary(byte[] bytes1, ComparisonRule condition, byte[] bytes2)
{
boolean equal = super.compareBinary(bytes1, ComparisonRule.EQUAL_TO, bytes2);
boolean passed = condition == ComparisonRule.EQUAL_TO ? equal : !equal;
@sonarqubecloud

sonarqubecloud Bot commented Nov 7, 2026

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants