Re-enable integration tests

[martin-helmich]

No description provided.

[martin-helmich]

@mfal As announced, this is something that I had difficulties with. In the current setup (Jest + ts-jest), oclif seems to have issues importing the concrete command files (which just hangs indefinitely, with the tests eventually timing out). I also tried other setups (like using Babel¹), but ran into other problems with those (for example, not processing the TSX files or difficulties with ESM imports).

Any thoughts?

Footnotes

1. https://jestjs.io/docs/getting-started#via-babel ↩

[gandie]

@martin-helmich brought this one back to life ( to make it a draft then 😆 ).
Would love to see such tests running again, as they would greatly increase my confidence in changes. First step was to rebase on top of current master and creating a plan how integration tests might look like in the future ( see plan document i've added ).

tl;dr : Use mockoon ( or something else ) plus already prepared environment variable switch to alter mittwald API base URL. This would allow real HTTP requests to run during tests, no mocking for that needed. By altering and specializing the mockoon schema we could provide HTTP responses matching our test cases ( if an example response does not fit already ).

[gandie]

Getting somewhere:

[run-all] execution complete: 185 commands
[run-all] statistics: successful=100, failed=85, total=185

Failed commands mostly come from missing arguments, the command discovery misses some.
Yet, at least two errors from current error log caught my eye already, will analyze them when fog has lifted.

[gandie]

Well fog has lifted massively. Current baseline:

[run-all] statistics: successful=153, failed=0, waived-skipped=35, total=188
[run-all] failure taxonomy summary:
[run-all]   ARG_MISUSE             count=  0 sample=-
[run-all]   INTERACTIVE_REQUIRED   count=  0 sample=-
[run-all]   RESOURCE_PRECONDITION  count=  0 sample=-
[run-all]   CONTRACT_SHAPE         count=  0 sample=-
[run-all]   COMMAND_BUG            count=  0 sample=-
[run-all]   DEPRECATED_ENDPOINT    count=  0 sample=-
[run-all] waiver summary:
[run-all]   ARG_MISUSE             count=  0 sample=-
[run-all]   INTERACTIVE_REQUIRED   count= 19 sample=app download, app exec, app get, app ssh, app upgrade
[run-all]   RESOURCE_PRECONDITION  count= 13 sample=app dependency versions, app upload, app version-info, container run, database mysql dump
[run-all]   CONTRACT_SHAPE         count=  0 sample=-
[run-all]   COMMAND_BUG            count=  3 sample=app list-upgrade-candidates, app open, app versions
[run-all]   DEPRECATED_ENDPOINT    count=  0 sample=-
[run-all] wrote classification catalog with 35 entries
 PASS  src/test/integration/run-all-commands.test.ts (415.21 s)
  integration: run all commands
    ✓ discovers and executes every command once (415127 ms)

Combination of schema patches ( which are all conserved as artifacts for upstream patch proposals! ) and real improvements ( replacing deprecated api client calls ) allowed to raise numbers up to a point i am quite happy with. Will take a look at the three remaining COMMAND_BUG classifications, which will probably turn out to be some other kind of problem. Tooling is also included to work on such topics, e.g. setting MW_TEST_CATEGORY in env allows to call focused runs on specific issue clusters. For deeper inspection of single commands MW_TEST_COMMAND_ID can be used to filter discovery to a single command. This also allows to inspect the specific request chains for each command in mockoon env, which is lovely for debugging or to set up specific scenarios where desired. "Hopelessly overengineered" is a matching term for this, yet it is also intended as exploration tool for the whole cli command landscape.