Performance of Unittests if you have plenty

[Niyol]

Hello,

First and foremost, I'd like to express my gratitude for your work on this framework.

I have a concern regarding the performance of unit tests, particularly when dealing with a large number of them. Presently, I have 90 unit tests, and the runtime for executing them is approximately 30 minutes.

As I intend to expand the test suite with more tests, it's becoming crucial to enhance the performance. One potential approach I'm considering is the ability to run test classes simultaneously.

Currently, I'm utilizing Xunit with the collection feature to execute all tests(one-by-one). However, I'm wondering if there are any additional optimizations I might have overlooked in this regard. Alternatively, are there any plans to implement features aimed at improving test execution performance?

Thank you for your attention to this matter.

[mark-abrams]

Hi @Niyol , thank you for your feedback. It is always great to see that people are finding benefits to using the framework.

Regarding the topic of performance - a very good point and something that I thought about a while ago but never progressed. Running the tests sequentially (one by one) does result in a long test execution time when there are many tests, although I have noticed that running the tests on a Linux build server takes approximately half the time that it takes to run the tests on a Windows build server. You can see this if you look at the builds for LogicAppUnit in GitHub which includes a set of 50 tests that run on both operating systems.

I can think of two other approaches:

1. Currently the framework creates the Functions runtime instance at the start of each test, then runs the test and then closes down the Functions runtime instance. This ensures that each test runs in a "clean" runtime environment but does create inefficiencies when running lots of tests. For example, 100 tests results in starting 100 runtime instances and closing down 100 runtime instances. One option might be for the testing framework to create a Functions runtime instance at the start of a set of tests in a class, via the ClassInitialize method (using MS Test terminology). This would ensure that tests in different classes are isolated from each other (by running in different Functions runtime instances), but tests in the same class run in the same instance and therefore perform quicker. I'm not sure what performance improvement would be achieved using this approach.
2. Another option is to ensure that running tests in parallel is supported. At the moment I'm not sure this would work, the framework expects only one test to be running at the same time. The framework could be improved to support parallel test execution, although I can see some complexities in this, especially relating to the mock HTTP server and making sure that requests from the workflows are routed to the correct test instance. I'd need to give this some careful thought.

I'll look into this further, I suspect option 1 might be easier to do.

[mark-abrams]

Hi @Niyol , sorry about the delay in investigating this issue.

I've been working on a change to the framework that will provide a new test configuration option to enable the reuse of the Test Runner instance across multiple tests in the same test class, i.e. having the same workflow definition. This would avoid the need to create a new instance of the Function runtime for each test, which consumes am significant proportion of the test execution time. By reusing the instance of the Test Runner, a new Function runtime instance is created once for the first test in the class, and all subsequent tests in the class will reuse the instance and therefore avoid creating new instances of the Function runtime.

I've got some issues with controlling the lifetimes of the internal classes in the testing framework but I'm hopeful these can be fixed. But the test performance improvements are looking very good.