This is a summary of my experiences in parallelizing JUnit functional tests that speeds up test execution time (by at least 30% for us), in a Jenkins CI pipeline.
Traditional testing methodology
We have traditionally run our JUnit functional tests through a centralized test infrastructure shared by all the development teams. A single Jenkins server manages all the Windows and Linux slaves. A test execution request results in Jenkins running a single Test Plan (a child class of JUnit TestCase) on a slave.
We needed to move to a model where teams are able to manage their own Jenkins infrastructure, and build, deploy and test jobs independently. This model requires that there be a way to run tests in parallel on each independent team Jenkins infrastructure.
Problems running tests in parallel
Running JUnit tests in parallel threads was tricky and did not work for us because our test cases and test framework had static variables all over the place. The tests and the framework were assuming the static variables were not initialized and would initialize them when running the tests. When we tried to run the tests in parallel, the static variables got initialized by the first test which interfered with the flow of subsequent tests running in a different thread.
Below are some known solutions to parallelize JUnit test execution that we explored.
Experimental ParallelComputer class
We investigated using the experimental ParallelComputer class provided in JUnit 4.12. A few examples are shown here. We found the following issues and risks with this approach:
- ParallelComputer is an experimental class and at the time of writing this, there is no ParallelComputer implementation in later versions of JUnit.
- The ParallelComputer class does not provide a solution for static variables that are used in frameworks and classes
- Rewriting all the test and framework classes to eliminate or reduce the use of static variables was too big of a task to undertake.
Use ClassLoader to Implement Parallelization
Using ClassLoader to implement parallelization:
- runs each test in its own thread
- each thread has its own ClassLoader that loads the test class and its dependent classes in its context
Having a different ClassLoader for thread provides a different name space for the shared static variables. This, combined with running each test in its own thread, lets you parallelize running JUnit tests.
The code example is here.
Here’s how it works:
- All JUnit tests are run in different threads.
- The number of tests to run in parallel is configured through command line parameter “maxThreads”
- The runner reads a file containing the list of the test cases to run with their full class names
- The JUnitParallelRunner uses a custom class loader (TestClassLoader) to load the legacy Junit test classes.
- Each thread does the following:
- Creates a new class loader and loads the test class.
- Calls the JunitCore to run the tests.
Using this approach introduces another problem – How do you separate the console output (i.e output from the System.out.println) for each test/thread? For this I have implemented a PrintStream that captures the console output for each individual threads in its own file. You can find my PrintStream implementation here. The PrintStream maintains a mapping of the Thread name and the corresponding file that it saves the console output to.
We have been testing this solution for couple of months with over two thousand tests with multiple teams.
Not only do the tests run green, but there is also a time saving that corresponds with the number of test cases. We have seen a savings of at least 30% in test execution time.
I hope this approach helps your team with parallelizing JUnit functional tests and speeding up the execution. Please share your experiences, suggestions, and comments below.