On the eve of Klocwork Hackathon 3, here’s a story on how the results of the last Hackathon went into production…
With the extensive amount of testing performed nightly on Klocwork, the test and verification team was often in the situation of waiting up to 24 hours for the test results of the latest build. As we strive to identify and resolve issues as fast as possible, an alternative method of execution had to be found.
We wanted to find a way to reuse the bulk of our testing infrastructure, as to not “reinvent the wheel”. Also, since our builds are generated via Jenkins, we looked for ways of integrating build and test environments. At our last Hackathon (a regular event where the product team gets together to code up new ideas), we tested a proposal out, a brand-new parallelization infrastructure that could run automated tests in parallel. After a few days of hacking away, we came up with a solution that reduced automated test times by 15% using eight VMs and 27% using four VMs, granting us the winning Hackathon entry! (and beat the team that did the Unreal Engine 4 analysis)
Making it so
What we have implemented now is a series of Jenkins jobs, triggered automatically by the availability of a new Continuous Integration (CI) build, that launch and execute our Linux-based tests across a farm of virtual machines. It re-uses the bulk of our existing test framework, minimizing the amount of refactoring required. Over 5000 tests that used to take 24 hours to execute now complete in under 4 hours. One of the most intriguing aspects of this solution is its scalability: execution time can be further reduced by adding more VMs to the farm. Also, test results are immediately imported into a web-based test repository as soon as their test suite completes, which means the first results can start being reviewed just minutes after execution.
With the success of this project, we’re looking at introducing the following changes over time:
· Adding Windows VMs to the farm, to execute our Windows-based tests as well
· Increased testing coverage: Add a mode of test execution that runs on several varied flavours and versions of operating systems
· Adding more Linux VMs to further reduce execution time