The adoption of a continuous integration (CI) process is quickly becoming a major initiative of many software development teams, including high-performance computing (HPC). CI coupled with continuous delivery (CD) forms the basis of DevOps and aims to improve the quality and performance of applications, deliver applications faster, improve end-customer experience, and improve collaboration between development and operations teams.
One major aspect of the CI process is the automation of testing and reporting failures. Our recent blog post, How to integrate memory leak detection into continuous integration, introduced how to incorporate the detection of memory leaks and illegal memory operations as part of the CI process to find these issues earlier in the development process. Too often, looking for memory leaks occurs late in the development cycle or worse, they’re reported by a customer.
This blog explains how to automatically detect memory leaks and integrate the process into the Jenkins CI system. If you are using TeamCity, GitLab CI, or another CI system, refer to the original blog post for links to other supported CI systems.
Jenkins and JUnit – An open source CI solution
There are several CI systems available for development teams, each with their own pros and cons. One of the popular ones is Jenkins. Jenkins is an open source project providing a self-contained automation server, which is used for automating different tasks including building a software application using a CI process. Many plugins are available for Jenkins, extending its capabilities and functionality. One plugin for automated testing is JUnit. The JUnit plugin provides capabilities to consume XML test reports that are generated as part of building the product. The XML reports contain information about the tests run during a Jenkins job. JUnit consumes the XML test information and provides graphical visualizations, using the JUnit graph plugin, of the historical test results.
To incorporate the leak detection results from the CodeDynamics HPC debugging platform into Jenkins, we’ll leverage the capabilities of JUnit by transforming leak detection log information into JUnit XML consumable files.
Adding CodeDynamics leak detection testing to Jenkins
There are a few simple steps to add leak detection to the Jenkins CI process.
Install the JUnit Plugin for Jenkins
Typically, the JUnit plugin is installed as part of the Jenkins installation. To double check, select Manage Jenkins from the dashboard, then select Manage Plugins. Click on the Installed tab and type “junit” in the filter box. JUnit Plugin should show up in the list. If it does not, click on the Available tab, search for “junit” again and install the plugin.
Creating a job to find leaks in Jenkins
With Jenkins and JUnit prepared, its time to test for memory leaks, assuming that a prior Jenkins job built our target application.
From your Jenkins dashboard, click on New Item.
For this session, we will create a Freestyle project to check the leaks on our application. Enter an item name for the project, click on Freestyle project, and then click the OK button.
Jenkins creates our new Freestyle project and presents a page with various options. For this example. we’re only concerned with the Build section. Click on the Build tab at the top of the page and then select Execute shell from the Add build step menu.
Jenkins will present the following form for providing a command to execute within a shell. It’s here that we are going to leak detect our application by running it under the control of CodeDynamics’s memory script interface, memscript. Enter the following command into the Command field for Execute shell.
/opt/toolworks/totalview/bin/memscript \ -event_action "termination_notification=>list_leaks" \ tx_local_leak
Note: The path to memscript may need to be adjusted depending on where CodeDynamics was installed.
In this scenario, we are running the target application under the control of memscript directly within Jenkins. This same technique can be applied to your test harness and the test harness would be run by Jenkins.
Our job is now configured to run our test application under memscript and detect leaks just before it exits. When Jenkins runs our job, memscript will generate a human-readable log file with textual information about any leaks found in the target application. The final step will be to extract the leak detection information from the log file and prepare it for Jenkins through the JUnit plugin.
Converting memscript .log files into JUnit XML files
Currently, memscript generates text log files that contains human-readable information about detected leaks, memory errors, and detailed information about all the memory issues. In order to bring this information into Jenkins, we are going to leverage JUnit’s capabilities to read test result information from XML files.
Using a handy Python script, available for download, we are going to convert memscript log file information into JUnit’s XML file format. When run, the script takes the name of the JUnit XML file to produce and one or more of the memscript generated log files to convert.
python memscript2junit.py memscript.xml *.log
For a single log file, the script would produce an XML file something like the following:
<?xml version="1.0" encoding="UTF-8" ?> <testsuites id="20140612_170519" name="Memory Tests" tests="1" failures="1"> <testsuite id="Memory Tests" name="Memory Tests" tests="1" failures="1"> <testcase id="tx_local_leak" name="tx_local_leak"> <failure message="tx_local_leak has 9 leaks totaling 450 bytes of leaked memory" type="MemoryLeak"> <![CDATA[ … Details from the memscript log file … ]]> </failure> </testcase> </testsuite> </testsuites>
To perform this conversion within Jenkins, simply add another Execute shell build step to the project and enter the memscript2junit.py command. I’ve placed my memscript2junit.py script in the Jenkins workspace directory so it is easy to locate. You may need to adjust paths based on where your memscript2junit.py script is located.
Our project is now ready to run, detect leaks in our application, and convert the memscript-generated log files into JUnit consumable XML files. The final step before we run our project is to tell Jenkins where to fetch the JUnit XML files and publish a JUnit test results report.
Publishing JUnit test results report with leak detection information
In order for JUnit to generate test reports, it needs to be told where to gather up the XML files with test result data. To do this, click on Add post-build action at the bottom of our project configuration and then select Publish JUnit test result report. As an aside, the XML files generated by memscript2junit.py are also compatible with the xUnit test reporting plugin as well.
The following form is displayed. This form allows you to specify where to pick up JUnit XML files. We can simply enter memscript.xml in the Test report XMLs field since we are generating the memscript.xml files into the Jenkins workspace area.
Finally, click Save to save our project.
Generating leak detection results in Jenkins
With our project configured and saved, all we have to do run it is click Build Now from the project dashboard.
Jenkins will run our project and show the most recent build in the Build History on the dashboard.
The build is marked as yellow, indicating there is a problem. Let’s explore the leak detection results.
Exploring leak detection results in Jenkins
Clicking on the build #1 link in the Build History section brings up a summary of the status.
There is one test failure in our tx_local_leak test application. Clicking on the link gives us details about the failure.
The Error Message reports that the test failed due to 9 leaks totaling 450 bytes of leaked memory. The “Stacktrace” area provides the full contents of the log file generated by memscript, including details about all the leaked memory.
The development team now can easily check their applications for memory leaks and have them automatically be reported in Jenkins as failed test results!
Incorporating CodeDynamics leak detection capabilities into your continuous integration environment provides many benefits and allows your team to check for memory leaks in your application and report them through Jenkins. The memscript2junit.py script makes it easy to convert memscript log files into XML files that are easily incorporated into the Jenkins CI process. The XML files are also compatible with the xUnit XML file format, allowing them to be easily incorporated into xUnit testing results too.
Although this blog focused on integration with Jenkins, integration with other CI systems that understand JUnit or xUnit XML files would follow a similar approach.
For more information, other CI solutions available from Rogue Wave Software, including incorporating static code analysis into your CI system using our Klocwork product, visit roguewave.com.