Code Coverage for C++

If you need to test a codebase, one of the first things you should consider is getting code coverage metrics, so that you know what is currently being tested and what isn't. This is one of the first tasks I set myself when I started my new job, three weeks ago.

At first I tried to use gcov, which comes standard with the gcc compiler. The first problem that I ran into with gcov is that it doesn't support shared libraries. So I spent a day or two convincing my Makefiles to build static binaries and working around the resulting link errors. Unfortunately, as soon as I ran the static binaries, they crashed due to memory corruption. This was with gcc 3.3.2. At this point, I looked for other alternatives.

There are a plethora of code coverage tools for Java, but very few for C++.

I couldn't use PurifyPlus because they don't support code coverage analysis for Linux. Also, my previous experience with Purify suggested that they really want to quit supporting UNIX altogether.

On SourceForge I found a project called covtool, but I didn't bother trying it out because its mailing list only had one question, dated October 2002, claiming that it doesn't work. That message never even received a reply.

Then I found Bullseye Coverage, and my search was over. It took me only a few hours to install Bullseye, build the code, run the tests, and get some useful coverage metrics. I had only two problems: one was a bug in our code that gcc hadn't noticed before, and the other was a bug in Bullseye's handling of an obscure aspect of C++ (the instantiation of a temporary object defined in the condition of an if statement when the class of the object has no default constructor). I filed a bug report with Bullseye on Friday evening, and they had fixed the bug by Monday.

The first thing I liked about Bullseye was how easy it was to get it working. It literally only took two or three hours, including the time to rebuild the code.

The next thing that impressed me were the summary reports -- building similar summaries using gprof would have taken me a day or two of Perl hacking.

Once I got used to the summary reports, I came to appreciate Bullseye's focus on condition/decision coverage, instead of just line coverage. It gives you a much better idea how much coverage you've achieved in a qualitative sense, as opposed to the merely quantitative statistics that you get from measuring line coverage.

Another thing that I like about Bullseye is that it can output its results in CSV format. I've already used this capability to write a Perl script that generates hyperlinked and colorized reports, for easier browsing.

Also, because Bullseye uses source-to-source instrumentation, it should be possible to use it in conjunction with ccache.

Posted on March 16, 2004 10:36 AM

More testing articles

Comments

Hi,

besides Bullseye Coverage there is an other very good code coverage tool for C and C++ (also availble for all targets in embedded systems): CTC++ from Testwell in Finland.

Product infos are available from http://www.testwell.fi or http://www.verifysoft.net/en_ctcpp.html

Klaus

Posted by: Klaus Lambertz at May 12, 2004 08:47 AM

The compile process with BullseyeCoverage with my g++2.9.6 seemed endless:-(, any idea?

Posted by: XieWei at February 9, 2006 09:53 PM

Is it free/open source?

Posted by: Zhang sf at September 7, 2006 07:14 AM

Zhang, no, it isn't. It's relatively cheap though, if you're associated with a company.

Posted by: Kim at September 7, 2006 06:31 PM

How about the price? And another question, we want to a Unit Test for our software. And we write some UT cases for it by using CppUnit an auto-testing framework. Now, we want to know, our test cases code coverage for the software(code). Can the BullseyeCoverage or CTC++ do this job? If the tools can be run together with CppUnit case running? We work in GNU Linux. Another question, we, in some possibility, maybe use a c++ compiler for xscale.

Posted by: Zhang sf at September 8, 2006 03:44 AM

Zhang, for more accurate responses to your questions, I encourage you to check out the Bullseye website. Specifically, the purchase, and platform pages.

When I puchased it, Bullseye cost $800 for the first year, and $100/year after that.

It should work with any unit test framework, since it recompiles the product code.

Posted by: Kim at September 8, 2006 03:21 PM

Kim, Thanks a lot.

Posted by: Zhang sf at September 11, 2006 05:58 AM

Any chance you still have the Perl script you referenced in this article lying around? I'm evaluating C++ code coverage systems for my project, and one thing we hoped to accomplish is having our automated build system run coverage tests and produce results accessible from a web browser. I could write the script myself of course, but if a perfectly round wheel already exists I see no need to craft a new one. :)

Thanks!

Posted by: David at November 1, 2007 04:04 PM

My experience with bullseye was extremely positive. we use visual studia and the integration was so simple, even a dummy like me could get it working in minutes. The output is also very easy to read with summaries etc

I attempted to trial the testwell stuff, but after speaking with someone there, I didnt hear anything back from them about trial license etc.

Posted by: roger balakrishnan at May 9, 2008 02:02 PM

My experience with bullseye was extremely positive. we use visual studia and the integration was so simple, even a dummy like me could get it working in minutes. The output is also very easy to read with summaries etc

I attempted to trial the testwell stuff, but after speaking with someone there, I didnt hear anything back from them about trial license etc.

Posted by: roger balakrishnan at May 9, 2008 02:03 PM

Does anyone here have any experience using Coverage Validator by Software Verify: http://www.softwareverify.com/cpp/coverage/index.html

I am trying to explore C++ coverage analysis tools that have no conflicts with Visual Studio.

Bullseye looks promising from the posts...How about Coverage Validator...any comments.

TIA :)

Posted by: Evian at May 15, 2008 07:21 PM

Hi,

Can i re-insatll the evaluation copy of Bulls Eye Software on the same PC,After the completion of evaluation period.

Posted by: Gaurav at November 11, 2008 12:56 AM

how to install bullseye code covereage tool on unix

Posted by: sudeep at January 16, 2009 06:34 AM

Hi ,

Anyone knows how to use coverage Broswer in Bull's Eye code coverage . My code is in Unix box and i can run and see the coverage in UNix box , but anyone knows how can i use Coverage Browser to see the result in windows .

Posted by: Shilpa at February 5, 2009 04:53 AM

Semantic Designs offers a test coverage tool that handles a variety of C++ dialects: ANSI, GNU, MSVC 6 and MS Studio 2005. It also offers similar test coverage tools for other languages, including C#, Java, PHP and COBOL.

http://www.semanticdesigns.com/Products/TestCoverage/CppTestCoverage.html