Running unit tests under valgrind

[Donald Allen]

On Mon, May 21, 2012 at 11:10 PM, John Ralls <jralls at ceridwen.us> wrote:
>
> On May 21, 2012, at 3:41 PM, Donald Allen wrote:
>
>> On Mon, May 21, 2012 at 6:22 PM, Phil Longstaff <phil.longstaff at yahoo.ca> wrote:
>>> Yes.  A lot of the leaks are in the unit test code.  But I've found 2 legitimate leaks as well.
>>
>> Do you guys run the whole system under valgrind (John is obviously
>> right about tests on the parts don't necessarily say anything about
>> the whole) prior to release as part of your QA process?
>
> Not routinely. Every so often someone will get motivated and run under valgrind for a while and chase down some of the leaks, but there are too many execution paths and not yet enough tests to be sure of exercising even a significant percentage of the program. We'll get that sorted eventually.

I would argue that waiting until you have what you think is adequate
test coverage is not a good strategy. Releases don't happen that often
and it sounds like running the whole thing under valgrind isn't that
difficult, so the cost of doing it is not high. And it just might turn
up something important, even without comprehensive test coverage. You
could make the gnucash+valgrind package available to some volunteers
whever you see fit during the release cycle (I'd be happy to be one of
them) to exercise the system and report the problems it turns up. So I
think it makes sense from a cost-benefit standpoint not to wait.

I would also argue that the comprehensive test suite, especially on a
volunteer project like this, could be a pipe dream. I've been involved
in projects where people were being paid for their work and all the
right things were said, but not done, about test coverage. It's like
documentation. Programmers like to program, not write tests, not write
documentation. I'm not saying it never happens, but its a lot more
difficult to make it happen than to get reams of code written.

/Don

>
> Regards,
> John Ralls
>