Re: Requirements for (level >=3) tests

* Boris Zbarsky wrote:
>Assume that you have a test case that runs a bunch of code.  The author 
>claims it exposes a remote code execution vulnerability, but you can't 
>reproduce any memory corruption or crashes or anything like that.

Then the test case does not expose a problem. My point is that it is
much harder to come up with test cases that expose problems than it is
to deconstruct them. My point is that the purpose of the test suite is
to expose problems, and not to make it easy to reason about the test
cases or to teach people how to write good style sheets or to fix the
problem a test might be exposing. I am saying that test suites are an
information gathering utility, not a learning resource.

>_That_ is closer to the situation I'm talking about, where you can't 
>even tell whether the testcase is exposing a bug or not.

I agree this happens and I agree that this is bad, but I want people to
submit test cases that expose problems, even if they do not use the best
coding standards, even if the test cases are difficult to analyze. I'd
want people to tell me new things, and care much less about them making
it easy to digest those things.
-- 
Björn Höhrmann · mailto:bjoern@hoehrmann.de · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

Received on Wednesday, 22 February 2012 23:22:17 UTC