Re: Best practice for RDF test manifests

Thanks Andy,

On 24/03/2021 12:02, Andy Seaborne wrote:
> How about this decision tree?
> AKA "It depends"
>
>
> Has it become a negative test?
>   Mark as negative test. (4)

I am not comfortable with changing a test from positive to negative. 
This means that an EARL report asserting a "PASS" would mean the exact 
opposite depending on the version of the test-suite that you are 
considering...

I have one such test -- I decided instead to withdraw the positive test 
and make a new negative test.

>
> Was the suite ever "published" with this test in it?
>   If not, remove. (1)
>
> Are there any reports that mention the test?
>   If not, remove. (1)
>
> Else
> (2)+(3).

makes sense :)

>
> Removing from the entries list is quite important.
>
>     Andy
>
> On 24/03/2021 10:44, Pierre-Antoine Champin wrote:
>> Hi RDF testers,
>>
>> I am currently updating the test-suite of RDF-star [1], and I am not 
>> sure  about the best way to go to remove a test from the suite.
>>
>> Here are the options that I can think of:
>>
>> 1) Removing the test completely from the manifest file
>> PROS: keeps the manifest clean (and I still have archives in git).
>> CONS: old EARL reports will point to something that does not exist 
>> anymore. Cool URIs don't change (or disappear).
>>
>> 2) Keep the test in the manifest.ttl file, but remove it from the 
>> entry list.
>> PROS: will be easily ignored by people running the test suite, but 
>> still findable from old EARL reports.
>> CONS: clutters the file.
>>
>> 3) Mark the test as Withdrawn (one of the statuses defined by [2]).
>> PROS: semantically clean and "sufficient".
>> CONS: clutters the file and the manifest entry list; forces users to 
>> look at the status when iterating over the entries.
>>
>> For the moment, I went for 2+3, to be on the safe side. Would you 
>> agree with this approach? Is there another option that I didn't think 
>> of?
>>
>> Thanks in advance.
>>
>>    pa
>>
>>
>> [1] https://w3c.github.io/rdf-star/
>>
>> [2] http://www.w3.org/2001/sw/DataAccess/tests/test-dawg#
>>
>>
>>
>