Re: placeholder test files

Thanks for this.  Note that, in parallel, I added what I consider complete
tests for a few things to the WAT tree yesterday as well.  These might be
massaged to use your updated definitions... I just did the definitions
inline.

My recommendation would be that you try to not make the tests too
sequential...  I will be the first to admit that I don't have all of the
context of the last N years of developing this spec, but I assume if there
are multiple ways to express body (for example) then it is assumed that
each of these is important. If that is the case, then:

   1. Have a separate test for each expression type (each feature)
   2. Make that test as simple as possible
   3. Don't worry about sequentially determining what to test.  You can
   test it all at once.

Such an architecture will make it very easy to identify which features are
supported by multiple implementations.  For example, in the case of body
and looking at your diagram, to test textualBody have an assertion that:

   - requires body
   - requires body be an object
   - require body have a property of value
   - require body NOT have a property of source nor type nor id
   - require the value property be a string (?)
   - then CHECK but do not fail if the optional components are not present
   (no requirements): language, format, processingLanguage, textDirection


That's it.  Have a test like this for EACH form of body.  The instructions
for the test say supply an annotation that is of the correct form for that
type.  An implementation is tested by supplying that. Rinse, Repeat.

If you want to see an example of this, check out
bodiesTargets/3.2.1-bodyValue.test.  That test simply asserts that there is
a bodyValue that is a string, and that there is NOT a body property.

I do not personally think it is critical that each of these tests also
ensure that all the required components of an annotation are present.  On
the other hand, if they are not present the test should not pass.  If
people feel it is critical those are included as well it is easy enough.  I
put up a test that does just that in annotations/3.1-model-musts.test.  The
contents of that could be pulled out into a single .json file and put in
/common, then referenced from each test as the first assertion.


On Wed, Jul 20, 2016 at 8:42 AM, Jacob Jett <jjett2@illinois.edu> wrote:

> Hi Shane,
>
> Tim, Janina, and I uploaded a number of new schema definitions for the
> various body objects yesterday. Our thinking is that the definitions would
> be used to test the body object and showcase exactly what the object is
> with respect to the Annotation vocabulary.
>
> We've also developed what we think is a good logical flow diagram for the
> test of the body object (see attached flow diagram). Although in
> retrospect, we probably want to test the *target* key first since, if
> it's absent then we already know that the annotation isn't valid or
> well-formed. We can reuse the attached flow diagram for target objects
> simply by eliminating the check for the *value* key (i.e., the
> TextualBody case) from the diagram.
>
> Another thing to note about the uploaded definitions--we've adopted a
> standardized and simple scheme for titles and descriptions.
>
> Tim will likely follow up this email later this afternoon with additional
> details. Will likely discuss some of this on Friday's call.
>
> Regards,
>
> Jacob
>
>
>
>
> _____________________________________________________
> Jacob Jett
> Research Assistant
> Center for Informatics Research in Science and Scholarship
> School of Information Science
> University of Illinois at Urbana-Champaign
> 501 E. Daniel Street, MC-493, Champaign, IL 61820-6211 USA
> (217) 244-2164
> jjett2@illinois.edu
>
> On Sun, Jul 17, 2016 at 9:30 AM, Shane McCarron <shane@spec-ops.io> wrote:
>
>> So... it turns out that the prefix "stub-" is reserved in WPT.  I am not
>> sure for what, but it is a special class of test files that can't currently
>> be selected from the UI.  So when I create the empty test files that need
>> populating, I am going to use the prefix "ph-" for placeholder.
>>
>> On a related note, the result of NOTRUN in WPT literally means that a
>> test was created and then nothing was run about it.  I am going to have the
>> ph-* tests do this so they show up in our reporting but.... I am sure that
>> is not what NOTRUN was intended for and we might get some pushback from the
>> powers that be.  Whoever they are.
>>
>> --
>> Shane McCarron
>> Projects Manager, Spec-Ops
>>
>
>


-- 
Shane McCarron
Projects Manager, Spec-Ops