Testing progress

We have had a lot of discussion in the past couple of weeks about how to do
testing.  I feel that we have a clear path forward that will permit
exercising the data model and the protocol.  There are a lot of open
questions, but we are making progress.

Specifically, I have established a "Project" within Spec-Ops around test
development, and made Web Annotation a task of that Project.  This is
mostly internal nonsense, and you all don't really care.

I set up a discussion list within Spec-Ops for the Project that anyone can
join [1].  There is also a wiki [2] where we will be  assembling concepts
as they develop.  I am doing this in Spec-Ops space just because it is
expedient and because this work applies across several projects that we are
or plan to be working on in the next couple of months.

Spec-Ops has also forked the Web Platform Tests (WPT) GitHub repo [3] so
that there is a place for us to put the tests and associated tools.  The
concept here is that there is a familiar environment in which to develop
the tests and tools, and of course to run the tests and capture results.

In a nutshell, we are looking at writing tests using a declarative grammar
that will be familiar to most of you since it looks just like the JSON you
have been writing in your spec.  A tool on the test server will parse the
test files and present information about the tests through the WPT
interface.  Test data will be analyzed (in the user agent or on the server)
and a result recorded.

This general model can be used not just for Web Annotation, but also for
other specifications that rely only JSON message passing (there are lots of
these).  But rest assured, we are not going sit around trying to craft the
perfect general case solution.  That's not how I like to operate.  Gregg
Kellogg (cc'd on this) has a lot of experience with this sort of testing,
and I am confident that we can leverage his knowledge to get something in
place quickly.

More on the call tomorrow.

[1] http://lists.spec-ops.io/listinfo.cgi/testdev-spec-ops.io
[2] https://wiki.spec-ops.io/wiki/AnnotationTesting
[3] https://github.com/Spec-Ops/web-platform-tests

P.S. It is very hard to write tests without implementations to test
against.  We can (and will) mock up some golden data to test the
infrastructure... but it would be great to have access to implementations
sooner than later.  Even buggy ones!

-- 
Shane McCarron
Projects Manager, Spec-Ops