update on Exit Criteria, was Re: ORG futures

On 04/13/2013 03:52 AM, Dave Reynolds wrote:
> As noted noted on Friday I find the agreed approach to CR exit 
> confusing, complex and unconvincing. It's just the best we could find.
>

I just spent a long time in a W3C staff meeting trying to get clarified 
guidance.   I regret not doing this before the F2F, although I suppose 
at that point I wouldn't have been able to push for clarification nearly 
as effectively.  Today I was able to give very specific examples of 
concerns.

Anyway, the guidance is very good news for getting to REC, setting a bar 
lower than I though it would be set.

1.  To exit CR, it's enough to show that multiple projects are using the 
vocabulary (in some way they want to use it, eg publishing data) and to 
have some methodology to verify that it's being used in a manner 
consistent with the spec.

2.  That methodology must be enough for the WG to satisfy itself that 
the project has used the vocabulary properly.   If the project involves 
one of the editors or active WG members, that may be assurance enough.  
In other cases, the WG may want someone in the WG to take a look at 
it.   The key is that the WG must be be satisfied about how the project 
is using the spec.

3.  It is not necessary that every vocabulary term be used.  If there 
are terms that clearly make sense but just don't happen to have been 
necessary to describe the situations the projects were trying to 
describe, that's not a reason to stop the vocabulary from proceeding.

I suggest to the chairs this is new information, and we should re-open 
the question of exit criteria, setting a somewhat lower bar.

I actually like Richard's plan -- but I think we should only do it if 
someone's really comfortable doing all that work in the timeframe we have.

(more below)

> For Data Cube we will take a different approach because we have strong 
> notions of well-formedness there (this was in IRC but I want to make 
> sure it is clear).
>
> So the question I've been thinking about is what to do about ORG. I 
> see a few of options so far:
>
> 1. Just do it
>
> Go ahead with the agreed approach. Put the time into trying to create 
> a sufficiently good set of probe queries and evaluation checklist, and 
> find a way to explain them clearly. It will take non-trivial work to 
> put this together in a form that would be understandable and 
> convincing at a transition meeting and would be understandable to 
> implementers.
> It might be doable, I've not yet ruled this one out.
>
> 2. Profiles
>
> As part of the exit criteria we define a few profiles of ORG (e.g. 
> core, memberships, history). We put everything not in the core profile 
> at risk. We define a set of conformance tests for the profiles which 
> include genuine tests[1] and probe queries (no separate checklists). 
> Our criteria are that each profile should have two implementations 
> that pass the tests and pass inspection-by-probe [2]. Oh, and 
> presumably still have each term in the profile have two examples of 
> usage.
>
> This approach would give us a structured way to exit without complete 
> implementations. By having constrained profiles we might be able to do 
> stronger tests which means that we could rely on the tests more. It's 
> more work than #1 but at least the work might feel useful.
>
> 3. Publish as a Note
>
> Vocabularies are not software and they need not be schemas. In 
> semantic web open world style they are modular things you can pick and 
> choose bundles of terms from. ORG provides a useful bundle of terms 
> for organizations which demonstrably fill in gaps and people are 
> finding useful. However, we are not yet ready to define schema-like 
> interoperability criteria for it. So stop battering out heads against 
> the REC track brick wall and simply walk through that tempting "make 
> it a Note" door.
>
>
> So some clarifying questions:
>
> (a) What's the constraints on marking things "At Risk" at CR stage. 
> Can we mark things "At Risk" which weren't marked that way at Last Call?
>

A very fine point, on which I also asked for clarification.   In fact, 
yes, you technically can add "At Risk" flags when going to CR.  The 
assumption is that folks who reviewed it during LC will at least notice 
those "At Risk" flags in the CR document, and can send in their comments 
and object then, if necessary.

(but hopefully we wont need to do this)

> (b) Would it be OK to have specific bundles of features in the CR 
> criteria?  We defined a notion of profiles at Last Call but obviously 
> didn't list any profile sub-sets of ORG itself.
>

I don't really understand this question.    Hopefully it's obsolete.

> (c) How problematic would it be if we moved ORG to a Note?
>

My impression is that there is a community of potential adopters that 
would appreciate having some vocabularies (like org) having this kind of 
stamp of approval.  I have no first-hand data on this, though.   We 
could pointedly ask everyone in the WG if they have any evidence of a 
need for these vocabs to be RECs.

Another issue here: as Phil mentioned, we're working hard to come up 
with an alternative to the REC-track for vocabularies -- a more 
scalable, fast, and cheap way produce vocabularies which are high 
quality and stable.    I'm fairly confident we'll succeed, so then it 
wont matter much whether org is a REC or a NOTE.   But still this effort 
might fail, and then we might wish we'd gotten a REC while we had the 
chance.

         -- Sandro




> Dave
>
> [1] These are implicitly going to making stronger closed-world 
> assumptions than the vocabulary strictly requires. This is in the 
> interests of jumping the "can test interoperability hurdle", not an 
> actual constraint on how people in general can use ORG.
>
> [2] Might want present the probe queries as a test consuming 
> application, that would be easier to explain.
>
>

Received on Tuesday, 16 April 2013 20:36:28 UTC