QA WG/IG F2F Tuesday March 2nd afternoon

QA face-to-face meeting 1-2 march | Dom scribing Tuesday afternoon
a "TestGL lite" proposal by [PC]
Going through the CP and GL trying to see what we can cut out
mar 02 13:24:28 --> karl (Snak@18.29.0.30) has joined #qa
(ref file [PC] sent me)
[PC]: defining the boundaries of the test suite is a pre-requisite of
building this test suite
[LR]: this should be precised in the intro of the new testGL
[PC]: the identity of specifications tested should probably be added to
the TS documentation
... we don't need to say what the TS developer is supposed to care about
[MS]: that's the scope of the TS, so it needs to be defined as for a
sepc
[PC]: this is only a design decision, there is nothing we can say about
it
[LH]: I'm happy if this info is in the TS doc
[DD]: let's keep in mind that the goal of these GL is to make the TS
more useful and more useable
[PC]: "define TS strategy" is only used in the process of creating the
TS
... it doesn't need to be present to assess if the TS is good or not
[LH]: still, it's our business to point that they should scope their
work
... but that doesn't need to be a checkpoints
... it's an implementation detail
(some sort of agreement on this approach)
(in [PC]'s text, "delete" means "doesn't need to appear as CP)
[PC]: Automation doesn't need to be there ; it's obvious that if you
can, you'll do it
... keeping stuff about the reproduceability, etc., since they are
characteristics of a good TS
... (will come back later to this)
... reviewing your TM seems trivial too, so doesn't need to appear at
the high level
... doc and packaging seems central to me
... publish the testing of the TS and mechanism for feedback is
low-level, should go away
... versionning seems central to me
... test results concistency seems important
... encouraging test results, I don't know, but looks like process req.
... out of this comes a list of principles (cf other doc)
(present in the room are the same as in the morning, minus Vivien, plus
SusanLesch)
[PC]: what seems central: "a collection of tests is not a test suite",
... the tests must be reproducible
[MB], [LR]: not sure to agree with that
[PC]: reproducibility is not necessary an automation
... it might be just instructions
... if there is no way to compare the results between 2 runs, you can't
use the TS
[LR]: often, the TS is only a set of files that needs to be rendered
[PC]: fine, then the instructions are simple
[KD]: can the instructions be part of the metadata?
[MB]: does it come from frustration of not being able to use a given TS?
[PC]: it comes from my frustration of seeing W3C TS which are not TS
... my TS come with a 96 pages manual
... the DOM TS comes with a well-defined way to run the test cases
[LH]: you're "how to execute" is instructions of use
[KD]: but should this be in the test case metadata? somewhere else?
[PC]: probably most of it is in the test cases themselves
... (and their metadata)
... some other belong to the TS level
[KD]: we had the comment that we should not impose a way on how to build
a test suite
[PC]: if W3C TS had enough info for me to use them as such, I wouldn't
have to wrap them up in my own harness
[MB]: still, that wouldn't take up the 96 pages you were referring to
[PC]: sure, these 96 pages explains how to use the harness
... if everything is meething that requirement, that's fine
... just dropping some test cases in directory doesn't make up a test
suite
... you need to know how they've been reviewed
... if they've been approved, if there is a way to get feedback
[DD]: but then testing should also be there to ensure its quality
[PC]: yes, but that doesn't need to be part of the normative
requirements
... it's an obvious way to ensure the quality of any product to test it
(and that's also true for a test suite)
[MB]: but how much is enough?
[PC]: enough is enough :)
... we don't define it, it's just goes in the way you assess how good
your test suite is
[MB]: but testing a test suite is easy when you have a reference
implementation
... but when you don't?
[PC]: it doesn't really matter, the point is to have enough that you
feel confident with your TS
[MS]: I guess this points that this may need to answered in TestGL
[PC]: I say it's out of scope
... that's why we dropped it from TestGL
[MS]: but we can still say that they should test their test suite
[PC]: but I don't think this should be normative
[MS]: we should address that later
[PC]: ok; to get back to the high principles, repeatability and
reproducibility seems central
... (testing the TS is a way to ensure that)
... versionning and a feedback loop seems also very important
... what about test results reporting?
[DHM]: I think it's out of scope, since it doesn't allow to judge a
given TS
... ie a TS  state doesn't change by the fact that its results are
reported or not
[PC]: OK, let's drop it from here, and make sure it's part of the QA
handbook
[KD]: is there a deadline to fix a bug?
[PC]: there is no contractual obligation
[KD]: OK, I was thinking about the issue with the RDF TS being a Rec,
and the timeline of publishing errata
[LR]: To get at a higher level, I think the idea is "maintain your test
suite" which would includes versionning and the feedback loop
[PC]: packaging and documentation could get integrated in "repeatibility
and reproducibility"
[MS]: but it also intefers with maintenance
[PC]: so, what do you think about this proposal? is there anything
missing?
[LR]: I like it; I'm not worried about things missing, since I think it
addresses the most important questions
mar 02 14:17:03 --> bernard (bernard@213.41.134.248) has joined #qa
[KD]: do we plan to make it enforceable in the future?
... Jeremy Carroll told me that he was expecting something more
constructive into this QA Framework
[PC]: wrt to test driven development of specification, the fact is that
we're assessing conformance TS
... if you're developing the TS at the same time as the spec, you're not
in conformance testing anymore
... testGL is about the final product
... you may still be able to use some of the CP
[DD]: we must insist on not judging the way it's done, only the result
[MS]: now that we have a clearer view on where we want to go is probably
the right time to ask ourselves what we want to make normative
[PC]: what this tells us is we can create normative requirements on the
Test Suite as a whole
... so it's possible to make TestGL normative
[LH]: the same way, SpecGL could made normative
[MS]: but when doing this, is still possible to check for the "what" and
not the "how"?
[PC]: I think I can define normatively the properties of a good TS
(good=conformaning to some TestGL conformance requriements)
... without addressing the how
... only check the properties
[DHM]: still, there are 2 questions
... - can we do it?
... - should we do it?
... It looks like we can
... but does that mean that we should
[DD]: it seems easier now than before
[LR]: still, it's not because it's possible that we should
[MS]: because it draws attention to it
[LR]: is that really the goal?
[MS]: we've heard that people didn't want to apply our GL for 2 reasons
... - because they were not usable
... - because they didn't care
... we're fixing the 1st point, but still the 2nd
[DHM]: but mandatoriness and normativiy are different issues
[LH]: they're bound, but making something normative doesn't make it
mandatory
[MS]: having normative requirements is critical also to get metrics
... someone should be able to say if he did good or wrong using our docs
[DHM]: there is also a question of priorities; we don't have that many
resources, and publishing normative requirements may spread our
resources further
[LR]: I think we need to have a way to assess if you did wrong or right
... it doesn't need necessarily to be very formal
[KD]: a good way to do this is to have a tool to check
[DHM]: but still, if it's that easy to extract requirements, we can
probably do that next
[PC]: they doesn't need to be incorporated in the same text; this was
what killed the usability in our current version of the framework
[LH]: still, we need to see what our goal is
... are we just defering the answer?
[KD]: the 1st goal is to get something useful
[DHM]: still, we need to get a plan
... when do we decide if we produce these normative requirements?
[DD]: we don't have to dismiss everything we did in the previous drafts
... there are probably things that we can keep from them
... and esp. the way we formulated the conformance requirements
[LH]: [MoS] is not mandatory, but using normative style
[PC]: I have no problem using normative style
[LH]: if we don't have any normative content, I fear we're making them
useless and ignored
[MS]: I agree with Lofton [!!!!!]
[DD]: again, we should not revert to saying nothing because of the
feedback we got
[KD]: agreed
... we can have normative prose as long as we don't scare people away
[DHM]: giving [WebArch] as an example
... lots of good and useful text
... plus some well-identified normative prose (constaints, good
practices)
[LR]: We should still target to explain what should be do, give examples
on how doing it
... and add as a bonus "if you've done it, you're compliant"
[LH]: so, there is consensus that normative content would be a good
thing
... still not clear about when
RESOLVED: we'll produce normative contents
[LH]: what about the plan? is there people thinking we should not trying
to tackle this for the next draft?
[LR]: I think we should first tackle the useful stuff
[MS]: but they are the focal points!
[LH]: I agree [!]
[LR]: but why not just keep what we have today?
[LH]: but that's the way [PC] proceeded this morning in fact
[PC]: indeed; the goal may be to keep the interesting CP we've found
this morning
... and integrate them in a better text
[DHM]: what I like in webarch is that you can read it for the sake of
getting good info about the WWW Architecture
... you can also read it just to assess how you're doing with a
technology you're developing
... that's what I like
[LH]: I like it too
[DHM]: we need to keep separate the aspect of having a conformance
model, how complex it should be, and how we should write it down in
prose
[MS]: what about SHOULD vs MUST in webarch?
[DHM]: SHOULD vs MUST doesn't only make sense in a context of a
conformance section
... which webarch doesn't have anyway
[MS]: still, we need to figure how we want to write our requirements
[LH]: my assumption, is that in simplifying, we're not going to
trivialize the goal
... just that we're trying to focus on the most important things, ie to
keep only on the P1 CP
... also, webarch is relying on a good faith conformance
... I think that could be our first cut goal
[DHM]: we do have to simplify the Conf. Req.
... and take this good faith conformance approach
... at least for the 1st cut, let's not bother about cheating
... and remove the too many details in our Conf Req
(agreement on this)
[LH]: is there agreeemnt on a one conformance level for the next cut?
(yes)
[LH]: and also not bothering about not leaving any wiggle room?
(yes)
(discussion about publishing the results of conformance to our doc,
binary state of conforming vs percentage of conformance)
(discussing about how Ops should be : not normative, a handbook, some CP
migrating to other docs ; e.g. Ops8.5 could move into TestGL)
(going through PC's slides for the plenary day)
[LR]: note that my guest topic session was Mary's presentation
[LH]: so, should we do a lightning talk?
AI KD to propose a CR assessment report by 2004-03-10
LH guest topic can be swapped with LR's (in May)
progress of the docs would probably be WD, before a LC WD
AI LH to check with Ian wrt WG Note vs WD by 2004-03-08
Plan for SpecGL:
- detailed outline by end of next week (2004-03-16), discussion on
Monday 22nd
(idem for TestGL)
(idem for OpsGL)
- a month after that, 1st WD -> Apr 22nd
(TR publication week after that)
this draft would likely be still incomplete
but the 1st structure should be agreed on
Next monday: looking at development schedule, comparing CR issues to our
decisions
Next F2F: AI for KD, OT, PC to look for a possible location for F2F by
2004-03-15
foreseen date: sometime in June

([PC]'s documents discussed above are attached to this mail)
-- 
Dominique Hazaël-Massieux - http://www.w3.org/People/Dom/
W3C/ERCIM
mailto:dom@w3.org