Re: Proposal on forming of W3C Test Group (action item A-2002-03-1-3)

Thanks for you comments Lofton, Daniel

I reply to them below (I've copied text where necessary)

On Monday, April 8, 2002, at 06:51 , Lofton Henderson wrote:

> DImitris,
>
> Thanks for the proposal and starting this dialog.  I have a couple of 
> comments in-line, to your and Kirill's comments...
>
> At 04:58 PM 4/4/02 +0200, Dimitris Dimitriadis wrote:
>> Hi Kirill
>> Thanks for your comments. As I noted, this was a primer, it is certain 
>> to change as more comments are made. My replies inline:
>>
>> On Thursday, April 4, 2002, at 04:44 , Kirill Gavrylyuk wrote:
>>> Hi, Dimitris!
>>> Have 2 initial questions/suggestions:
>>> - Why should this be independent group? If you're saying that
>>> ...it is important that there is a group that can take over after the
>>> current QA WG finishes the more process-oriented work.....
>>> Isn't it what the original QA WG was chartered for? Why not just have
>>> the Testing Group as a task force inside QA WG?
>
> I'm a little uncertain about the scope of the activities and 
> deliverables of the proposed group, so maybe defining it more 
> closely -- Kirill's second point below -- should be a prerequisite to 
> deciding about organization.
>
> But what I think I'm reading is that you're proposing that the scope 
> includes the 4th parts of the Frameworks ("Test Materials"), and 
> related QAWG charter deliverables (e.g., a couple of the last bullets 
> at [1]).  If so, here are a couple of observations:
>
> -- I'm not sure about the characterization of "jobs of two groups 
> different in principle".  I view it more as a practical necessity that, 
> with limited resources, we are tackling a big set of problems in a 
> logical order:  Process, Specs, then Test Materials.  It could be that 
> more parallel development is possible, but I don't see that we have the 
> resources.
>
[dd] I could envision there being a distinction between the two groups 
even in the future: one would be more technically oriented, and with 
possible representation in the other WGs, whereas the other would deal 
with issues of conformance claims, branding and the other things we 
haven't dealt with just yet.

> -- I myself view the "Test" as the most fun part, and hope that we (or 
> at least I) will be able to spend more time there, after we have some 
> momentum and maturity on the first parts.
>
[dd] Count me in

>> In order to facilitate information exhcange between other interested 
>> parties (other WGs) I think it's simpler to go directly to a WG 
>> instead of going to a task force set up in a WG (in my experience, at 
>> least).
>>
>> Of course, and especially given that formal aspects are not that 
>> important after all, setting up a task force may be another option to 
>> consider. I lean toward separating the two, though, especially insofar 
>> as enhancing the two group's (overlapping but not identical) goals is 
>> concerned.
>
> On the one hand, at the QA Workshop we did leave the door open to 
> creating more WGs within the QA Activity.  On the other hand, there is 
> a lot of overhead.  A good part of QAWG's energy for its first few 
> months was absorbed in logistics, procedures, methods, tools (e.g., 
> Issues list, doc technologies, etc), Web site, establishing 
> liaisons, ...
>
> For that reason alone, I'd favor an initial "task group" approach, 
> unless it could be demonstrated that having a separate new WG under the 
> QA Activity would, by itself, lead to a big increase in new members.  
> I.e., will lots of people join and be active in a new Test WG, who 
> wouldn't join a "Test Task Group" within QAWG?
>
[dd] That's a very good point that I hadn't thought of at all. I 
indicated in my reply to Kirill that the structure of the group and its 
organizatorial status is not the biggest issue, but I can envision 
another way in which it is easier for a separate test WG to function: 
communication with other WGs. I think it would be simply more efficient 
to have two separate bodies within the QA activity that people could 
report progress to and ask assistance from. Again, I do not stress "WG" 
here, it could be any other kind of constellation.

>
>>> - I'd empathize somewhere in the proposal what are the specific
>>> deliverables of the Testing Group and what is the expected term of
>>> functioning. You're saying "to aid in producing test materials" and 
>>> "to
>>> help in producing specification authoring tools", but I'd like it to
>>> have more finite term and measurable goals.
>> [dd] Great suggestion, thanks. I left it out because I wanted the WG 
>> to consider even the basic draft, but I will incorporate this 
>> information in the next iteration.
>
> I'm interested in these details, also.
>
> Thanks for the effort to start sorting this out.  Whatever the ultimate 
> organizational details, it clearly is the next major piece of the QA's 
> work.  It also relates to a question that I have been thinking about 
> for a while:  with our current QAWG scope and near-future plans, are 
> there omissions that interest people?  I.e., are there topics and 
> subjects that people, including potential new participants, might want 
> to work on?
>
[dd] Test Group, intraWG communications, setting up W3C frameworks for 
both isolated specificaiton testing but also interoperability testing, 
unambigous reporting language (the work that has been done with EARL) 
test packaging (could be an "appendix" of sorts to the actual spec, like 
the validator coudl be viewed as an integral part of the various xxML 
specs.

Daniel's comments:

Like Kyrill, I'd also favor integration in the QA WG, either as a task
force or as part of our regular work.

I understand your points about this group doing different things, more
TS dev and technique oriented, but I think it's OK to have a WG doing
different things, all related: the WAI Web Content Guidelines WG, for
instance, didn't create a new group to work on their techniques, etc,
and on the other hand, there's a cost in creating a WG at W3C, with
resources identified: staff contact, chair training, director's
approval, charter, reports at AC meeting, etc. and those resources are
getting really scarced nowadays.

[dd] The limited resources issue is of course very important. However, 
as things exponentially grow, I think we and the W3C will progressively 
identify the need for a test and design oriented 
group-force-constellation, especially given that we will probably start 
reporting on levels of conformance at some point (or at least reply to 
such claims made).

So I'd favor first: see how much of what you're proposing fit into the
existing QA WG framework deliverables, and if it doesn't. create a
task force within the QA WG to specifically work on that.

[dd] OK, I could, as I promised Kirill, form a more detailed proposal 
and do a quick mapping between (what I think) needs to be done and what 
is already taken into account in QA as it is now.

/Dimitris


> -Lofton.
>
> [1] http://www.w3.org/QA/WG/charter#Scope
>
>
>>> Thanks
>>>
>>> -----Original Message-----
>>> From: Dimitris Dimitriadis [mailto:dimitris@ontologicon.com]
>>> Sent: Thursday, April 04, 2002 5:20 AM
>>> To: www-qa-wg@w3.org
>>> Subject: Proposal on forming of W3C Test Group (action item
>>> A-2002-03-1-3)
>>>
>>>
>>> QA WG,
>>>
>>> Below please find my outline of a proposal for forming a dedicated 
>>> Test
>>> Group within or in parallel to the QA activity. For simplicity's 
>>> sake I
>>> write in in email form, after receiving comments I will circulate a 
>>> more
>>>
>>> usual html draft.
>>>
>>> Forming a QA Test Group
>>>
>>> Rationale: Simplify production of Test Suites (with the required 
>>> minial
>>> set of quality requirements), enhance coordination between WGs and 
>>> allow
>>>
>>> for easier asserting of cross-specification functionality. Also to
>>> increase the practice of the guidelines produced by the QA WG.
>>>
>>> Introduction: Most of the QA-related test activities that have been
>>> produced up until this date in connection to W3C specs have either 
>>> been
>>> inernally produced within the particular specification's WG, or
>>> incorporated in coordination with external parties. This has lead to
>>> quality testing frameworks on the one hand, but in some cases very
>>> different and incosistent, on the other.
>>>
>>> Proposal: According to the work that this WG produces, the W3C and its
>>> WGs shouls produce test suites that are checked against the same
>>> guidelines and checkpoints. This implies that they should share some
>>> basic functionality and design. In order to achieve this and ease the
>>> burden on the WGs that are to produce the Test Suites, I propose that
>>> the W3C form a special Test Group which has the characteristics below:
>>>
>>> 1. Have full and normative knowledge of the various QA-related
>>> frameworks within the W3C, especially with regard to tests and
>>> conformance issues (as opposed to specification authoring, for 
>>> example)
>>> 2. Help WG representatives to produce Test Material (mentioned, but 
>>> not
>>> formalized so far in the QA WG work)
>>> 3. Aid in producing Test Materials to be used for testing
>>> interdependencies between implementations of specifications
>>> 4. Help in producing specification authoring tools that allow easier
>>> generation of test materials (particulalry important given the current
>>> idea of enhancing granularity of schemas used to write W3C
>>> specifications)
>>>
>>> Organization: The Test Group should ideally be another (technically
>>> oriented) WG within the QA activity. The reason it shoudl be a WG is
>>> that it needs a chair, for coordination, W3C staff allocated in order 
>>> to
>>>
>>> make sure the technical architecture is there, as well as W3C member
>>> organizations and/or invited experts.
>>>
>>> Also, it is important that there is a group that can take over after 
>>> the
>>>
>>> current QA WG finishes the more process-oriented work. It is certain
>>> that there will be many issues witht the testing frameworks that will 
>>> be
>>>
>>> produced, and even more certain that various issues will arise in
>>> connection with conformance claims that will be made by various
>>> implementors.
>>>
>>> Concluding, I not that one was of viewing this group's work (but not
>>> exhaustive) is as the more techically oriented parts of the current QA
>>> WG documents: making what we say there actually happen.
>>>
>>> I look forward to the WG's comments.
>>>
>>> Kind regards,
>>>
>>> /Dimitris
>>
>

Received on Monday, 8 April 2002 13:37:42 UTC