Re: WebTV Help for Getting Engaged in W3C Test Effort

Bryan,
thanks for starting this.

Robin,Tobie, testing folks,
during the workshop that Bryan mentioned, we discussed testing of we specs,
and various organizations and company, while agreeing that having a common
pool of test cases would be ideal, wanted to first get a better
understanding of the of the W3C testing effort, to make sure this meets the
requirements of those organizations/companies and that any effort they
would put into contributing to it wouldn't be wasted.


In no particular order, here a set of questions I've heard from various
people+some comments from me. Can you help address them? I would like to
invite other IG participants to chime in if I forgot something:

1. the first question was about "where to find information on the W3C
testing setup and material". Bryan tried to answer with the mail below. In
short it seems to me that the starting point is
http://testthewebforward.org/docs/. Please chime in if anything needs to be
added

2. TTWF website points to this post (
http://testthewebforward.org/blog/2013/02/20/testing-the-open-web-platform.html)
under "Want to learn more about our plans?". Is that post still 100%
accurate? If not, would be good to get an update post/page about what TTWF
is and what the plans around testing are.

3. One other question raised was about process: how you submit tests, how
do their get reviewed approved etc. AFAIKS this is answered here:
http://testthewebforward.org/docs/review-process.html. One thing that is
not clear though is timing information and who does what. If an
organization/company was to submit 100s of test, who would be reviewing
them? Is there any guarantee those would be reviewed at all? Or would your
recommend to whoever intends to contribute tests to also contribute reviews?

4. Let's assume some organizations/companies decide the contribute to the
W3C effort. What are the plans when it comes to maintaining the test that
gets submitted? Are there processes in place to make sure that if spec
changes, tests are "invalidated"? In other words, how can I know, at any
given time, if a test suite for a given spec is still valid? And who is in
charge to check that tests are still valid when a spec gets updated? Also,
are there way to "challenge" a test, i.e. to say that a given (approved)
test is in fact invalid?

5. IIRC not all WGs are using the process/tools from TTWF. Is this
documented somewhere? Will these other groups continue with their tool for
the time being or is there any plan to merge the various efforts at some
point?

6. Do the test include metadata that easily allow to (at the very list)
extract relevant test for a given spec? Are these
mandatory/checked/maintained?

7. Some people expressed an interest to organize a TTWF event (or something
similar) dedicated for TV, to better discuss and understand the needs of
the TV industry. Do you think this would be doable? Is there already a
calendar or next events and/or open dates? How would you recommend we go
about this? (and do you think it would be useful in the first place?

/g


On Wed, Apr 16, 2014 at 4:23 PM, SULLIVAN, BRYAN L <bs3131@att.com> wrote:

>  Hi all,
>
>
>
> In the 4th Web & TV Workshop, one of the action items that I said I would
> take was to help disseminate useful information to WebTV members that want
> to get engaged in the testing effort at W3C, as a first step toward closing
> the gaps in WebTV-usecase test assets and methodology noted in the
> workshop. This is addressing a specific need of the WebTV IG, but should be
> useful for any community of interest (e.g. WebMob) or new contributors at
> large.
>
>
>
> This information will be provided on the test-infra list (
> http://lists.w3.org/Archives/Public/public-test-infra/), and for more
> convenient/long-term use on one or more of:
>
> ·         the Testing Wiki (https://www.w3.org/wiki/Testing) – note the
> “not maintained” banner for now, but I expect that it will get maintained
> soon as part of this effort, unless the places below take precendence
>
> ·         the TTWF site (http://testthewebforward.org/) if/once I figure
> out how we can add communities of interest (e.g. WebTV, Mobile, etc) to
> TTWF. TTWF should be the preferred public home for such info I think.
>
> ·         The W3C github repository (http://testthewebforward.org/) in
> particular the readme.md (
> https://github.com/w3c/web-platform-tests/blob/master/README.md) or some
> other place on the github site, for more detailed help in getting engaged.
>
>
>
> The point of putting this info on one of these more permanent/usable
> places is to prevent people who want to get engaged, from having to
> rediscover this info each time (similar to forum FAQ/sticky posts) and bug
> the list etc with newbie questions. We can also add a “who to ask” list if
> questions aren’t answered there and direct people to the test-infra list as
> a backup.
>
>
>
> Our intent to do this is not to put a management layer on the testing
> effort, rather to make it easier for people to get engaged and to organize
> as a community of interest around closing the gaps that matter to them.
> This may include reviewing existing tests, adding new ones, building
> metadata e.g. test runner scripts that focus test runs on specific features
> or link tests to specific features/assertions where that info is missing
> from tests, sharing test results, etc.
>
>
>
> Any other info that people have on “how to get engaged” info and where
> that is documented, is requested as a response to this post.
>
>
>
> Thanks,
>
> Bryan Sullivan | Service Standards | AT&T
>
>
>