RE: Test suite for the Presentation API (was: Test Facilitator for the Presentation API)

Hi Anssi, Louay,

> From: Kostiainen, Anssi
> Sent: Wednesday, December 16, 2015 7:33 PM
> To: Bassbouss, Louay <louay.bassbouss@fokus.fraunhofer.de>; Zhang,
> Zhiqiang <zhiqiang.zhang@intel.com>
> Cc: public-secondscreen@w3.org
> Subject: Test suite for the Presentation API (was: Test Facilitator for the
> Presentation API)
> 
> Hi All, Louay, Zhiqiang,
> 
> > On 15 Dec 2015, at 19:09, Bassbouss, Louay
> <louay.bassbouss@fokus.fraunhofer.de> wrote:
> >
> > I added a new Testing section [1] in the wiki that describes how to setup
> the test environment and how to write tests for the Presentation API. As you
> know, we will use the Web Platform Tests Project [2] as a Test Suite for the
> Presentation API. I already submitted a new PR [3] with initial folder
> structure and IDLharness for the Presentation API. once this PR is merged in
> the main repository we will have an additional folder “presentation-api” in
> the web-platform-tests project similar to other W3C API. inside this folder
> we will have two subfolders “controlling-ua” and “receiving-ua” to test the
> Presentation API on controlling and receiving user agents. The initial version I
> submitted contains WebIDL tests of the Presentation API (each of the
> “controlling-ua” and “receiving-ua” contains a test file “idlharness.html”).
> After the PR is merged, you are welcomed to submit tests ;). I will share with
> you more details about test strategy of the Presentation API in the next days.
> Looking forward for your feedback and comments.

Great start.

> Louay - thank you for bootstrapping the testing effort for the Presentation
> API! When the initial test suite PR has landed, I suggest you update the spec
> adding a pointer to the test suite to give it more visibility (otherLinks in
> respecConfig).
> 
> Some questions:
> 
> - In this initial contribution it is proposed the test suite tracks the latest spec
> published on TR. If we follow this practice, we should make sure we publish
> new WDs frequently enough to make the TR spec a good synchronisation
> point (Echidna makes that very easy). Or alternatively, we could agree we
> write tests against the latest ED spec. A downside is ED may have more spec
> churn, so writing tests against it may be more risky for people who are not
> following the spec development closely. However, my assumption is that
> most of the test contributions would come from the participants of this
> group though, so that would not be a major issue. Your thoughts?

Tests based on ED looks good to me.

> - What are your thoughts, should we use Critic for this spec's test suite or just
> stick with the GH based workflow? I'm not very familiar with Critic, so I'm
> interesting in hearing feedback from people who are more familiar with Critic
> (Zhiqiang?) on cases when it is particularly useful.

Both Critic and GH are OK and acceptable, feel free to use your familiar one or both. Anyone is welcome to review the tests.

> Zhiqiang - at our recent F2F you indicated interested to contribute to testing
> of this spec. Could you please review [3] and provide feedback in the PR (or
> in Critic if you think that'd be a helpful tool to use). Feel free to merge the PR
> after review.

Yes, I will review it, but please expect this is to be done later next week. If someone reviews it before that time and at least leave a LGTM comment, I can help merge it without my review.
 
Thanks,
Zhiqiang

Received on Thursday, 17 December 2015 01:28:02 UTC