[testing] HTML Media Capture tests/pull request review

I’ve taken a look a the HTML Media Capture tests and issue thread related to them [1].

The issue is not so much with the tests themselves, but rather with how we are handling the implementation requirements related to security and privacy in our specifications. 
The executive summary is that we make SHOULD statements related to security and privacy to give leeway to implementations based on circumstances or to capture broad requirements on implementations. This results in some ambiguity as to whether tests are required or appropriate for these statements. In some cases these statements are quite clear and can be tested yet others might be hard to test in any case (but still relevant), e.g. “Implementors SHOULD take care of additional leakage of privacy-sensitive data from captured media” 

The concern Tobie noted is related to this larger issue. 

I did not run “capture_reflect.html” so that one might need more review; have we experience running this test?

We currently have no tests visible for HTML Media capture.

My proposal is that we accept the pull request for the following reasons:

1. Make a test draft baseline visible for further comment and involvement.

2. Provide tests based on the current approach of the specification.

I believe the current Security and Privacy Considerations section with its "shoulds" represents  a compromise between mandating security/privacy items and recognizing the need for implementer judgement. I suspect we will need to accept the ambiguity related to testing but manage it by making clear the core tests versus these ones that provide guidance. (I thought that was what Tobie was suggesting in thread about adding metadata but might have misread that). The tests that relate to this do have "<meta name="flags" content="interact should”>” but maybe we need an “optional-to-test” flag.

Ultimately I think we need a document that says: here are the mandated tests, and here are additional tests and rules related to security/privacy, not sure the framework supports such notion, or is there some metadata that allows this?

Thoughts? Are there better examples of addressing security/privacy in a meaningful way in such specifications?

regards, Frederick

Frederick Hirsch
Nokia

[1] https://github.com/w3c/web-platform-tests/pull/306

Statements of concern and tests in pull request:

(a) The user agent SHOULD NOT enable any device for media capture, such as a microphone or camera, until a user interaction giving implicit consent is completed.  A user agent SHOULD also provide an indication when such an input device is enabled and make it possible to terminate such capture. 

tests for explicit consent -  capture_audio.html, capture_audio_deny.html, capture_image.html, capture_image_deny.html

no test for indications to user

(b) Similarly, the user agent SHOULD allow the user:

	• to select the exact media capture device to be used if there exists multiple devices of the same type (e.g. a front-facing camera in addition to a primary camera).
	• to disable sound capture when in the video capture mode.

capture_image_multiple_device.html and capture_video.html, capture_video_deny.html

(c) This specification builds upon the security and privacy protections provided by the <input type="file"> [HTML5] and the [FILE-API] specifications; in particular, it is expected that any offer to start capturing content from the user’s device would require a specific user interaction on an HTML element that is entirely controlled by the user agent.

Implicit in the tests

(d) Implementors SHOULD take care of additional leakage of privacy-sensitive data from captured media. For instance, embedding the user’s location in a captured media metadata (e.g. EXIF) might transmit more private data than the user might be expecting.

No test, depends on details of information and implementation

——

Received on Tuesday, 18 March 2014 21:38:30 UTC