Re: Test suite for Activity Streams 2.0

Comments inline...

On Mon, Oct 19, 2015 at 1:06 PM, Evan Prodromou <evan@e14n.com> wrote:
[snip]
>
> There's a definition of the suite-to-wrapper interfaces here:
>
> Consumer interface:
> https://github.com/w3c-social/as2test/blob/master/README.md#consumer-interface
> Producer interface:
> https://github.com/w3c-social/as2test/blob/master/README.md#producer-interface
>

These are generally a good start but are certainly not expansive enough.

> I don't think every implementation will implement both consuming and
> producing; that's why there are two separate test suites and interfaces. So,
> I'm not crazy about the round-trip test.

In the code I wrote, the roundtrip is largely to normalize the data
and ensure that the activitystrea.ms module is working against the
normative @context. It allows us to ask questions based on the
normative context without necessarily worrying about the different
allowed input variations.

> Passing the AS2 JSON to the consumer wrapper on stdin means that there can
> be a variety of storage options on the test suite side. Right now, they're
> just class attributes on some pyunit test case classes. It also means that
> wrapper writers can skip all the path management and file I/O issues.

Yep. Implementing stdin support is on my todo list.

> I like JSON Pointer, but I'm not sure that it works well for
> multi-implementation consumer tests.
>
> Firstly, we'll need to have JSON Pointer implementations in each of the
> implementation languages. I don't know the state of JSON Pointer in
> different languages, and I don't really want to have to depend on them,
> since we'd probably be using some advanced features.
> Second, I think a lot of our tests on the implementations won't actually be
> queryable by JSON Pointer. The first tests I've done use example 1 from
> activitystreams-core and ask a consumer what the actor ID is. I think that a
> consumer that can answer that question understands AS2 better than a
> consumer that knows what "/actor" is.
>

That's fair. Although, as far as I'm aware, there are JSON Pointer
implementations available for ruby, python, php, java, javascript and
c#. It's not that difficult of a thing to implement.

> So, I'd like to stick with the simple interfaces for now. They're going to
> require a couple dozen more command-line options, but I think the model
> works OK.
>
> -Evan
>
>
> On 2015-10-14 09:03 AM, James M Snell wrote:
>
> Evan,
>
> I took at quick look at the driver and it looks like a great start.
> Just to throw some additional ideas into the pot, I took some time
> yesterday to put together an alternative:
> https://github.com/jasnell/asint
>
> asint tests the JS activitystrea.ms impl by first performing a
> roundtrip deserialization-serialization then applying a JSON Pointer
> to select output.
>
> So, for instance, if I have an input file test.json:
>
> {
>   "@context": [
>     {"foo": "urn:foo:", "bar": "foo:bar"},
>     "http://www.w3.org/ns/activitystreams#"
>   ],
>   "@id": "http://example.org/foo",
>   "@type": "Like",
>   "displayName": "I like this",
>   "actor": {
>     "@type": "Person",
>     "displayName": "James"
>   },
>   "bar": 1
> }
>
> I can perform a number of inspections on the file:
>
> $ asint -i test.json -p /@type
> Like
> $ asint -i test.json -p /displayName
> I like this
> $ asint -i test.json -p /actor/displayName
> James
> $ asint -i test.json -p /urn:foo:bar
> 1
>
> By having the tool perform a full deser/ser roundtrip, we can make
> sure that the input was understood correctly by the activitystrea.ms
> JS implementation. The JSON Pointer used to interrogate is based on
> the normalized, normative AS 2.0 @context so the results should be
> predictable.
>
> Thoughts?
>
> - James
>
> On Tue, Oct 13, 2015 at 1:49 AM, Evan Prodromou <evan@e14n.com> wrote:
>
> Christopher Webber and I and others have been discussing the best way to do
> a test suite for Activity Streams 2.0 (AS2). We've got an initial version
> here:
>
>     https://github.com/w3c-social/a2test
>
> The suite can test both AS2 consumer and producer implementations. It uses a
> simple command-line interface to give a simple cross-language testing
> framework. We considered using a Web interface but we decided that this
> would require too much setup to run a test, and would also delve into dicey
> territory around the Social API.
>
> There's a long README about the interface that I won't duplicate here.
>
>     https://github.com/w3c-social/a2test/blob/master/README.md
>
> I started work on testing wrappers for jasnell's awesome activitystreams.js
> library. The wrappers are here:
>
>     https://github.com/evanp/activitystreams.js-test
>
> As of right now, they pass the test suite. James, I'd be happy to see this
> merged into activitystreams.js or leave it separate if that's what you'd
> prefer.
>
> Cwebber is working on a Python library for AS2, visible here:
>
>     https://github.com/w3c-social/activipy
>
> I hope to see it passing the test suite soon!
>
> The test suite itself is the bare minimum needed to prove out the interface.
> We'll need to expand it significantly to cover all the examples in AS2 core
> and vocabulary, as well as edge cases and implied properties. But it's on
> the right track, I think.
>
> Thanks to everyone who offered suggestions. And please let us all know if
> you're implementing an AS 2.0 library and are going to use this test suite
> on it.
>
> -Evan
>
>
>

Received on Tuesday, 20 October 2015 16:34:14 UTC