Re: Test suite for Activity Streams 2.0

Hey, James. Sorry for the wall of text here.

  * This is /not/ just a test for activitystreams.js. It's a test suite
    for all implementations, in many different programming languages. I
    did a wrapper for activitystreams.js so we could test it against the
    global test suite. My expectation is that cwebber's Python
    implementation and future implementations in Go, Ruby, PHP, etc.
    will all use the same test suite (with appropriate language-specific
    wrappers).
      o Test suite: https://github.com/w3c-social/as2test/
      o Wrapper for activitystreams.js to make it work with the test
        suite: https://github.com/evanp/activitystreams.js-test
  * There's a definition of the suite-to-wrapper interfaces here:
      o Consumer interface:
        https://github.com/w3c-social/as2test/blob/master/README.md#consumer-interface
      o Producer interface:
        https://github.com/w3c-social/as2test/blob/master/README.md#producer-interface
  * I don't think every implementation will implement both consuming and
    producing; that's why there are two separate test suites and
    interfaces. So, I'm not crazy about the round-trip test.
  * Passing the AS2 JSON to the consumer wrapper on stdin means that
    there can be a variety of storage options on the test suite side.
    Right now, they're just class attributes on some pyunit test case
    classes. It also means that wrapper writers can skip all the path
    management and file I/O issues.
  * I like JSON Pointer, but I'm not sure that it works well for
    multi-implementation consumer tests.
      o Firstly, we'll need to have JSON Pointer implementations in each
        of the implementation languages. I don't know the state of JSON
        Pointer in different languages, and I don't really want to have
        to depend on them, since we'd probably be using some advanced
        features.
      o Second, I think a lot of our tests on the implementations won't
        actually be queryable by JSON Pointer. The first tests I've done
        use example 1 from activitystreams-core and ask a consumer what
        the actor ID is. I think that a consumer that can answer that
        question understands AS2 better than a consumer that knows what
        "/actor" is.

So, I'd like to stick with the simple interfaces for now. They're going 
to require a couple dozen more command-line options, but I think the 
model works OK.

-Evan

On 2015-10-14 09:03 AM, James M Snell wrote:
> Evan,
>
> I took at quick look at the driver and it looks like a great start.
> Just to throw some additional ideas into the pot, I took some time
> yesterday to put together an alternative:
> https://github.com/jasnell/asint
>
> asint tests the JS activitystrea.ms impl by first performing a
> roundtrip deserialization-serialization then applying a JSON Pointer
> to select output.
>
> So, for instance, if I have an input file test.json:
>
> {
>    "@context": [
>      {"foo": "urn:foo:", "bar": "foo:bar"},
>      "http://www.w3.org/ns/activitystreams#"
>    ],
>    "@id": "http://example.org/foo",
>    "@type": "Like",
>    "displayName": "I like this",
>    "actor": {
>      "@type": "Person",
>      "displayName": "James"
>    },
>    "bar": 1
> }
>
> I can perform a number of inspections on the file:
>
> $ asint -i test.json -p /@type
> Like
> $ asint -i test.json -p /displayName
> I like this
> $ asint -i test.json -p /actor/displayName
> James
> $ asint -i test.json -p /urn:foo:bar
> 1
>
> By having the tool perform a full deser/ser roundtrip, we can make
> sure that the input was understood correctly by the activitystrea.ms
> JS implementation. The JSON Pointer used to interrogate is based on
> the normalized, normative AS 2.0 @context so the results should be
> predictable.
>
> Thoughts?
>
> - James
>
> On Tue, Oct 13, 2015 at 1:49 AM, Evan Prodromou <evan@e14n.com> wrote:
>> Christopher Webber and I and others have been discussing the best way to do
>> a test suite for Activity Streams 2.0 (AS2). We've got an initial version
>> here:
>>
>>      https://github.com/w3c-social/a2test
>>
>> The suite can test both AS2 consumer and producer implementations. It uses a
>> simple command-line interface to give a simple cross-language testing
>> framework. We considered using a Web interface but we decided that this
>> would require too much setup to run a test, and would also delve into dicey
>> territory around the Social API.
>>
>> There's a long README about the interface that I won't duplicate here.
>>
>>      https://github.com/w3c-social/a2test/blob/master/README.md
>>
>> I started work on testing wrappers for jasnell's awesome activitystreams.js
>> library. The wrappers are here:
>>
>>      https://github.com/evanp/activitystreams.js-test
>>
>> As of right now, they pass the test suite. James, I'd be happy to see this
>> merged into activitystreams.js or leave it separate if that's what you'd
>> prefer.
>>
>> Cwebber is working on a Python library for AS2, visible here:
>>
>>      https://github.com/w3c-social/activipy
>>
>> I hope to see it passing the test suite soon!
>>
>> The test suite itself is the bare minimum needed to prove out the interface.
>> We'll need to expand it significantly to cover all the examples in AS2 core
>> and vocabulary, as well as edge cases and implied properties. But it's on
>> the right track, I think.
>>
>> Thanks to everyone who offered suggestions. And please let us all know if
>> you're implementing an AS 2.0 library and are going to use this test suite
>> on it.
>>
>> -Evan
>>
>>