Re: Micropub test suite

The tests for updates, deletes, undeletes and querying are now complete!

The implementation report template now references the specific tests that
correspond with each feature.
https://github.com/w3c/Micropub/blob/master/implementation-reports/CLIENT-TEMPLATE.md

You'll notice there are still a couple that are left as self-reported,
because they are things that micropub.rocks can't really test. For example,
I can't tell whether a client considers an update to have been successful.

As you work through the tests on micropub.rocks, it actually fills out the
implementation report for you. It then gives you the raw text you can copy
and paste into an implementation report.

----
Aaron Parecki
aaronparecki.com
@aaronpk <http://twitter.com/aaronpk>


On Tue, Mar 7, 2017 at 4:00 PM, Aaron Parecki <aaron@parecki.com> wrote:

> *Discovery*
> The discovery test does not have a number, but by doing the setup process
> with the sign-in flow, you will pass this test. The process provides lots
> of feedback during the sign-in flow if you do something wrong:
>
>    1. https://media.aaronpk.com/Screen-Shot-2017-03-07-07-09-50.png
>    2. https://aaronparecki.com/2017/02/08/19/image-1.png
>    3. https://aaronparecki.com/2017/02/08/19/image-2.png
>    4. https://media.aaronpk.com/Screen-Shot-2017-03-07-07-10-14.png
>
> *Authentication*
> Similarly, there is no test number, but by using any of the tests, you
> will check off the boxes depending on how the request is made. The "scope"
> item is checked off as soon as the authorization process begins.
>
> *Syntax*
> Each item in syntax has a corresponding test. I've updated the report
> template to show the test number that corresponds with each checkbox.
>
> *Creating Posts*
> Each item in creating posts has a corresponding test, with the exception
> of "recognizes HTTP 201 and 202..." which is checked off after you create a
> post with any of the other tests and the client redirects to the post. In
> the report template, each item now references the corresponding test number.
>
> *Media Endpoint*
> There is a test for the first item, but not the second.
>
> *Updates, Deletes, Undeletes*
> No tests for these yet.
>
> *Querying*
> There are tests for the two endpoint queries. Tests for querying post
> information are still in progress.
>
> All the tests are pretty picky about syntax. I've already had to make a
> couple of fixes to Quill in order to pass the tests. Things like when you
> send a Micropub JSON request, the values have to be arrays. Here's all the
> error messages I return for the tests: https://github.com/
> aaronpk/micropub.rocks/blob/master/app/ClientTests.php#L501
>
> I spent today adding tests for querying and media endpoints, as well as
> making micropub.rocks fill out the implementation report template for you.
> Now as you work through the tests, it will check off all the boxes as you
> go.
>
>
> ----
> Aaron Parecki
> aaronparecki.com
> @aaronpk <http://twitter.com/aaronpk>
>
>
> On Mon, Mar 6, 2017 at 8:37 PM, Sandro Hawke <sandro@w3.org> wrote:
>
>> Excellent.   Can you characterize which features have automated tests and
>> which do not?     Is there anything boundary testing about them, testing
>> the kind of thing someone might get wrong, or middle-of-the-road
>> probably-everyone-will-pass tests?
>>
>>      -- Sandro
>>
>>
>>
>> On 03/06/2017 07:50 PM, Aaron Parecki wrote:
>>
>> We now have a client test suite for creating posts! Also the good news is
>> that the test suite passes itself, as demonstrated in the below video.
>>
>>
>> This is live on the micropub.rocks website now! You can read more about
>> the details here:
>>
>> https://aaronparecki.com/2017/03/06/14/day-76-micropub-rocks-client-tests
>>
>>
>>
>>
>> ----
>> Aaron Parecki
>> aaronparecki.com
>> @aaronpk <http://twitter.com/aaronpk>
>>
>>
>> On Mon, Mar 6, 2017 at 8:11 AM, Sandro Hawke <sandro@w3.org> wrote:
>>
>>> Apparently I wasn't paying enough attention at this point of the meeting
>>> last week.  Sorry for not catching this.
>>>
>>> To answer the question: I'm pretty sure we did not explicitly, with full
>>> consideration, agree to relax our previous plan of having complete test
>>> suites.  I see how that's implicit in the decision we made, and I recall
>>> Aaron mentioning it, so maybe everyone else thought it through, but in the
>>> mix of all the things going on during that meeting, I didn't put 2+2
>>> together.
>>>
>>> I agree we should be consistent on this.   In general, I'd say every
>>> constraint in the spec ought to have a few tests.   That's not a constraint
>>> of W3C process though -- the WG is free to set a different bar for
>>> interoperability and confirming implementations -- but we probably do need
>>> to be rational and consistent in setting that bar.
>>>
>>> So, which way do we want to go on this?
>>>
>>> And Aaron, how much of a burden would it be to finish that test suite
>>> (or can we recruit someone else to do it?)
>>>
>>>     -- Sandro
>>>
>>>
>>>
>>> On 03/06/2017 10:20 AM, Amy G wrote:
>>>
>>> Given the resolution about advancing Micropub to PR at the last meeting,
>>> did the working group decide that we don't need actually need complete test
>>> suites to progress to PR so long as there are reports? Does this also apply
>>> to LDN, WebSub and ActivityPub?
>>>
>>> On 6 March 2017 at 23:05, Aaron Parecki <aaron@parecki.com> wrote:
>>>
>>>> Correct, I have a start to the client tests but I haven't launched it
>>>> on the site yet. The implementation reports for clients are all self
>>>> reported.
>>>> On Mon, Mar 6, 2017 at 6:49 AM Amy G <amy@rhiaro.co.uk> wrote:
>>>>
>>>>> Hola,
>>>>>
>>>>> Just to confirm - there are no tests for Micropub clients right? You
>>>>> can only test a server implementation with the current test suite?
>>>>>
>>>>> Amy
>>>>>
>>>>
>>>
>>>
>>
>>
>