Re: experimental vc-jwt 2.0 test suite

That is correct, but needs to be documented better.

Of course the implementation could be just calling another registered
implementation behind the scenes in that case... perhaps we give
reproducible builds of the test report a "higher rating".

Ideally we would see implementations in go, java, rust, javascript,
typescript, python and c... before we see people registering "conformance
results" from a black box implementation.

This should be less hard, given the new JWT structures.

We also need to consider adding negative tests... injecting "emoji",
misconfigured mandatory and optional fields, etc...

OS

On Mon, Apr 17, 2023 at 8:30 AM Mike Prorock <mprorock@mesur.io> wrote:

> Cool - it looks like also, while docker is there (and would be a preferred
> way to execute for us) there is no real mandate to even use docker to test
> an implementation, as long as you checked in the appropriate key and output
> in the outputs folder you can get a test report.  Am I reading that code
> right?
>
> Mike Prorock
> mesur.io
>
> On Mon, Apr 17, 2023, 07:23 Orie Steele <orie@transmute.industries> wrote:
>
>> Inline:
>>
>> On Mon, Apr 17, 2023 at 8:01 AM Mike Prorock <mprorock@mesur.io> wrote:
>>
>>> Orie,
>>> Thanks for this.  I believe we can work with this approach, and we will
>>> contribute an implementation and help clean up and improve.  It looks like
>>> a second pass of test(s) could then run on the outputs of this once VC-JWT
>>> is verified, to then verify against the core data model.  Is that your
>>> thinking?
>>>
>>>
>> Yes, there are a couple different ways to process the results / grade
>> them.
>>
>> The idea is to allow implementation to produce both issuance and
>> verification results for analysis / grading.
>>
>>
>>> Also, it looks like this could be extended so that tests could run in an
>>> independent environment for those with proprietary implementations.  Is
>>> that a correct read as well?
>>>
>>>
>> Yes, building on docker also gives us some tools to help consider some of
>> the tougher parts of cryptographic testing such as FIPS.
>>
>> -
>> https://ubuntu.com/blog/building-and-running-fips-containers-on-ubuntu18-04
>>
>> There are some more advanced features we could consider:
>>
>> - publishing a conformance image that makes evaluation of the test suite
>> easier, friendlier for private environments.
>> - publishing a GitHub Action so that conformance evaluation can be
>> dropped into any CI pipeline on GitHub (even private repos).
>>
>> Some of the current pathing / data flows might need to be touched up to
>> make these easier in the future.
>>
>> I am currently mounting a data directory and assuming some pathing, which
>> feels a bit awkward.
>>
>> Mike Prorock
>>> mesur.io
>>>
>>> On Sun, Apr 16, 2023, 17:48 Orie Steele <orie@transmute.industries>
>>> wrote:
>>>
>>>> I put this together today:
>>>>
>>>> - https://transmute-industries.github.io/vc-jwt-test-suite
>>>> - https://github.com/transmute-industries/vc-jwt-test-suite
>>>>
>>>> It's a test suite built on docker, github actions and github pages.
>>>>
>>>> There are some rough parts to it, and it obviously does not run core
>>>> data model conformance tests yet.
>>>>
>>>> This test suite has no JSON-LD processing dependency.
>>>>
>>>> I believe the core data model can be tested without doing any JSON-LD
>>>> processing, but perhaps we will see if that is truly the case.
>>>>
>>>> Regards,
>>>>
>>>> OS
>>>>
>>>> --
>>>> *ORIE STEELE*
>>>> Chief Technical Officer
>>>> www.transmute.industries
>>>>
>>>> <https://www.transmute.industries>
>>>>
>>>
>>
>> --
>> *ORIE STEELE*
>> Chief Technical Officer
>> www.transmute.industries
>>
>> <https://www.transmute.industries>
>>
>

-- 
*ORIE STEELE*
Chief Technical Officer
www.transmute.industries

<https://www.transmute.industries>