WebID TestSuite Comments

hi,

the webIDTestSuite [1] came up again (see mail with subject WebIDRealm supports RDFa) and i think
it's well worth an own thread.

following are some comments and suggestions.

1. the page is kind of out of date and should be updated (see rsa#modulus example)
2. i personally think the output of the test as graph is a little complicated, first to produce
   and second to parse and interpret.
3. as i understand the workflow of the test it is suboptimal (please correct me if i'm wrong) :
   when interpreting the testresult, one has to infer from earl#outcome of a result that something
   went wrong - what exactly went wrong is not known.
4. i'm not sure if it is useful to return passed tests att all ("no message is a good message").
   i only would be interested in tests that fail (in case of a failure, i want to know what exactly went wrong)
5. i'm not sure if the test really tests what needs to be tested. see [2] for example. 
   if a validator is about to be tested it's not the validator's fault if the webID is not available.
   it is a prerequisite that the webID is available. currently the => endpoint's test <= has a "not passed"
   in the result if the webID is not available.
6. many of the tests don't actually test the validator but other things. see [3] for example. 
   this test tests a certificate.
7. so the real test for a validation endpoint looks kind of different to me : 
   "in the case of a valid certificate and a valid profile (where one of the webIDClaims in the 
   certificate points to said valid profile) a validation-endpoint must authenticate"
   (not authorize to do something).
   7.1. a valid certificate must be tested on its own. maybe this page [4] could help.
   7.2. in case of a valid certificate (if there's something wrong there's no need to proceed)
        the profile must be tested (that actually also not the work of a validation endpoint),
        a reference test page, that determines if a profile is ok and valid would be ideal.
   7.3. in case the profile is also considered ok and spec-compliant one can try to use
        said certificate and the profile to login at a certain validation-endpoint.
        we then only need to have the yes/no answer from a validation-endpoint to be able
        to see if the validation-endpoint is spec-compliant.
   7.4. how to test all these things? 
        7.4.1. have a page with certificate-validation. again maybe this helps somehow [4] or points to other resources about cert-testing
        7.4.2. have a page with profile-validation
        7.4.3. create n-certificates (some valid, some not), with related n-profiles (some valid, some not)
        7.4.4. create a list of expectations for these certificates (the not-valid are expected to be rejected)
        7.4.5. foreach certificate
                  validate certificate 
                     if valid
                        validate profile
                           if profile not available
                              log and next
                           if valid
                              use certificate and profile against validation endpoint => answer must "accepted"
                           if not valid
                              use certificate and profile against validation endpoint => answer must "rejected"
                     if not valid
                              use certificate and profile against validation endpoint => answer must "rejected"
   7.5. as to why an expectation of a certain certificate wasn't fullfilled does not necessarily have to be a community effort.
        if i run the test and see that i let pass a certificate that is expected to be rejected it is the implementor's duty
        to find out why.

i the hope that at least something is considered usefull..

wkr http://www.turnguard.com/turnguard

[1] http://www.w3.org/2005/Incubator/webid/wiki/Test_Suite
[2] http://www.w3.org/2005/Incubator/webid/earl/RelyingParty#profileGet
[3] http://www.w3.org/2005/Incubator/webid/earl/RelyingParty#certificateProvidedSAN
[4] http://csrc.nist.gov/groups/ST/crypto_apps_infra/pki/pkitesting.html


--
| Jürgen Jakobitsch,
| Software Developer
| Semantic Web Company GmbH
| Mariahilfer Straße 70 / Neubaugasse 1, Top 8
| A - 1070 Wien, Austria
| Mob +43 676 62 12 710 | Fax +43.1.402 12 35 - 22

COMPANY INFORMATION
| http://www.semantic-web.at/

PERSONAL INFORMATION
| web   : http://www.turnguard.com
| foaf  : http://www.turnguard.com/turnguard
| skype : jakobitsch-punkt

Received on Thursday, 5 January 2012 09:59:55 UTC