Re: Giving up on XML DSig => JSON

Response inline below..

On Aug 29, 2013, at 11:32 AM, Anders Rundgren <anders.rundgren@telia.com>
 wrote:

> On 2013-08-29 18:49, Jim Klo wrote:
>> Hi Anders,
>> I did some work very similar some time back for Learning Registry [1].  There are some known issues with the solution we chose and upon investigation of your doc, seems to suffer from similar problems.
> 
> Hi Jim,
> This sounds interesting!
> 
>> My first question might be is what is your indented use case for this?
> 
> Primary replacing this:  http://webpki.org/papers/keygen2/keygen2.junit.run.html
> which is a part of:  https://code.google.com/p/openkeystore
> 

Not sure if I fully understand what's going on here - but looks like JUnit test cases that are provisioning some PKI - little hard to follow for me; I'm no crypto expert.

>> Like JOSE this seems to have very limited use due to a number of issues I'll outline below.
> 
> Well, I have done some shortcuts/redefinitions of JSON which should make this scheme usable for a wide class of web-based security-protocols, where the _message_ is the core and a signature is only there to vouch for the message's authenticity.
> 

Yes, that's essentially what we did - take some shortcuts… digitally signing the hash of our canonical bencoded representation was a 'shortcut' for storage and provided a check to ensure that you canonicalized correctly.  Our goal however was to ensure that maintain that any JSON parser could still natively access data, and try to use well known algorithms to aid in validation of authenticity and check for forgery. Known sha256 isn't perfect - but good enough… Bencode isn't perfect - but good enough at the time.

We've had several discussions on steps to improve - but most things are equally complex.  Crypto is already unnecessarily too hard for most  smart developers to understand - adding complex steps for validation of portable data structures is just another notch in that got added - that I'd like to make even more simple.

> That is, I don't expect this scheme to work with arbitrary JSON parsers.  I wrote a suitable parser from scratch that preserves property order as well as the base elements (int, bool, string) in text format.  If you do that canonicalization gets absolutely trivial:
> https://code.google.com/p/openkeystore/source/browse/library/trunk/src/org/webpki/json/JSONParser.java
> 300 lines of code seems supportable.  it is brand new so it may have some "warts".
> 

I suspect you could have also just modified something like Jackson and modified toString() and map storage to use a modified JSONObject as well. Regardless, the parser seems straight forward… the magic is in your JSONObject… where you have to provide a crutch to retain ordering via a LinkedHashMap to maintain key ordering - not available in all solutions without some effort…

I'm more interested in serialization of simple typed objects via JSON, less concerned about more generic container like implementations:

	Foo.java
	public class Foo {

		int alpha;
		String beta;

	}


	Foo.py
	class Foo(object):
		def __init__(self, a, b)
			self.alpha = a
			self.beta = b

Using Java reflection wouldn't necessarily walk properties in the same order as say Python dir(.) would. Producing a canonical output would require some kind of 'crutch' code, IMO. I'm just looking for simplicity. :)

Ideally, at least for our use case which is a federated metadata referratory, having the ability to access our envelope data (which includes the crypto signature) is pretty important for stream processing as well as minimizing storage requirements. Because data is publicly federated, the signature is primarily there to minimize forgery from trusted signatories.

Here's a sample: http://node01.public.learningregistry.net/obtain?by_doc_ID=true&request_ID=15a0b923465249fd80b8b3c0f9f7879e

documents[0].document[0] is our envelope, documents[0].document[0].resource_data is the metadata, everything else is api wrapper.

> IBM has apparently patented (!!!) a JSON signature scheme ( http://patents.justia.com/patent/20100185869 ) which does canonicalization the hard ("right" if you prefer that...) way. I won't try that because then the advantage of JSON becomes limited.
> 

I've not seen this before… Looks remarkably like XML variants of the same thing. It's amazing what the USPTO grants nowadays… but that's a topic for another discussion…

>> JSON isn't guaranteed to serialize across systems consistently. These are equivalent expressions:
>> // fig 1
>> {
>> "alpha": 1,
>> "beta": "kappa"
>> }
>> and
>> // fig 2
>> {
>> "beta": "kappa",
>> "alpha": 1
>> }
> 
> It would not validate using JSON schema ( http://json-schema.org/ ) and that is where I started (with my 12 years of XSD in the back).
> 

How so?

{
	"$schema": "http://json-schema.org/draft-04/schema#",
	"type": "object",
	"properties": {
		"beta": {
			"type": "string"
		},
		"alpha": {
			"type": "integer"
		}
	},
	"required": [ "alpha", "beta" ],
	"additionalProperties": false
}

As far as I know there is no mechanism in the language to specify key ordering. 

Cheers,

- Jim

> I.e. I consider JSON properties equivalent to XML elements rather than XML attributes.
> But that's of course *my* (re)definition of JSON :-)
> 
> Cheers
> Anders
> 
>> JOSE gets around this because the data is contained within the JWT packaging. Additionally JOSE does signature validation of the base64 encoded material - no the JSON entities.  Like you've mentioned this isn't suitable for usage as a data model.
>> The challenge is there's no good way (that I've uncovered) to canonicalize JSON objects and retain use as a parsed data model that serializes between platforms consistently.  Ultimately what you've done is created a canonical format that just treats the entire entity as a string. This wouldn't necessarily work well as moving the data between Java, JavaScript, Ruby, Python, etc… dictionary keys are not ordered consistently… say my system receives fig 1 above and stores it for processing, the key order isn't guaranteed when parsing it from it's native format.  In our use case we use CouchDB which stores documents as JSON.  While CouchDB just happens to be consistent in storing/retrieving dictionaries in the same order as they submitted - there's not guarantee of that.  Clients connecting to the DB also may not reconstruct key order as well depending upon their JSON parsers.  Thus in order to make your solution work, Like JOSE, the canonical format must be retained
> indefinitely in a portable format.
>> I've covered dictionaries… and FWIW XML shares the same problem with element attributes. Other challenges in signing JSON are brought upon by floating point and boolean values.  Floating point precision is problematic unless encoded as strings as floating point precision is not portable across platforms. JavaScript, Erlang, and Python could introduce error when moving objects between systems.  Booleans have the same issue - some languages internally store 1 and 0 others actually have native true and false.
>> Ultimately we chose to use Bencode [2] as part of a canonicalization solution which unfortunately cannot handle floats very well, however in meeting I had with BitTorrent folks some time ago we discussed Tagged Netstrings [3] as a more viable solution.
>> IMO, the only way to make signing JSON objects reasonable is to have a solution that is trivial to canonicalize the format for reconstruction of the signed object and permits native serialization of the JSON. I don't want to have to keep some string blob around because I need to preserve the canonical format.
>> To a certain extent the solution to this lies in the development, implementation, and adoption of s  "CJSON" or Canonical JavaScript Object Notation, where the representation of the object is already canonical and requires no transformations.  Alas, there seems to be very little interest for this, since most of the current work is like JOSE where signatures are only needed short term for one time validation of data sent over the wire.
>> [1] http://docs.learningregistry.org/en/latest/spec/Identity_Trust_Auth_and_Security/index.html#identity-and-digital-signatures
>> [2] https://wiki.theory.org/BitTorrentSpecification#Bencoding
>> [3] http://tnetstrings.org/
>> Jim Klo
>> Senior Software Engineer
>> Center for Software Engineering
>> SRI International
>> t.    @nsomnac
>> On Aug 28, 2013, at 11:30 PM, Anders Rundgren <anders.rundgren@telia.com> wrote:
>>> Since Google doesn't support XSD or XML DSig in Android I began looking at other alternatives.
>>> There were none :-( Now there is :-)
>>> 
>>> https://openkeystore.googlecode.com/svn/resources/trunk/docs/Enveloped-JSON-Signatures.pdf
>>> 
>>> Comments are welcome!
>>> 
>>> Cheers
>>> Anders
>>> 
> 
>