Re: [EXTERNAL] [jfraichot@learningmachine.com] Re: VC API: handling large documents client to server

So why not move to CBOR and COSE instead of JWT?

Leonard

From: Manu Sporny <msporny@digitalbazaar.com>
Date: Thursday, February 10, 2022 at 10:07 PM
To: Nikos Fotiou <fotiou@aueb.gr>, 'Orie Steele' <orie@transmute.industries>, 'Julien Fraichot' <Julien.Fraichot@hyland.com>
Cc: 'Mike Prorock' <mprorock@mesur.io>, 'W3C Credentials CG' <public-credentials@w3.org>
Subject: Re: [EXTERNAL] [jfraichot@learningmachine.com] Re: VC API: handling large documents client to server
On 2/10/22 2:30 PM, Nikos Fotiou wrote:
> Can you please provide some more information about why “base64url encoding
> is a major problem for signatures over complex data”. At least to me, it is
> not obvious.

By default, base64 encoding in JWTs leads to 133% message size increase when
the JWT is sent over a network connection and when being stored in a database.

This is because HTTP uses gzip/deflate compression by default, except that
when base64 is used, the compression algorithm goes into a "worst case" mode
and it results in the opposite of what you want to happen (duplication of
data, not compression).

This means that exchanging base64 data across HTTP takes 1.33x more space, and
storing it takes 1.33x more storage than a clear-text JSON representation.
While this might not sound like a lot, it adds up when you're doing tens of
millions of these sorts of transfers/storage a day.

This article explains why:

https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstackoverflow.com%2Fquestions%2F38124361%2Fwhy-does-base64-encoded-data-compress-so-poorly&amp;data=04%7C01%7Clrosenth%40adobe.com%7Cc828117c60a54f8f4c1808d9ecd0ff22%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637801204648672812%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=a9mi1jp9uhnje%2Fv9k%2B5yVUcOuDLeBwplKD%2FwohTHYfI%3D&amp;reserved=0

The problem gets even worse when you have to chain signatures together,
storage requirements start multiplying in the wrong direction.

I will note that JSON-LD encoded and Data Integrity protected Verifiable
Credentials don't suffer from this problem (except for where they use base64
to encode an image) and take advantage of standard HTTP compression available
today. Due to JSON-LD's semantics, it is possible to do the appropriate
(efficient) binary conversion when you go to the wire or to disk.

As Orie mentioned, there could be fixes for this that the original proponents
of JWS did not include:

https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailarchive.ietf.org%2Farch%2Fmsg%2Fjose%2Fl2e0SCf57V-9s2HWG-sJRhXV23I%2F&amp;data=04%7C01%7Clrosenth%40adobe.com%7Cc828117c60a54f8f4c1808d9ecd0ff22%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637801204648672812%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=buYaw%2BXbPyS8gzVQqhIpGsZm96X1HY%2F9PiDBLaA1mjI%3D&amp;reserved=0

... and even though it's defined for JWE, most JWT libraries don't support any
sort of feature in an interoperable way:

https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdvsekhvalnov%2Fjose2go%2Fissues%2F25%23issuecomment-539958151&amp;data=04%7C01%7Clrosenth%40adobe.com%7Cc828117c60a54f8f4c1808d9ecd0ff22%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637801204648672812%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=SVXcKQg9SyH%2BgSphBkGO3tut7aYMtZnrEW4yH7NFuSU%3D&amp;reserved=0

... and (depending on how it's implemented) the VC-JWT2 stuff is probably not
going to be backwards compatible OR is going to be specific to VC-JWT even if
it is backwards compatible... not that that's a big deal because until
recently, there wasn't a good test suite for VC-JWT and so there was no such
thing as provable interop for VC-JWT.

All that said, this is a curiosity that many developers don't care about. They
either don't know (most developers), or if they know, they just pay the 133%
over original size penalty and move on because they're more concerned about
"other more pertinent to them" things in their system.

-- manu

--
Manu Sporny - https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fin%2Fmanusporny%2F&amp;data=04%7C01%7Clrosenth%40adobe.com%7Cc828117c60a54f8f4c1808d9ecd0ff22%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637801204648672812%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=TvnnKqwfV1NQk2PfQ5dVhmv6%2Baw7BPhQte7ykBvvUMM%3D&amp;reserved=0
Founder/CEO - Digital Bazaar, Inc.
News: Digital Bazaar Announces New Case Studies (2021)
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.digitalbazaar.com%2F&amp;data=04%7C01%7Clrosenth%40adobe.com%7Cc828117c60a54f8f4c1808d9ecd0ff22%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637801204648672812%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=7UbkrOikKMyM5rJimXiBRZ2UuBpy6y9u4KiZ9kCERxE%3D&amp;reserved=0