Re: [EXTERNAL] [jfraichot@learningmachine.com] Re: VC API: handling large documents client to server

Thanks Manu for the great write up,

It’s true that Blockcerts historically has made heavy use of base64 content as part of the data, and it was a partial problem since we didn’t have much transport. Now that we are considering the problem it’s great to hear what could be the industry standard, so I will try out the different things you suggest.

From: Mike Prorock <mprorock@mesur.io>
Date: Thursday, 10 February 2022 at 15:49
To: Manu Sporny <msporny@digitalbazaar.com>
Cc: W3C Credentials CG <public-credentials@w3.org>
Subject: [EXTERNAL] [jfraichot@learningmachine.com] Re: VC API: handling large documents client to server
CAUTION: This email originated from outside of Hyland. Do not click links or open attachments unless you recognize the sender and know the content is safe.

+1 manu - great notes - I will note from our side that we utilize hashlink (and related approaches) when dealing with larger binary data that needs to be referenced in a credential - e.g. image of a product, etc.

Mike Prorock
CTO, Founder
https://mesur.io/



On Thu, Feb 10, 2022 at 9:39 AM Manu Sporny <msporny@digitalbazaar.com<mailto:msporny@digitalbazaar.com>> wrote:
On 2/10/22 3:01 AM, Julien Fraichot wrote:
> Basically when calling a verification API from a (browser) client, there
> might be times where the documents could be quite large (few MBs). I am
> wondering if there are some strategies to reduce the payload that would
> also be standard when dealing with a VC API complying service?

If you have a VC that is several MB in size, I would expect it to struggle in
the ecosystem. Yes, they are legal, in the same way that attaching a 250MB
slide deck to an email is legal -- while the SMTP protocol allows for it, most
mail servers will reject the message as too large.

Typically, these large VCs happen because people are embedding base-encoded
images directly into a VC. Instead, VC creators should consider modelling
their data differently -- e.g., use a hashlink, or some other way of creating
a cryptographic hyperlink:

https://datatracker.ietf.org/doc/html/draft-sporny-hashlink

> I am researching gzipping on the client and tried more exotic approaches to
> no avail, so I’d be willing to hear what the people have thought on the
> matter.

You get gzip compression during transmission (more or less) for free these
days, but that's not really going to save you given that you're probably
base-encoding the raw binary data. Quite counter-intuitively, doing that makes
using gzip expand the file size instead of reducing it:

https://stackoverflow.com/questions/38124361/why-does-base64-encoded-data-compress-so-poorly

This is another reason the JOSE/JWT stack, when used with VCs, harm wire-level
protocols -- everything is base64 encoded, and thus it effectively destroys
any ability to compress data on the wire.

Typical solutions to this problem require that you put the binary data outside
of the VC, if at all possible. This works well for common static images such
as logos. It is also possible to split the VC into two VCs... one with the
machine-readable data from the issuer (with a digital signature) and one with
the image data from any source (without a digital signature, since, if
hashlinked, the signature will verify the validity of the image data). That
latter approach can be more privacy preserving AND more complex than many
might feel is necessary.

Selective disclosure schemes (such as BBS+) are another way to deliver a
subset of the information to a verifier without having to send the image
payload data.

I expect this to be an active area of innovation for the next few years with a
few proposals on standard design patterns that all industries could use. This
problem appears most often with identification cards that have biometric
images embedded in them.

-- manu

--
Manu Sporny - https://www.linkedin.com/in/manusporny/
Founder/CEO - Digital Bazaar, Inc.
News: Digital Bazaar Announces New Case Studies (2021)
https://www.digitalbazaar.com/

-----------------------------------------  Please consider the environment before printing this e-mail -----------------------------------------  

CONFIDENTIALITY NOTICE: This message and any attached documents may contain confidential information from Hyland Software, Inc. The information is intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, or an employee or agent responsible for the delivery of this message to the intended recipient, the reader is hereby notified that any dissemination, distribution or copying of this message or of any attached documents, or the taking of any action or omission to take any action in reliance on the contents of this message or of any attached documents, is strictly prohibited. If you have received this communication in error, please notify the sender immediately by e-mail or telephone, at +1 (440) 788-5000, and delete the original message immediately. Thank you.