Datashards and Secure Data Hubs... how to achieve convergence?

As promised, here's that rant.  It's not actually all that ranty.  ;)

So having seen Datashards and DB's work on Secure Data Shards, and
knowing that I occasionally contract for DB, you might think that maybe
DB also had something to do with Datashards.  That's not the case, other
than that I've been trying to convince them of the ideas for the last
few years (we do talk a lot, but I haven't been paid for any work at all
in this area).  Over time we've come closer, but our work has happened
seperately, and we keep checking in and seeing that we come to many of
the same conclusions for different reasons, but not all of them.  So the
ideas are close, but they aren't yet aligned.  I hope at RWoT we can get
some better convergence.

There is one major difference that, in truth, barely matters (except
that it kind of does):

 - A difference in how data is serialized.  We can discuss this, it's
   not too important though.  It's easy enough to get convergence on
   this issue.

There are also several things that I used to argue for that we've
already come to convergence on:

 - The need for symmetric encryption on content-addressed content
   (well, at least we agree it's needed for some things; I'll argue
   for all CAS things later)
 - The need to chunk content into 32kb chunks

Yay, convergence!

Then there are several different differences which actually matter:

 - Datashards not only specifies an immutable system, but also a mutable
   system that layers on top of the immutable data.  The design is
   mostly taken from Tahoe-LAFS (which has done almost everything right
   except for making their system ~universal in a way that can be
   applicable on the web... in Tahoe you have to run things on specific
   clusters).

 - Datashards has two universal URI schemas that can exist *outside* of
   a data hub.  idsc: is for immutable data, mdsc: is for mutable data.
   This is, I would argue, a nice feature.  You can use these for
   everything you would normally put an https: link; this includes
   json-ld contexts and linking to cat photos and all sorts of things.
   (This is what I mean by Datashards building "universal web
   primitives".)

 - Datashards can exist *anywhere*.  You can put them on a usb key, you
   can stick them in a key-value web store, whatever.  I'm arguing that
   Secure Data Hubs are a good idea, except that they should just be
   "a common web interface to store your Datashards".  That's a big
   feature because then you have a URI so you can refer to those same
   objects univerally, the same way you an refer to an http uri, etc.

 - I think DB isn't yet convinced that we need symmetric encryption for
   *all* content-addressed content.  Again, I refer you to the podcast
   for why this is, but I think that content-addressed peer to peer
   systems that don't do symmetric encryption and chunking by default
   not only don't have a nice privacy layer by default, but they put
   node operators at risk.  (Imagine if you worked for a volunteer
   postal delivery service, but instead of packages and letters being
   wrapped in opaque paper, they were wrapped in syran wrap.  You might
   start to feel like you need to make moral judgements for what you're
   delivering, and even worse, making moral judgements may become so
   commonplace that eventually a law is enacted telling all postal
   service volunteers that they *must* examine contents before
   delivering them, and are likewise liable for their contents.
   Sometimes it's better to know less.)

We disagree on the four above points somewhat, but I'll argue that it's
really little to no extra cost (it may even be less) to switch the
Secure Data Hubs approach to the way I'm describing (being *a* place to
store Datashards as opposed to being content which can only live at that
node and can't be conveniently referenced outside of it), and the gains
are very great.

But, RWoT is a nice place for us to work on resolving these differences,
which I am convinced are really not so large, but the benefits of
resolving them are very high.  I look forward to working through them at
RWoT. :)

 - Chris

Christopher Lemmer Webber writes:

> Since I intend to bring up both in a significant way at Rebooting Web of
> Trust, it may be useful to hear that we put out two episodes on the
> podcast I co-host: one about Datashards:
>
>   https://librelounge.org/episodes/26-announcing-datashards.html
>
> and one about OcapPub:
>
>   https://librelounge.org/episodes/25-managing-spam-and-hate-speech-on-the-fediverse.html
>
> More about Datashards:
> https://github.com/WebOfTrustInfo/rwot9-prague/blob/master/topics-and-advance-readings/datashards-rationale.md
>
> More about OcapPub:
> https://gitlab.com/spritely/ocappub/blob/master/README.org
>
> You may wonder how Datashards and Secure Data Hubs can fit together.
> For that, I will follow up to this email with a rant. ;)

Received on Saturday, 17 August 2019 13:36:07 UTC