Re: Reference implementation anti-pattern (was: Re: New Work Item Proposal: Universal Wallet 2020)

I normally only lurk, but as a Java User Group coordinator for many years,
it is worth noting the definition of a "reference implementation" (RI) in
Wikipedia comes from the Java Community Process (JCP) in the early 2000's -
this worked because there was a large organisation (SUN Microsystems)
funding the development of the RI alongside the Specification and
Conformance Test Suite (this in turn feed into professional services and
commercial licences for productised versions of the reference
implementations of the Java technologies - but in the end it did not work
out so well for SUN :( ).


I agree that applying the term "reference implementation" outside the JCP
framework (and funding) is problematic (RI is one part of three elements)
.  Also worth noting the Characteristics of a Reference Implementation in
the source cited in Wikipedia, is from a presentation where the term "*Gold
standard*" is *italicised* ; and so I would assume it was presented with
some context, now lost.


For what it's worth; for me, "*model implementation*" fits for an open
source implementation intended to "illustrate" a specification for others
to emulate, be inspired, and/or extend. This differs from an implementation
not opened up to others, usually for commercial reasons, and flags that the
implementation is intended to follow the specification, rather then exceed
it, again usually for commercial/competitive reasons.

Definition of "*model*" : an example for imitation or emulation
https://www.merriam-webster.com/dictionary/model

/derek

(hope this just provides some paint to the bikeshed, rather then adding a
new room)


On Thu, 25 Jun 2020, 10:54 pm Manu Sporny, <msporny@digitalbazaar.com>
wrote:

> On 6/24/20 4:45 PM, Wayne Chang wrote:
> > Manu, could you please elaborate on your concerns about calling
> > certain code a "reference implementation"? What drawbacks are there,
> > and what else could we call it? "Example implementation"? "Conforming
> > implementation"?
>
> Thanks to Leonard, DanB, Dmitri, Kim, and DanielH's comments, which do a
> much better job than I would have elaborating on why using the term
> "reference implementation" is problematic.
>
> The biggest issue is the "gold standard" language in the Wikipedia
> definition, which I don't disagree with.
>
> The problem that we've seen play out time and time again in the global
> standards process over the last 25+ years is that "reference
> implementation" for algorithms work out well. Reference implementations
> for non-trivially complex systems, like what is being proposed in
> Universal Wallet 2020, end up with software that is:
>
> 1. Not maintained since it's often unfunded work.
> 2. Not suitable for production use (see #1 above), and thus is
>    dangerous if developers use it as a reference.
> 3. Politically unstable because of the power dynamics -
>    developers think that features implemented in the
>    reference implementation are blessed by the group, when
>    it may be that the developer that did the reference
>    implementation just through the optional feature was neat.
>    To combat this, committees form around reference
>    implementations and suck energy away from the spec.
> 4. Always behind the state of the market, because of #1 above
>    and the aforementioned need to "design by committee".
>
> Fundamentally, it was the failure of reference implementations that led
> to conformance test suites and ensuring that the specification was the
> gold standard (and not some piece of software).
>
> To put it another way, reference implementations for non-algorithmic
> standards have been an anti-pattern for a very long time. You'll note
> that many of the folks speaking against it have been around the block
> several times in the standards space -- they are speaking from
> experience, possibly having proposed the same thing in the past and
> having it blow up in their face. I know that's what's happened to me in
> every reference implementation attempt I've participated in. :)
>
> So, what do we call it? An implementation... because that's all it is.
> If it's good, it's a conforming implementation... hopefully, one of many.
>
> -- manu
>
> PS: Also, I'll note that this was far down on our list of concerns. We
> don't need to bikeshed it now. :)
>
> --
> Manu Sporny - https://www.linkedin.com/in/manusporny/
> Founder/CEO - Digital Bazaar, Inc.
> blog: Veres One Decentralized Identifier Blockchain Launches
> https://tinyurl.com/veres-one-launches
>
>