Re: Is there a pure JS implementation of the WOFF2 Brotli codec?

On 15/7/15 17:46, Mike Kamermans wrote:
> Yeah,that's the conclusion we had to draw too - debugging the running
> process revealed that the emscripten'd library was allocating three
> massive array buffers (~313MB each), just by loading the library, so
> that's either a nasty emscripten bug, or could be a deferred allocation
> pattern that works in C++ but turns into instant allocation in JS.

Note that Brotli makes use of a very large static array, of which there 
are separate copies in the encoder and decoder sources; see

   https://github.com/google/brotli/blob/master/dec/dictionary.h
   https://github.com/google/brotli/blob/master/enc/dictionary.h

If the emscripten'd library is eagerly loading this into an array, and 
doing so separately for the encoder and decoder, that could easily 
account for at least a couple of massive array allocations. (Why three, 
though? No idea offhand.)

JK

>
> That said, a pure JS implementation would be great as "reality check"
> too, so that for raw speed a (fixed) emscripten'd library is great, but
> for sanity, a secondary implementation is available to hold emscripten
> accountable to.
>
> - Mike
>
> On Tue, Jul 14, 2015 at 7:37 PM, Robert O'Callahan <robert@ocallahan.org
> <mailto:robert@ocallahan.org>> wrote:
>
>     On Wed, Jul 15, 2015 at 11:25 AM, Mike Kamermans <nihongo@gmail.com
>     <mailto:nihongo@gmail.com>> wrote:
>
>         On the off-chance that anyone knows, fontkit
>         (https://github.com/devongovett/fontkit) is currently using an
>         emscripten'ed version of the Brotli codec for handling WOFF2
>         decoding, which turns what is a 34MB footprint process without
>         WOFF2 decoding into a ~1000MB footprint process with... making
>         it rather unusable for dealing with WOFF2 files in any
>         meaningful production setting.
>
>
>     That overhead sounds way too high. I guess something is wrong with
>     emscripten or some other part of the workflow.
>
>     Rob
>     --
>     lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe
>     uresyf toD
>     selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
>     rdsme,aoreseoouoto
>     o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or
>     rnea lurpr
>     .a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt
>     sstvr  esn
>
>

Received on Wednesday, 15 July 2015 17:07:52 UTC