RE: Simulation Results from the Adobe Dataset

This is great work guys, and very good news – thank you for sharing!
We will discuss this in details at our WG call next week. The numbers obviously speak loud and clear in favor of PFE, but I was surprise to see that on slow connections transferring whole font (large one, I might add!) in a single request could be a better option than PFE – this is completely counter-intuitive until one starts thinking of all underlying details.

Thank you very much!
Vlad


From: Garret Rieger <grieger@google.com>
Sent: Wednesday, April 8, 2020 8:42 PM
To: w3c-webfonts-wg (public-webfonts-wg@w3.org) <public-webfonts-wg@w3.org>
Subject: Simulation Results from the Adobe Dataset

Adobe provided us with data about the performance of their progressive font enrichment system. I was able to use that data to run the simulations for all of the different font transfer methods and then compare those to how Adobe's system performs.

You can find the results of the simulations attached to this email. The various transfer methods are scored on a two different metrics:
1.     Percent of bytes transferred vs. the optimal strategy. Lower is better.
2.     The total cost from the simulation (details on the cost function<https://protect-us.mimecast.com/s/WzLnC68vyBsr0ZBphpG6St>) for each set of network parameters. Lower is better.
My takeaways from the data:

  *   The 3 progressive enrichment strategies (Optimal, Patch subset, and Adobe's solution) all produce significant reduction in bytes transferred compared to current state of the art transfer methods. For example unicode range transferred 7.6 times more bytes than patch/subset.
  *   Both Adobe's solution and Patch/Subset are pretty close to optimal with Patch/Subset transferring an additional 19%, and Adobe's solution an additional 45% versus the optimal transfer bytes.
  *   Looking at the total cost for each method and network config it looks like PFE methods are usually beating unicode range and whole font methods except on connections with very high RTT's. This makes sense to me since PFE methods are typically going to result in more network requests. In a high RTT environment limiting the number of requests by using a single large font seems to be the winning strategy.
So overall these results are pretty good looking and validate that progressive font enrichment approaches could materially improve the font loading experience for users.

Some notes on the setup for the simulations:

  *   As noted these simulations use exclusively CJK fonts. In particular it uses the set of open source CJK families available on adobe's service the majority of which are from either the Source Han Sans/Serif or Noto Sans/Serif CJK collections.
  *   The network parameters used for the different network configurations are those defined here: https://github.com/googlefonts/PFE-analysis/blob/master/analysis/network_models.py<https://protect-us.mimecast.com/s/htisC73wzpCAVnJ8uWox1e>
  *   The cost function used for scoring the simulation results is the one here: https://github.com/googlefonts/PFE-analysis/blob/master/analysis/cost.py<https://protect-us.mimecast.com/s/l3mxC82xAMh6PBK2S2CYlD>
  *   All of the CFF fonts were de-subroutnized before being used in the simulation. This produced the lowest overall transfer sizes.
I'm currently working on running simulations on our own internal data set that covers a number of languages beyond CJK. I hope to be able to publish those results in the near future.