- From: Norm Tovey-Walsh <norm@saxonica.com>
- Date: Wed, 13 Jul 2022 09:12:13 +0100
- To: ixml <public-ixml@w3.org>
- Message-ID: <m28rox8lsy.fsf@Hackmatack.fritz.box>
Hello,
For your amusement, here are the results I got running Michael’s recent
performance tests (the a-star tests) through my Earley and GLL parsers.
These results are unscientific on two fronts: first, I ran the tests
from an old version of the test-catalog.xml file that Michael
subsequently corrected in the PR, and second, I did this on a laptop,
running them in parallel, so there could be a lot of variation depending
on when the laptop decided to throttle the CPU to keep it cool, etc.
I ran each test individually, starting a new VM, so that latter tests
wouldn’t be penalized by GC from previous tests. Given that my
implementation isn’t multi-threaded and my machine has several cores, I
expect that running them in parallel had little impact on the performance
(modulo CPU throttling, other processes, etc.)
Cells marked “-” indicate that the test didn’t complete. The “% diff”
column is an attempt to indicate how much faster the GLL parser is.
Negative numbers indicate that Earley was faster by that percent.
|------------+------------+-------------+-----------|
| test | Earley | GLL | % diff |
|------------+------------+-------------+-----------|
| base | 0.34s | 0.40s | -15.00 |
| G.d2.m.lc | 0.35s | 0.41s | -14.63 |
| G.d2.m.sc | 0.35s | 0.53s | -33.96 |
| G.d2.m.ws | 0.38s | 0.46s | -17.39 |
| G.d2.s.lc | 0.38s | 0.51s | -25.49 |
| G.d2.s.sc | 0.36s | 0.48s | -25.00 |
| G.d2.s.ws | 0.48s | 0.55s | -12.73 |
| G.d3.m.lc | 1.87s | 0.59s | 316.95 |
| G.d3.m.sc | 0.69s | 1.48s | -53.38 |
| G.d3.m.ws | 4.38s | 9.29s | -52.85 |
| G.d3.s.lc | 25.54s | 0.58s | 4403.45 |
| G.d3.s.sc | 2.09s | 2.89s | -27.68 |
| G.d3.s.ws | 52.65s | 58.31s | -9.71 |
| G.d4.m.lc | 17m6.83s | 1.28s | 80221.09 |
| G.d4.m.sc | 1m9.04s | 5m16.61s | -78.19 |
| G.d4.m.ws | 56m47.80s | 1h40m22.77s | -43.42 |
| G.d4.s.lc | - | 1.21s | |
| G.d4.s.sc | 15m57.37s | 12m24.77s | 128.55 |
| G.d4.s.ws | - | - | |
| G.b1.m.lc | 0.35s | 0.50s | -30.00 |
| G.b1.m.sc | 0.34s | 0.48s | -29.17 |
| G.b1.m.ws | 0.35s | 0.48s | -27.08 |
| G.b1.s.lc | 0.36s | 0.73s | -50.68 |
| G.b1.s.sc | 0.37s | 0.65s | -43.08 |
| G.b1.s.ws | 0.37s | 0.74s | -50.00 |
| G.b2.m.lc | 0.37s | 0.79s | -53.16 |
| G.b2.m.sc | 0.35s | 0.69s | -49.28 |
| G.b2.m.ws | 0.41s | 0.81s | -49.38 |
| G.b2.s.lc | 0.37s | 0.72s | -48.61 |
| G.b2.s.sc | 0.43s | 0.73s | -41.10 |
| G.b2.s.ws | 0.38s | 0.72s | -47.22 |
| G.b3.m.lc | 0.39s | 0.71s | -45.07 |
| G.b3.m.sc | 0.37s | 0.80s | -53.75 |
| G.b3.m.ws | 0.39s | 0.76s | -48.68 |
| G.b3.s.lc | 0.39s | 0.71s | -45.07 |
| G.b3.s.sc | 0.36s | 0.78s | -53.85 |
| G.b3.s.ws | 0.43s | 1.01s | -57.43 |
| G.b4.m.lc | 0.37s | 0.76s | -51.32 |
| G.b4.m.sc | 0.38s | 0.91s | -58.24 |
| G.b4.m.ws | 0.45s | 1.20s | -62.50 |
| G.b4.s.lc | 0.56s | 0.91s | -38.46 |
| G.b4.s.sc | 0.41s | 0.89s | -53.93 |
| G.b4.s.ws | 0.85s | 1.82s | -53.30 |
| G.b5.m.lc | 0.43s | 0.68s | -36.76 |
| G.b5.m.sc | 0.41s | 0.87s | -52.87 |
| G.b5.m.ws | 0.79s | 1.47s | -46.26 |
| G.b5.s.lc | 2.01s | 0.60s | 334.83 |
| G.b5.s.sc | 0.50s | 1.28s | -60.94 |
| G.b5.s.ws | 2.92s | 5.90s | -50.51 |
| G.b6.m.lc | 0.91s | 0.87s | 104.60 |
| G.b6.m.sc | 0.50s | 1.79s | -72.07 |
| G.b6.m.ws | 2.22s | 5.77s | -61.53 |
| G.b6.s.lc | 7.35s | 0.89s | 825.84 |
| G.b6.s.sc | 1.13s | 2.08s | -45.67 |
| G.b6.s.ws | 15.08s | 24.39s | -38.17 |
| G.b7.m.lc | 4.20s | 1.10s | 381.82 |
| G.b7.m.sc | 0.95s | 3.26s | -70.86 |
| G.b7.m.ws | 9.08s | 28.88s | -68.56 |
| G.b7.s.lc | 53.02s | 0.85s | 6237.65 |
| G.b7.s.sc | 3.69s | 5.33s | -30.77 |
| G.b7.s.ws | 1m55.86s | 1m58.11s | -1.91 |
| G.b8.m.lc | 19.20s | 0.80s | 2400.00 |
| G.b8.m.sc | 2.54s | 13.02s | -80.49 |
| G.b8.m.ws | 1m1.27s | 2m13.37s | -54.06 |
| G.b8.s.lc | 7m27.34s | 0.76s | 58860.53 |
| G.b8.s.sc | 21.49s | 20.56s | 104.52 |
| G.b8.s.ws | 15m10.14s | 12m37.35s | 120.17 |
| G.b9.m.lc | 3m15.25s | 1.00s | 19525.00 |
| G.b9.m.sc | 9.82s | 1m12.58s | -86.47 |
| G.b9.m.ws | 8m39.05s | 15m9.09s | -42.90 |
| G.b9.s.lc | 1h1m59.03s | 0.93s | 399895.70 |
| G.b9.s.sc | 2m29.46s | 2m11.77s | 113.42 |
| G.b9.s.ws | 2h20m8.87s | 1h48m20.07s | 129.37 |
| G.b10.m.lc | 19m38.45s | 1.38s | 85394.93 |
| G.b10.m.sc | 1m15.29s | 6m39.73s | -81.16 |
| G.b10.m.ws | 56m59.30s | 1h43m44.15s | -45.06 |
| G.b10.s.lc | - | 1.34s | |
| G.b10.s.sc | 19m55.89s | 14m20.23s | 139.02 |
| G.b10.s.ws | - | - | |
|------------+------------+-------------+-----------|
As I said durning the call, I’ve done no optimization on my GLL parser
and it’s consistently a bit slower then Earley, except when it’s orders
of magnitude faster :-)
Be seeing you,
norm
--
Norm Tovey-Walsh
Saxonica
Received on Wednesday, 13 July 2022 08:22:11 UTC