Monday 17 March 2008

Benchmarking Mono-Compiled LSL

After Robby's response to my post about listener lag, I had the idea to see just what sort of speed improvements Mono will bring. I made a fairly simple benchmarking script that does the following tasks:
  • Integer addition (7000 times)

  • Float addition (7000 times)

  • Integer multiplication (7000 times)

  • Float multiplication (7000 times)

  • Integer division (7000 times)

  • Float division (7000 times)

  • Grow a list by repeatedly inserting [0] (300 times)

  • Randomize a list and sort it (100 times, list size 300)

  • Break a string into a list of words and check whether each word is equal to "hello" or "world" (200 times)

The tasks are pretty basic. That last one is generally what a listener would do. I chose the number of times each would be repeated so that every task would take about 10 seconds.

In doing this, I discovered that the performance of standard LSL scripts is extremely variable. I configured the numbers so that each task would take 10 about seconds; at this point I was at my house on the normal grid. But when I logged into the beta grid and ran the script in a Mono sim (without compiling to Mono), the arithmetic tasks each took about 3 seconds. Thinking maybe for some reason normal LSL scripts also ran faster on Mono sims, I tried running it at various other sims on the Beta grid. In some it ran at 10 seconds, others 3 seconds, others 17 seconds. I couldn't really see a pattern (Havok1 vs. Havok4, Mono vs. non-Mono), although it was always 3 seconds on the Mono sims, but I did get 3 seconds once on a non-Mono Havok4 sim. The numbers fluctuated between sims, but if I ran it multiple times on the same sim, I would get about the same results. Are all the sims on the beta grid Tier 5? It probably just comes down to how much other stuff is going on in the sim. But the last 3 tasks were always about the same time...very strange.

Anyway, here are the results. For the non-Mono numbers, I chose the ones closest to 10 seconds.

TaskNon-MonoMono% IncreaseTimes faster
Adding integers12.3564650.1333509166%92.662
Adding floats12.6435000.06676118838%189.385
Multiplying integers12.3109930.06660118385%184.847
Multiplying floats12.5562960.04441528170%282.704
Dividing integers12.3553540.06672718416%185.163
Dividing floats12.5559550.04492527848%279.487
List insertion10.0187190.000000N/AN/A
List randomize/sort8.9050110.000000N/AN/A
String tokenize9.6408860.1388146845%69.452
Total103.3431790.56159318302%184.018
















So the claims that Mono is over 1000 times faster don't seem to hold up. But still...this is just amazing. I laughed out loud when the benchmark finished in under a second when I had just waited over a minute and a half for the previous one to finish. I especially like how the list operations were too fast to measure. I am going to make some changes tomorrow so that I can actually get some numbers that I can analyze. I also want to add some more tests that better simulate "real-world usage". If you have any ideas, please tell me.

2 comments:

Anonymous said...

I should think that another thing to test would be listens. Actual open and unfiltered listens.

Create an announcer object that announces a predefined "start message", a thousand bogus messages, and a predefined "finish message".

Then create a listener object that tracks the amount of time between the start and finish messages, and sets it as the hovertext or announces it on the debug channel or something for easy reference. To really test it, they should all listen on the same channel with no filter parameters set.

Then see if there's any significant difference in timing when there's one listener object as opposed to, say, 40 of them.

Then again, it's nearly my bedtime and that idea may only sound useful to my own tired brain, but it would seem that it may have some slight chance of demonstrating something interesting.

If it does demonstrate something interesting, then I'd go and add filters for the start and finish messages to the listener script, and see if there's any significant change.

Anonymous said...

Interesting numbers, by the way, and much more in line with what limited testing I did. I admit that I was less rigorous in my own testing after discovering that Mono is still horribly broken, and even today (several updates after the last time I tested) it was absolutely not up to the level required to compile and run C:SI scripts.

To see what I mean, check out this comment I left on my own blog after testing on a Mono sim this morning : http://www.daikonforge.com/webtab/2008/03/15/csi-weapons-breaking/#comment-573