IE9, FF4 beta in real-world benchmark

Posted on by Ben Dilts

Most browser benchmarks are isolated, artificial tests that can be gamed by browser vendors optimizing those specific cases.  With only those benchmarks to go on, we were skeptical that IE9′s beta would actually outperform other modern browsers in real-world applications.

To separate hype from reality, we built our first browser benchmarking tool based in Lucidchart itself.

Here is a quick video of one test run of the benchmark.  The benchmark works by simply dragging a part of the diagram around the page for five seconds.  We measured the total frames per second over that period, plus how the frame rendering time was split between actual javascript execution and the “inter-frame” period.

Javascript engines have become so fast that the majority of time is now spent in between render calls for all the browsers we tested.  This is the time the browser spends actually drawing the page we set up during our render code, along with whatever other miscellaneous things may delay the next frame from starting.

Numbers and analysis are after the break.

Lucidchart FPS BenchmarkLucidchart Render Time Benchmark

Update for angry Opera torch & pitchfork mob: I re-ran this benchmark with Opera 10.62, and it showed about a 20% improvement in inter-frame performance, with Javascript speed about the same.  That took its average FPS from 63 to 76.  It’s still solidly in (a very respectable) second place.

The big picture

Chrome is the obvious winner in this benchmark.  While Opera, Safari, and the IE9 beta approach Chrome in raw Javascript speed, those browsers drop far behind by taking too long to actually draw the image set up by the Javascript and get back into rendering the next frame.

Surprisingly, Firefox 4.0 Beta 6 came in behind all other browsers except for IE8–even coming in behind its predecessor, Firefox 3.6.  While we expect its performance to improve before release, it was surprising to see this result.  It should be noted that no Firefox 4.0 Beta 6 test run came in faster than any test run by another modern browser.

IE9 Beta 1 performed surprisingly well (with a few caveats; see the notes below).  In fact, it performs well enough that Lucidchart will finally be able to remove the obnoxious warning to IE users that they’re getting an inferior experience in Lucidchart.

Testing methods

These tests were conducted on a Windows 7 x64 laptop running a Core i7 Q720 CPU with 4GB of DDR3 RAM.

For each browser tested, we closed and relaunched the browser before each trial, and the window size was consistent across all trials.  The document for the benchmark was very similar to our standard business process template, of moderate complexity.  We ran four trials for each browser, and the results shown here are averaged across those trials.

Some final notes on Firefox 4 Beta 6

In several trials on Firefox 4 Beta 6, performance absolutely tanked due to the browser doing some unknown background work.  These trials were discarded for the purpose of this data, but it should be noted that Firefox 4 may have inconsistent performance for certain types of modern web applications, like Lucidchart.

Some final notes on IE9 Beta 1

When we first attempted the benchmark in IE9 beta 1, we got results only about twice as fast as in IE8, and many times slower than all the other browsers tested.  This actually led us to fix a minor flaw in our rendering algorithm that had us performing many more DOM operations that we needed to during rendering.  Removing that flaw let IE9 catch up to most of the other browsers (Chrome excluded).

So while Javascript and rendering performance in IE9 are respectable, DOM manipulation seems to be nearly as slow as ever.  This benchmark should be taken for what it is–one representative example of one modern web application’s performance in these browsers.  IE9 performance does drop precipitously under certain workloads.

Run the benchmark yourself

This benchmark can be run by anyone in Lucidchart.  First, sign up for a free account here.

Then create a document that you’d like to benchmark.  Right-click the block you’d like to have dragged around the page by the benchmark, and click “Bring to Front.”  The benchmark uses the topmost block for its test.

Then click the tools menu, and select “Benchmark Performance.”

This entry was posted in Tech Talk. Bookmark the permalink.

51 Responses to IE9, FF4 beta in real-world benchmark

  1. Pingback: Tweets that mention IE9, FF4 Beta In Real-World Benchmark -- Topsy.com

  2. Pingback: IE9, FF4 Beta In Real-World Benchmark | TheComputerSoftware.Net

  3. Pingback: World Wide News Flash

  4. Pingback: IE9, FF4 Beta In Real-World Use Face-Off | JetLib News

  5. Boris says:

    The way the intra-frame time is being measured here is broken. See http://news.slashdot.org/comments.pl?sid=1791422&cid=33617588

    The short story is that the “1ms” interval is not actually respected by any browser, so you’re just measuring what browsers clamp it to, not how long browsers are taking to actually _do_ something.

  6. sgunhouse says:

    Not certain why you’re testing with an old version of Opera. The latest stable release is 10.62, the latest unstable build (not technically a beta, but not really a snapshot as other companies use that term) is 10.70 build 9049.

  7. Sachin G says:

    I do not understand why do you always compare opera 10.53 with chrome 6 instead of opera 10.62?? And then people claim chrome is faster and better blah blah …

  8. Salford Waxing says:

    What, specifically, did you change to make IE9 faster? Was it something along these lines
    http://mir.aculo.us/2010/08/17/when-does-javascript-trigger-reflows-and-rendering/
    ?

  9. MarkG says:

    I see what you did there. You found out Opera 10.62 won, so decided to use an older version of Opera in the tests…

    Anyone that thinks this benchmark has ANY credibility because they have cherrypicked new development versions vs older slower versions of other browsers is frankly, a fool.

    Opera 10.62 is 50% faster than Opera 10.5

    http://www.itwire.com/business-it-news/technology/40189-opera-1060-claims-faster-javascript-execution

  10. Pingback: Android OS news » IE9, FF4 Beta In Real-World Use Face-Off

  11. WOFall says:

    I’m also disappointed to see you using Opera 10.53, where 10.6× has quite noticeable speed improvements. 10.53 was released around the same time as Chrome 5.
    Your basic methodology is skewed if you lucky-dip the browser versions to test. If we’re to presume you just used whatever you happened to have installed, then why, for instance, should we think you used factory settings?

  12. Pingback: Comienza la batalla de las betas IE9 beta contra Firefox 4.0 | MuyWindows

  13. Lennie says:

    You seem to be comparing release versions with beta versions.

    For example you should not be testing Firefox Beta already, it doesn’t matter what performance it has now. Because the beta does not yet include the new JavaScript engine which will be in the release version.

    You should rerun the test in a few weeks when the beta does include the engine. If you want to be notified when this is, just mail me.

  14. Pingback: Batalla entre Internet Explorer 9 y Firefox 4.0 : Noticias2D

  15. gwlaa says:

    this comparison is not making sense because older version of opera is used.

  16. Phil Dokas says:

    I’d be very curious to see the performance of a build of Firefox from http://nightly.mozilla.org/js-preview.html run on this benchmark. The “JS Preview Builds” are where the Firefox Jägermonkey team’s output is enabled. For more information you can read this blog post by one of its devs. JM is not yet available in the mainline 4.0 betas but if I understand correctly it is slated for the 4.0 release.

  17. Pingback: إكسبلورر 9 يتفوق على فايرفوكس 4 | تيدوز

  18. lol says:

    this test is a joke

    chrome 6 and more than a half a year old opera, 10.53

    I lol’d, go away

  19. Pingback: [LC] IE9, FF4 Beta In Real-World Benchmark. - Overclock.net - Overclocking.net

  20. Pingback: إكسبلورر 9 يتفوق على فايرفوكس 4

  21. Opera Man says:

    Hey thanks for using the wrong version of Opera, pretty much invalidates all of your data… do it again and use the correct version.

    http://opera.com

  22. Tyler Rasmussen says:

    The one thing IE9 claims to have done right that the average frames per second test reveals Chrome has done wrong is IE9 has supposedly capped its fps at 60. Whether capped at 60 or 75 (with an advanced option to change the cap or remove it), I don’t know many (if any) monitors that refresh at a faster rate. Chrome is wasting processing power at nearly 120 fps.

  23. monty says:

    You should test Opera 10.62, not old and slower 10.53 version

  24. monty says:

    Oh, and BTW – it’s not “Real-World Benchmark” when you use high-end machine (seriously, how many people have i7?) to run it.

  25. Tom says:

    Opera fanbasementdwellers. Sigh. I love how you’re all so adamant that his data are useless because it’s an older version of Opera, or that it’s an intentional plot to make Opera look bad. w3schools.net browser statistics have Opera at 2.3%, a relatively small userbase. Perhaps he should just leave Opera out next time? I personally don’t know anyone who uses Opera, possibly because Opera users are as intolerable as you.

    I’m sure you were too infuriated to notice, but you were given a means to test whatever browser(s) you want at the very end of the post.

  26. Ben Dilts says:

    I’ve updated the post with a note on Opera 10.62. To those who think I intentionally used an “old, slow” version of Opera to oppress that minority: You’re right, everyone really is out to get you. (WTF?)

    To those who more politely requested an updated benchmark, I retested with 10.62, and saw the average FPS improve from 63 to 76. It’s still solidly in second place, so I won’t update the graphs until the next time I run a set of benchmarks.

  27. richtaur says:

    Really good stuff. I’m not surprised at all to see Chrome in 1st but I am disappointed to see Firefox 4 so low. Thanks for putting this together!

  28. Boris says:

    Why not adding konqueror to this benchmark ?

  29. Geoff Strickler says:

    How about testing one a more mainstream machine or laptop (since represents more >50% of users now)? I presume you had a DirectCompute capable GPU in the machine, which IE9 will use for it’s rendering. I’m glad that MS is finally doing some much needed work on IE, but you need to provide more information and/or test on a mainstream dual-core desktop or laptop for this to be of any value to the public.

  30. Pingback: Internet Explorer 9 Beta Released | Full On Design

  31. Kris Tilford says:

    You should also run these benchmarks under other OS’ such as Ubuntu Linux and Mac OS X. You can install both these onto your laptop easily enough, and both support Chrome, Opera, and Firefox. Safari under OS X seems very different to me than under Windows, so I believe the OS will make a difference in the benchmark scores, but only a real world test will tell.

  32. Justin says:

    Boris is one of the star Mozilla hackers. If he says your benchmark isn’t measuring what you think it’s measuring, he’s probably right. Come join us on IRC if you want to know more…

  33. Andrew Cartwright says:

    @Tom, @Ben with all respect, the people posting about Opera 10.53 were in fact quite polite and have a very good point.

    Ben it seems somewhat perplexing that you would go to the effort to run the tests using the latest versions of certain browsers (in point IE9, FF4 which are beta and not even release) and not others and then post graphs based on these results.
    In a good experiment all variables should be controlled properly, this oversight does in fact bring into question the veracity of your methods.

    In addition your justification for not updating the graph to better reflect the corrected results is very odd and appears to be more of a reaction to being queried over why you did not use the most recent released versions of certain browsers (in this case Opera) in your tests. Surely it’s not that hard to update a couple of graphs?

    Either way, good article and good work.

  34. AnnoLoki says:

    Ben Dilts – I had to drop you a quick note to say good work so that the response to your article wasn’t entirely the whingers that have posted so far. What they have failed to recognise is that you DID include the version numbers of the browsers, and that the comparisons were accurately between those versions. You haven’t posted results of “Opera vs FF” etc claiming for them to be the latest versions, because in this field, doing so is futile, things change so quickly. People not liking what you’re saying is their own problem, it doesn’t make it untrue. Don’t let the emotional reactionaries with nothing better to do in their lives than whinge about the publication of facts get to ya, they’re not the most correct just because they’re the loudest :-) Anyway, just thought I’d try offer some balance.

  35. Ed says:

    @Tom “opera fanbasement dwellers” FTW.

    Your headline says “real world”, the real world doesn’t use Opera. Spare yourself the grief from these paranoid idiots and omit Opera next time, it is not a significant browser.

  36. Mikey P says:

    Nice effort. Can we either stick with released browsers or betas?
    Please include Chrome 7 and Opera 10.70.

    Basically you’re taking the two best browsers (One out of date version – Opera) and comparing everyone elses (Bar Apples appalling effort of a browser) betas.

    Reason unknown. Nor do I give a flying ape shit for these crap results. Compare apples with apples or oranges with oranges. Not apples with oranges.

  37. mhenriday says:

    Ben, glad to see that you ran the benchmark with the latest version of Opera ! But shouldn’t that courtesy also have been extended to Chrome ? The latest dev version of that browser is 7.0.517.8 dev ; even though Chrome 6 led the pack in your two tests, readers might have been interested in seeing how the latest and greatest version fared, given that you chose to test instable versions of both FF and IE….

    Henri

  38. thenonhacker says:

    Oh look! IE8 is slow in rendering according to the 2nd graph. So where is IE9 there? Ok, so you removed IE9 and retained IE8 to make it look bad right? Nice try.

  39. Phil Dokas says:

    Just as a small followup to my previous comment, Jägermonkey will be in Firefox 4.0b7 per http://hacks.mozilla.org/2010/09/firefox-4-recent-changes-in-firefox/. I’d love to see it tested alongside these other browsers.

  40. Eagle says:

    I tried latest browser versions in benchmark (ctrl+shift+B) “sales process” with i5 750:

    Opera 10.7 – 124fps
    Chrome 7 – 187fps
    Firefox 4b7pre (which already jagermonkey and HWA enabled) – 54fps
    Firefox 3.6.8 – 74fps

    Phil Dokas
    jagermonkey is already in 4b7pre and it don’t make firefox any faster in this test )))

    Opera and Chrome is best. Why most people used bad browsers?

  41. lucideer says:

    Warning: Pitchfork wielding comment below – ignore if desired

    For those criticising Opera users for being “reactionary”, etc. etc. I think it’s worth pointing out two fairly valid points. Firstly, of the 8 Opera-user comments above, only two are mildly rude, most are well-worded, informative and not in any way offensive, over the top (nor do they demonstrate the sarcasm of the italicised update to the blog post).

    Secondly, despite the fact you’re comparing an unstable Firefox (4) and an unstable IE9, to the stable Opera 10.6 and not the unstable 10.7, there’s been no rebuke from this apparently “angry, pitch-fork-wielding mob” – can you honestly say that if you’d used old version of ANY other browser (or even just the stable version of Firefox instead of 4) that there wouldn’t have been angry offensive comments of the same sort from “that” crowd here?

    I suppose what I’m getting at is… any other user-group would have reacted in the same way, or most likely in a far less polite, more reactionary way – is the sarcastic pitchforks comment in the blog update really necessary? Being a “marginalised minority” of users is one thing, being name-called in otherwise respectable blog-posts is another.

  42. physician assistant says:

    nice post. thanks.

  43. WP Themes says:

    Nice post and this mail helped me alot in my college assignement. Thanks you for your information.

  44. Pingback: Visit Beauty of the Web: IE9 Beta Released - Page 2 - Digit Technology Discussion Forum - Tech Discussion Forums in India

  45. yosef019 says:

    where is the gpu rending??
    which is beter in hardware not software???

  46. Izkata says:

    Whatever the reason behind IE9′s bug that caused you to change your test, the fact is you *changed your test* to make IE9 score better. And your goal was to create a benchmark test, not just a website.

    How many sites in the wild are using the version that kills IE9′s score? Is it more likely to happen than the “cleaned” version?

  47. pavan says:

    nomatter what the results shows… FF rocks as every1 thinks and feels differnetly and and hv diifferent style and d oly way 2 satisfy is using addons wich oly FF can deliver and none can do it as good as FF!! others myt hv but nt as much as ff!! also the diff in performance is just beyond the scope of human perception. and the aesthetic appeal also matters and the new features in ff jags up all the others !! so overall w8 for FF4 stable and c how it jags up all the others!!

  48. Pingback: IE9, FF4 Beta In Real-World Benchmark - DotA Forums

  49. Worker says:

    Hola,
    Interesante, no va a continuar con este artнculo?

    Worker

  50. IE9 actually runs very good when compared to mozilla or chrome. I went back to IE after the 9′th version went out.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>