At Lucidchart, we really care about making a great application that is easy to use and intuitive for first time users, yet powerful enough for the sophisticated diagrams and mockups that many people need to create. That means that we want Lucidchart to be fast, even when a diagram gets very large and complex. And because we run inside the browser, we are subject to the limitations of each browser vendor.
Internally, we’ve developed a benchmark that runs inside of the Lucidchart diagramming application. Our benchmark simulates many tasks that users typically do while diagramming, but on a much larger scale.
For example, we connect a single block to dozens of others with lines, then rotate the entire diagram around dozens of times; we also add text to shapes and lines, move and resize objects, and so forth. Our entire benchmark includes sixteen different sections, each corresponding to a set of tasks that represent different actions users perform when using Lucidchart.
What does this benchmark measure?
We designed our benchmark to determine if a new release of our software is going to slow down the user experience. Not only do we measure overall runtime, but we actually instrument a number of critical functions so that we know, for example, how many times a piece of text is re-rendered. If those numbers go up, then we know we’ve introduced a new bug that hurts performance.
We also use the benchmark to compare the performance of different browsers. With the recent release of Internet Explorer 10 on Windows 8 and some initial reviews of its performance (for example: here, here and here), we were interested to see how IE10 performance stacks up against the current versions of other browsers.
Most browser benchmarks, such as SunSpider, the V8 benchmark suite, and so on, focus on important but narrow low-level functionality, such as mathematical calculation speed. Our benchmark is not directly comparable to these other benchmarks, as we’re attempting to measure user-visible speed of an entire graphical application. We are measuring a much wider variety of performance characteristics to answer questions such as:
- How quickly does the canvas API execute?
- How does the browser perform when we are rapidly changing the size of canvases?
- How quickly can a very complex DOM including tens of thousands of elements be laid out and rendered?
- How quickly can we reorder, re-Z-index, re-style, render to, and move around thousands of elements in the DOM?
Enough already. Which browser is fastest?
The overall run time for the entire benchmark test in each browser is as follows:
We ran our latest benchmark on the same Windows 8 laptop. Each test was run on a fresh launch of the browser in question. Unfortunately, we were unable to test IE9 under the same conditions to see how it stacks up against IE10, since you can’t install IE9 on Windows 8.
Chrome still holds first place, but by a surprisingly small margin over IE10. Firefox repeatedly refused to complete the entire benchmark, crashing hard every time we attempted it. We gathered timing information for the parts of the benchmark that it did complete, and on those portions it performed comparably to IE10.
Is IE10 really that fast?
On a few of the sections, IE10 handily beats Chrome on timing. For example, in a test where we simulate typing 1500 characters into one of our custom rich-text editors, IE finished in 4.8 seconds vs. Chrome at 10.3 seconds. In a test of rotating a few dozen different shapes simultaneously, IE finished in 8.3 seconds vs. Chrome’s 17.3 seconds.
This chart shows the amount of time taken by each browser for each section of the benchmark (excluding Firefox, which could not complete the benchmark).
Chrome still holds the crown as the fastest overall performer for Lucidchart. IE10 made a surprisingly strong showing, performing far better than any other browser (though, under a more reasonable load, Firefox actually does quite well).
There is one more strange thing about IE10 that I should mention. IE9 often maxes out one CPU core when a Lucidchart diagram is open, even when the user is completely idle. IE10 is somewhat better, but on our test machine it still used about 50% of one core when the user is idle. No other browser has any detectable CPU load under the same conditions. Whatever the cause may be, this is the one issue that keeps us from recommending IE10 as a first-class experience for Lucidchart.