Selenium IDE: The Good, the Bad, and the Ugly

Here at Lucid Software, we are always trying to release high quality code to users as quickly as possible. For every release, we spend three to five days doing a full regression to make sure the code is user ready. Three to five days is not as fast as possible. In an effort to release more quickly, we are always on the lookout for new software to help us improve our testing process. We encourage all members of the team to look around for new and interesting tools that will make testing at Lucid faster and more complete. This summer, as part of this effort to find new tools, one of our interns began experimenting with Selenium IDE.

Selenium IDE is an add-on for Firefox that allows anyone to create macros through commands and then translate them into reproducible steps with the Selenium Firefox Driver. Commands include clicking on a element, typing into a field, and dragging an element from one location to another. The add-on can be downloaded from the Mozilla app store with tutorials on how to get started.

Screenshot of Selenium IDE add on
Selenium IDE

To understand how we were interested in incorporating Selenium IDE into our development cycle, you first need to understand where our testing infrastructure stands as a company. At Lucid Software we have 13 full-time testers on two teams: a team of nine manual testers and a team of four testers dedicated to maintaining and improving our UI Automation framework. Lower level tests, such as integration and unit tests, are the responsibility of each developer. Additionally, all developers write some UI tests for new features they implement. As a relatively young company only five years old, we currently only have about half of our regression tests automated. Automating more of the regression tests is one of our top priorities company wide.

As part of this effort, we began experimenting with Selenium IDE. From our investigation and experimentation, we found out a great deal about Selenium IDE; hence, we present The Good, The Bad, and The Ugly.

The Good

  1. Fast setup. As a browser add-on, Selenium IDE can be set up very quickly. We have had four or five different developers and QA members try the tool, and all of them were able to get going in under 10 minutes.
  2. Simple interface. Selenium IDE doesn’t require a CS degree or knowledge of programming languages to set up most tests. In fact, it’s mainly just point and click. There is a whole list of possible asserts to look for when it comes to debugging, and it highlights red where the test fails. Overall, the tool is built for any web user with even the simplest understanding of testing.
  3. Quick turnaround time. Compared to writing Selenium tests with our framework, Selenium IDE was incredibly fast. We timed writing a simple test to make sure our Lucidchart editor loads. Even with most of the methods written and available in our homegrown Selenium framework, it took about ten minutes from the time the test was started until we were able to run it. With Selenium IDE, it took two.

The Bad

  1. Only runs in Firefox. About 25% of traffic to our sites is on Firefox, while almost 70% comes through Chrome. While it’s good for us to have tests in both Firefox and Chrome, we want our tests to mimic the workflows of the majority of our users.
  2. Working with Canvas elements is hard. I mean really hard. In both our products, the canvas element is about 80% of the editor’s real estate. With our homegrown Selenium solution, we make use of the Actions object to move things around on the canvas, click at locations, and drag and drop blocks. Without access to the Actions object in Selenium IDE, this became just a matter of trying to click at the correct offset from the canvas object to make selections and move them.
  3. Lack of conformity to a standard. Selenium IDE is an open source project, and it shows. The list of commands and syntaxes varies widely. Some just use the command field and a target. Others require the target to be listed in the value field. Still others require you to add specific fields or take some away. While the versatility is good, the lack of standard means that any tester needs to spend a lot of time learning the different commands and what they expect in order to do anything complicated. The simplicity and ease of the product is lost through poor adherence to standards. This problem will likely become worse as more commands are added.
Comic about how long it takes to fix tests being a negative side effect of automation testing
Source: http://simply-the-test.blogspot.com/2009_05_01_archive.html

The Ugly

  1. The waiting game. This is the number one complaint around the web with Selenium in any fashion. With Selenium IDE, it is even worse. Waiting on a site to execute Javascript requires the browser to wait for a particular action. In Selenium IDE, it tries to execute as quickly as possible. For example, as soon as a button gets added to the DOM, it tries to click the button, even if the browser is still calculating position and the button won’t work yet. You need to wait until the button is fully set up, but Selenium IDE moves on as soon as the button is added to the DOM.  In order to work around this problem, the options in Selenium IDE are using a loop and building out some retry code or simply putting in a wait command to tell it to wait a little longer.
  2. Access to the API. One of the advantages of our home grown Selenium framework is direct access to backend APIs. We use these to create users, add users to teams, create documents, and clean up at the end of the test. With Selenium IDE, we don’t have this access, so every single test creates a new user in our test environment that never gets cleaned up.
  3. Reliability. To be completely frank, simple tests flake out around 5% of the time. Once we ramped up a little and tried to do some complex testing, we found we were getting false positives about 20-25% of the time. On these tests, we spent hours trying to fine-tune waiting periods, change the speed of the test, and click on more clear components. Even with hours of effort, the best we could get was about a 10% fail rate.

The Verdict

Selenium IDE is a user-friendly, easy-to-use tool. You can set it up very quickly, and anyone can use it. At Lucid Software, we have decided to add Selenium IDE to our toolset for our Quality Assurance Specialists, but only in a limited capacity at the discretion of each team member. It was not reliable enough or full-featured enough for us to warrant porting out old UI tests forward or even making it a primary tool in automation. The cost to maintain was just too high.

XKCD about automation efforts
Source: https://xkcd.com/1319

Our Quality Assurance Specialists have found use for it in repeating common tasks that they do every day. They use it to set up team situations, set up a new account with specific A/B tests, or even just clean up old testing documents by going through all the documents and deleting them. These are all simple tasks that can fail without hurting anything, as the QA Specialist can see what failed and simply finish the process manually. Selenium IDE accomplishes the task faster and allows the specialist to focus on other things while the process is being run.

16 Comments

  1. Why just rely on the IDE. IDE is good for a quick test and as you pointed out simple and easier to setup. However it does not serve as a full fledged Automation Suite. I recommend using Selenium Web Driver instead, which is way better and the scripts from IDE can be quickly converted as per the need.

  2. >> The waiting game. This is the number one complaint around the web with Selenium…

    >> Access to the API. One of the advantages of our home grown Selenium framework….

    etc etc etc.

    All of these complaints about free software and not word one of how you, as a money making business, did anything to improve the situation.

    How much code have you contributed? Have you given the project any money for bug bounties or one off improvements?

  3. As a company, over the last year we have contributed to selenium several times. We also are active on the forums logging bugs, helping users work through issues, and answering questions on stack overflow. You are correct, we have never contributed to Selenium IDE. We had one intern spend a Summer experimenting with it and reported our findings for others who are interested in using it. The post was not to degrade or demean Selenium IDE in anyway. We tried it, it wasn’t what we hoped for, this is why.

  4. Hi Jared! Great article! I agree with the short-comings of Selenium IDE. We actually built a tool that automates testing of your local web sites or web application that addresses a lot of the issues you mentioned (the waiting game, browser compatibility). It just requires you to drop an html-fragment into your site and is totally java script based, so it runs on any browser. Also there is a feature where you can report bugs directly in the tool and the bug becomes sharable over the URL which is really useful for distributed teams.

    Moreover, it’s free to use. Sign-up here if you want to try it out: http://boozang.com. We recently launched so we would really appreciate any feedback.

  5. Because we experienced that Selenium IDE lacks a lot of base functionality, but overall Selenium/WebDriver is a great concept, we built our own IDE for Firefox, the Script Developer. We share it freely for test automation purposes with everyone – https://www.xceptance.com/en/xlt/features.html

    No license, no sign-up, no email required, just download. You can also download from the Mozilla Marketplace – https://addons.mozilla.org/en-US/firefox/addon/xceptance-script-developer/

    Feedback and questions are welcome at any time – our forum is open: https://ask.xceptance.de/

  6. Dmitry PashkevichDecember 9, 2016 at 10:49 am

    Thanks for sharing, Rene! We’ll check it out!

  7. Hello,

    I can’t believe no one has corrected you regarding firefox and selenium IDE. FYI, just make your test scripts in selenium IDE, save the test script and you can run it on chrome and IE as long as you ran it from command line. You also need to download selenium standalone server and chrome or IE driver to work with selenium. Just google the steps and you will be rewarded. If a farmer like me can do it, so can you !

  8. *Only runs in Firefox.*
    Regarding this one, it’s seems like you can actually use WebDrive playback to run tests from Selenium IDE in Chrome (according to this: http://bit.ly/2mWvlXI and this: http://bit.ly/2n6jExM). This only provides you with part of the functionality, but it’s better than nothing:)

  9. Steve PaulMarch 25, 2017 at 5:49 am

    @Anton, excellent point. I’m busy with frontend dev (Angular2) and the IDE is actually pretty awesome.

    Even as a developer, I prefer the ease of using the IDE to select elements and type commands to protractor. As ergonomic as protractor is, coding can never be as ergonomic as the IDE. Every now and then I have a little waitForCondition javascript which I execute, but most of the time its very declarative.

    I can’t vouch for the Lucid Charts HTML5 canvas usecase, however for CRUD apps, think of the IDE the most efficient tool for humans to translate UI actions into test cases. Those test cases can be exported (as code) and run on some test server as part of a suite against the webdriver.

  10. Selenium has a strong reputation on the IT market, so it will be hard to replace it anyway. But new ui testing tools offer you new possibilities and makes your testing process easier and faster. They cannot definitely replace a human check, but they help to make your testing more efficient.

    I would like to share our team’s experience. We have been using our own UI testing tool, Screenster (http://screenster.io), to test ours and our customers’ web apps. It has proven itself to be a helpful alternative to Selenium for visual/CSS testing tasks.
    Screenster is a test automation tool which performs screenshot-based comparison of different versions of your web pages. First it creates a visual baseline for a page, taking a screenshot for each user action. During the next run it takes a new screenshot at each step, compares it with the one from baseline and highlights differences.

    Summing it up, Screenster has the following advantages:

    Visual baseline: screenshots are captured for each user step during test recording
    Screenshot based comparison: Screenster compares images captured during a playback to those from the baseline and highlights all differences

    Smart CSS selectors: tester can select CSS elements on the screenshots and perform actions with them – e.g. mark them as ignore regions to exclude from further comparison

    Advanced test maintenance options: it is possible to add or delete steps, modify actions on each step, override tests with new parameters and so on.

    Screenster is available for free download or can be tried on its Demo server.

  11. Selenium IDE has stopped working since Firefox 55 (https://medium.com/@briananderson2209/in-search-of-the-selenium-ides-successor-33bc7eaefe7f). Selenium IDE is a legacy project dating back to the days of Selenium RC. The last time it got updated was in 2015. I don’t think someone will pick up the project and port it to Chrome.

  12. Dmitry PashkevichSeptember 15, 2017 at 4:23 pm

    Thanks for your comment @present88. Please note that this post was written a year ago 🙂

  13. Interesting article’s title 🙂
    By the way, there are some alternatives to Selenium IDE you may want to know:
    http://selenium-ide.weebly.com/

  14. After this, I got a reasonable thought regarding selenium ide.till now I searched for selenium and other robotization apparatuses, old however great one.If anyone has thought regarding this AnyAUT.com tool let me know. I found that one interesting.

  15. […] According to a user, when you use the software for complex testing, you may get false positives 25 percent of the time. And it didn’t do better with the simple tests either. The user claimed to experience flake out 5 percent of the time. […]

  16. Thanks for sharing.
    I will definitely give Selenium IDE a try.

Your email address will not be published.