experiments

The Multi-Armed, Multi-Fingered Bandit: Implementing a Bandit Algorithm in a Multiple-Payout Scenario

August 23, 2016 Tanner Thompson 1 Comment

A/B tests—whether you love them or hate them, if you’ve ever tried to sell something on your website, you’ve probably heard of them. That’s because they’re a straightforward way to find out what’s profitable and what’s not. However, the word testing tends to make some people (*cough* management *cough*) understandably…

Disaster-proof A/B Testing, Pt. 1: Managing metrics

June 23, 2016 Brian Campbell 0 Comments

Almost every product change made here at Lucid is first validated by an A/B test. Over the years, we’ve made a lot of mistakes while running A/B tests and writing the code for managing them. In this series of posts, we’ll be sharing what we learned from these mistakes and…

Disaster-proof A/B Testing, Pt. 2: Ensuring a consistent experience (on 750+ arms)

June 22, 2016 Brian Campbell 0 Comments

When setting up an A/B testing framework, pay special attention to how users are assigned to different arms of a test (and which metrics you track). If you aren’t careful, you’ll give the same user a different experience each time they go through a tested flow. This creates a problem…

Testing our homepage with KissMetrics

November 20, 2014 Brian Pugh 1 Comment

We recently used KissMetrics to measure the performance of our upcoming homepage test, which I wrote about on the KissMetrics blog. In that post, I mentioned my method of working backwards from your desired results to determine which reports to run. Once you know the latter, you can work out…

Research-Driven Redesign

October 7, 2014 Matt Swensen 1 Comment

April 2014 was a time of transition for our rapidly growing team here at Lucid Software. Having just received a round of funding, the office was abuzz with excitement at the prospective growth and change. As part of this transitory phase, we recognized that our homepage was dated and in…