What A/B testing solution should I use for my Rails App?

Update Jan 2015: I wrote a monster guide on getting started with A/B testing, which you may find useful.

So you probably visited this page in your quest for an A/B testing solution for your Rails application. There’s a plethora of options, but the bottom line you are probably asking is: Should I roll my own/use an existing gem? Or should I use third-party software like Optimizely or Visual Website Optimizer?

I’ve used both, and in the rest of this article I’m going to chat about what works and what doesn’t. This is surprisingly a complex question.

Who is the Primary User?

If you work at a startup, you need an A/B testing solution, the first step you should determine is, who will be using this?

If it’s the marketing team (and they aren’t technical), you need to choose a solution that they can use and manage. Fortunately, the landscape of software has evolved quite a bit, so it’s now possible to run experiments without writing code. I’ll chat more about this later.

If you’re an engineer running the tests, you could probably whip up your own solution quickly.

The bottom line is that the software needs to be geared towards the end-user who will be using it the most.

How often will you be running the A/B Test?

This is very self-explanatory, but if you don’t receive enough traffic to warrant a change at least once a week, I would recommend rolling your own solution or using a popular gem. Unfortunately, the state of A/B testing gems isn’t that great, but I’ve found a couple that I enjoy:

Split – this gem is pretty popular, and uses Redis to store which variation is presented to the user. It also has the ability to mount a dashboard, so you can select a winner directly from a GUI.
Simple ABs – I really like this gem, it uses cookies to keep track of the variation presented to the user, and is extremely simple to setup.

I personally wouldn’t worry about tracking a winner (even though it has it built-in) with the Simple ABs gem – instead I would track it in Mixpanel or Kissmetrics. Just setup a funnel and tag each variation with an parameter (read my post on this.)

How much are you willing to spend?

This shouldn’t be a huge concern, simply because A/B testing is one of the best ways to make your company more money. If you are just starting out and cash is a problem, use one of the gems listed above. If you are a growing company, spend the money and get software you can trust.

Here’s the pricing for some of the popular 3rd party options:

Optimizely – starts at $20/month (2000 visitors)
Visual Website Optimizer – $50/mo (10,000 visitors)

Importance of Sample Size

This is extremely important – oftentimes you will run a test, and there’s no clear winner. So what do you do? You probably let it keep running so you can effectively determine statistical significance.

That’s a good idea in theory, but it can cost some serious $$. In my opinion, this is the single-biggest reason why I recommend building your own solution.

Here’s the tradeoff you might face:

You need a solution that enables the marketing department to run tests without requiring engineering resources.
You need a solution where you aren’t worrying about hitting quota (or exceeding it) every month.

No technical experience necessary?

I will agree that the state of AB testing has gotten much better over the past few years, but the messaging I believe is a little misleading.

First, simple text changes and button colors CAN be performed without a knowledge of code using software like Optimizely, but the more complex the experiment gets, the more likely using your own solution will work better.

The startup world loves to talk about how button colors can increase conversions by X percent, but the majority of tests with significant learnings are results of tests with more than one variable (also known as a multi-variate test.)

Most savvy marketers start off experimenting with what I would call “polar opposite” tests – it’s like throwing something against the wall and hoping it sticks. After running a few of these tests, you can take the winner and start performing A/B tests (like button colors.)

Oftentimes these tests require more engineering, which is why I lean towards using your own solution.

In conclusion

It’s because of the reasons above that I would recommend using your own solution/existing gem.

If you have enough traffic to warrant a change every few days, you should hire someone who can manage everything around setting up, running, and tracking performance of your test.

Lastly, my goal isn’t to complain about A/B testing software, but instead raise awareness of important questions you should ask. I hope you found this article informative!