At the Forge

It's nice to have many people visit your Web site. It's even better when people don't just come to your site, but also enjoy your content. But, best of all is when visitors to your site do what you would like them to do—sign up for your newsletter, register for your SaaS application or buy one of your products.

The rate at which visitors become customers is called the “conversion rate”, and it's probably the top priority for Web-based businesses. If you can convert 10% of your visitors into customers, you're doing twice as well as the person who can convert 5% of visitors into customers.

What leads people to convert more often? That's a question to which I'm still learning the answer, as is an entire industry of Internet marketers and “conversion optimization” experts. The thing is, it's often difficult to know what will work. Should you use a green button or a red one? Should your headline be a question or a statement?

One of the most popular, and effective, ways to check the effectiveness of your copy is to do “split testing”, sometimes known as “A/B testing”. Although I heard about A/B testing years ago, I have only recently actually started to apply it.

In this article, I describe a simple way to run split tests on your Web pages. The examples I show here are written in Ruby, but there are A/B testing libraries for many different languages. There also are third-party businesses that can help you with A/B testing, either on your own Web site or on your mailing lists, lead pages and other products.

The most important thing to keep in mind when you're doing split testing is that you're trying to get users to do something. What that something is depends on your site. The goal at Amazon is to get you to buy things. The goal at Google is to get you to click on ads. And the goal for a mailing-list subscription form is to get people to sign up.

Once visitors have achieved your goal, they have been “converted” into customers. The goal is to increase the number of such conversions—giving you more customers, subscribers or users of your system. The key insight with split testing is that by changing the text, graphics and even layout of your page, the number of conversions will change as well. The question is, which of your various ideas will work best?

Split testing uses the scientific method, backed by simple statistics, to try to answer that question. In science, you can test a hypothesis by using a control and an experiment. Split testing works the same way. You take your existing text and an alternate text, and display them to your users. One of them probably will work better than the other. You analyze the numbers to understand which worked better, and then you use the best one. Then you start all over again, trying to find another improvement via split testing.

In order for split testing to work, you need to do several things:

Regular readers of this column know that I tend to use Ruby on Rails for most of my Web development work. Rails isn't the only Web framework written in Ruby, although it certainly is the most widely used. All modern Ruby-based Web frameworks use “Rack” to communicate with the HTTP server software that invokes them. This is analogous to the Python world's WSGI. The idea in both of these cases is that the HTTP server doesn't need to know much about the Ruby or Python application (or framework) and vice versa. This means libraries that know how to work with Rack can operate with any framework, not just with Rails. In the specific case of Ruby, Rack-sensitive gems thus can work with Rails, Sinatra or anything else that follows the Rack API.

The Split gem, written by Andrew Nesbitt (with a number of contributors), is one such package of code, able to provide split-testing execution and analysis to any Rack-compatible application server. It is based on a number of earlier split-testing gems, including the well-known A/Bingo gem by Patrick Mackenzie. The idea behind Split is you identify your conversion target, automatically try two or more variations on users, collect statistics on which of them actually managed to achieve more conversions, and then start to test things again.

Split takes care of much of this work for you, allowing you to concentrate on the portion of your site you want to change, rather than the mechanics of setting up experiments and reporting their results. Split also implements a number of features that ensure the statistical power of your results. For example, it will involve each participant in only a single experiment at a time, and it will compare results only after at least 30 people have seen it. These options can be changed, of course, but the defaults are more than good enough for most basic tests that you'll want to run.

You install split as you would any other Ruby gem:

gem install split -V

Note that I use rvm, the Ruby version manager, which allows me to work with multiple versions of Ruby at any given time. One side effect of using rvm is that gems are installed in my own home directory, rather than on the system. As a result, I don't need to preface the gem command with sudo; depending on your system configuration, you might need to do so.

After installing the gem, you also will need to install and start Redis, an extremely fast and popular key-value store. I have used Redis in a number of projects through the years, and I never cease to be amazed by its utility for caching. In the case of Split, Redis is used to keep track of the experiments you are running. You can start Redis by executing:

redis-server

Now that you have installed the gem and Redis, you're ready to perform some experiments. Let's create a simple Web site using Sinatra, in which the page displays a link. While Sinatra has a reputation for being simple and for allowing you to create a single-page application, I've generally gone for a configuration that uses several files: the application itself, a Rack configuration file (setup.ru) and a Gemfile to list all the Ruby gems I want to use (via the “Bundler” management gem). These are shown in Listings 1, 2 and 3.

source 'https://rubygems.org'

gem "sinatra"
gem 'split', github: 'andrew/split'

Let's review them briefly. The setup.ru file is the way that Rack runs your application. If you want to run the application without any external HTTP server, you can use the rackup command:

rackup config.ru

Rack then will run an HTTP server (defaulting to the built-in WEBrick server that comes with Ruby), acting as the glue between the server and your application. The config.ru basically bootstraps your application, loading the gems listed in the Gemfile (thanks to Bundler), then loading your application and the special code needed to run the Split dashboard.

Then you do something interesting. Rather than simply running your application, you tell Rack that it should route requests to different places. Anything starting with a / should go to LinkSplit.new, meaning a new instance of the Sinatra application. But, anything starting with /split will be routed to a completely separate Sinatra application, Split::Dashboard.new, part of the Split gem.

The Sinatra application defines two different URLs, both of which are available via HTTP GET requests. The first, /foo, displays the contents of the foo.erb file (located in views/foo.erb), as shown in Listing 4. While this file seems, on the surface, to be no different from any other ERb (embedded Ruby) document, it contains the telltale sign of a split test:

<% ab_test("click_text", "Click on me!", 
 ↪"Click here!") do |click_text| %>
    <%= click_text %>
<% end %>

The ab_test method, loaded by the line:

helpers Split::Helper

in linksplit.rb does several things at once: it defines a split test (the first parameter), and it provides the two alternatives that you are interested in testing. In this example, you can see that you're testing whether “Click on me!” (the control) converts more frequently than “Click here!” (the experiment). You then pass a block to the ab_test method. The block can contain any text you want, and it can be as long or short as you want. The key thing to realize is that the block parameter (click_text in this case) will contain one of the two text strings.

<html>
  <head>
    <title>Foo</title>
  </head>
  <body>
    <h1>Foo</h1>
    <p>
      <a href="/bar">
        <% ab_test("click_text", "Click on me!", 
         ↪"Click here!") do |click_text| %>
          <%= click_text %>
        <% end %>
      </a>
    </p>
  </body>
</html>

The above split test, thus, will compare the efficacy of two different links. But, you easily could switch the class of an HTML tag (thus giving different styles, including colors and fonts), a different location, the addition of a picture or anything else.

Once this is in place, the Split gem will produce an experiment, displaying one of these text strings to each of your users. Typically, you'll want to show them equally. There are ways to change the ratios, so that you're showing your experimental text to only a small proportion of your users.

However, showing these different texts isn't quite enough. You also need to be able to tell Split when visitors have “converted”—that is, when they have achieved the goal that you set out. In this particular case, you want to know which text is more effective at getting users to click on the link. So, you should report a conversion back to Split when the user has clicked on the link. In other words, in the Sinatra application's handler for the “/bar” method, you invoke the “finished” method, passing it the name of the experiment you want to indicate was finished:

finished('click_text')

Once you have put a split test in place, you can sit back and wait for users to come to your site. Some will get the first test, and some will get the second. How do you know which was more effective? By using the Split dashboard, which you connected to /split when you set up routing for Rack. It will show the percentage of users who responded to each of your experimental texts, so you can see which was more effective. Split also will tell you the confidence that it has in the results of this test. As a general rule, the more people you test, the more confident you can be in the results. But, statistics also show that even a (surprisingly) small sample size can give you interesting and meaningful results.

The dashboard is useful in several ways. First, it allows you to look at your various experiments, seeing how many people are viewing each of your experimental texts, and how many did and didn't complete the test. It also allows you, after examining the data, to use one of the two experimental texts permanently. This is useful in several ways. It means that a nonprogrammer can control and decide on each text. But, it also means that you don't have to go back into the code and remove your experiment when it is over. You can take some time, letting the Split system handle it for you. You also can reset the statistics, allowing you to try new ideas or throw out old ones.

Split testing is a powerful tool for helping ensure that your goals are met—whether those goals are selling something on-line or just getting people to sign up for your mailing list. The Split gem works with any Rack-powered Web site, and it makes it extremely easy to implement and then act on your optimization experiments. I encourage you to try some experiments on your own Web applications and see if you can find ways to get your users to convert into customers. If so, congratulations! You've discovered that modern, on-line marketing requires an understanding of programming as much as an understanding of the customer.