At the Forge

The field of statistics typically has had a bad reputation. It's seen as difficult, boring and even a bit useless. Many of my friends had to take statistics courses in graduate school, so that they could analyze and report on their research. To many of them, the classes were a form of nerdy, boring torture.

Maybe it's just me, but after I took those courses, I felt like I was seeing the world through new eyes. Suddenly, I could better understand the world around me. Newspaper articles about the government and scientific and corporate reports made more sense. I also could identify the flaws in such reports more easily and criticize them from a position of understanding.

Much of the power of statistics lies in the creation of a “model”, or a mathematical description of reality. A model is a caricature of sorts, in that it doesn't represent all of reality, but rather just those factors that you think will affect the thing you're trying to understand. A model lets you say that given inputs A, B, C and D, you can know, more or less, what the output will be.

Sometimes, the goal of a statistical model is to predict a value—for example, given a certain size and neighborhood, you can predict the price of a house. Or, given someone's age, weight and where they live, you can predict his or her likelihood of getting a certain disease.

Often, the goal is to predict a category—for example, in an upcoming election, for whom are people likely to vote? Taking into account where they live, what level of education they've received, their ethnic background and a few other factors, you can often predict for whom people will vote before they know it themselves.

During the past few years, there has been a huge amount of buzz around the terms “big data”, “data science” and “machine learning”. As these buzzwords continue to gain acceptance, many statisticians are wondering what the big deal is. And to be honest, their complaint makes some sense, given that “machine learning” is, more or less, a computerized version of the predictive models that statisticians have been creating for decades.

Now, why am I telling you this? Because I actually do believe that machine learning is a game-changer for huge parts of our lives. Just as my perspective was changed when I learned statistics, giving me tools to understand the world better, many businesses are having their perspectives changed, as they use machine learning to understand themselves better. Everything from online shopping, to the items you see in your social-network feeds, to the voice-recognition algorithms in your phone, to the fraud detection used by your credit-card company is being affected, boosted and (hopefully) improved via machine learning.

This means that no matter what sort of software development you do, you would be wise to gain as much experience as you can with machine learning. Its benefits might not be obvious to you at once or even be applicable to your work right away. But machine learning is becoming ubiquitous, and there is no shortage of ways in which to use it.

So with this article, I'm starting a series on machine learning and some of the ways your organization can take advantage of it. I'll look at a number of problems, many of which are common on web applications, that can benefit from using machine learning. Along the way, I hope you'll get lots of ideas for the sorts of analysis and uses that machine learning can bring to your applications.

If you're completely new to the world of machine learning, I encourage you to read the free Geek Guide I wrote on the subject, published by Linux Journal and available at geekguide.linuxjournal.com/content/machine-learning-python-0.

If you have ever invested money, you'll undoubtedly remember that the fund in which you invested, or the broker with whom you worked, warned you that “past performance is no guarantee of future results” or words to that effect. That's because we, as living beings, are conditioned to assume that if the world worked a certain way in the past, then it'll likely work a certain way in the future. For most of us, most of the time, this is a good way to live our lives.

Machine learning works on this principle, that the past is a good indicator of the future. We create a machine-learning model, telling the model that given a set of inputs, we got a particular output. One such piece of information is unlikely to give us anything useful. But, several hundred samples later, the model can start to make some predictions. Several thousand, or even million, samples later, and your predictions can potentially be quite accurate.

If you run an online store, machine learning can help you to understand your customers better. For example, if you know that customers who bought products X and Y also bought product Z, you can send email promotions to such people who haven't yet bought Z, knowing that a proportion of them will respond positively.

More nefariously, you also could raise the price on product Z when those people visit your site, knowing that a fair number of such people are likely to buy it anyway.

(And by how much should you raise the price? Assuming enough traffic, you can try different numbers on different people, until you figure out the optimal setting.)

If you have additional information about your customers, such as their age, gender or where they live, a machine-learning algorithm can help you determine even more about them—from what they're likely to buy, to how often they'll visit your store. You also can keep track of things they thought about buying, but later removed from their shopping carts.

Take the famous story of Target, which sent a “so you're expecting” promotion to a teenage girl in the United States. It turns out that Target's machine-learning systems had correctly identified that based on her purchasing habits, she was likely to be pregnant. The only problem was that this teenager in question, who was indeed pregnant, hadn't told her parents.

I recently spoke with the CTO of a new online marketplace for a specific type of consulting, in which customers and consultants would communicate, with a goal toward solving the problem. After each session, each of the participants would then indicate how satisfied they were. The CTO wanted to know where machine learning could help; I told him that over time, they could accumulate a huge amount of data regarding which types of customers got along best with which sorts of consultants—allowing them to make increasingly good recommendations and be better matchmakers.

And speaking of “matchmakers”, every modern online dating site uses machine learning. They know a lot about their users, and they use that data to try to predict which people in their database are likely to be the greatest success. You could say that these dating sites, thanks to their machine-learning systems, know more about people's dating preferences than people could explicitly say about themselves.

One classic example of machine learning is a recommendation engine. I have been shopping on Amazon since it first opened, so I've provided that company with a great deal of data about myself and the things I like to purchase. When it suggests that I might be interested in a product, the odds are good that I either have it already, or that I considered it or that I would indeed be interested in it.

If you run an e-commerce site, you can use machine learning to similar ends. You can create a model that identifies which products are similar to which other products. Then, you can go through someone's purchase history, finding unpurchased products that are similar to the ones that they already have bought.

Nowadays, sites like Amazon often have access not only to your purchase history, but also to the ratings you gave to various products. In this way, sites can suggest not only the products that you're likely to buy, but also those that you're most likely to enjoy as well.

Another way to handle recommendations is to look at people, rather than products. Instead of telling me what I'm likely to buy based on my past purchases, a site could tell me what I'm likely to buy, based on my friends' purchasing habits. If you have access to friends' recommendations, the combination of a friend's purchase and a high rating from that friend might make a product especially attractive—and, thus, the object of a special promotion.

Of course, many of the heaviest and largest users of machine learning are financial firms. You can be sure that credit-card companies and other payment companies, such as PayPal, spend a great deal of time and effort on machine-learning algorithms that identify when someone might be committing fraud. If I use my credit card to buy something unusual or in a country I haven't visited before, my credit-card company sometimes will contact me to make sure the transaction is legitimate.

As you can imagine, such machine-learning models take a wide variety of inputs to try to determine whether the transaction is legitimate or fraudulent. These checks often have to take place in real time, which makes them particularly impressive in my book, given the amount of data that they have to deal with in a given moment. But, there inevitably will be some false positives and false negatives in such a system. If you're like me, the false negatives continue to haunt you for a long time afterward. After all, although I understand how these companies are just trying to do their jobs, it's maddening to be stuck at the supermarket checkout, trying to convince your credit-card company that you're simply trying to buy milk and bread in Stockholm.

“FinTech” companies are popping up all over, and much of what they do is an application of machine learning to finance (hence the name, of course). Whether it's identifying fraud or looking for investment opportunities, such models can crunch more data more quickly than people—and can draw upon millions of previous examples, rather than the dozens that humans can keep in their heads.

As you can see, machine learning offers an incredible variety of solutions, providing opportunities for new types of analysis. It's definitely worth sitting down to learn something about machine learning. In upcoming articles, I plan to walk you through how to solve some of the problems described in this column using open-source languages and tools. I hope you'll soon see that in data science and machine learning, the coding is the easy part. The hard part is thinking about how to build your model, what to include and which algorithm would be the most appropriate.