Introduction

Support Vector Machine is one of the most performant off-the-shelf supervised machine learning algorithms. This means that when you have a problem and you try to run a SVM on it, you will often get pretty good results without many tweaks. Despite this, because it is based on a strong mathematical background, it is often seen as a black box. In this book, we will go under the hood and look at the main ideas behind SVM. There are several Support Vector Machines, which is why I will often refer to SVMs. The goal of this book is to understand how they work.

SVMs are the result of the work of several people over many years. The first SVM algorithm is attributed to Vladimir Vapnik in 1963. He later worked closely with Alexey Chervonenkis on what is known as the VC theory, which attempts to explain the learning process from a statistical point of view, and they both contributed greatly to the SVM. You can find a very detailed history of SVMs here.

In real life, SVMs have been successfully used in three main areas: text categorization, image recognition, and bioinformatics (Cristianini & Shawe-Taylor, 2000). Specific examples include classifying news stories, handwritten digit recognition, and cancer tissue samples.

In the first chapter, we will consider important concepts: vectors, linear separability, and hyperplanes. They are the building blocks that will allow you to understand SVMs. In Chapter 2, instead of jumping right into the subject, we will study a simple algorithm known as the Perceptron. Do not skip it—even though it does not discuss SVMs, this chapter will give you precious insight into why SVMs are better at classifying data.

Chapter 3 will be used to step-by-step construct what is known as the SVM optimization problem. Chapter 4, which is probably the hardest, will show you how to solve this problem—first mathematically, then programmatically. In Chapter 5, we will discover a new support vector machine known as the Soft-margin SVM. We will see how it is a crucial improvement to the original problem.

Chapter 6 will introduce kernels and will explain the so called “kernel trick.” With this trick, we will get the kernelized SVM, which is the most-used nowadays. In Chapter 7, we will learn about SMO, an algorithm specifically created to quickly solve the SVM optimization problem. In Chapter 8, we will see that SVMs can be used to classify more than one class.

Every chapter contains code samples and figures so that you can understand the concepts more easily. Of course, this book cannot cover every subject, and some of them will not be presented. In the conclusion, you will find pointers toward what you can learn next about SVMs.

Let us now begin our journey.