One vs One, One vs Rest with SVM for multi-class classification
In machine learning, binary classification algorithms become one of the most important and used algorithms when things come into the accuracy part of modelling. In most cases, we can see a support vector machine is a preferable option for data scientists in their projects. One thing which lags here is that these binary classifiers are not capable of performing multiclass classification. One-vs-rest and One-vs-one are two methods that make binary classifiers work as multiclass classifiers. In this article, we are going to discuss the one-vs-one and one-vs-rest or one-vs-all method with a support vector machine for multi-class classification. The major points to be discussed in the article are listed below. Table of contents Binary classification to multiclass classification What is One-vs-Rest (OvR) or One-vs-All(OvA)? Implementation of  One-vs-Rest (OvR) What is One-vs-One (OvO)? Implementation of  One-vs-One (OvO) Let’s first discuss how Binary classifiers can be used for multi-class classification. Binary classification to multiclass classification Generally, we see the usage of algorithms like SVM and logistic regression in binary classification problems in which using these algorithms we are required to predict one class out of two for any sample data. Predicting one class out of two classes is the reason to consider classification as binary classification. In one of our articles, we have discussed that a support vector machine is the most used method for classification because it can create non-linear boundaries by projecting the data with higher dimensions in the space using its nonlinear function. But these models are only capable of dealing with binary classification problems. In multiclass classification problems, we need to deal with more than two classes which means the algorithm which we are using should be capable of working with multiple classes. There are various models available for this and some methods are also available that can make support vector machines capable of dealing with more than two classes.  We call these methods heuristic methods. There are two types of heuristic methods: One-vs-Rest (OvR) or One-vs-All(OvA) One-vs-One (OvO) Let’s take a closer look at One-vs-Rest (OvR) or One-vs-All(OvA). Are you looking for for a complete repository of Python libraries used in data science, check out here. What is One-vs-Rest (OvR) or One-vs-All(OvA)? We can think of One-vs-Rest (OvR) or One-vs-All(OvA) as an approach to making binary classification algorithms capable of working as multiclass classification algorithms. This approach mainly splits the multiclass data as binary classification data so that the binary classification algorithms can be applied to convert binary classification data. The procedure of conversion of the data can be understood using an example of iris data where we have three classes as follows: Setosa Versicolor Virginica The converted data as binary classification data will look like the following: Setosa vs [Versicolor, Virginica] Versicolor vs [Setosa, Virginica] Virginica vs [Setosa, Versicolor] By looking at the conversion we can think that there is a requirement of three models but with the large datasets creating three models can be a tough and non-accurate approach to modeling. Here One-vs-Rest (OvR) or One-vs-All(OvA) comes to save us where binary classifiers can be trained to predict any class as positive and other classes as negative. We mainly use binary classifiers that can give membership probability or probability-like scores because argmax of these scores can be utilized to predict a class out of multiple classes. Let’s see how we can implement binary classifiers using the One-vs-Rest (OvR) or One-vs-All(OvA) method for a multiclass classification problem. Implementation of One-vs-Rest (OvR) For implementing the One-vs-Rest (OvR) method the scikit-learn libraries provide a method called OneVsRestClassifier under the multiclass package. In this article, we are going to see how we can implement a support vector classifier for the multiclass classification of IRIS data using the OneVsRestClassifier. Let’s import libraries first. from sklearn import datasets from sklearn.svm import SVC from sklearn.multiclass import OneVsRestClassifier Here in the above, we can see that we have just called SVC models, the OneVsRestClassifier model, and iris data. Let’s make independent and target variables using the iris data. iris = datasets.load_iris() X = iris.data y = iris.target Let’s instantiate the SVC model. svc = SVC() Instantiating the One-Vs-Rest Classifier for the SVC model. o_vs_r = OneVsRestClassifier(svc) Let’s train the model o_vs_r.fit(X, y) Predicting values from the model. yhat = o_vs_r.predict(X) yhat Output: Here in the above, we can see that the support vector classifier is predicting 3 classes and we did not require to use three binary classifiers.  Let’s take a closer look at One-vs-One (OvO). What is One-vs-One (OvO)? This method can also be considered as an approach to making binary classification algorithms capable of working as multiclass classification algorithms.it is similar to the One-Vs-Rest method because it also works based on splitting the data but the splitting behavior of this method is different from the One-Vs-Rest method. This method includes the split of data for each class where each class has every other class as its opponent. We can again use the iris data set to understand the data split behavior of this method. In the iris, we have the following classes: Setosa Versicolor Virginica Split iris data using the One-vs-One (OvO) methods will look like the following: Setosa vs Versicolor Setosa vs Virginica Versicolor vs Virginica Virginica vs Versicolor Here we can see using this data we have binary classification data that includes one class with every other class. We can find out the number of data split using the following formula Split of data = (number of classes X (number of classes – 1))/2 Other functions of this method are similar to the One-vs-Rest method. Let’s see how we can implement a support vector classifier for multiclass classification using the One-vs-One method. Implementation of One-vs-One (OvO) To implement this method we can use the scikit-learn library where the OneVsOneClassifier method is provided under the multiclass package. In this section, we will look at how we can apply this method to the support vector classifier to classify the IRIS data. Let’s import the library from sklearn import datasets from sklearn.svm import SVC from sklearn.multiclass import OneVsOneClassifier Let’s call the data. iris = datasets.load_iris() X = iris.data y = iris.target Instantiating support vector classifier svc = SVC() Instantiating One-Vs-One method o_vs_o = OneVsOneClassifier(svc) Training the model o_vs_o.fit(X, y) Predicting from model yhat = o_vs_o.predict(X) yhat Output: Here we can see our support vector classifier can predict more than two classes using the One-vs-One method. Final words In this article, we have discussed the two heuristic methods that make a binary classifier capable of working on multi-class data. These methods can help us in applying binary classification methods to a multiclass classification using the split of the data. References Link for the codes
In machine learning, binary classification algorithms become one of the most important and used algorithms when things come into the accuracy part of modelling. In most cases, we can see a support vector machine is a preferable option for data scientists in their projects. One thing which lags here is that these binary classifiers are […]
One-vs-One and One-vs-Rest are two methods that can help binary classifiers to work on multiclass classification problem
[]
["data science", "scikit-learn", "Go", "machine learning", "TPU", "programming_languages:R", "AI", "Python", "programming_languages:Python", "R"]
["AI", "machine learning", "data science", "scikit-learn", "TPU", "Python", "R", "Go", "programming_languages:Python", "programming_languages:R"]
Yugesh is a graduate in automobile engineering and worked as a data analyst intern. He completed several Data Science projects. He has a strong interest in Deep Learning and writing blogs on data science and machine learning.