A hands-on guide to principal component regression in Python
Principal Components Regression (PCR) is a technique for analyzing multiple regression data that suffer from multicollinearity. PCR is derived from Principal Component Analysis (PCA). So, it is PCA applied to a regression algorithm that has multicollinear features. Principal components regression reduces errors in regression estimates by adding a degree of bias and by doing so, it will be possible to provide more reliable estimates. In this article, we will put focus on implementing PCR on a regression problem. Following are the topics covered. Table of contents About the architecture of PCAAbout multicollinearity in regressionAbout Principal Components Regression (PCR)Implementing PCR in Python Let’s start with the architect PCA from which PCR is derived. About the architecture of PCA Principal Component Analysis (PCA) is the analysis of principal features of the data. The analysis is done by reducing the dimensionality of the feature space. In other words, it is a tool to reduce the features from the data to get only the required features or principal components for the learner. PCA has three major components which help to reduce dimensionality: The covariance matrix is the measure of how much the variables are associated with each other.The eigenvectors are the directors in which the data is dispersed.The eigenvalues are the relative importance of the directions. About multicollinearity in regression From the name, it is clear that the collinearity between the independent variables in a regression problem is stated as multicollinearity in regression. The reasons behind curing a multicollinear regression problem are: Understanding of the significance of features for the regression learner.Instability in estimating the coefficientOverfitting of the learner Multicollinearity in regression could be vandalized with the help of Principal Component Regression (PCR). Let’s understand how does PCR control multicollinearity. About Principal Components Regression (PCR) The Principal Component Regression (PCR) algorithm is an approach for reducing the multicollinearity of a dataset. Although multi-variate linear regression can fit well on the test set, there is normally a high-variance problem with it. For this reason, PCR adds a small bias to the model, such that it aims to maintain a high level of accuracy, but reduce the variance substantially. This is achieved by applying the PCA to the features before the training. Let’s try to learn PCR by implementing it on data. Implementing PCR in Python The objective of this regression model is to predict the salary of the player based on different features. Importing packages import numpy as np import pandas as pd Reading the data set salary=pd.read_csv("Hitters.csv") df=salary.copy() df.dropna(inplace=True) df.shape,salary.shape ((263, 20), (322, 20)) This data set is taken from the Kaggle repository, to use this dataset the links are given in references. Copying the dataset to another data frame for further pre-processing so that the original data frame remains unchanged. Then drop all the missing values and at last check the number of rows and columns of the original and copied data frames. There are two categorical columns and the rest are continuous features, needed to encode the categorical features for further utilization. Encoding categorical features: df = pd.get_dummies(df,columns=['League', 'Division', 'NewLeague']) df.head() Encoded the data using the pandas get_dummies function.  The original features which are used for creating the dummies are replaced with a total of six new columns as shown in the above image. Since the regression problem is to predict the salary of the players so the dependent variable is “Salary”. y=df['Salary'] X=df.drop(['Salary'],axis=1) There are a total of 23 columns in this data frame out of which only a few are important for the analysis and prediction. There is a requirement for feature elimination which would be performed by  PCA. Applying PCA from sklearn.decomposition import PCA pca=PCA() X_red = pca.fit_transform(scale(X)) PCA is been imported from the sklearn library and stored in a variable for easier applications. The PCA is fitted with the independent variables for dimensionality reduction. The percentage of variance in the dependent variable is explained by adding each principal component to the model. np.cumsum(np.round(pca.explained_variance_ratio_, decimals = 4)*100)[0:5] The above output is explained as: By using the first principal component, we can explain 32.62% of the variation in the dependent variable.By using the second principal component, we can explain 53.02% of the variation in the dependent variable.Similarly, by using others we can explain 68.82%, 78.07%,85.39%, 89.31% The conclusion of this is that we need to use a total of five principal components in the regression learner. Now, the principal components are decided it’s for split the dataset into a 70:30 ratio for train and test for training and testing the learner. X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=42) Let’s build the final model using five principal components in the linear regression model. X_red_train = pca.fit_transform(scale(X_train)) X_red_test = pca.transform(scale(X_test))[:,0:5] lm = LinearRegression() pcr = lm.fit(X_red_train[:,0:5], y_train) y_pred = pcr.predict(X_red_test) The final model is built and trained on the train data set and also predicted the salary using the test dataset. Let’s check how good the model has performed. np.round(np.sqrt(mean_squared_error(y_test, y_pred)),2) 397.52 The root means square error (RMSE) is low approximately 398 this can be improved by tunning the model further I would leave that to you. Nutshell Applying Principal Component Analysis before the regression can reduce multicollinearity and helps in better prediction with fewer features in lesser time. PCR also reduces the chances of overfitting learners. References Link for the above codePaper released on PCR
Principal Components Regression (PCR) is a technique for analyzing multiple regression data that suffer from multicollinearity. Principal components regression reduces errors in regression estimates by adding a degree of bias
The Principal Component Regression (PCR) algorithm is an approach for reducing the multicollinearity of a dataset.
["overfitting", "PCA", "regression analysis"]
["Go", "NumPy", "TPU", "programming_languages:R", "AI", "R", "regression analysis", "Python", "Aim", "programming_languages:Python", "PCA", "overfitting", "Pandas"]
["AI", "Aim", "Pandas", "NumPy", "TPU", "Python", "R", "Go", "programming_languages:Python", "programming_languages:R"]
Sourabh has worked as a full-time data scientist for an ISP organisation, experienced in analysing patterns and their implementation in product development. He has a keen interest in developing solutions for real-time problems with the help of data both in this universe and metaverse.