A beginner&#8217;s guide to stacking ensemble deep learning models
Most data science practitioners are starving for higher accuracy from their models and neural networks are regularly giving us state-of-the-art performances. In this article, we are going to see a method applying that we can boost the accuracy level of our modelling procedure. This method can be considered a way to join multiple machine learning models. We call this method a stacking ensemble. The major points to be discussed in the article are listed below. Table of contents What is stacking Implementing stacking ensemble deep learningData preparation Model preparation Stacking models Evaluating results Let’s start with understanding stacking. What is stacking? We can consider stacking as a process of ensembling multiple machine learning models together. There can be various methods for ensemble models such as bagging, boosting, and stacking is one of them. When talking about the bagging method, it works by applying multiple models with high variance and these high variances are averaged to decrease variance. And boosting work by building multiple incremental models so that bias can be decreased. Stacking is very different from these ensemble methods because this method focuses on exploring the space of different models applied to the same problem. Basically, the idea behind this method is to handle a machine learning problem using different types of models that are capable of learning to an extent, not the whole space of the problem. Using these models we can make intermediate predictions and then add a new model that can learn using the intermediate predictions. The final model that is learned using the intermediate predictions can be considered as stacked on top of the intermediate models.  Using such methods we can improve our performances and always we can get a model which is better than the intermediate model. The below picture can explain the stacking method. Image source The above picture represents that a final classifier is stacked on top of three intermediate classifiers. In this article, we are going to see how we can do stack ensembling with deep learning models. Let’s start the implementation. Are you looking for a complete repository of Python libraries used in data science, check out here, Implementation In this section, we will look at how we can stack an MLP classifier on top of some neural networks. This whole section includes the usage of packages from TensorFlow, Keras, and sci-kit learn libraries. Let’s start with preparing data for the process. Data preparation In this article, we will try to stack classification models. For this, we are going to make toy data using the sklearn provided make_classification  function. Let’s see how we can do that. from sklearn.datasets import make_classification X, y = make_classification() print(X.shape, y.shape) Output: Let’s split the dataset into training and test sets. from sklearn.model_selection import train_test_split X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=0) Let’s check what our toy data looks like. print(X_train) Output: print(y_train) Output: Model preparation Now in this example of stacking, we are going to use neural networks as our intermediate models so we require importing some of the packages from Keras. from tensorflow import keras from keras import layers from keras.constraints import maxnorm from keras.models import Sequential from keras.layers import Dense, Dropout, Activation, Flatten, Input In this article, we are going to use the stacking facility provided by scikit learn so we are required to import a wrapper provided in Keras for SciKit-Learn so that it can be compatible with scikit learn facilities. Lets import KerasClassifier. from keras.wrappers.scikit_learn import KerasClassifier Defining a model function def create_model (): # create model model = Sequential() model.add(Dense(20, input_dim=20, activation='relu')) model.add(Dropout(0.2)) model.add(Flatten()) optimizer= keras.optimizers.RMSprop(lr=0.001) model.add(Dense(units = 1, activation = 'sigmoid')) # Compile model model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=[keras.metrics.AUC(), 'accuracy']) return model In the above, we have defined a simple function for creating models in which we have added 2 dense layers and a dropout and flatten function. Let’s make some classifiers using the above function. Defining models NN_clf1=KerasClassifier(build_fn=create_model, epochs=5, batch_size= 32) NN_clf1._estimator_type = "classifier" NN_clf2=KerasClassifier(build_fn=create_model, epochs=10, batch_size= 32) NN_clf2._estimator_type = "classifier" NN_clf3=KerasClassifier(build_fn=create_model, epochs=15, batch_size= 32) NN_clf3._estimator_type = "classifier" NN_clf4=KerasClassifier(build_fn=create_model, epochs=20, batch_size= 32) NN_clf4._estimator_type = "classifier" NN_clf5=KerasClassifier(build_fn=create_model, epochs=25, batch_size= 32) NN_clf5._estimator_type = "classifier" Here we have defined 5 models where we have altered the number of epochs we can also use some other methods to perform alteration. Stacking models Let’s start by combining the above-defined models. intermediate = [('NN1', NN_clf1), ('NN2', NN_clf2), ('NN3', NN_clf3), ('NN4', NN_clf4), ('NN5', NN_clf5)] intermediate Output: Here we can see that we have combined all the above-defined neural networks. Now we are ready to stack these models. In this article, we are going to use MLPClassifier as the final model that will learn from the learning of the intermediate neural networks. from sklearn.ensemble import StackingClassifier from sklearn.neural_network import MLPClassifier clf = StackingClassifier(estimators=intermediate, final_estimator=MLPClassifier()) clf.fit(X_train, y_train) Output: Here we can see the final details of the model where it is representing estimators or intermediate models and final models. Evaluating the final model Let’s check for accuracy. print("Stacking model score: %.3f" % clf.score(X_test, y_test)) Output Here is our final accuracy. Here we can see an example of stacking deep learning models. Let’s check the probability score. clf.predict_proba(X_test) Output: Here we can see the probability of a sample being part of class 0 or 1. This article represents the idea that if applying neural networks in a stacking manner, we can also use base models like random forest and logistic regression models as the intermediate or final models. Final words In this article, we have discussed stacking which is a process of ensembling multiple machine learning models together in a stacked manner and we also see how we can stack models from sklearn on top of the neural networks. References Link to the codes
the idea behind stack ensemble method is to handle a machine learning problem using different types of models that are capable of learning to an extent, not the whole space of the problem. Using these models we can make intermediate predictions and then add a new model that can learn using the intermediate predictions.
Stacking ensemble is a joining process of multiple machine learning models together to boost accuracy level of our modelling procedure.
["AI (Artificial Intelligence)", "bagging", "Boosting", "Data Science", "Data Scientist", "Deep Learning", "Machine Learning"]
["data science", "scikit-learn", "machine learning", "bagging", "Keras", "AI", "neural network", "TensorFlow", "ML", "Boosting", "Machine Learning", "RAG", "deep learning", "Deep Learning", "Data Science", "Data Scientist", "AI (Artificial Intelligence)"]
["AI", "machine learning", "ML", "deep learning", "neural network", "data science", "TensorFlow", "Keras", "scikit-learn", "RAG"]
Yugesh is a graduate in automobile engineering and worked as a data analyst intern. He completed several Data Science projects. He has a strong interest in Deep Learning and writing blogs on data science and machine learning.