How to deploy and monitor your Keras model with Comet?
Comet is an online experimentation platform for artificial intelligence and machine learning used by Researchers and Data Scientists which provides an easy platform for tracking the ML models and comparing the results of the model on the server. Comet provides an excellent dashboard to visualize the various parameters of the model deployed in the interface and keeps track of the model internally. So this article provides a brief overview of how to deploy a Keras model on the online platform of Comet and make suitable interpretations from the model. Table of Contents Introduction to CometImplementing a Keras model from scratchLogging in to the Comet InterfaceIntegrating the Keras model into the comet interfaceObtaining predictions from comet interfaceMounting the model in the comet serverVisualizing the model deployed on the serverSummary Let’s first briefly discuss about the Comet. Introduction to Comet Comet is an online API that serves as an online platform to deploy machine learning or deep learning models and continuously validate all the stages of a model deployment lifecycle until production. The model on the platform can be deployed as public or can be limited for certain access for analyzing the parameters of the model deployed on their platform. With the concerned access, comet provides the flexibility to access the model at any time on the server and monitor the models deployed on their server. Are you looking for a complete repository of Python libraries used in data science, check out here. Implementing a Keras model from scratch Keras module provides useful datasets for deep learning model development and here we have used the MNIST Digits dataset for model development and deployment of the model developed on the Comet server. So initially the dataset was acquired and split into train and test and the split data was visualized using subplots. Later a simple Sequential model was built with appropriate input dimensions and activation functions. The dependent features of the dataset were suitably reshaped and the target variables were encoded according to the number of digits present in the MNIST dataset. After suitable preprocessing the model was compiled with categorical_crossentropy as the loss function and accuracy as the metrics to evaluate from the model developed. Also, the model was validated for its training and testing accuracy and loss. The complete steps to follow to implement a basic digit classification model for the MNIST dataset are shown below. import numpy as np import tensorflow as tf import matplotlib.pyplot as plt import random from tensorflow.keras.datasets import mnist from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense, Dropout from tensorflow.keras.callbacks import EarlyStopping from tensorflow.keras.preprocessing import image from tensorflow.keras.optimizers import RMSprop from tensorflow.keras.utils import to_categorical (X_train,y_train),(X_test,y_test)=mnist.load_data() print('Number of training records',X_train.shape) print('Number of testing records',y_test.shape) plt.figure(figsize=(5,10)) for i in range(9): plt.subplot(3,3,i+1) num = random.randint(0, len(X_train)) plt.imshow(X_train[num], cmap='gray', interpolation='none') plt.title("Class {}".format(y_train[num])) #plt.tight_layout() plt.axis('off') plt.show() X_test_original=X_test.copy() # preprocess and normalize the data X_train = X_train.reshape(60000, 28*28) X_test = X_test.reshape(10000, 28*28) X_train1=X_train/255. X_test1=X_test/255. ## Encoding the target variable y_train=to_categorical(y_train,10) y_test=to_categorical(y_test,10) With all these suitable preprocessing model building was taken up as shown below. Model Building model=Sequential() model.add(Dense(units=512,activation='relu',input_dim=784)) model.add(Dense(units=256,activation='relu')) model.add(Dense(units=125,activation='relu')) model.add(Dense(units=10,activation='softmax')) model.summary() model.compile(loss='categorical_crossentropy',optimizer='adam',metrics=['accuracy']) model_res=model.fit(X_train1,y_train,epochs=5,validation_data=(X_test,y_test)) Evaluating the model performance plt.figure(0) plt.plot(model_res.history['accuracy'],label='training accuracy') plt.plot(model_res.history['val_accuracy'],label='testing accuracy') plt.title('Accuracy') plt.xlabel('epochs') plt.ylabel('accuracy') plt.legend() plt.show() plt.figure(1) plt.plot(model_res.history['loss'],label='training loss') plt.plot(model_res.history['val_loss'],label='testing loss') plt.title('Loss') plt.xlabel('epochs') plt.ylabel('loss') plt.legend() plt.show() model.evaluate(X_train,y_train) model.evaluate(X_test,y_test) Logging in to the Comet Interface Firstly the comet module can be easily installed in the working environment using the pip commands and the module can be loaded using the import statements. So once when the module is successfully installed in the working environment a random name has to be given to the model that would be deployed onto the comet server using the init() built-in function. So a link will be generated in that particular run instance and a unique API Key will be generated which can be stored in the form of a string character in the working environment and utilized if required. The steps involved are shown below. import comet_ml from comet_ml import Experiment comet_ml.init('comet-keras') Initiating the comet interface keras_exp=Experiment(project_name='comet-keras',api_key=API_key) So now this comet interface can be used to train, test, and evaluate the parameters of the model on the comet server. Now the interface and the login with the appropriate API key can be validated using the display() function as shown below. keras_exp.display() Integrating the Keras model in the comet interface The integration of the model into the server starts with primarily hashing the training dataset using the log_dataset_hash() function and the experiment can be trained on the comet interface using the train() inbuilt function of the comet. The steps involved in hashing the dataset and loading the model are shown below. keras_exp.log_dataset_hash(X_train) with keras_exp.train(): model_res=model.fit(X_train1,y_train,batch_size=32,epochs=10,validation_data=(X_test1,y_test)) Evaluating model parameters on the Comet interface Now using the test() inbuilt function of the comet the model deployed on the server can be evaluated on various parameters. Here the steps involved to evaluate the training and testing loss and accuracy are shown below. with keras_exp.test(): test_loss, test_accuracy = model.evaluate(X_test1, y_test) train_loss, train_accuracy = model.evaluate(X_train1, y_train) metrics = {'test_loss':test_loss,'test_accuracy':test_accuracy,'train_loss':train_loss,'train_accuracy':train_accuracy} } Logging various metrics in the Comet Interface The log_metrics() inbuilt function is utilized to deploy or log in the metrics used for evaluating the model on the comet server. keras_exp.log_metrics(metrics) Now let us see how to obtain predictions from the model deployed onto the comet interface. Obtaining predictions from comet interface For obtaining the predictions from the comet interface the log_figure() function is used which provides outputs for the model’s prediction for each digit being classified as shown below. But for obtaining predictions the original TensorFlow model has to be used and to visualize the model performance the inbuilt function of Comet can be used. model_pred = model.predict(X_test1[:10]).argmax(-1) # remember our copy of x_test? This is where it comes into play again! # Since we flatted x_test during pre-processing, we need to preserve an unflattened version plt.figure(figsize=(16,8)) for i in range(10): plt.subplot(1, 10, i+1) plt.imshow(X_test_original[i], interpolation='nearest') plt.text(0, 0, model_pred[i], color='black', bbox=dict(facecolor='white', alpha=1)) plt.axis('off') keras_exp.log_figure('Predicted MNIST Digits from Comet Interface',plt) By clicking on the web link from the plots obtained the output of the model deployed on the server can be downloaded if required for documentation along with the REST api information. Now the experiment on the comet server has to be ended to visualize how the comet interface has monitored the model deployed on its server Logging out the model deployed on the server Now the experiment carried out on the server has to be ended using the end() inbuilt function. While ending the experiment on the server, several information pertaining to the Data, Metrics used to evaluate the model, and various other information will be provided as shown below. keras_exp.end() Mounting the model in the comet server The model can be saved in an HDF5 (h5) format and the model can be mounted onto the comet server using log_model() function which may facilitate using the model weights for any parameter evaluation or using the model weights for any other tasks. The steps involved to mount the model on the comet server are shown below. model.save('keras-comet-new.h5') keras_exp.log_model('Saved Keras model',"/content/keras-comet-new.h5") Visualizing the model deployed on the server The display() function of the comet can be used to visualize the parameters and to monitor the model functionality on the server. keras_exp.display() The entire code that is being used to deploy on the server can be seen in the code section of the server where the entire code is made visible to the person with access. All the experiments carried out on the server with its logging time and the parameters evaluated can be visualized in the metrics pane as shown below. Along with this, the hardware resource usage can also be visualized in the server as shown below. Summary So this article provides a brief overview of how to develop a Keras model and deploy it on the Comet server to continuously monitor the model. Easily interpretable reports and graphs are produced with respect to the model performance on the server and the model’s memory consumption for each of the hardware resources used in the premise can also be visualized using the system metrics. So by using the Comet server the model can be made available on any platform and utilized accordingly to monitor the model on the servers.
A detailed implementation of usage of Comet platform for deploying and monitoring a model.
A detailed overview of how to integrate deep learning models Keras on the comet server and monitor the model's parameters and performance where in a complete overview of developing a model from the base and deploying it on the server to evaluate the model parameters and monitor the model's performance on the server is emphasized.
["Deep Learning", "Keras", "Machine Learning", "Tensorflow"]
["data science", "NumPy", "artificial intelligence", "Keras", "machine learning", "AI", "ML", "Machine Learning", "Ray", "deep learning", "Deep Learning", "TensorFlow", "Tensorflow"]
["AI", "artificial intelligence", "machine learning", "ML", "deep learning", "data science", "Ray", "TensorFlow", "Keras", "NumPy"]
Darshan is a Master's degree holder in Data Science and Machine Learning and an everyday learner of the latest trends in Data Science and Machine Learning. He is always interested to learn new things with keen interest and implementing the same and curating rich content for Data Science, Machine Learning,NLP and AI