A guide to Orchest for building ML pipelines
Creating pipelines is one of the most important procedures in any application development task. We can define a pipeline as an arrangement of files that are interconnected and running them in a flow provides the final goal of the pipeline. There are a variety of platforms that are available to provide the service for building or making pipelines. Orchest is one of them and recently developed platforms that understand the requirement of users and provides a one-place interface for building pipelines. In this article, we are going to discuss the Orchest tool for building machine learning pipelines. The major points to be discussed in the article are listed below. Table of contents What is Orchest?Joining Orchest Building pipelineStep 1: Data gatheringStep 2: Data preprocessingStep 3: Defining modelStep 4: Collecting accuracy Let’s begin with understanding what Orchest is. What is Orchest? Orchest is a tool that can be utilized for building machine learning pipelines. Using this library we can build pipelines in a visualized manner using the Orchest provided user interface.  One of the best things about this tool is it doesn’t require any third-party integration or DAGs.Using this tool is very easy and we can use Jupiter lab or VSCode with this tool to build our machine learning models in pipeline settings. This tool provides its facilities with various languages like python,  R, and Julia. Pipeline build under Orchest contains steps that we used to build the model and every step contains an executable file and the UI of Orchest makes them interconnected with each other using the nodes. This tool creates a representation that tells us about the data flow from one step to another step. We can pick, drop and connect steps very easily with this tool and this feature makes the Orchest user-friendly. Visualization of the pipeline progress helps us in making the data flow accurate and also we can debug the codes if we find any errors. To install this toolkit in our environment we are required to have a Docker Engine version more than or equal to 20.10.7. To install this toolkit in macOS and Linux operating systems we need to clone and install Orchest in the environment that we can do using the following code git clone https://github.com/orchest/orchest.git && cd orchest ./orchest install After cloning and installing we can start it using the following lines of codes: ./orchest start With the above-given features, Orchest also gives various other integrations like we can build a web app using the streamlit or we can write codes to ask for data from Postgres SQL. In this article, we are aiming to make a pipeline using Orchest. So that we can understand how it processes. Let’s start by implementing a pipeline that can classify iris data. Are you looking for for a complete repository of Python libraries used in data science, check out here. Joining Orchest Before getting started with the Orchest pipeline we are required to know how we can sign in or make an account with Orchest. For doing so we can go through this page. After making an account we can have various options using which a data pipeline can be built easily. For this article, we are using a free space where we get to practice our projects. As a free space, we get 50 GB volume with 2 vCPUs and 8 GiB specifications instance. Here you can get the whole pipeline. The overview of this pipeline will look like the following if not changed anything. Let’s just start with the process. First of all, to make a pipeline we are required to click on the create pipeline button that we get in the pipeline tab after initiating our instance. After creating a pipeline and giving it a name we will get a black page as given in the below image. Here the new step button is for making our steps. Every step will hold a file that can be of R, Python, Julia extension files. In my pipeline, I have used the python language. Let’s take a look at how I make a pipeline for iris classification. Building pipeline Step 1: Data gathering The below image is for the step that helped the pipeline to get the data from sklearn. Under this step, we have got an ipynb file that can be opened in the jupyter notebook. In real-time we will also get a jupyter file for writing codes that can be accessed by clicking on the step. The following codes have been used for completing this step. import orchest import pandas as pd from sklearn.datasets import load_iris df_data, df_target = load_iris(return_X_y=True) orchest.output((df_data, df_target), name="data") In the above code, we are required to focus on the first line and the last line. Orchest we have imported that is already installed in the notebooks environment and Orchest.output() helps us in exporting the output of a current step to the next step and it is necessary to link the next step using the arrow that we get in the pipeline interface. Step 2: Data preprocessing This step consists of the codes for preprocessing the data that we get in the first step. The following image is of the second step. Under this step, we can find the following codes import orchest from sklearn.model_selection import train_test_split from sklearn.pipeline import Pipeline from sklearn.preprocessing import MinMaxScaler data = orchest.get_inputs() X, y = data["data"] scaler = MinMaxScaler() X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20) X_train = scaler.fit_transform(X_train) X_test = scaler.transform(X_test) orchest.output((X_train, y_train, X_test, y_test), name="training_data") In this step, we have used the orchest.get_input() module to import the data from the previous step and split the data and again passed the split data for the next step. Step 3: Defining model In this step, we can find that we have combined three steps. We have modelled the data from preprocessed steps using three models ( logistic regression, decision tree, random forest). The below image is the representation of this step. The code we have pushed in these steps is similar. We just changed the model so in the below I have posted the codes of only one model. import numpy as np import orchest from sklearn.linear_model import LogisticRegression from sklearn.metrics import mean_squared_error # Retrieve the data from the previous step. data = orchest.get_inputs() X_train, y_train, X_test, y_test = data["training_data"] model = LogisticRegression() model.fit(X_train, y_train) from sklearn.metrics import accuracy_score y_pred = model.predict(X_test) test_accracy = accuracy_score(y_test, y_pred) Orchest.output(test_accracy, name="logistic-regression-accuracy") In the above codes, we can see how the data is collected from the previous step and how the final results from the model are pushed for the next step. Step 4: Collecting accuracy After defining and fitting the model step 4 is our final step that will collect the accuracy of all the models. The below image is the image of our final pipeline. Codes in the final steps are as follows: import orchest data = orchest.get_inputs() for name, value in data.items(): if name != "unnamed": print(f"\n{name:30} {value}") Output: Here we can see the final output. We can also check this using the log button in the pipeline interface. Here we can also get the information if any component of the pipelines needs bug fixes or has errors. Final words In this article, we have discussed what Orchest is and found that is an easy way to build the machine learning pipeline with it. We also looked at an example that can be followed for building machine learning pipelines. After logging in you can see this example pipeline using this link. References Orchet documentation  Iris pipeline
Orchest is a tool that can be utilized for building machine learning pipelines. Using this library we can build pipelines in a visualized manner using the Orchest provided user interface.  One of the best things about this tool is it doesn’t require any third-party integration or DAGs.
Orchest is a tool that can be utilized for building machine learning pipelines in a visualized manner using its user interface.
["AI (Artificial Intelligence)", "Data Science", "Data Scientist", "Deep Learning", "iris dataset python", "Machine Learning", "pipelines in machine learning"]
["data science", "NumPy", "iris dataset python", "machine learning", "AI", "ML", "Machine Learning", "docker", "Aim", "Jupyter", "Streamlit", "pipelines in machine learning", "Deep Learning", "Data Science", "Data Scientist", "AI (Artificial Intelligence)", "Pandas"]
["AI", "machine learning", "ML", "data science", "Aim", "Streamlit", "Jupyter", "Pandas", "NumPy", "docker"]
Yugesh is a graduate in automobile engineering and worked as a data analyst intern. He completed several Data Science projects. He has a strong interest in Deep Learning and writing blogs on data science and machine learning.