Why Is There A Shortage Of MLOps Engineers?
MLOps, or machine learning operations, is emerging as one of the hottest fields. In the last four years, the hiring for machine learning and artificial intelligence roles has grown 74% annually. MLOps engineer, also known as DevOps for machine learning, covers a whole gamut of machine learning tasks starting from data integration to training and managing infrastructure to deploying. Source: Arrikto (MLOps pipeline) DevOps Vs MLOps MLOps and DevOps engineers require different skill sets. Firstly, developing machine learning models do not need a software engineering background as the focus is mainly on the proof of concept/prototyping. Secondly, MLOps are more experimental in nature compared to DevOps. MLOps calls for tracking different experiments, feature engineering steps, model parameters, metrics, etc. MLOps is not limited to unit testing. Various parameters need to be considered, including data checks, model drift, analysing model performance, etc. Deploying machine learning models is easier said than done as it involves various steps, including data processing, feature engineering, model training, model registry and model deployment. Lastly, MLOps engineers are expected to track data distribution with time to ensure the production environment is consistent with the data it is being trained on. The AI reality Last year, AI/ML research hit the doldrums in the wake of the pandemic; tech giants like Google slowed down hiring AI researchers and ML engineers, and Uber laid off their AI research and engineering team. According to a 2020 Stanford study, the AI sector saw an increase in private investments despite the pandemic. China surpassed the US in scholarly work on AI the same year, the report added. The demand for data scientists and machine learning engineers is at an all-time high. Machine learning engineers topped LinkedIn’s Emerging Jobs ranking, with a recorded growth of 9.8 times in five years (2012-2017). Similarly, data scientist jobs witnessed an 8.5x surge. Source: LinkedIn (Top 20 Emerging Jobs) MLOps engineers: Short in supply MLOps engineering, being a fledgling field, is witnessing a shortage of experienced professionals. Other factors for the shortage include: Lack of clarity in role and responsibility of MLOps engineer at the organisational level, especially startupsMultiple platforms and tools to learnShortage of dedicated courses for MLOps engineers Scribble Data CEO Venkata Pingali said companies are facing attrition in MLOps staff. “Ability to handle the loss of staff and associated knowledge is becoming a key requirement to design systems and processes,” said Pingali. We asked Pingali how companies could deal with attrition, he said whatever MLOps engineers build should be person independent, documented, and reproducible. Moreover, the companies should keep evaluating individual staff. Enterprises had to downsize at the beginning of Covid breakout, which hampered the deployment of AI projects globally. Plus, companies are still stuck with the traditional method while interviewing an MLOps engineer. “Companies should ignore the resume and should start giving realistic but simplified hands-on assignments to see how the potential staff handles problems,” said Pingali. An ideal MLOps engineer should have good discipline, architectural thinking, and tooling agility. Also, experience matters. Who can pivot? A software engineer can easily transition to the MLOps engineer role. However, they need to understand the nuances of data science. For this, software engineers’ mindset has to change – not necessarily the tooling. That is hard, Pingali said. Essential skills for MLOps engineer MLOps engineers need a strong foundation across the machine learning pipeline, tools, framework, and processes to deploy machine learning models throughout the project’s lifecycle systematically. The core skills a machine learning operations (MLOps) engineer should possess are: Expertise in cloud architecture/DevOps to recommend enterprise-grade solutions for operationalising AI analytics. An in-depth understanding of cloud platforms (AWS, Azure, etc.), AI lifecycle, and business problems to develop end-to-end (data/Dev/ML) Ops pipelines.Experience in project governance, alongside customer-facing experience of discovery, assessment, execution and operations.
MLOps, or machine learning operations, is emerging as one of the hottest fields. In the last four years, the hiring for machine learning and artificial intelligence roles has grown 74% annually. MLOps engineer, also known as DevOps for machine learning, covers a whole gamut of machine learning tasks starting from data integration to training and […]
MLOps engineering, being a fledgling field, is witnessing a shortage of experienced professionals.
["machine learning jobs", "ML jobs"]
["machine learning jobs", "ML jobs", "data science", "machine learning", "artificial intelligence", "AWS", "AI", "ML", "MLOps", "analytics", "model registry", "Azure"]
["AI", "artificial intelligence", "machine learning", "ML", "data science", "analytics", "MLOps", "model registry", "AWS", "Azure"]
Amit Raja Naik, known as the 'AI Human,' serves as the editor at AIM Media House, where he leads a team of talented tech journalists who are driving and shaping technology conversations across India and around the world.