How to build the next-generation data lake
A data lake is a fundamental component in the digital transformation roadmap for most companies. But, the ability to get data ready and available for generating insights in a governed manner is still one of the most complex, costly and time-consuming processes. While data lakes have been around for many years now, new tools and technologies are evolving and a new set of capabilities are added to data lakes to make them more cost-effective and improve their adoption. Manish Gupta, head of Azure Big Data engineering practice at Tiger Analytics and Muthu Govindarajan, head of AWS & GCP Big Data, in the presentation titled,  next-generation data lake spoke about the trends in the industry such as self-service based data ingestion/processing and delivery; analytics workbench for business teams; automated data classification & PII discovery; low/no code approach for data modernisation & migration programs etc. “The whole landscape of what you can do with data lakes is changing”, said Manish. Over the last few years, new capabilities have come up, said Muthu. He divided it into four segments: Segment 1: What’s been there? Leveraging big data and cloudDataOps- CI/CD automation and platform automationConfigurable reusable frameworks Segment 2: What is getting better? Granular and efficient updates and deletesGranular data access controlsAnalytics workbench (playground, lab setup for data scientist with automated data provisioning on-demand) Segment 3: what’s creating impact? Configurable self service-based data ingestion/ processing and deliveryLow/ no-code approach for data modernisation & migration programsVirtual lakehouse technologiesData platform and observability Segment 4: Cool trends Enterprise metadata knowledge graphsContainerised data engineering workloadsEvolving hybrid cloud architectures patterns Data lake management Manish spoke in detail about improving the speed and agility of data lake management: Self-service data lake managementIntelligent data catalogue and data discoveryImproving SQL Query & BI performance “One of the key factors that can help bring agility in the supply chain is about bringing self-service capabilities to the data. This means enabling all types of users to easily manage and govern data in a cloud lake environment themselves,” Manish said. It is important to create a robust and scalable pipeline for supporting a data lake and bring in the right automation and self-service capabilities. The self-service capabilities bring in higher agility and operational efficiency, allow users to leverage best practices in a reusable way with governance at the core, democratise data and reduce the burden on IT teams. The first step is to create a cloud platform and infra services, on top of which reusable components are built, followed by data pipelines and orchestration, APIs for data and platform management, and lastly self-service interfaces and data lake users. The goal of the business is to deliver data trusted by the user. “This is a step beyond data quality,” said Madhu. Observability works in three different stages. The first stage is basic data health monitoring, followed by advanced monitoring with prediction and data platform observability. Basic data health monitoring is a critical component in every data platform consisting of key capabilities like data catalogue, self-service data discovery, configurable tools, data health dashboards, case management, crowdsource data asset rating, alerts, and actions. He discussed how users could ensure quality and platform governance through two key ideas. Data observability Platform observability On the observability end, some value generators are important to make reliable data available on time to consumers. These include monitoring data flow and environment, monitoring performance, monitoring data security, analysing workloads, predicting issues and optimising reliable data delivery. REGISTER HERE TO ACCESS THE CONTENT
The whole landscape of what you can do with data lakes is changing.
The whole landscape of what you can do with data lakes is changing.
["AI Tool", "data lakes"]
["knowledge graphs", "Go", "GCP", "data lakes", "AI", "AWS", "R", "RAG", "analytics", "SQL", "AI Tool", "Azure"]
["AI", "analytics", "RAG", "knowledge graphs", "AWS", "Azure", "GCP", "R", "SQL", "Go"]
Avi Gopani is a technology journalist that seeks to analyse industry trends and developments from an interdisciplinary perspective at Analytics India Magazine. Her articles chronicle cultural, political and social stories that are curated with a focus on the evolving technologies of artificial intelligence and data analytics.