Multimodal AI: The Next Frontier in Artificial Intelligence
AI has evolved beyond traditional single-data-source models to incorporate multiple data sources across various modalities. As noted by Anindya Sengupta, Client Partner, AI Client Services, and Abhijit Guha, Principal Data Scientist, AI Client Services at Fractal, the development of multimodal AI—systems that can process and integrate data from multiple modalities like text, images, audio, and video—has brought AI closer to mimicking human capabilities. For many years, AI was primarily limited to processing structured data. Guha noted that about five years ago, AI primarily relied on structured sources, or often required extensive “data munging” for unstructured sources before it could be fed into models. However, AI has evolved, and so has its ability to process and interpret more complex forms of unstructured data, such as human language, images, and video. Multimodal AI, as Anindya explained, “works with text, image, and video”. The concept involves taking various types of inputs from different faculties (e.g., vision, hearing, and speech) and combining them into a unified model that can mimic human-like processing and decision-making. Guha elaborated on this integration: “Multimodal AI is moving closer to how humans experience the world by integrating inputs from multiple sources—text, video, images—and processing them together.” The Mechanics of Multimodal AI While the underlying philosophy of AI—rooted in machine learning and number crunching—remains intact, multimodal AI introduces new challenges and techniques for integrating different data types. Guha explained that multimodal AI converts all inputs—text, image, or video—into numbers, which are then processed using machine learning algorithms. He describes three fusion techniques commonly used in multimodal AI: Early Fusion – This method involves merging different data sources at the initial stage before applying algorithms, allowing the model to process the data. Late Fusion – Here, data from various sources is processed independently, with the results being combined only at the end. Hybrid Fusion – This approach combines aspects of both early and late fusions, depending on the specific requirements of the data. Each technique has its strengths, and its application depends on the nature of the data and the specific task at hand. Real-World Applications of Multimodal AI The insurance industry offers a compelling example of multimodal AI’s capabilities. Anindya recounted a project where AI was used to assess if certain insurance claims were fraudulent. Previously, models only considered structured data like claim history and customer information. However, by integrating unstructured data, such as handwritten notes from claim adjusters, the accuracy of the model improved dramatically. “With structured data alone, we saw the accuracy score, or KS, hovering around 50-56,” Anindya explained. “But when we combined that with unstructured data, the KS jumped to 75-76.” This significant improvement illustrates the power of combining multiple data sources. Beyond insurance, multimodal AI is making waves in other sectors too. In human resource management, Fractal is developing an interview bot capable of analysing not only a candidate’s words, but also their body language and speech patterns. The bot can assess pauses, eye contact, and even detect potential fraud during virtual interviews. In the automotive sector, Fractal has partnered with companies to generate marketing content by combining images of cars with automatically generated captions. Similarly, in healthcare, Fractal’s Vaidya.ai integrates multiple data types—including prescription text, medical images, and X-rays—to provide healthcare professionals with a holistic understanding of the patient’s conditions. Challenges and Ethical Considerations While multimodal AI’s potential is vast, its development is not without challenges. According to Anindya, one of the biggest hurdles is the lack of large annotated datasets. “You can have the algorithms, but without the right data in a usable format, AI models will struggle to deliver accurate results.” Data processing also presents technical challenges. Handling large volumes of video and audio data requires significant computational power and storage, making these systems costly to develop and deploy. Guha added that while multimodal applications are in high demand, the cost of creating such systems remains a barrier to widespread adoption, especially in the B2C space. Ethical concerns also come into play, particularly when it comes to data security and bias. AI models are trained on data collected from human society, which inherently carries biases. Anindya acknowledged this, explaining that at Fractal, they focus heavily on ensuring their tools are certified ‘Responsible AI’. The goal is to develop models that minimise bias and follow strict ethical guidelines, even as they evolve to become more human-like. The Future of Multimodal AI Multimodal systems will likely become more prevalent across industries. According to Guha, one of the key advancements multimodal AI offers is “increasing the coverage of what AI can do.” By integrating multiple faculties—vision, language, and speech—AI can handle a wider array of tasks, offering more comprehensive solutions. In the coming years, multimodal AI will transform the landscape of artificial intelligence, bringing it significantly closer to replicating human cognition and decision-making.
Multimodal AI transforms how industries process unstructured data by integrating text, images, and video for enhanced decision-making.
Multimodal AI transforms how industries process unstructured data by integrating text, images, and video for enhanced decision-making.
["AI", "Editors Picks", "Multimodal"]
["Go", "machine learning", "artificial intelligence", "AI", "RAG", "responsible AI", "Editors Picks", "Multimodal", "Aim", "Ray", "multimodal AI", "R"]
["AI", "artificial intelligence", "machine learning", "multimodal AI", "Aim", "Ray", "RAG", "R", "Go", "responsible AI"]
Mohit writes about AI in simple, explainable, and often funny words. He's especially passionate about chatting with those building AI for Bharat, with the occasional detour into AGI.