Forget Mixture of Experts, Mixture of Agents is Here
Just as we began getting comfortable with the mixture of experts (MoE) method, the mixture of agents (MoA) approach started to gain prominence. MoA takes the concept of specialisation a notch higher by leveraging the collective strengths of multiple LLMs. Unlike MoE, which operates within a single model, MoA employs a layered architecture where each layer comprises several LLM agents. “While mixture of experts is an innovative approach to overcome hardware restrictions, mixture of agents goes one step further in providing flexibility and depth, which is not possible with MoE,” Arjun Reddy, the co-founder of Nidum.AI, told AIM. For countries like India, where computational resources and data availability can be limiting factors, MoA offers a practical and scalable solution. MoA can achieve state-of-the-art results without the need for extensive computational power or data by utilising open-source models and focusing on collaboration rather than individual model performance. [CL] Mixture-of-Agents Enhances Large Language Model CapabilitiesJ Wang, J Wang, B Athiwaratkun, C Zhang, J Zou [Duke University & Together AI] (2024)https://t.co/G0MwggzhDt– Recent advances in large language models (LLMs) show great capabilities in language tasks. However,… pic.twitter.com/yKbAHBWJKL— fly51fly (@fly51fly) June 10, 2024 Recent research highlights the transformative potential of MoA. A study by Together AI demonstrates how MoA significantly enhances the capabilities of LLMs by constructing a layered architecture, where each layer comprises multiple agents. These agents collaboratively generate responses by utilising outputs from the previous layer, leading to state-of-the-art performance on benchmarks like AlpacaEval 2.0, MT-Bench, and FLASK. For instance, the MoA model achieved a score of 65.1% on AlpacaEval 2.0, outperforming GPT-4 Omni’s 57.5%. The Rise of MoA OpenAI is exploring the MoA framework through its multi-agent debate technique. This method involves multiple independent agents simultaneously attempting to solve the same problem proposed by the user. Each agent retains its solution in memory, and the system synthesises these solutions to arrive at a final response. In a post on X, Together AI explains how MoA works and how it can be implemented in just 50 lines of code, showcasing the approach’s simplicity and effectiveness. Together Mixture-Of-Agents in 3 minutes! We go over:◆ Explaining how Together Mixture-Of-Agents works ◆ Implementing MoA in just 50 lines of code ◆ Discussing good use cases to leverage MoA◆ Showing off results from tests we ranpic.twitter.com/JC7yGPSyeX— Together AI (@togethercompute) July 17, 2024 A research by Ajith’s AI Pulse elaborates on the MoA’s layered architecture, where each layer includes multiple LLM agents. Each agent processes the outputs of agents from the previous layer, refining and enhancing the response iteratively. This collaborative process enables the model to leverage the strengths of different LLMs, resulting in improved performance. The rise also favours general audience as they can create their own local mixture of agents all by using the Llama index-pack, giving a glimpse of how flexible and effective MoA is. Research paper titled ‘Mixture of Agents: A New Paradigm for Large Language Models’ provides a comprehensive theoretical foundation for the MoA framework. Exploring how the collaboration of multiple agents leads to improved performance metrics, enhanced accuracy, and scalability. «any layer, regardless of its position, can be used to compute a token as long as it possesses the needed processing capabilities»Mixture of Agents is so quaint. Transformer Soup is the future. https://t.co/DitcBxNCyA pic.twitter.com/Vg7rPk1pNB— Teortaxes▶️ (@teortaxesTex) July 10, 2024 Mixture of Agents in Action This innovative approach is already being harnessed in various cutting-edge applications, demonstrating its potential to revolutionise the field. For instance, the integration of MoA with Grok has shown remarkable improvements in AI performance, surpassing even GPT-4 in speed and efficiency. Notably, Andrej Karpathy has also shared his insights on MoA in his recent posts, discussing how people will take the Llama 3.1 405B, and distil and convert it into a small agent for narrow tasks and applications. This points towards a growing community of AI enthusiasts and professionals actively exploring the potential of MoA. Huge congrats to @AIatMeta on the Llama 3.1 release!Few notes:Today, with the 405B model release, is the first time that a frontier-capability LLM is available to everyone to work with and build on. The model appears to be GPT-4 / Claude 3.5 Sonnet grade and the weights are…— Andrej Karpathy (@karpathy) July 23, 2024 NVIDIA has demonstrated the use of AI agents to optimise supply chains through their cuOpt microservice. This system uses multiple LLM agents to handle complex optimisation tasks, transforming natural language queries into optimised plans. This approach exemplifies how MoA can be applied to real-world problems, enhancing efficiency and decision-making processes in large-scale operations.
While mixture of experts is an innovative approach to overcome hardware restrictions, mixture of agents goes one step further in providing flexibility and depth.
While mixture of experts is an innovative approach to overcome hardware restrictions, mixture of agents goes one step further in providing flexibility and depth.
["Agents", "MOE"]
["Go", "Agents", "TPU", "OpenAI", "AI", "Scala", "RAG", "GPT", "Aim", "Claude 3.5", "R", "MOE"]
["AI", "OpenAI", "Claude 3.5", "Aim", "RAG", "TPU", "R", "Go", "Scala", "GPT"]
A software engineer who loves to experiment with new-gen AI. He also happens to love testing hardware and sometimes they crash. While reviving his crashed system, you can find him reading literature, manga, or watering plants.