Open-Source LLM360 Unveiled by Cerebras Systems, Petuum and MBZUAI
AI supercomputer company Cerebras Systems, AI company Petuum, and Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) launched LLM360, a framework for creating open-source large language models (LLMs). Developed in partnership with MBZUAI’s Institute of Foundation Models, LLM360 empowers developers by providing detailed insights and methodologies, promising to simplify, expedite, and reduce costs in the development of LLMs. Two open-source large language models are released : Amber, a 7 billion parameter English-language model trained on 1.2 trillion tokens, and CrystalCoder, a 7 billion parameter model, trained on 1.4T tokens designed for English language and coding tasks. Both the models are released under the Apache 2.0 license. There is also another model Diamond with 65 billion parameters which is set to release soon. These models are trained on the Condor Galaxy 1 supercomputer, built by G42 and Cerebras systems. Both the models are built on Meta’s LLaMA architecture and Amber is said to perform similarly to LLaMA-7B, OpenLLaMA-v2-7B and outperforms Pythia-6.7B. Source: LLM360 Blog CrystalCoder undergoes meticulous training, incorporating a thoughtful blend of text and code data to enhance its effectiveness in both domains. Notably, the introduction of code data occurs early in the pretraining stage, distinguishing it from Code Llama 2, which relies solely on code data during fine-tuning on Llama 2. Furthermore, CrystalCoder is specifically trained on Python and web programming language, strategically designed to elevate its capabilities as a programming assistant. UAE Heading Towards AI Dominance With the recent AI developments, UAE is working towards becoming an AI superpower. Following TII’s Falcon and demographic-specific Jais large language model, UAE has been also rallying for open-source models to promote research initiatives. With the recent AI company, A171 that was launched a few weeks ago, UAE looks to even take on AI giant OpenAI.
LLM360 releases Amber and CrystalCoder models that are built on Meta’s Llama architecture
LLM360 releases Amber and CrystalCoder models that are built on MEt’as Llama architecture
["Cerebras", "Cerebras systems", "Falcon", "Jais", "LLaMA", "Meta", "Python", "TII"]
["Go", "Meta", "artificial intelligence", "Cerebras", "OpenAI", "AI", "LLaMA", "llm_models:Llama", "Jais", "Python", "TII", "Cerebras systems", "Falcon", "programming_languages:Python", "foundation models", "R"]
["AI", "artificial intelligence", "foundation models", "OpenAI", "Python", "R", "Go", "Cerebras", "llm_models:Llama", "programming_languages:Python"]
As a rare blend of engineering, MBA, and journalism degree, Vandana Nair brings a unique combination of technical know-how, business acumen, and storytelling skills to the table. Her insatiable curiosity for all things startups, businesses, and AI technologies ensures that there's always a fresh and insightful perspective to her reporting. She now hosts her tech segment 'Point Break' on AIM Tv.