Soket AI Labs Unveils Pragna-1B, Multilingual Indic Language Model
Soket AI Labs has introduced Pragna-1B, India’s first open-source multilingual model designed to cater to the linguistic diversity of the country. Available in Hindi, Gujarati, Bangla, and English, Pragna-1B represents a significant step towards inclusive AI technology, transcending linguistic barriers and enhancing user engagement across diverse linguistic landscapes. Click here to check out the model on Hugging Face. Pragna-1B features a Transformer Decoder-only model with 1.25 billion parameters and a context length of 2048 tokens. Pragna-1B’s training involved 1.25 billion parameters, focusing on Hindi, Bangla, and Gujarati, processing approximately 150 billion tokens. It is engineered for efficient deployment on-device, Pragna-1B delivers state-of-the-art performance for vernacular languages in a small form factor. Despite its modest parameter count, Pragna-1B’s performance matches that of larger 7 billion parameter models, offering comprehensive multilingual support for English, Hindi, Bangla, and Gujarati. The model is meticulously trained on curated datasets specifically designed to encompass the Indian context, Pragna-1B ensures accurate and culturally relevant outputs. Pragna-1B is a decoder-only transformer model inspired by TinyLlama, with the following specifications: Layers: 22 Attention Heads: 32 Context Length: 2048 Hidden Dimension: 2048 Expansion Dimension: 5632 Vocabulary Size: 69632 Pragna employs a Byte-Pair Encoding (BPE) tokenizer, specifically trained for handling Indian languages, achieving a vocabulary size of 69,632. Soket AI Labs created “Bhasha,” a series of high-quality datasets specifically designed for training Indian language models. Bhasha-wiki: Includes 44.1 million articles translated from English Wikipedia into six Indian languages. Bhasha-wiki-indic: A refined subset of Bhasha-wiki, focusing on content relevant to India. Bhasha-SFT: Facilitates the development of language models for various NLP tasks. The researchers are also experimenting with a Mixture of experts model, expanding the languages, along with different architectures for increasing optimisation.
Pragna-1B features a Transformer Decoder-only model with 1.25 billion parameters and a context length of 2048 tokens.
Pragna-1B features a Transformer Decoder-only model with 1.25 billion parameters and a context length of 2048 tokens.
["India AI", "Soket AI Labs"]
["India AI", "Hugging Face", "TPU", "programming_languages:R", "AI", "Soket AI Labs", "RAG", "NLP", "ai_applications:NLP", "ai_frameworks:Hugging Face", "llm_models:Llama", "R"]
["AI", "NLP", "Hugging Face", "RAG", "TPU", "R", "llm_models:Llama", "ai_frameworks:Hugging Face", "programming_languages:R", "ai_applications:NLP"]
Mohit writes about AI in simple, explainable, and often funny words. He's especially passionate about chatting with those building AI for Bharat, with the occasional detour into AGI.