Cognitive Lab Introduces Tokenizer Arena for Devanagari Text
When it comes to building Indic language models, Tokenization has been one of the most discussed topics as it is very different for each model. Looking at this, Cognitive Lab has introduced Tokenizer Arena on Hugging Face. The arena lets users compare different tokenizers simultaneously and is built on top of the transformer js library. Click here to check it out. https://twitter.com/adithya_s_k/status/1788670846136930506 The arena hosts several models such as Gemma, Mistral, all versions of Llama, GPT-3, GPT-4, Grok-1, Claude, Phi-3, and Command R. This is ideal for developers who are trying to build Indic LLM models on top of existing open source models for tokenizing on devanagari text, which is very different from English language. Recently, Google Researchers from India created the IndicGenBench dataset for multilingual benchmarking of different LLMs on 29 Indic languages, 13 scripts, and 4 language families. Adithya S Kolavi, the founder of Cognitive Lab recently created the Indic LLM Leaderboard for measuring different Indic LLMs rising in the country. The team also released Ambari, the first bilingual Kannada model built on top of Llama 2. The Indic LLM Leaderboard offers support for 7 Indic languages, including Hindi, Kannada, Tamil, Telugu, Malayalam, Marathi, and Gujarati, providing a comprehensive assessment platform. Hosted on Hugging Face, it initially supports 4 Indic benchmarks, with plans for additional benchmarks in the future.
The arena hosts several models such as Gemma, Mistral, all versions of Llama, GPT-3, GPT-4, Grok-1, Claude, Phi-3, and Command R.
Cognitive Lab has introduced Tokenizer Arena on Hugging Face for comparing different tokenizers simultaneously for Devanagari text.
["cognitive computing human capital", "Cognitive Lab", "Devanagari Text Generation"]
["Go", "Hugging Face", "programming_languages:R", "Devanagari Text Generation", "llm_models:Claude", "GPT", "Cognitive Lab", "ai_frameworks:Hugging Face", "llm_models:Llama", "cognitive computing human capital", "R", "llm_models:GPT"]
["Hugging Face", "R", "Go", "GPT", "llm_models:GPT", "llm_models:Claude", "llm_models:Llama", "ai_frameworks:Hugging Face", "programming_languages:R"]
Mohit writes about AI in simple, explainable, and often funny words. He's especially passionate about chatting with those building AI for Bharat, with the occasional detour into AGI.