Browsing by Author "Md Hasibullah Hasib"

Now showing 1 - 1 of 1
  • Thumbnail Image
    Item
    Open Access
    Enhancing Bengali Language Processing: LoRA-Driven Adaptation in BLOOM
    (North South University, 2023) Md Hasibullah Hasib; Samira Ali; Irfanuddin Ahmed; Dr. Nabeel Mohammed (NbM); 1811451042; 2011423042; 1813476642
    Enhancing Bengali Language Processing: LoRA-Driven Adaptation in BLOOM In order to optimize a multilingual language model specifically for Bangla, this paper presents a novel method that uses LoRA (Low-Rank Adaptation) to significantly reduce model parameters without compromising performance. The article describes a thorough process that starts with quantizing the original model and ends with the use of LoRA configurations, which incorporate trainable rank decomposition matrices into the Transformer architecture while maintaining pre-trained weights. The astounding outcomes of this optimization show a significant decrease in the number of trainable parameters from 3 billion to 4.9 million, or just 0.16% of the original model. Interestingly, performance remains comparable to the larger, non-optimized version despite this reduction, suggesting that the model’s effectiveness is unaffected. Optimizing the LoRA-configured model on various Bangla datasets demonstrates the model’s flexibility and expertise even more. The Bangla2B+ Dataset, a proprietary conversational dataset, and a corpus of over 300,000 articles were used to tackle next-word prediction, question answering, and conversational tasks. Evaluation metrics showed a significant improvement, and perplexity scores decreased, outperforming the original model’s performance. These developments demonstrate the effectiveness of LoRA in lowering computational complexity and improving the model’s language comprehension abilities when it comes to the Bangla language. The work presents a new paradigm for effective and proficient language modeling in low-resource languages, demonstrating how LoRA can be used to simplify models for better performance without sacrificing computational efficiency. Given the dearth of multilingual models specifically designed for Bengali language processing, we aim to close a significant gap in language processing but also spur the development of customized NLP solutions in vital industries, improving accuracy and accessibility in domains centered around Bengali language.