DSL-13-SRMAP/Te-BERT_WR

Telugu-BERT_WR

Model Description

Telugu-BERT_WR is a Telugu sentiment classification model built on Telugu-BERT (L3Cube-Telugu-BERT), a Transformer-based BERT model pretrained exclusively on Telugu text by the L3Cube Pune research group.

The base model is pretrained on Telugu OSCAR, Wikipedia, and news corpora using the Masked Language Modeling (MLM) objective. Being tailored specifically for Telugu, Telugu-BERT captures language-specific vocabulary, syntax, semantics, and idiomatic expressions more effectively than multilingual models such as mBERT and XLM-R.

The suffix WR denotes With Rationale supervision. This model is fine-tuned using both sentiment labels and human-annotated rationales, enabling stronger alignment between predictions and human-identified evidence.

---

Pretraining Details

• Pretraining corpora:
  • Telugu OSCAR
  • Telugu Wikipedia
  • Telugu news data
• Training objective:
  • Masked Language Modeling (MLM)
• Language coverage: Telugu only

---

Training Data

• Fine-tuning dataset: Telugu-Dataset
• Task: Sentiment classification
• Supervision type: Label + rationale supervision
• Rationales: Token-level human-annotated evidence spans

---

Rationale Supervision

During fine-tuning, human-provided rationales are incorporated alongside sentiment labels. In addition to the standard classification loss, an auxiliary rationale loss guides the model to align its attention or explanation scores with annotated rationale tokens.

This supervision improves:

• Interpretability of sentiment predictions
• Alignment between model explanations and human judgment
• Plausibility of generated explanations

---

Intended Use

This model is intended for:

• Explainable Telugu sentiment classification
• Rationale-supervised learning experiments
• Monolingual Telugu NLP research
• Comparative evaluation against label-only (WOR) baselines

Telugu-BERT_WR is particularly suitable for pure Telugu text analysis when sufficient labeled data and human rationales are available.

---

Performance Characteristics

Rationale supervision enhances explanation quality and human alignment, while preserving the strong sentiment classification capability of Telugu-BERT.

Strengths

• Deep understanding of Telugu vocabulary and syntax
• Superior handling of nuanced and idiomatic sentiment expressions
• Human-aligned explanations through rationale supervision

Limitations

• Not designed for cross-lingual or multilingual tasks
• Requires annotated rationales, increasing annotation cost
• Performance depends on availability of sufficient Telugu training data

---

Use in Explainability Evaluation

Telugu-BERT_WR is well-suited for evaluation with explanation frameworks such as FERRET, enabling:

• Faithfulness evaluation: How well explanations support the model’s predictions
• Plausibility evaluation: How closely explanations align with human rationales

---

References

• Joshi et al. (2022). Telugu-BERT. EMNLP.