Cross Model Knowledge Distillation: Enhancing Whisper Small via Teacher Generated Pseudo Labels Modeller Arasi Bilgi Damitma: Ögretmen Model Destekli Türetilmis Etiketler ile Whisper Small Modelinin Gelistirilmesi
34th Signal Processing and Communications Applications Conference, SIU 2026, İstanbul, Türkiye, 7 - 10 Temmuz 2026, (Tam Metin Bildiri)
- Yayın Türü: Bildiri / Tam Metin Bildiri
- Doi Numarası: 10.1109/siu71813.2026.11636607
- Basıldığı Şehir: İstanbul
- Basıldığı Ülke: Türkiye
- Anahtar Kelimeler: fine-tuning, knowledge distillation, pseudo-labels
- TED Üniversitesi Adresli: Evet
Özet
This study examines the Cross-Model Knowledge Distillation method to enhance the performance of the Whisper Small model on TBMM general assembly records. Within the scope of the research, the Whisper Large-v3 model was used as the "teacher"to generate pseudo-labels for the dataset, and these labels served as the primary data source in the fine-tuning process of the Whisper Small model. Experimental results indicate that the base model's WER decreased from 27.07% to 15.97% with the pseudo-labeling strategy, while the CER dropped from 7.68% to 3.89%. The achieved 15.97% WER value exhibits a success quite close to the 15.55% results from manual corrections. These findings demonstrate that teacher model-supported derived labels provide an effective and sustainable alternative to high-cost and time-consuming manual labeling processes in low-resource and technical domains such as parliamentary terminology.