LLMs Get Smarter from Targeted Synthetic Multilingual Data

Language-specific competency (LSC) is the phenomenon of a language model performing better or worse depending on the language of the prompt. We introduce HOTFIXR, a framework that generates multilingual synthetic training data by identifying and addressing a model’s cross-lingual weaknesses. Our approach yields approximately 6.2% improvement in in-distribution performance, a 3.7% reduction in performance degradation on out-of-distribution tasks, and a 7.1% improvement on out-of-distribution languages. This work contributes to the efforts of making LLMs multilingually proficient for real-world applications that require multilingual capabilities.