The Japanese Society for Artificial Intelligence

Large Language Models (LLMs) have set new benchmarks in various fields, achieving higher task performance with smaller training datasets. This success is largely attributed to the pretrain-finetune paradigm: models are first pretrained on extensive unlabeled corpora to develop general language understanding and then fine-tuned on smaller labeled datasets for specific tasks. While effective for many languages and tasks, this approach remains challenging for lower-resource languages, where labeled task data is scarce. Even Japanese, a higher-resource language, is held back by the relative scarcity of task-specific datasets. However, leveraging the wealth of English-language resources through cross-linguistic training offers a promising solution. This study investigates the cross-linguistic generalization capabilities of LLMs by fine-tuning a monolingual English model and its continually pretrained Japanese counterpart on English task datasets and evaluating them on comparable Japanese tasks. Our findings reveal that much of the task-specific knowledge imparted during fine-tuning transcends language boundaries, positing cross-lingual fine-tuning as a powerful strategy for enhancing LLM performance in lower-resource languages.