[5O1-IS-1] Knowledge engineering

9:30 AM - 9:45 AM JST(12:30 AM - 12:45 AM UTC)

[5O1-IS-1-03] LLM Enrichment by Leveraging Domain-Specific Knowledge During Model Pre-training towards Full and Parameter Efficient Fine-Tuning

〇Mohammad Golam Sohrab¹ (1. National Institute of Advanced Industrial Science & Technology)

regular

Keywords:

LLM,Transfer Learning,Parameter Efficient Fine-Tuning,UMLS,Bioinformatics

This work introduces a transfer learning architecture that integrates domain-specific adaptation of model pretraining towards full-parameter and parameter efficient fine-tuning (PEFT). To better capture the awareness of knowledge in language model pre-training, a concept-aware a.k.a. entity-aware span masking denoising objective in the biomedical domain is introduced in the text-to-text model like T5 through a continual pretraining stage where medical concepts from unified medical language systems are masked. In continual pretraining, the model is initialized from the publicly available umt5-xl, which is a 3B parameter large language model. During full-parameter fine-tuning (FFT), the model can be computationally very expensive to fine-tune all the model parameters. To further facilitate this study, the most popular low-rank adaptation for PEFT is introduced into biomedical datasets. We are focused on measuring whether this concept-aware masking can enrich the downstream tasks by leveraging the knowledge during pre-training. Experiment results show that the FFT shows its dominance over the PEFT.