[16p-K505-1~17] 23.1 Joint Session N "Informatics"

4:00 PM - 4:15 PM JST(7:00 AM - 7:15 AM UTC)

[16p-K505-12] Benchmark for multi-modal LLM in Materials Science

〇Michiko Yoshitake¹, Yuta Suzuki², Ryo Igarashi¹, Yoshitaka Ushiku¹, Keisuke Nagato³ (1.OSX, 2.Osaka Univ., 3.Univ. Tokyo)

Keywords:

LLM,benchmark,multi-modal

For the evaluation of multi-modal LLMs, a benchmark Q&A dataset with figures for materials science that can not be solved without the interpretation of the meaning of figures are constructed. Q&A were taken from common university textbook, while figures are modified without altering the essence of questions and the correct answers to avoid copy right problems. The examples of figure modification and some answers from LMMs will be presented.