講演情報
[PPS02-P20]月ミッション計画立案に特化したLLMについて、Fugaku-LLMとChatGPTのデータ学習、セッション、RAGでの言語差異比較
*新井 遥大1、出村 裕英1 (1.会津大学)
キーワード：
検索拡張生成、大規模言語モデル、ミッション計画立案、Fugaku-LLM、ChatGPT、月
In recent years, we have more opportunities to use large language models (LLMs) such as ChatGPT. However, general LLMs often suffer from hallucinations and factual errors, and we are seeing a shift from general LLMs to Domain-Specific LLMs (DSLMs). In addition, Vision Language Models (VLMs) have attracted attention for extending LLM capabilities by incorporating image modalities.
The scope is to develop a mission planning system for DSLM, specialized in schematic diagrams. Therefore, in this research, a Retrieval-Augmented Generation (RAG) system is established using existing LLMs and two language datasets, and the RAG outputs for each LLM and each language dataset are compared to investigate language-dependent differences in LLM behavior as a RAG system for DSLM development.
The two RAG systems are established using Fugaku-LLM and GPT-4.1. The RAG vector database is created using FAISS, and the system is implemented with LangChain. The Japanese dataset is Lunar Sciencebook, and the English dataset is Lunar Sourcebook. I would like to explain the structure of RAG using an architecture diagram during the poster session.
I compared the outputs of each LLM during Japanese and English sessions, when the LLM was trained on these data. The inputs consist of 30 questions about the Lunar mission in both Japanese and English for each evaluation axis. Then, I calculated the ratio of Japanese and English responses that matched among the questions.
This research focuses on whether the English and Japanese answers match, rather than whether the answers are correct. Whether the answers match is determined by me.
As a result, Fugaku-LLM excels at processing Japanese data, as evidenced by its ratio. However, GPT-4.1 provided accurate responses regardless of the language it was trained in.
In addition, the RAG system is evaluated using RAGAS(RAG Assessment) with the following parameters. Faithfulness, Answer_relevancy, Context_precision and Context_recall. These parameters can be obtained by inputting the text of the questions and answers submitted to RAG.
The values for faithfulness and answer_relevancy are generally high, so the RAG system can generate appropriate responses to questions based on the retrieved context. RAG learning data shows an increase in value compared to the default GPT-4.1, indicating that training the model improves its ability to search for context using context_score.
This study discussed the following points. The language capabilities of LLMs influence their responses based on differences in the language of the learning data. The retrieval-based question-answering system has been properly constructed and functions as intended. The performance of the LLM can be improved by increasing the amount of data
I would like to use highly performant models, larger datasets, and more specific data for lunar mission planning for future work.
The scope is to develop a mission planning system for DSLM, specialized in schematic diagrams. Therefore, in this research, a Retrieval-Augmented Generation (RAG) system is established using existing LLMs and two language datasets, and the RAG outputs for each LLM and each language dataset are compared to investigate language-dependent differences in LLM behavior as a RAG system for DSLM development.
The two RAG systems are established using Fugaku-LLM and GPT-4.1. The RAG vector database is created using FAISS, and the system is implemented with LangChain. The Japanese dataset is Lunar Sciencebook, and the English dataset is Lunar Sourcebook. I would like to explain the structure of RAG using an architecture diagram during the poster session.
I compared the outputs of each LLM during Japanese and English sessions, when the LLM was trained on these data. The inputs consist of 30 questions about the Lunar mission in both Japanese and English for each evaluation axis. Then, I calculated the ratio of Japanese and English responses that matched among the questions.
This research focuses on whether the English and Japanese answers match, rather than whether the answers are correct. Whether the answers match is determined by me.
As a result, Fugaku-LLM excels at processing Japanese data, as evidenced by its ratio. However, GPT-4.1 provided accurate responses regardless of the language it was trained in.
In addition, the RAG system is evaluated using RAGAS(RAG Assessment) with the following parameters. Faithfulness, Answer_relevancy, Context_precision and Context_recall. These parameters can be obtained by inputting the text of the questions and answers submitted to RAG.
The values for faithfulness and answer_relevancy are generally high, so the RAG system can generate appropriate responses to questions based on the retrieved context. RAG learning data shows an increase in value compared to the default GPT-4.1, indicating that training the model improves its ability to search for context using context_score.
This study discussed the following points. The language capabilities of LLMs influence their responses based on differences in the language of the learning data. The retrieval-based question-answering system has been properly constructed and functions as intended. The performance of the LLM can be improved by increasing the amount of data
I would like to use highly performant models, larger datasets, and more specific data for lunar mission planning for future work.