Presentation Information
[PPS02-P20]Language differences between Fugaku-LLM & ChatGPT in learning, sessions, and RAGs on LLMs for Lunar mission planning
*Haruto Arai1, Hirohide Demura1 (1.The university of Aizu)
Keywords:
RAG,LLM,Mission planning,Fugaku-LLM,ChatGPT,The Moon
In recent years, we have more opportunities to use large language models (LLMs) such as ChatGPT. However, general LLMs often suffer from hallucinations and factual errors, and we are seeing a shift from general LLMs to Domain-Specific LLMs (DSLMs). In addition, Vision Language Models (VLMs) have attracted attention for extending LLM capabilities by incorporating image modalities.
The scope is to develop a mission planning system for DSLM, specialized in schematic diagrams. Therefore, in this research, a Retrieval-Augmented Generation (RAG) system is established using existing LLMs and two language datasets, and the RAG outputs for each LLM and each language dataset are compared to investigate language-dependent differences in LLM behavior as a RAG system for DSLM development.
The two RAG systems are established using Fugaku-LLM and GPT-4.1. The RAG vector database is created using FAISS, and the system is implemented with LangChain. The Japanese dataset is Lunar Sciencebook, and the English dataset is Lunar Sourcebook. I would like to explain the structure of RAG using an architecture diagram during the poster session.
I compared the outputs of each LLM during Japanese and English sessions, when the LLM was trained on these data. The inputs consist of 30 questions about the Lunar mission in both Japanese and English for each evaluation axis. Then, I calculated the ratio of Japanese and English responses that matched among the questions.
This research focuses on whether the English and Japanese answers match, rather than whether the answers are correct. Whether the answers match is determined by me.
As a result, Fugaku-LLM excels at processing Japanese data, as evidenced by its ratio. However, GPT-4.1 provided accurate responses regardless of the language it was trained in.
In addition, the RAG system is evaluated using RAGAS(RAG Assessment) with the following parameters. Faithfulness, Answer_relevancy, Context_precision and Context_recall. These parameters can be obtained by inputting the text of the questions and answers submitted to RAG.
The values for faithfulness and answer_relevancy are generally high, so the RAG system can generate appropriate responses to questions based on the retrieved context. RAG learning data shows an increase in value compared to the default GPT-4.1, indicating that training the model improves its ability to search for context using context_score.
This study discussed the following points. The language capabilities of LLMs influence their responses based on differences in the language of the learning data. The retrieval-based question-answering system has been properly constructed and functions as intended. The performance of the LLM can be improved by increasing the amount of data
I would like to use highly performant models, larger datasets, and more specific data for lunar mission planning for future work.
The scope is to develop a mission planning system for DSLM, specialized in schematic diagrams. Therefore, in this research, a Retrieval-Augmented Generation (RAG) system is established using existing LLMs and two language datasets, and the RAG outputs for each LLM and each language dataset are compared to investigate language-dependent differences in LLM behavior as a RAG system for DSLM development.
The two RAG systems are established using Fugaku-LLM and GPT-4.1. The RAG vector database is created using FAISS, and the system is implemented with LangChain. The Japanese dataset is Lunar Sciencebook, and the English dataset is Lunar Sourcebook. I would like to explain the structure of RAG using an architecture diagram during the poster session.
I compared the outputs of each LLM during Japanese and English sessions, when the LLM was trained on these data. The inputs consist of 30 questions about the Lunar mission in both Japanese and English for each evaluation axis. Then, I calculated the ratio of Japanese and English responses that matched among the questions.
This research focuses on whether the English and Japanese answers match, rather than whether the answers are correct. Whether the answers match is determined by me.
As a result, Fugaku-LLM excels at processing Japanese data, as evidenced by its ratio. However, GPT-4.1 provided accurate responses regardless of the language it was trained in.
In addition, the RAG system is evaluated using RAGAS(RAG Assessment) with the following parameters. Faithfulness, Answer_relevancy, Context_precision and Context_recall. These parameters can be obtained by inputting the text of the questions and answers submitted to RAG.
The values for faithfulness and answer_relevancy are generally high, so the RAG system can generate appropriate responses to questions based on the retrieved context. RAG learning data shows an increase in value compared to the default GPT-4.1, indicating that training the model improves its ability to search for context using context_score.
This study discussed the following points. The language capabilities of LLMs influence their responses based on differences in the language of the learning data. The retrieval-based question-answering system has been properly constructed and functions as intended. The performance of the LLM can be improved by increasing the amount of data
I would like to use highly performant models, larger datasets, and more specific data for lunar mission planning for future work.