[2S01e] 10. ASPIRE - Part 6

6:03 PM - 6:06 PM JST(9:03 AM - 9:06 AM UTC)

[2ASPR-26] Provenance-Preserving Framework for Integrating Heterogeneous Ecosystem 

Monitoring Data With Large Language Models

○Jaronchai Dilokkalayakul 1, , Takeshi Obayashi 1,2 (1 Graduate School of Information Science, Tohoku University , 2 WPI-AIMEC, Tohoku University)

Retrieval-augmented generation (RAG) has emerged as an important approach for 

enabling large language models (LLMs) to answer questions using external evidence rather 

than relying only on their internal parameters. By retrieving relevant information from a 

knowledge source and providing it to an LLM as context, RAG can improve factual 

grounding, reduce unsupported generation, and support more transparent question 

answering. However, applying RAG to heterogeneous multimodal scientific data remains 

technically challenging, particularly in the context of ecosystem monitoring. Unlike 

ordinary text corpora, which usually consist of documents with relatively consistent 

structure and language, ecosystem monitoring data are produced by different 

instruments, analytical pipelines, and observational methods. These data often include 

structured CTD measurements describing physical and chemical water-column 

conditions, metagenomic abundance tables representing biological community 

composition, and satellite-derived remote sensing products describing surface 

environmental conditions. Each modality differs in format, spatial and temporal 

resolution, measurement assumptions, uncertainty, and scientific meaning. 

These differences create several obstacles for the direct application of conventional 

RAG pipelines. First, the data cannot simply be treated as interchangeable text documents, 

because each source has its own internal structure and interpretation. A CTD profile, for 

example, represents a vertical measurement of environmental conditions at a specific 

time and location, while a metagenomic sample represents biological signals derived from 

sequencing and taxonomic classification. Satellite sea surface temperature data, in 

contrast, provide gridded remote observations that may cover broader spatial and 

temporal scales. Second, scientific interpretation often depends on preserving metadata 

such as sampling time, location, depth, instrument type, processing method, and source 

file. If this contextual information is lost during preprocessing or embedding, the retrieved 

evidence may become misleading even if it appears semantically relevant. Third, 

LLM-generated answers must remain traceable to their original data sources so that 

researchers can verify the evidence, inspect the underlying observations, and reproduce 

the reasoning process. 

This work proposes a provenance-preserving multimodal RAG framework designed 

to integrate heterogeneous ecosystem monitoring data into a retrieval-oriented 

representation for source-grounded question answering. The framework converts 

different data modalities into structured evidence units while retaining their original 

provenance and modality-specific context. Rather than flattening all information into 

generic text, the system represents each evidence unit with both narrative descriptions 

and structured metadata. This allows CTD casts, metagenomic samples, and 

satellite-derived summaries to be retrieved and interpreted together while preserving 

their distinct scientific roles. The framework is intended to support questions that require 

cross-modal reasoning, such as identifying biological observations that coincide with 

warmer surface conditions or comparing environmental measurements with changes in 

taxonomic composition. 

A central feature of the proposed system is explicit traceability. Each data unit is 

linked to its original source, preprocessing history, and relevant metadata, allowing 

generated answers to be connected back to the evidence used during retrieval. This 

provenance-preserving design helps address one of the major risks of applying LLMs to 

scientific analysis: the possibility of plausible but unsupported responses. By maintaining 

source references throughout retrieval, reasoning, and answer generation, the system 

enables users to evaluate whether an answer is supported by the underlying data. This is 

especially important in scientific contexts, where interpretability and reproducibility are 

not optional but necessary conditions for trustworthy analysis. 

The framework also includes an LLM-based reasoning layer that synthesizes 

retrieved evidence into human-readable responses. Instead of replacing traditional 

scientific analysis, the LLM functions as an interface for exploring complex datasets 

through natural language. Retrieved evidence is provided to the model together with 

structured context, allowing the response to reflect both semantic relevance and scientific 

metadata. The system therefore supports a workflow in which researchers can ask 

high-level questions, inspect retrieved evidence, and trace conclusions back to specific 

observations. 

Overall, this study presents a framework for traceable, interpretable, and 

reproducible LLM-assisted analysis of heterogeneous multimodal scientific datasets. Its 

contribution lies not simply in applying RAG to ecosystem monitoring data, but in 

designing a retrieval structure that respects the differences between data modalities while 

preserving provenance. By integrating CTD measurements, metagenomic data, and 

satellite-derived observations into a source-grounded question-answering system, the 

proposed framework provides a foundation for more transparent use of LLMs in 

environmental and ecosystem research.