Presentation Information
[4Yin-A-53]Construction of a Web UI Operation Dataset with Ambiguous Natural Language Instructions Containing Degree Adverbs
〇Yamamoto Hiromu1 (1. Waseda University)
Keywords:
Ambiguous instruction understanding,fuzziness,Web UI operation,alignment
This study addresses the problem of determining human-like amounts of Web UI operations in response to vague instructions containing degree adverbs, such as “scroll down a little.” Because degree adverbs do not specify quantitative thresholds, their interpretation can vary widely, and thus a framework is needed to quantitatively estimate “how much to operate” in web operation agents and assistive systems. To this end, we collected 1,664 operation logs (26 participants × 64 trials) for four common UI actions—scrolling, sliding, page navigation, and map zooming—and constructed a dataset that captures operation-amount distributions for each adverb. We further fine-tuned a Japanese LLM (Llama-3-ELYZA-JP-8B) for an operation-amount generation task, showing that the success rate of predictions falling within the human distribution (IQR) improved from 17.5% before training to 50.0% after training.