Presentation Information
[5Yin-A-15]Exploring Feedback-Driven Text Optimization for User-Level Fixed-Allocation A/B Testing
〇Masashi Ueno1, tetsu sato1, Naohiro Ikahata1 (1. PERSOL CAREER CO., LTD.)
Keywords:
LLM,text optimization
User-level fixed-allocation A/B testing is a method that assigns each user to a specific group for the entire duration of an experiment and compares the performance of the groups. Attempts to optimize text in accordance with evaluation outcomes by repeatedly conducting fixed-allocation A/B tests have traditionally relied on manual trial and error, which incurs substantial human effort. In this study, with the aim of automating sequential A/B testing, we investigate a method for optimizing text with respect to an evaluation signal under the condition that the evaluation is provided only as numeric values, by iteratively running A/B tests. We propose a hybrid approach that combines TextGrad, which is used in prompt optimization, with a genetic algorithm. Using opinions extracted from the Perspectrum dataset which collects diverse perspectives on a given claim as optimization targets, we optimize text based on evaluation signals from an LLM-as-a-judge and evaluate the performance of the proposed method. The results show that the proposed method outperforms baseline approaches in later generations.