[2M4-GS-11a] AI and Society

2:15 PM - 2:30 PM JST(5:15 AM - 5:30 AM UTC)

[2M4-GS-11a-04] Social Bias Evaluation via Localization Grounded in Laws and Regulations

〇Masahiro Kaneko^2,3, Hiroshi Matsuda^1,3, Hisami Suzuki³, Satoshi Sekine³ (1. Megagon Labs, Tokyo, Recruit Co., Ltd., 2. MBZUAI, 3. NII)

Keywords:

AI safety,social bias,stereotype,discrimination

Large language models (LLMs) risk generating content that reflects discriminatory social biases, making systematic evaluation essential. However, what constitutes "discriminatory social bias" depends on social context, and defining universal value standards is difficult. Therefore, safety assurance should be grounded in value criteria that are socially agreed upon within each specific context. Prior studies on bias benchmark construction have addressed cross-national differences by employing local annotators for localization. Yet such approaches rely heavily on annotators’ subjective judgments and do not necessarily guarantee that safety criteria reflect nationally agreed standards. In this study, we propose a legal-based localization framework that uses statutes and relevant case law as the minimum socially agreed criteria for discriminatory bias in a given country. Focusing on Japan, we collect cases identified as discriminatory in the domains of employment and healthcare provision, and construct a social bias dataset, JLawBias. We conduct a preliminary evaluation of six Japanese LLMs to demonstrate the effectiveness of the proposed approach.