[5J2-OS-31a] OS-31

12:45 PM - 1:00 PM JST(3:45 AM - 4:00 AM UTC)

[5J2-OS-31a-04] Software Engineering for Treating LLMs as Lightweight General-Purpose ComponentsInsights from an On-device LLM Case Study

〇Takuya Oikawa¹ (1. Tably Inc.)

Keywords:

LLM,Software Engineering,On-device LLM,Modularization

The adoption of large language models (LLMs) is influencing design principles and development practices in software engineering. While many applications use LLMs via APIs, the design implications of treating LLMs as lightweight general-purpose components have not been sufficiently organized.

The objective of this paper is to identify design patterns for integrating on-device LLMs as lightweight general-purpose components and to clarify how these patterns differ from conventional modularization models. As a case study, we analyze a web application that employs an on-device LLM, using Gemini Nano as an example.

Through this analysis, we identify three distinctive design characteristics: (1) the necessity of cooperative design with rule-based components, (2) treating fallback mechanisms as part of the normal execution flow rather than exceptions, and (3) a shift in assumptions regarding reuse and caching due to the non-deterministic nature of LLM outputs. Based on these observations, this paper discusses how such design patterns may prompt a reconsideration of modularization, abstraction, and reuse in software engineering, and provides discussion points for software engineering in the LLM era.