Presentation Information
[B-6-54]Cost-Efficiency Evaluation of Node Configurations in Distributed LLM Inference Using Cloud GPUs
◎△Takeru Kono1, Atuya Kanno2, Nagisa Nasu2, Kaichi Taguti2, Takahiro Koita2, Akihito Kohiga1 (1. Doshisha Univ., 2. Graduate School of Doshisha Univ.)
Keywords:
Mesh-LLM,AmazonWebService,Large Language Model / LLM,Distributed Inference,EC2Instance
In recent years, GPU acceleration has become essential for large language model (LLM) inference, while the procurement of high-performance GPU servers requires substantial initial investment. Cloud GPUs, such as those provided by AWS EC2, offer a practical alternative by allowing computational resources to be allocated only when needed. This study evaluates the cost efficiency of different node configurations for distributed LLM inference using Mesh-LLM on AWS EC2. Specifically, we compare a single large instance configuration with a multiple small instance configuration under the same total number of GPUs. In the evaluation, the GPU generation is unified to A10G, the model is fixed to QWEN2.5-14B with GGUF Q4 quantization, and AIME 2025 is used as the inference task. Processing time, throughput, and total cost are measured to analyze the relationship between performance degradation caused by inter-node communication and differences in hourly instance pricing. Through this evaluation, we aim to clarify which node configuration provides superior cost efficiency for distributed LLM inference in cloud GPU environments.