LLanMER 2025

The First International Workshop on Large Language Model-Oriented Empirical Research

Trondheim, Norway

held in conjunction with The ACM International Conference on the Foundations of Software Engineering (FSE), June 23 - June 27, 2025

News: The program schedule for LLANMER 2025 is now available!

News: Congratulations to the authors of accepted workshop papers! 🎉

News: The LLANMER workshop will be held Friday June 27, 2025 from 14:00-17:30!


Introduction

Large language models (LLMs) are very large deep learning models pre-trained on vast amounts of data. They can answer questions and assist users with different tasks such as composing emails, essays, and code. Ever since ChatGPT's launch in November 2022, researchers have conducted various studies or created tools to integrate LLMs into (1) the current practices of software development as well as maintenance, (2) the training of next-generation software engineers, and (3) human-computer interactions to facilitate better software/hardware usage.

Lots of research questions remain open regarding the methodologies for conducting empirical research with LLMs. For instance, what is the best usage of LLMs in different scenarios, what are rigorous measurements for LLM results, and what are the potential impacts of LLM-oriented research on ethics, economy, energy, as well as environment? All these questions are critical for researchers to conduct responsible, reliable, and reproducible empirical research. Thus, we organize this workshop focusing on methodologies for conducting empirical research with LLMs. This workshop intends to provide a venue for researchers and practitioners to share ideas, discuss obstacles and challenges in LLM-oriented empirical research, brainstorm solutions, and establish collaborations to define reliable LLM-oriented empirical methodologies in cost-efficient ways. To achieve that goal, our workshop will include a keynote talk, paper presentations, and a panel.

Areas of interest include but are not restricted to:

  • Methodologies: How should we leverage LLMs to solve real-world problems? In the problem-solution procedure, how can we reveal, measure, and address the hallucination issues of LLMs?
  • Measurements: How do we precisely measure the effectiveness of LLM usage and evaluate results? How can we ensure the reproducibility and representativeness of evaluation results? How can we evaluate LLM-based approaches in a scalable way?
  • Analytics: How do we compare different usage of LLMs? How do we ensure a fair comparison given the existence of randomness and hallucination issues?
  • Ethical Aspect: What approaches can we take to ensure that the LLM-oriented empirical research does not violate ethical regulations or raise ethical concerns?
  • Economy Aspect: What is the cost comparison between different usage of LLMs? How are LLM-based approaches compared with non-LLM-based approaches in terms of effectiveness, performance, runtime cost, and financial cost?
  • Energy Aspect: What is the energy consumption of different usage of LLMs? What kinds of LLM-oriented approaches are energy-saving solutions or energy-consuming solutions? How can we optimize the energy consumption by distinct LLM usage without compromising the effectiveness significantly?
  • Environment Aspect: What potential impacts LLM usage can introduce to our environment or society? How does that impact personal privacy, intellectual property, technology accessibility, and copyright?

Important dates

Paper submissions: February 25th, 2025 March 4, 2025 (extended)
Paper notifications: March 25th, 2025
Paper camera-ready: April 24th, 2025
Workshop date: June 27th, 2025

Program

Welcome and Opening Remarks (14:00 - 14:15)

  • Workshop Organizers

Session 1: LLMs for SE (14:15pm - 15:30pm)

  • [14:15-14:30] How Well Do ChatGPT Models Maintain Software?
    Md Mahir Asef Kabir and Sk Adnan Hassan
  • [14:30-14:45] Decoding CI/CD Practices in Open-Source Projects with LLM Insights
    Łukasz Chomątek, Jakub Papuga, Przemyslaw Nowak, and Aneta Poniszewska-Marańda
  • [14:45-15:00] Towards LLM-Based Automatic Playtest
    Yan Zhao and Chiawei Tang
  • [15:00-15:15] LLPut: Investigating Large Language Models for Bug Report-Based Input Generation
    Alif Al Hasan, Subarna Saha, Mia Mohammad Imran, and Tarannum Shaila Zaman
  • [15:15-15:30] Automatic Proof Generation: Fine-tuning and RAG in Reasoner vs. Math LLMs
    Juan Carlos Recio Abad, Ruben Saborido, and José Francisco Chicano García

Coffee Break (15:30 - 16:00)

Session 2: SE for LLMs (16:00pm - 16:45pm)

  • [16:00-16:15] From Prompts to Properties: Rethinking LLM Code Generation with Property-Based Testing
    Dibyendu Brinto Bose
  • [16:15-16:30] Research: A Tool and Workflow for Benchmarking LLM-Based Code Improvements
    Zeineb Rejiba, Raphael Eidenbenz, and Johnny Borkhoche
  • [16:30-16:45] The Impact of Hyperparameters on Large Language Model Inference Performance: An Evaluation of vLLM and HuggingFace Pipelines
    Matias Martinez

Session 3: LLM Applications in Education and Engineering (16:45 - 17:15pm)

  • [16:45-17:00] From Words to Wisdom: LLMs Summarizing Instructional Content
    Łukasz Chomątek, Wojciech Słabosz, and Aneta Poniszewska-Marańda
  • [17:00-17:15] Smart Building Operations and Virtual Assistants Using LLM
    Reachsak Ly, Alireza Shojaei, and Xinghua Gao

Closing Remarks (17:15 - 17:30pm)

  • Workshop Organizers

Submission details

In each submission, the first keyword should be either Research or Experience to imply the paper type. All paper submissions must conform to the FSE 2025 Format and Submission Guidelines. We call for two types of submissions to the workshop:

  • Long (up to 6 pages), or Short (up to 3 pages) Research Papers, plus at most 2 pages for references and well-marked appendices. These papers present research work in early stage. Position papers with exceptional visions will also be considered.
  • Long (up to 6 pages), or Short (up to 3 pages) Experience Papers, plus at most 2 pages for references and well-marked appendices. These submissions should report experience on the application or assessment of LLMs in a non-trivial setting.

All the submissions must not have been published elsewhere or under review elsewhere when being considered for LLanMER 2025. Similar to FSE, LLanMER will employ a double-blind review process. Thus, no submission may reveal its authors’ identities. The authors must make every effort to honor the double-anonymous review process.

Please submit your papers through the following HotCRP link:

https://llanmer25.hotcrp.com/

For accepted papers (except for talk abstracts), authors are required to prepare their final submissions for the FSE 2025 companion proceedings based on the suggestions provided by reviewers, and one author is expected to attend the workshop and present the paper.

Organizers

Na Meng

Virginia Tech, USA

Xiaoyin Wang

The University of Texas at San Antonio, USA

Chris Brown

Virginia Tech, USA

Technical Program Committee

Waad Aldndni

Northern Border University, Saudi Arabia

Marvin Muñoz Barón

Technical University of Munich, Germany

Sebastian Baltes

University of Bayreuth, Germany

Chris Brown

Virginia Tech,
USA

Davide Fucci

Blekinge Institute of Technology, Sweden

Shu Lin

Institute of Automation Chinese Academy of Sciences, China

Seongmin Lee

Max Planck Institute for
Security and Privacy, Germany

Na Meng

Virginia Tech,
USA

Francisco Servant

University of Málaga,
Spain

Christoph Treude

Singapore Management University, Singapore

Xiaoyin Wang

The University of Texas at San Antonio, USA

Ying Zhang

Wake Forest University,
USA

Xi Zheng

Macquarie University,
Australia