Responsible instructor: Dr. Tibor Guzsvinecz, Associate Professor, PhD
The course explores the opportunities and challenges of adaptive interactions based on Large Language Models (LLMs) within various digital reality environments. The focus is on natural language interactions, personalized experiences, support for learning and rehabilitation, and the impact of dynamically generated content on user experience and sense of presence.
The course provides theoretical foundations for the integration of LLMs, including the concept of language model-mediated social presence, and introduces both qualitative and quantitative methodologies to assess the effectiveness of such interactions in XR environments. Experimental design, conducting user studies, and critical evaluation of prototypes are also key components of the course.
Topics:
- Introduction to the functioning of LLMs
- Natural language interactions in XR
- Presence and immersion with language support
- Persona generation using LLMs for XR environments
- Implementation in digital realities
- Risks, ethics, and cognitive load
- Prototype presentation and critical analysis
References:
Bozkir, E., Özdel, S., Lau, K. H. C., Wang, M., Gao, H., & Kasneci, E. (2024). Embedding large language models into extended reality: Opportunities and challenges for inclusion, engagement, and privacy. In Proceedings of the 6th ACM Conference on Conversational User Interfaces (pp. 1-7). https://doi.org/10.1145/3640794.366556
Gao, H., Xie, Y., & Kasneci, E. (2025). PerVRML: ChatGPT-Driven Personalized VR Environments for Machine Learning Education. International Journal of Human–Computer Interaction. https://doi.org/10.1080/10447318.2025.2504188
Hutson, J. (2024). Combining Large Language Models and Immersive Technologies to Represent Cultural Heritage in the Metaverse Context. In Augmented and Virtual Reality in the Metaverse (pp. 265-281). Cham: Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-57746-8_14
Rahimi, F., Sadeghi-Niaraki, A., & Choi, S. M. (2025). Generative AI Meets Virtual Reality: A Comprehensive Survey on Applications, Challenges, and Future Direction. IEEE Access. https://doi.org/10.1109/ACCESS.2025.3574779
Sánchez-Berriel, I., Pérez-Nava, F., & Pérez-Rosario, L. (2025). Natural Interaction in Virtual Heritage: Enhancing User Experience with Large Language Models. Electronics, 14(12), 2478. https://doi.org/10.3390/electronics14122478
Tang, Y., Situ, J., Cui, A. Y., Wu, M., & Huang, Y. (2025, April). LLM Integration in Extended Reality: A Comprehensive Review of Current Trends, Challenges, and Future Perspectives. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (pp. 1-24). https://doi.org/10.1145/3706598.3714224