Keyword: Large language models

Jia, Ye, Sin, Zackary P. T., Wang, Xiangzhi Eric, Li, Chen, Ng, Peter H. F., Huang, Xiao, Dong, Junnan, Wang, Yaowei, Baciu, George, Cao, Jiannong, Li, Qing

2024 IEEE International Conference on Metaverse Computing, Networking, and Applications (MetaCom) (2024)

Edu-metaverse is a specialized metaverse dedicated for interactive education in an immersive environment. Its main purpose is to immerse the learners in a digital environment and conduct learning activities that could mirror reality. Not only does it enable activities that may be difficult to perform in the real world, but it also extends the interaction to personalized and CL. This is a more effective pedagogical approach as it tends to enhance the motivation and engagement of students and it increases their active participation in lessons delivered. To this extend, we propose to realize an interactive virtual teaching assistant called NivTA. To make NivTA easily accessible and engaging by multiple users simultaneously, we also propose to use a CAVE virtual environment (CAVE-VR) as a "metaverse window" into concepts, ideas, topics, and learning activities. The students simply need to step into the CAVE-VR and interact with a life-size teaching assistant that they can engage with naturally, as if they are approaching a real person. Instead of text-based interaction currently developed for large language models (LLM), NivTA is given additional cues regarding the users so it can react more naturally via a specific prompt design. For example, the user can simply point to an educational concept and ask NivTA to explain what it is. To guide NivTA onto the educational concept, the prompt is also designed to feed in an educational KG to provide NivTA with the context of the student’s question. The NivTA system is an integration of several components that are discussed in this paper. We further describe how the system is designed and implemented, along with potential applications and future work on interactive collaborative edu-metaverse environments dedicated for teaching and learning.

Zhuohang Jiang, Pangjing WU, Ziran Liang, Chen, Peter Q, Xu Yuan, Ye Jia, Jiancheng Tu, Chen Li, Peter H. F. Ng, Qing Li

KDD 2025 Datasets and Benchmarks Track (2025)

Structure reasoning is a fundamental capability of large language models (LLMs), enabling them to reason about structured commonsense and answer multi-hop questions. However, existing benchmarks for structure reasoning mainly focus on horizontal and coordinate structures (\emph{e.g.} graphs), overlooking the hierarchical relationships within them. Hierarchical structure reasoning is crucial for human cognition, particularly in memory organization and problem-solving. It also plays a key role in various real-world tasks, such as information extraction and decision-making. To address this gap, we propose HiBench, the first framework spanning from initial structure generation to final proficiency assessment, designed to benchmark the hierarchical reasoning capabilities of LLMs systematically.HiBench encompasses six representative scenarios, covering both fundamental and practical aspects, and consists of 30 tasks with varying hierarchical complexity, totaling 39,519 queries.To evaluate LLMs comprehensively, we develop five capability dimensions that depict different facets of hierarchical structure understanding. Through extensive evaluation of 20 LLMs from 10 model families, we reveal key insights into their capabilities and limitations: 1) existing LLMs show proficiency in basic hierarchical reasoning tasks; 2) they still struggle with more complex structures and implicit hierarchical representations, especially in structural modification and textual reasoning. Based on these findings, we create a small yet well-designed instruction dataset, which enhances LLMs' performance on HiBench by an average of 88.84\% (Llama-3.1-8B) and 31.38\% (Qwen2.5-7B) across all tasks.The HiBench dataset and toolkit are available

Ye Jia, Chen Li, Zackary P. T. Sin, Wang, Xiangzhi Eric, Jiongning Lian, Peter H. F. Ng, Xiao Huang, George Baciu, Cao, Jiannong, Qing Li

IEEE International Conference on Metaverse 2025 (2025)

This study examines the impact of an AI-powered Virtual Teaching Assistant (NivTA) within a VR-based Edu-Metaverse (K-Cube), highlighting the roles of social presence, trust, and engagement in shaping learning outcomes. Grounded in Social Presence Theory, the Uses and Gratifications framework, and the Cognitive-Affective Theory of Learning with Media (CASTLE), our AI-Assisted Immersive Learning Framework emphasizes both cognitive and affective dimensions. A user study with 21 participants in a Cave Automatic Virtual Environment (CAVE) setting collected quantitative and qualitative data on trust, social presence, engagement, workload, and learning performance. Partial Least Squares Structural Equation Modeling revealed that heightened social presence fosters trust, which in turn drives behavioral, cognitive, and affective engagement. Notably, cognitive social presence was directly linked to better knowledge test scores, while confidence in test responses stemmed primarily from all forms of engagement. Overall, these findings underscore the significance of nurturing trust and social presence to enhance learner engagement and outcomes in AI-driven immersive educational environments.