This research aims to develop an open XR collaboration platform that induces high-quality immersion and social co-presence to enhance user collaboration quality. The platform is based on Scene Graph and utilizes digital twins to manage collaborative interactions of spatial models, static/dynamic objects, sound, hand gestures, and haptic elements through functions like creation, deletion, movement, and transformation. The study focuses on developing high-quality immersive elements and enhancing social co-presence by emphasizing various technologies. The research covers the composition and integration of 3D objects in the spatial domain, optimal alignment of scanned data and CAD models, sound reproduction techniques, application of hand gestures using reinforcement learning in a physics simulator, high-quality multi-modal haptic technology supporting wearable interfaces, avatar mesh-point cloud interaction, force-based interaction, deep learning models for 3D clothing reconstruction, adaptive spatial configuration between multiple VR clients and single AR hosts, CG avatar face and costume reconstruction with texture extraction, as well as the development of evaluation metrics and factors for enhancing social co-presence in XR.

In this project, we study a scene graph-based 3D scene reconstruction method for the dynamic digital twin. The dynamic digital twin track and update dynamic movement of physical objects in the indoor environment in real-time. In addition, the project studies XR interworking technology in which users in the field and remote users experience and manipulate the same environment directly or indirectly through realistic rendering-based augmented and virtual reality. Finally, this project will enable various realistic non-face-to-face collaboration applications. To implement this system, we intend to organically converge specialized technologies in various fields, such as computer vision, computer graphics, and indoor GIS.

For the effective application of smart glasses to everyday life, anyone can use them intuitively, reflect surrounding situations and individual characteristics, and ultimately research and development of WISE (Wearable Interface for Sustainable Enhancement) beyond the functional use of the interface. Dynamic interfaces reflecting the overall state of the user’s attention and intention, all surrounding spaces and objects are recognized as semantic objects to access related information, and the object is used as an input tool for each characteristic.

Our primary objective is to develop an XR Twin-based Space-telling Authoring Tool tailored for art museum curators. This tool aims to empower curators with the capability to virtually curate exhibitions by replicating exhibition spaces and arranging artworks within a virtual environment. To achieve this, our approach encompasses cutting-edge visualization, data transformation, simulation, and visitor information analysis technologies, all meticulously integrated to streamline pre-planning and enable online showcasing. Through this cultural technology research, our ultimate goal is to optimize exhibition curation practices and significantly enhance the overall museum experience.