Publications

Abstract: Electrocardiogram (ECG) signals are frequently utilized for detecting important cardiac events, such as variations in ECG intervals, as well as for monitoring essential physiological metrics, including heart rate (HR) and heart rate variability (HRV). However, the accurate measurement of ECG traditionally requires a clinical environment, thereby limiting its feasibility for continuous, everyday monitoring. In contrast, Photoplethysmography (PPG) offers a non-invasive, cost-effective optical method for capturing cardiac data in daily settings and is increasingly utilized in various clinical and commercial wearable devices. However, PPG measurements are significantly less detailed than those of ECG. In this study, we propose a novel approach to synthesize ECG signals from PPG signals, facilitating the generation of robust ECG waveforms using a simple, unobtrusive wearable setup. Our approach utilizes a Transformer-based Generative Adversarial Network model, designed to accurately capture ECG signal patterns and enhance generalization capabilities. Additionally, we incorporate self-supervised learning techniques to enable the model to learn diverse ECG patterns through specific tasks. Model performance is evaluated using various metrics, including heart rate calculation and root minimum squared error (RMSE) on two different datasets. The comprehensive performance analysis demonstrates that our model exhibits superior efficacy in generating accurate ECG signals (with reducing 83.9\% and 72.4\% of the heart rate calculation error on MIMIC III and Who is Alyx? datasets, respectively), suggesting its potential application in the healthcare domain to enhance heart rate prediction and overall cardiac monitoring. As an empirical proof of concept, we also present an Atrial Fibrillation (AF) detection task, showcasing the practical utility of the generated ECG signals for cardiac diagnostic applications. To encourage replicability and reuse in future ECG generation studies, we have shared the dataset and will also make the code as publicly available.

Abstract: This paper presents the first systematic investigation of how congruence in visual self-representation influences self-perception and behavior. We span a continuum from the physical self through avatars with graded self-similarity to clearly dissimilar avatars in virtual reality (VR). In a 1x4 within-user study, participants completed movement and quiz tasks in either physical reality or a digital twin environment in VR, where they embodied one of three avatars: a photorealistic self-similar avatar, a dissimilar same-gender avatar, or a dissimilar opposite-gender avatar. Subjective measures included presence, sense of embodiment, self-identification, and perceived change, and were complemented by an objective movement metric of behavioral change. Compared to physical reality, VR, even with a self-similar avatar, produced lower presence, a weaker sense of embodiment, and reduced self-identification, revealing a persistent gap in visual congruence. Within VR, self-similar avatars enhanced body ownership, self-location, and self-identification relative to dissimilar avatars. Conversely, dissimilar avatars produced measurable behavioral changes compared with self-similar ones. Gender cues, however, had little impact in gender-neutral tasks. Overall, the findings show that photorealistic self-similar avatars reinforce embodiment and self-identification. However, VR still falls short of achieving congruence with physical reality, underscoring key challenges for avatar realism and ecological validity.

Abstract: Integrated Virtual Reality (IVR) systems are central to avatar-mediated use cases in Virtual Reality (VR), reconstructing users' movements on avatars. They differ primarily in their tracking architectures, which determine how completely and accurately users' movements are captured and reconstructed on avatars. Many current IVR systems reduce user-worn hardware, trading reconstruction accuracy against cost and setup complexity, yet their impact on user experience and task performance across use cases remains underexplored. We compared three reduced user-worn IVR systems. Each system has distinct technical approaches: (1) Captury (markerless outside-in optical tracking), (2) Meta Movement SDK (markerless inside-out optical tracking), and (3) Vive Trackers (marker-based outside-in optical tracking with IMUs). In a 3x5 mixed-design, participants performed five tasks, simulating different use cases, to probe distinct aspects of these systems. No system consistently outperformed the others. Meta excelled in hand-based, fast-paced interactions, while Captury and Vive performed better in lower-body tasks and during full-body pose observation. These findings underscore the need to evaluate reduced user-worn IVR systems within the specific use case. We offer practical guidance for system selection based on use-case demands and released our tasks as an open-source, extensible framework to support future evaluations for selecting IVR systems.

Abstract: Social Virtual Reality (SVR) allows users to meet and build relationships through embodied avatars and real-time interaction in virtual spaces. While embodiment can strengthen social connections and presence, it can also intensify negative encounters, making SVR particularly vulnerable to harassment. Despite frequent reports of verbal, visual, and "physical" violations in SVR, little is known about how harassment reshapes users' self-perception, including their sense of embodiment, self-identification, closeness, and avatar customization preferences. We conducted a controlled experiment with 52 participants who experienced either a neutral or a harassment condition in a scenario modeled after real SVR incidents. Participants perceived the harassing peer as significantly more negative, annoying, and disturbing than the neutral peer. Contrary to prior reports, harassment did not significantly affect well-being measures, including emotional state, self-esteem, and physiological arousal, within this controlled scenario. However, participants reported stronger bodily change, attributed more of their own attitudes and emotions to their avatars, and increased interpersonal distance when personal space was invaded. Self-reported coping strategies included ignoring, stepping back, using humor, and retaliating. Notably, avatar customization preferences shifted across conditions. Participants in the neutral condition favored personalized avatars, whereas those in the harassment condition more frequently preferred anonymity in public spaces. Together, these findings demonstrate that harassment in SVR not only exploits embodiment but also reshapes self-perception. We further contribute methodological insights into how harassment can be ethically and reproducibly studied in controlled SVR-like experiments.

Abstract: The global prevalence of mental health disorders has created a substantial treatment gap. To support clinicians and increase access to care, researchers in the field of Artificial Intelligence (AI) and Virtual Reality (VR) have investigated technology-mediated psychotherapy for years. However, research about stakeholders' concerns and their readiness to use AI in psychotherapy remains scarce. This study focuses on a user-centered approach to accommodate patients' concerns and, based on the results, implement measures to foster self-disclosure and trust towards an embodied AI therapist in VR. First, we conducted an online study with mental health patients ($N = 152$), which identified data autonomy and transparency as their primary ethical concerns. In a subsequent in-person VR study ($N = 90$) we compared effects of increased data autonomy and transparency on self-disclosure and trust towards an embodied AI therapist. Results indicated that higher data autonomy led to greater self-disclosure, while transparency had no significant effect. Manipulating data autonomy and transparency did not affect perceived trust, though exploratory calculations revealed that women reported significantly higher trust levels than men. These findings illuminate patients' priorities and provide implications for technical designs for AI-driven mental health care.

Abstract: We present CECA, a configurable framework for embodied conversational AI agents in Unity-based extended reality (XR) applications. CECA employs a client–server architecture to decouple agent logic from game engine–based embodiment. Built on LiveKit Agents, our approach integrates speech-to-text (STT), large language models (LLMs), and text-to-speech (TTS) into a unified, streaming voice-to-voice pipeline configured via metadata rather than code changes. We outline how this architecture flexibly integrates local and cloud AI providers while mitigating limited provider SDK support in Unity. Finally, we highlight opportunities for future work, including multi-agent scenarios, higher-level templates for XR research, and systematic user studies.

Abstract: Media bias is a multifaceted problem, leading to one-sided views and impacting decision-making. A way to address digital media bias is to detect and indicate it automatically through machine-learning methods. However, such detection is limited due to the difficulty of obtaining reliable training data. Human-in-the-loop-based feedback mechanisms have proven an effective way to facilitate the data-gathering process. Therefore, we introduce and test feedback mechanisms for the media bias domain, which we then implement on NewsUnfold, a news-reading web application to collect reader feedback on machine-generated bias highlights within online news articles. Our approach augments dataset quality by significantly increasing inter-annotator agreement by 26.31% and improving classifier performance by 2.49%. As the first human-in-the-loop application for media bias, the feedback mechanism shows that a user-centric approach to media bias data collection can return reliable data while being scalable and evaluated as easy to use. NewsUnfold demonstrates that feedback mechanisms are a promising strategy to reduce data collection expenses and continuously update datasets to changes in context.

Abstract: Facial expressions are crucial for many eXtended Reality (XR) use cases, from mirrored self exposures to social XR, where users interact via their avatars as digital alter egos. However, current XR devices differ in sensor coverage of the face region. Hence, a faithful reconstruction of facial expressions either has to exclude these areas or synthesize missing animation data with model-based approaches, potentially leading to perceivable mismatches between executed and perceived expression. This paper investigates potential effects of the coverage of facial animations (none, partial, or whole) on important factors of self-perception. We exposed 83 participants to their mirrored personalized avatar. They were shown their mirrored avatar face with upper and lower face animation, upper face animation only, lower face animation only, or no face animation. Whole animations were rated higher in virtual embodiment and slightly lower in uncanniness. Missing animations did not differ from partial ones in terms of virtual embodiment. Contrasts showed significantly lower humanness, lower eeriness, and lower attractiveness for the partial conditions. For questions related to self-identification, effects were mixed. We discuss participants' shift in body part attention across conditions. Qualitative results show participants perceived their virtual representation as fascinating yet uncanny.

Abstract: Virtual Reality Exposure Therapy (VRET) has become an effective, customizable, and affordable treatment for various psychological and physiological disorders. Specifically, it is used to treat specific anxiety disorders, such as acrophobia or arachnophobia, for decades. However, to ensure a positive outcome for patients, we must understand and control the effects potentially caused by the technology and medium of Virtual Reality (VR) itself. This article specifically investigates the impact of the Plausibility illusion (Psi), as one of the two theorized presence components, on the fear of heights. In two experiments, 30 participants each experienced two different heights with congruent and incongruent object behaviors in a 2 x 2 within-subject design. Results show that the strength of the congruence manipulation plays a significant role. Only when incongruencies are strong enough will they be recognized by users, specifically in high fear conditions, as triggered by exposure to increased heights. If incongruencies are too subtle, they seem to be overshadowed by the stronger fear reactions. Our evidence contributes to recent theories of VR effects and emphasizes the importance of understanding and controlling factors potentially assumed to be incidental, specifically during VRET designs. Incongruencies should be controlled so that they do not have an unwanted influence on the patient's fear response.

Abstract: Interpupillary distance (IPD) is a crucial characteristic of head-mounted displays (HMDs) because it defines an important property for generating a stereoscopic parallax, which is essential for correct depth perception. This is why contemporary HMDs offer adjustable lenses to adapt to users' individual IPDs. However, today's Video See-Through Augmented Reality (VST AR) HMDs use fixed camera placements to reconstruct the stereoscopic view of a user's environment. This leads to a potential mismatch between individual IPD settings and the fixed Inter-Camera Distances (ICD), which in turn can lead to perceptual incongruencies, limiting the usability and potentially the applicability of VST AR in depth-sensitive use cases. To investigate this incongruency between IPD and ICD, we conducted a 2x3 mixed-factor design empirical evaluation using a near-field, open-loop reaching task comparing distance judgments of Virtual Reality (VR) and VST AR. We also explored improvements in reaching performance via perceptual calibration by incorporating a feedback phase between pre- and post-phase conditions, with a particular focus on the influence of IPD-ICD differences. Our Linear Mixed Model (LMM) analysis showed a significant difference between VR and VST AR, a significant effect of IPD-ICD mismatch, as well as a combined effect of both factors. This novel insight and its consequences are discussed specifically for depth perception tasks in AR, eXtended Reality (XR), and potential use cases.

Abstract: This paper investigates if and how self-similarity and having motor control impact sense of embodiment, self-identification, and body weight perception in Augmented Reality (AR). We conducted a 2x2 mixed design experiment involving 60 participants who interacted with either synchronously moving virtual humans or independently moving ones, each with self-similar or generic appearances, across two consecutive AR sessions. Participants evaluated their sense of embodiment, self-identification, and body weight perception of the virtual human. Our results show that self-similarity significantly enhanced sense of embodiment, self-identification, and the accuracy of body weight estimates with the virtual human. However, the effects of having motor control over the virtual human movements were notably weaker in these measures than in similar VR studies. Further analysis indicated that not only the virtual human itself but also the participants' body weight, self-esteem, and body shape concerns predict body weight estimates across all conditions. Our work advances the understanding of virtual human body weight perception in AR systems, emphasizing the importance of factors such as coherence with the real-world environment.

Abstract: In this work, we explored asymmetric collaboration under two distinct tasks: collaborative sorting and conversational talking tasks. We answer the research question of how different tasks impact the user experience in asymmetric interaction. Our mixed design compared one symmetric and one asymmetric interaction and two tasks, assessing self-perception (presence, embodiment), other-perception (co-presence, social presence, plausibility), and task perception (task load, enjoyment). 52 participants collaborated in dyads on the two tasks, either using head-mounted displays (HMDs) or one participant using an HMD and the other a desktop setup. Results indicate that differences in social presence diminished or disappeared during the purely conversational talking task in comparison to the sorting task. This indicates that differences in how we perceive a social interaction, which is caused by asymmetric interaction, only occur during specific use cases. These findings underscore the critical role of task characteristics in shaping users’ social XR experiences and highlight that asymmetric collaboration can be effective across different use cases and is even on par with symmetric interaction during conversations.

Abstract: In social virtual reality (VR), harassment persists as a pervasive and critical issue. Prior work emphasizes its perceived realness and emotional impact. However, the influence of avatar design, particularly the role of self-similarity, remains underexplored. Self-similar avatars can enhance user identification and engagement, yet potentially intensify the psychological and physiological effects of harassment. Existing studies often rely on interviews or user-generated content, lacking systematic analysis and controlled comparisons. To address these gaps, we present a process for creating realistic VR harassment scenarios. We built a scenario based on existing literature and validated it with expert reviews and user feedback. We propose a 2 x 2 between-subjects design to systematically examine users' emotional and physiological states, their identification with avatars, and the effects of avatar self-similarity. The study design will deepen the understanding of harassment dynamics in VR. Additionally, it can provide actionable insights for designing safer, more inclusive virtual environments that promote user well-being and foster equitable communities.

Abstract: Body image distortions in anorexia nervosa pose significant therapeutic challenges, requiring innovative interventions. Virtual Reality (VR) and Augmented Reality (AR) technologies offer promising solutions, yet stakeholder preferences, from therapists and patients, remain unexplored. This methodological proposal outlines focus groups to compare VR and AR mirror exposures using personalized and body-weight-modifiable avatars in anorexia nervosa therapy. Therapists will evaluate therapeutic potential, risks, and practicality, while adolescent patients will assess comfort, stress responses, and usability. The findings aim to advance the user-centered integration of VR and AR into anorexia nervosa therapy, addressing critical treatment gaps.

Abstract: Body weight issues can manifest in low self-esteem through a negative body image or the feeling of unattractiveness. To explore potential interventions, the pilot study examined whether embodying a photorealistically personalized avatar with enhanced attractiveness affects self-esteem. Participants in the manipulation group adjusted their avatar's body weight to their self-defined ideal, while a control group used unmodified avatars. To confirm the manipulation, we measured the perceived avatars' attractiveness. Results showed that participants found avatars at their ideal weight significantly more attractive, confirming an effective manipulation. Further, the ideal weight group showed a clear trend towards higher self-esteem post-exposure.

Abstract: In co-located extended reality (XR) experiences, headsets occlude their wearers’ facial expressions, impeding natural conversation. We introduce two techniques to mitigate this using off-the-shelf hardware: compositing a view of a personalized avatar behind the visor (“see-through visor”) and reducing the headset’s visibility and showing the avatar’s head (“head substitution”). We evaluated them in a repeated-measures dyadic study (N = 25) that indicated promising effects. Collaboration with a confederate with our techniques, compared to a no-avatar baseline, resulted in quicker consensus in a judgment task and enhanced perceived mutual understanding. However, the avatar was also rated and commented on as uncanny, though participant comments indicate tolerance for avatar uncanniness since they restore gaze utility. Furthermore, performance in an executive task deteriorated in the presence of our techniques, indicating that our implementation drew participants’ attention to their partner’s avatar and away from the task. We suggest giving users agency over how these techniques are applied and recommend using the same representation across interaction partners to avoid power imbalances.

Abstract: This systematic review investigates the current state of research on multimodal fusion methods, i.e., the joint analysis of multimodal inputs, for intentional, instruction-based human-computer interactions, focusing on the combination of speech and spatially expressive modalities such as gestures, touch, pen, and gaze. We examine 50 systems from a User-Centered Design perspective, categorizing them by modality combinations, fusion strategies, application domains and media, as well as reusability. Our findings highlight a predominance of descriptive late fusion methods, limited reusability, and a lack of standardized tool support, hampering rapid prototyping and broader applicability. We identify emerging trends in machine learning-based fusion and outline future research directions to advance reusable and user-centered multimodal systems.

Abstract: This study explores how people interact in dyadic social eXtended Reality (XR), focusing on two main factors: the animation type of a conversation partner’s avatar and how immersed the user feels in the virtual environment. Specifically, we investigate how 1) idle behavior, 2) AI-generated gestures, and 3) motion-captured movements from a confederate (a controlled partner in the study) influence the quality of conversation and how that partner is perceived. We examined these effects in both symmetric interactions (where both participants use VR headsets and controllers) and asymmetric interactions (where one participant uses a desktop setup). We developed a social XR platform that supports asymmetric device configurations to provide varying levels of immersion. The platform also supports a modular avatar animation system providing idle behavior, real-time AI-generated co-speech gestures, and full-body motion capture. Using a 2×3 mixed design with 39 participants, we measured users’ sense of spatial presence, their perception of the confederate, and the overall conversation quality. Our results show that users who were more immersed felt a stronger sense of presence and viewed their partner as more human-like and believable. Surprisingly, however, the type of avatar animation did not significantly affect conversation quality or how the partner was perceived. Participants often reported focusing more on what was said rather than how the avatar moved.

Abstract: Virtual reality (VR) exercise applications are promising tools, e.g., for at-home training and rehabilitation. However, existing applications vary significantly in key design choices such as environments, embodiment, and virtual coaching, making it difficult to derive clear design guidelines. A prominent design choice is the use of embodied virtual coaches, which guide user interaction and provide feedback. In a user study with 76 participants, we investigated how different levels of performance specificity in feedback from an embodied virtual coach affect intermediate factors, such as VR experience, motivation, and coach perception. Participants performed lower-body movement exercises, i.e., Leg Raises and Knee Extensions, commonly used in knee rehabilitation. We found that highly performance-specific feedback led to higher scores compared to medium specificity for perceived realism, as well as the anthropomorphism and sympathy of the virtual coach, but did not affect motivation. Based on our findings, we propose the design suggestion to include precise, performance-specific details when creating feedback for a virtual coach. We observed a descriptive pattern of higher scores in the low specificity condition compared to the medium condition on most measures, which raises the possibility that less specific feedback may, in some cases, be perceived more positively than moderately specific feedback. These findings provide valuable insights into how design choices impact relevant intermediate factors that are crucial for maximizing both workout effectiveness and the quality of the virtual coaching experience.

Abstract:

Abstract: Immersive 3D mid-air sketching systems liberate users from the confines of traditional 2D sketching canvases. However, complications from perceptual challenges in Virtual Reality (VR), combined with the ergonomic and cognitive challenges of sketching in all three dimensions in mid-air lower the accuracy and aesthetic quality of 3D sketches. This paper explores how color-to-depth and opacity cues support users to create and perceive freehand, 3D strokes in room-scale sketching, unlocking a full 360° of freedom for creation. We implemented three graphic depth shader cues modifying the (1) alpha, (2) hue, and (3) value levels of a single color to dynamically adjust the color and transparency of meshes relative to their depth from the user. We investigated how these depth cues influence sketch efficiency, sketch quality, and total sketch experience with 24 participants in a comparative, counterbalanced, 4 x 1 within-subjects user study. First, with our graphic depth shader cues we could successfully transfer results of prior research in seated sketching tasks to room-scale scenarios. Our color-to-depth cues improved the similarity of sketches to target models. This highlights the usefulness of the color-to-depth approach even for the increased range of motion and depth in room-scale sketching. Second, our shaders assisted participants to complete tasks faster, spend a greater percentage of task time sketching, reduced the feeling of mental tiredness and improved the feeling of sketch efficiency in room-scale sketching. We discuss these findings and share our insights and conclusions to advance the research on improving spatial cognition in immersive sketching systems.

Abstract: This paper addresses the critical importance of standards and documentation in kinematic research, particularly within Extended Reality (XR) environments. We focus on the pivotal role of motion data, emphasizing the challenges posed by the current lack of standardized practices in XR user motion datasets. Our work involves a detailed analysis of 8 existing datasets, identifying gaps in documentation and essential specifications such as coordinate systems, rotation representations, and units of measurement. We highlight how these gaps can lead to misinterpretations and irreproducible results. Based on our findings, we propose a set of guidelines and best practices for creating and documenting motion datasets, aiming to improve their quality, usability, and reproducibility. We also created a web-based tool for visual inspection of motion recordings, further aiding in dataset evaluation and standardization. Furthermore, we introduce the XR Motion Dataset Catalogue, a collection of the analyzed datasets in a unified and aligned format. This initiative significantly streamlines access for researchers, allowing them to download partial or entire datasets with a single line of code and without the need for additional alignment efforts. Our contributions enhance dataset integrity and reliability in kinematic research, paving the way for more consistent and scientifically robust studies in this evolving field.

Abstract: Self-exposure using modulated embodied avatars in virtual reality (VR) may support a positive body image. However, further investigation is needed to address methodological challenges and to understand the concrete effects, including their quantification. We present an iteratively refined paradigm for studying the tangible effects of exposure to a modulated self in VR. Participants perform body-centered movements in front of a virtual mirror, encountering their photorealistically personalized embodied avatar with increased, decreased, or unchanged body size. Additionally, we propose different body size estimation tasks conducted in reality and VR before and after exposure to assess participants' putative-elicited perceptual adaptations.

Abstract: Depending on their formality, clothes do not only change one's appearance, but can also influence behavior and cognitive processes. Shoes are a special aspect of an outfit. Besides coming in various degrees of formality, their structure can affect human gait. Avatars used to embody users in immersive Virtual Reality (VR) can wear any kind of clothing. According to the Proteus Effect, the appearance of a user's avatar can influence their behavior. Users change their behavior in accordance to the expected behavior of the avatar. In our study, we embody 39 participants with a generic avatar of the user's gender wearing three different pairs of shoes as within condition. The shoes differ in degree of formality. We measure the gait during a 2-minute walking task during which participants wore the same real shoe and assess selective attention using the Stroop task. Our results show significant differences in gait between the tested virtual shoe pairs. We found small effects between the three shoe conditions with respect to selective attention. However, we found no significant differences with respect to correct items and response time in the Stroop task. Thus, our results indicate that virtual shoes are accepted by users and, although not eliciting any physical constraints, lead to changes in gait. This suggests that users not only adjust personal behavior according to the Proteus Effect, but also are affected by virtual biomechanical constraints. Also, our results suggest a potential influence of virtual clothing on cognitive performance.

Abstract: Our study investigated how virtual agents impact users in challenging VR environments, exploring if nonverbal animations affect social pressure, positive encouragement, and trust in 30 female participants. Despite showing signs of pressure and support during the experimental trials, we could not find significant differences in post-exposure measurements of social pressure and encouragement, interpersonal trust, and well-being. While inconclusive, the findings suggest potential, indicating the need for further research with improved animations and a larger sample size for validation.

Abstract: Previous work on cyberbullying has shown that the number of victims is increasing and the need for prevention is exceptionally high among younger school students (5th-9th grade). Due to the omnipresence of cyberattacks, victims can hardly distance themselves psychologically, thus experience an intrusion in almost all areas of life. The perpetrators, on the other hand, feel the consequences of their actions even less in cyberspace. However, there is a gap between the need and the existence of innovative prevention programs tied to the digital reality of the target group and the treatment of essential aspects of psychological distance. This article explores the design space, feasibility, and effectiveness of a unique VR-based cyberbullying prevention component in a human-centered iterative approach. The central idea is reflected in creating a virtual personal space invasion with virtual objects associated with cyberbullying making the everyday intrusion of victims tangible. A pre-study revealed that harmful speech texts in bright non-removable message boxes best transferred the psychological determinants associated with a personal space invasion to virtual objects contextualized in cyberbullying scenarios. Therefore, these objects were incorporated into a virtual prevention program that was then tested in a laboratory study with 41 participants. The results showed that the intervention could trigger cognitive dissonance and empathy. In the second step, the intervention was evaluated and improved in a focus group with the actual target group of children and adolescents. The improved application was then evaluated in a school workshop for five days with 100 children and adolescents. The children understood the metaphor of virtual space invasion by the harmful text boxes and reported the expected psychological effects. They also showed great interest in VR. In summary, this paper contributes to the innovative and effective prevention of cyberbullying by using the potential of VR. It provides empirical evidence from a laboratory experiment and a field study with a large sample from the target group of children and adolescents and discusses implications for future developments.

Abstract: Spatial User Interfaces along the Reality-Virtuality continuum heavily depend on accurate depth perception. However, current display technologies still exhibit shortcomings in the simulation of accurate depth cues, and these shortcomings also vary between Virtual or Augmented Reality (VR, AR: eXtended Reality (XR) for short). This article compares depth perception between VR and Video See-Through (VST) AR. We developed a digital twin of an existing office room where users had to perform five depth-dependent tasks in VR and VST AR. Thirty-two participants took part in a user study using a 1×4 within-subjects design. Our results reveal higher misjudgment rates in VST AR due to conflicting depth cues between virtual and physical content. Increased head movements observed in participants were interpreted as a compensatory response to these conflicting cues. Furthermore, a longer task completion time in the VST AR condition indicates a lower task performance in VST AR. Interestingly, while participants rated the VR condition as easier and contrary to the increased misjudgments and lower performance with the VST AR display, a majority still expressed a preference for the VST AR experience. We discuss and explain these findings with the high visual dominance and referential power of the physical content in the VST AR condition, leading to a higher spatial presence and plausibility.

Abstract: Text input is desirable across various eXtended Reality (XR) use cases and is particularly crucial for knowledge and office work. This article compares handwriting text input between Virtual Reality (VR) and Video See-Through Augmented Reality (VST AR), facilitated by physically aligned and mid-air surfaces when writing simple and complex sentences. In a 2x2x2 experimental design, 72 participants performed two ten-minute handwriting sessions, each including ten simple and ten complex sentences representing text input in real-world scenarios. Our developed handwriting application supports different XR displays, surface alignments, and handwriting recognition based on digital ink. We evaluated usability, user experience, task load, text input performance, and handwriting style. Our results indicate high usability with a successful transfer of handwriting skills to the virtual domain. XR displays and surface alignments did not impact text input speed and error rate. However, sentence complexities did, with participants achieving higher input speeds and fewer errors for simple sentences (17.85 WPM, 0.51% MSD ER) than complex sentences (15.07 WPM, 1.74% MSD ER). Handwriting on physically aligned surfaces showed higher learnability and lower physical demand, making them more suitable for prolonged handwriting sessions. Handwriting on mid-air surfaces yielded higher novelty and stimulation ratings, which might diminish with more experience. Surface alignments and sentence complexities significantly affected handwriting style, leading to enlarged and more connected cursive writing in both mid-air and for simple sentences. The study also demonstrated the benefits of using XR controllers in a pen-like posture to mimic styluses and pressure-sensitive tips on physical surfaces for input detection. We additionally provide a phrase set of simple and complex sentences as a basis for future text input studies, which can be expanded and adapted.

Abstract:

Cybersickness is still a prominent risk factor potentially affecting the usability of virtual reality applications. Automated real-time detection of cybersickness promises to support a better general understanding of the phenomena and to avoid and counteract its occurrence. It could be used to facilitate application optimization, that is, to systematically link potential causes (technical development and conceptual design decisions) to cybersickness in closed-loop user-centered development cycles. In addition, it could be used to monitor, warn, and hence safeguard users against any onset of cybersickness during a virtual reality exposure, especially in healthcare applications. This article presents a novel real-time-capable cybersickness detection method by deep learning of augmented physiological data. In contrast to related preliminary work, we are exploring a unique combination of mid-immersion ground truth elicitation, an unobtrusive wireless setup, and moderate training performance requirements. We developed a proof-of-concept prototype to compare (combinations of) convolutional neural networks, long short-term memory, and support vector machines with respect to detection performance. We demonstrate that the use of a conditional generative adversarial network-based data augmentation technique increases detection performance significantly and showcase the feasibility of real-time cybersickness detection in a genuine application example. Finally, a comprehensive performance analysis demonstrates that a four-layered bidirectional long short-term memory network with the developed data augmentation delivers superior performance (91.1% F1-score) for real-time cybersickness detection. To encourage replicability and reuse in future cybersickness studies, we released the code and the dataset as publicly available.

Abstract: Recent research shows that visualizing linguistic bias mitigates its negative effects. However, reliable automatic detection methods to generate such visualizations require costly, knowledge-intensive training data. To facilitate data collection for media bias datasets, we present News Ninja, a game employing data-collecting game mechanics to generate a crowdsourced dataset. Before annotating sentences, players are educated on media bias via a tutorial. Our findings show that datasets gathered with crowdsourced workers trained on News Ninja can reach significantly higher inter-annotator agreements than expert and crowdsourced datasets with similar data quality. As News Ninja encourages continuous play, it allows datasets to adapt to the reception and contextualization of news over time, presenting a promising strategy to reduce data collection expenses, educate players, and promote long-term bias mitigation.

Abstract: This paper introduces “Motion Passwords”, a novel biometric authentication approach where virtual reality users verify their identity by physically writing a chosen word in the air with their hand controller. This method allows combining three layers of verification: knowledge-based password input, handwriting style analysis, and motion profile recognition. As a first step towards realizing this potential, we focus on verifying users based on their motion profiles. We conducted a data collection study with 48 participants, who performed over 3800 Motion Password signatures across two sessions. We assessed the effectiveness of feature-distance and similarity-learning methods for motion-based verification using the Motion Passwords as well as specific and uniform ball-throwing signatures used in previous works. In our results, the similarity-learning model was able to verify users with the same accuracy for both signature types. This demonstrates that Motion Passwords, even when applying only the motion-based verification layer, achieve reliability comparable to previous methods. This highlights the potential for Motion Passwords to become even more reliable with the addition of knowledge-based and handwriting style verification layers. Furthermore, we present a proof-of-concept Unity application demonstrating the registration and verification process with our pretrained similarity-learning model. We publish our code, the Motion Password dataset, the pretrained model, and our Unity prototype on https://github.com/cschell/MoPs

Abstract: This work presents a study where we used incongruencies on the cognitive and the perceptual layer to investigate their effects on perceived plausibility and, thereby, presence and spatial presence. We used a 2x3 within-subject design with the factors familiar size (cognitive manipulation) and immersion (perceptual manipulation). For the different levels of immersion, we implemented three different tracking qualities: rotation-and-translation tracking, rotation-only tracking, and stereoscopic-view-only tracking. Participants scanned products in a virtual supermarket where the familiar size of these objects was manipulated. Simultaneously, they could either move their head normally or need to use the thumbsticks to navigate their view of the environment. Results show that both manipulations had a negative effect on perceived plausibility and, thereby, presence. In addition, the tracking manipulation also had a negative effect on spatial presence. These results are especially interesting in light of the ongoing discussion about the role of plausibility and congruence in evaluating XR environments. The results can hardly be explained by traditional presence models, where immersion should not be an influencing factor for perceived plausibility. However, they are in agreement with the recently introduced Congruence and Plausibility (CaP) model and provide empirical evidence for the model's predicted pathways.

Abstract: The Field of View (FoV) is a central technical display characteristic of Head-Mounted Displays (HMDs), which has been shown to have a notable impact on important aspects of the user experience. For example, an increased FoV has been shown to foster a sense of presence and improve peripheral information processing, but it also increases the risk of VR sickness. This article investigates the impact of a wider but inhomogeneous FoV on the perceived plausibility, measuring its effects on presence, spatial presence, and VR sickness as a comparison to and replication of effects from prior work. We developed a low-resolution peripheral display extension to pragmatically increase the FoV, taking into account the lower peripheral acuity of the human eye. While this design results in inhomogeneous resolutions of HMDs at the display edges, it also is a low complexity and low-cost extension. However, its effects on important VR qualities have to be identified. We conducted two experiments with 30 and 27 participants, respectively. In a randomized 2x3 within-subject design, participants played three rounds of bowling in VR, both with and without the display extension. Two rounds contained incongruencies to induce breaks in plausibility. In experiment 2, we enhanced one incongruency to make it more noticeable and improved the shortcomings of the display extension that had previously been identified. However, neither study measured the low-resolution FoV extension's effect in terms of perceived plausibility, presence, spatial presence, or VR sickness. We found that one of the incongruencies could cause a break in plausibility without the extension, confirming the results of a previous study.

Abstract: This paper investigates the integration of motion capture and virtual reality (VR) technologies in competitive ballroom dancing (slow walz, tango, slow foxtrott, viennese waltz, quickstep), aiming to analyze posture correctness and provide feedback to dancers for posture enhancement. Through qualitative interviews, the study identifies specific requirements and gathers insights into potentially helpful feedback mechanisms. Using Unity and motion capture technology, we implemented a prototype system featuring real-time visual cues for posture correction and a replay function for analysis. A validation study with competitive ballroom dancers reveals generally positive feedback on the system’s usefulness, though challenges like cable obstruction and bad usability of the user interface are noted. Insights from participants inform future refinements, emphasizing the need for precise feedback, cable-free movement, and user-friendly interfaces. While the program is promising for ballroom dance training, further research is needed to evaluate the system’s overall efficacy.

Abstract: Immersive 3D sketching systems empower users with tools to create sketches directly in the air around themselves, in all three dimensions, using only simple hand gestures. These sketching systems have the potential to greatly extend the interactive capabilities of immersive learning environments. The perceptual challenges of Virtual Reality (VR), however, combined with the ergonomic and cognitive challenges of creating mid-air 3D sketches reduce the effectiveness of immersive sketching used for problem-solving, reflection, and to capture fleeting ideas. We contribute to the understanding of the potential challenges of mid-air sketching systems in educational settings, where expression is valued higher than accuracy, and sketches are used to support problem-solving and to explain abstract concepts. We conducted an empirical study with 36 participants with different spatial abilities to investigate if the way that people sketch in mid-air is dependent on the goal of the sketch. We compare the technique, quality, efficiency, and experience of participants as they create 3D mid-air sketches in three different tasks. We examine how users approach mid-air sketching when the sketches they create serve to convey meaning and when sketches are merely reproductions of geometric models created by someone else. We found that in tasks aimed at expressing personal design ideas, between starting and ending strokes, participants moved their heads more and their controllers at higher velocities and created strokes in faster times than in tasks aimed at recreating 3D geometric figures. They reported feeling less time pressure to complete sketches but redacted a larger percentage of strokes. These findings serve to inform the design of creative virtual environments that support reasoning and reflection through mid-air sketching. With this work, we aim to strengthen the power of immersive systems that support midair 3D sketching by exploiting natural user behavior to assist users to more quickly and faithfully convey their meaning in sketches.

Abstract: This work evaluates how the asymmetry of device configurations and verbal communication influence the user experience of social eXtended Reality (XR) for self-perception, other-perception, and task perception. We developed an application that enables social collaboration between two users with varying device configurations. We compare the conditions of one symmetric interaction, where both device configurations are Head-Mounted Displays (HMDs) with tracked controllers, with the conditions of one asymmetric interaction, where one device configuration is an HMD with tracked controllers and the other device configuration is a desktop screen with a mouse. In our study, 52 participants collaborated in a dyadic interaction on a sorting task while talking to each other. We compare our results to previous work that evaluated the same scenario without verbal communication. In line with prior research, self-perception is influenced by the immersion of the used device configuration and verbal communication. While co-presence was not affected by the device configuration or the inclusion of verbal communication, social presence was only higher for HMD configurations that allowed verbal communication. Task perception was hardly affected by the device configuration or verbal communication. We conclude that the device in social XR is important for self-perception with or without verbal communication. However, the results indicate that the device configuration only affects the qualities of social interaction in collaborative scenarios when verbal communication is enabled. To sum up, asymmetric collaboration maintains the high quality of self-perception and interaction for highly immersed users while still enabling the participation of less immersed users.

Abstract: In social virtual reality (social VR), users are threatened by potential cybercrimes, such as identity theft, sensitive data breaches, and embodied harassment. These concerns are heightened by the increasing interest in the metaverse, the advancements in photorealistic 3D user reconstructions, and the rising incidents of online privacy violations. Designing secure social VR applications that protect users while enhancing their experience, acceptance and trust remains a challenge. This article investigates potential identity management solutions in social VR, and their impacts on usability and user acceptance. We developed a social VR prototype with novel and established countermeasures, including motion biometric verification, and conducted a study with 52 participants. Our findings reveal diverse preferences for identity management and underscore the importance of authenticity, autonomy, and reciprocity. Key findings include: passive verification is favored for pragmatic user experience, while active verification is preferred for its hedonic quality; continuous or periodic verification strengthens users’ confidence in their privacy; and while user awareness promotes authentic engagement, it may also diminish the willingness to disclose personal information. This research not only offers foundational insights into the evaluated scenarios and countermeasures, but also sheds light on the designs of more trustworthy and inclusive social VR applications.

Abstract: The sense of presence and the sense of embodiment are two fundamental qualia, pivotal to many virtual reality experiences. Empirical research indicates a notable interdependence between these two qualia, where manipulations designed to affect one often exhibit a concurrent influence on the other. Existing theories on the development of qualia in virtual reality make no or only insufficient statements on this deep interdependence. In this work, we present a novel theoretical perspective on this connection. Based on existing theories, we argue that all the fundamental cues influencing one quale have the potential to impact the other one too. We present three studies ($n=42, n=42, n=32$) that generally support this novel perspective. Among other things, they show that traditional spatial presence cues such as head-tracking and passive depth cues (stereoscopy, linear perspective, etc.) can potentially also serve as embodiment cues. Conversely, they show that typical embodiment cues such as the visuotactile and visuoproprioceptive synchrony of a virtual hand are also spatial presence cues. The cues only differ in terms of how strongly they influence the respective quale. This novel perspective not only enhances our understanding of fundamental mechanics of virtual reality but it can also guide the development of more effective measurement instruments.

Abstract: Virtual Reality (VR) allows us to replace our visible body with a virtual self-representation (avatar) and to explore its effects on our body perception. While the feeling of owning and controlling a virtual body is widely researched, how VR affects the awareness of internal body signals (body awareness) remains open. Forty participants performed moving meditation tasks in reality and VR, either facing their mirror image or not. Both the virtual environment and avatars photorealistically matched their real counterparts. We found a negative effect of VR on body awareness, mediated by feeling embodied in and changed by the avatar. Further, we revealed a negative effect of a mirror on body awareness. Our results indicate that assessing body awareness should be essential in evaluating VR designs and avatar embodiment aiming at mental health, as even a scenario as close to reality as possible can distract users from their internal body signals.

Abstract: Mixed Reality (MR) applications along Milgram's Reality-Virtuality (RV) continuum motivated a number of recent theories on potential constructs and factors describing MR experiences. This paper investigates the impact of incongruencies on the sensation/perception and cognition layers to provoke breaks in plausibility, and the effects these breaks have on spatial and overall presence as prominent constructs of Virtual Reality (VR). We developed a simulated maintenance application to test virtual electrical devices. Participants performed test operations on these devices in a counterbalanced, randomized 2x2 between-subject design in either VR as congruent, or Augmented Reality (AR) as incongruent on the sensation/perception layer. Cognitive incongruency was induced by the absence of traceable power outages, decoupling perceived cause and effect after activating potentially defective devices. Our results indicate significant differences in the plausibility ratings between the VR and AR conditions, hence between congruent/incongruent conditions on the sensation/perception layer. In addition, spatial presence revealed a comparable interaction pattern with the VR vs AR conditions. Both factors decreased for the AR condition (incongruent sensation/perception) compared to VR (congruent sensation/perception) for the congruent cognitive case but increased for the incongruent cognitive case. The results are discussed and put into perspective in the scope of recent theories of MR experiences.

Abstract: This article compares two state-of-the-art text input techniques between non-stationary virtual reality (VR) and video see-through augmented reality (VST AR) use-cases as XR display condition. The developed contact-based mid-air virtual tap and wordgesture (swipe) keyboard provide established support functions for text correction, word suggestions, capitalization, and punctuation. A user evaluation with 64 participants revealed that XR displays and input techniques strongly affect text entry performance, while subjective measures are only influenced by the input techniques. We found significantly higher usability and user experience ratings for tap keyboards compared to swipe keyboards in both VR and VST AR. Task load was also lower for tap keyboards. In terms of performance, both input techniques were significantly faster in VR than in VST AR. Further, the tap keyboard was significantly faster than the swipe keyboard in VR. Participants showed a significant learning effect with only ten sentences typed per condition. Our results are consistent with previous work in VR and optical see-through (OST) AR, but additionally provide novel insights into usability and performance of the selected text input techniques for VST AR. The significant differences in subjective and objective measures emphasize the importance of specific evaluations for each possible combination of input techniques and XR displays to provide reusable, reliable, and high-quality text input solutions. With our work, we form a foundation for future research and XR workspaces. Our reference implementation is publicly available to encourage replicability and reuse in future XR workspaces.

Abstract: We report a mixed-design study on the effect of facial animation method (static, synthesized, or tracked expressions) and its synchronization to speaker audio (in sync or delayed by the method’s inherent latency) on an avatar’s perceived naturalness and plausibility. We created a virtual human for an actor and recorded his spontaneous half-minute responses to conversation prompts. As a simulated immersive interaction, 44 participants unfamiliar with the actor observed and rated performances rendered with the avatar, each with the different facial animation methods. Half of them observed performances in sync and the others with the animation method’s latency. Results show audio synchronization did not influence ratings and static faces were rated less natural and less plausible than animated faces. Notably, synthesized expressions were rated as more natural and more plausible than tracked expressions. Moreover, ratings of verbal behavior naturalness differed in the same way. We discuss implications of these results for avatar-mediated communication.

Abstract: Currently, there is an ongoing debate about the influencing factors of one's extended reality (XR) experience. Plausibility, congruence, and their role have recently gained more and more attention. One of the latest models to describe XR experiences, the Congruence and Plausibility model (CaP), puts plausibility and congruence right in the center. However, it is unclear what influence they have on the overall XR experience and what influences our perceived plausibility rating. In this paper, we implemented four different incongruencies within a virtual reality scene using breaks in plausibility as an analogy to breaks in presence. These manipulations were either located on the cognitive or perceptual layer of the CaP model. They were also either connected to the task at hand or not. We tested these manipulations in a virtual bowling environment to see which influence they had. Our results show that manipulations connected to the task caused a lower perceived plausibility. Additionally, cognitive manipulations seem to have a larger influence than perceptual manipulations. We were able to cause a break in plausibility with one of our incongruencies. These results show a first direction on how the influence of plausibility in XR can be systematically investigated in the future.

Abstract: When blending virtual and physical content, certain incongruencies emerge from hardware limitations, inaccurate tracking, or different appearances of virtual and physical content. They restrain us from perceiving virtual and physical content as one experience. Hence, it is crucial to investigate these issues to determine how they influence our experience. We present a virtualized augmented reality simulation that can systematically examine single incongruencies or different configurations.

Abstract: This paper introduces Reality Stack I/O (RSIO), a versatile and modular framework designed to facilitate the development of extended reality (XR) applications. Researchers and developers often spend a significant amount of time enabling cross-device and cross-platform compatibility, leading to delays and increased complexity. RSIO provides the essential features to simplify and unify the development of XR applications. It enhances cross-device and cross-platform compatibility, expedites integration, and allows developers to focus more on building XR experiences rather than device integration. We offer a public Unity reference implementation with examples.

Abstract:

Abstract: In 2021 we presented the Virtual Mission Control Room (VMCR) on the verge from fun educational project to testing ground for remote cooperative mission control. Since then, we successfully participated in ESA's 2022 campaign "New ideas to make XR a reality", which granted us additional funding to improve the VMCR software and conduct usability testing in cooperation with the chair of human-computer-interaction. In this paper and the corresponding poster session we give an update on the current state of the project, the new features and project structure. We explain the changes suggested by early test users and ESA to make operators feel more at home in the virtual environment. Subsequently, our project partners present their first suggestions for improvements to the VMCR as well as their plans for user testing. We conclude with lessons learned and and a look ahead into our plans for the future of the project.

Abstract: A novel theoretical model recently introduced coherence and plausibility as the essential conditions of XR experiences, challenging contemporary presence-oriented concepts. This article reports on two experiments validating this model, which assumes coherence activation on three layers (cognition, perception, and sensation) as the potential sources leading to a condition of plausibility and from there to other XR qualia such as presence or body ownership. The experiments introduce and utilize breaks in plausibility (in analogy to breaks in presence): We induce incoherence on the perceptual and the cognitive layer simultaneously by a simulation of object behaviors that do not conform to the laws of physics, i.e., gravity. We show that this manipulation breaks plausibility and hence confirm that it results in the desired effects in the theorized condition space but that the breaks in plausibility did not affect presence. In addition, we show that a cognitive manipulation by a storyline framing is too weak to successfully counteract the strong bottom-up inconsistencies. Both results are in line with the predictions of the recently introduced three-layer model of coherence and plausibility, which incorporates well-known top-down and bottom-up rivalries and its theorized increased independence between plausibility and presence.

Abstract: ABSTRACT The development of embodied Virtual Reality (VR) systems involves multiple central design choices. These design choices affect the user perception and therefore require thorough consideration. This article reports on two user studies investigating the influence of common design choices on relevant intermediate factors (sense of embodiment, presence, motivation, activation, and task load) in a VR application for physical exercises. The first study manipulated the avatar fidelity (abstract, partial body vs. anthropomorphic, full-body) and the environment (with vs. without mirror). The second study manipulated the avatar type (healthy vs. injured) and the environment type (beach vs. hospital) and, hence, the avatar-environment congruence. The full-body avatar significantly increased the sense of embodiment and decreased mental demand. Interestingly, the mirror did not influence the dependent variables. The injured avatar significantly increased the temporal demand. The beach environment significantly reduced the tense activation. On the beach, participants felt more present in the incongruent condition embodying the injured avatar.