Interacting with computer systems with speech is more natural than conventional interaction methods. It is also more accessible since it does not require precise selection of small targets or rely entirely on visual elements like virtual keys and buttons. Speech also enables contactless interaction, which is of particular interest when touching public devices is to be avoided, such as the recent COVID-19 pandemic situation. However, speech is unreliable in noisy places and can compromise users’ privacy and security when in public. Image-based silent speech, which primarily converts tongue and lip movements into text, can mitigate many of these challenges. Since it does not rely on acoustic features, users can silently speak without vocalizing the words. It has also been demonstrated as a promising input method on mobile devices and has been explored for a variety of audiences and contexts where the acoustic signal is unavailable (e.g., people with speech disorders) or unreliable (e.g., noisy environment). Though the method shows promise, very little is known about peoples’ perceptions regarding using it, their anticipated performance of silent speech input, and their approach to avoiding potential misrecognition errors. Besides, existing silent speech recognition models are slow and error prone, or use stationary, external devices that are not scalable. In this dissertation, we attempt to address these issues. Towards this, we first conduct a user study to explore users’ attitudes towards silent speech with a particular focus on social acceptance. Results show that people perceive silent speech as more socially acceptable than speech input but are concerned about input recognition, privacy, and security issues. We then conduct a second study examining users’ error tolerance with speech and silent speech input methods. Results reveal that users are willing to tolerate more errors with silent speech input than speech input as it offers a higher degree of privacy and security. We conduct another study to identify a suitable method for providing real-time feedback on silent speech input. Results show that users find an abstract feedback method effective and significantly more private and secure than a commonly used video feedback method. In light of these findings, which establish silent speech as an acceptable and desirable mode of interaction, we take a step forward to address the technological limitations of existing image-based silent speech recognition models to make them more usable and reliable on computer systems. Towards this, first, we develop LipType, an optimized version of LipNet for improved speed and accuracy. We then develop an independent repair model that processes video input for poor lighting conditions, when applicable, and corrects potential errors in output for increased accuracy. We then test this model with LipType and other speech and silent speech recognizers to demonstrate its effectiveness. In an evaluation, the model reduced word error rate by 57% compared to the state-of-the-art without compromising the overall computation time. However, we identify that the model is still susceptible to failure due to the variability of user characteristics. A person's speaking rate, for instance, is a fundamental user characteristic that can influence speech recognition performance due to the variation in acoustic properties of human speech production. We formally investigate the effects of speaking rate on silent speech recognition. Results revealed that native users speak about 8% faster than non-native users, but both groups slow down at comparable rates (34-40%) when interacting with silent speech, mostly to increase its accuracy rates. A follow-up experiment confirms that slowing down does improve the accuracy of silent speech recognition. The method yields the best accuracy rate when speaking at 0.75x of the actual speaking rate. These findings highlight the importance of considering speaking rate in silent speech-based interfaces. Finally, we evaluate the effectiveness of the modality in an actual computer system. Particularly, we study the feasibility of using silent speech as a hands-free selection method in eye-gaze pointing on computer systems. Results revealed that silent speech is significantly better than other hands-free selection methods, namely dwell and speech, in terms of performance, usability, and perceived workload.

Text and numeric input and editing are prevalent in our daily digital interactions. Although text entry has been meticulously investigated by the human-computer interaction (HCI) community, numeric input and editing have not been as thoroughly examined. Consequently, the methods employed on mobile and wearable devices, such as smartwatches, are inefficient compared to their desktop counterparts. This inefficiency is primarily due to the smaller screen real estate of these devices, which cannot display all relevant information, the absence of haptic feedback provided by physical keyboards and keypads, and an excessive reliance on precise target selection, which is difficult on smaller screens. This dissertation investigates, designs, and evaluates novel methods for text and numeric input and editing for mobile and wearable devices by leveraging the existing capabilities of smartphones and smartwatches. The first part of this dissertation proposes three new numeric input and editing methods for smartwatches based on gestures, tilting, and force. The proposed methods enable users to actively switch between slow-and-steady and fast-and-continuous increments and decrements of numeric values during the input process. Results reveal that the gesture-based method yields a significantly faster input rate and is perceived as faster, more accurate, and the least mentally and physically demanding compared to the other methods. The second part explores the possibility of using a force-based approach to target selection on smartwatches and its use in character entry. The initial study identifies the most comfortable range of force on smartwatches. The subsequent study compares the performance of tap and force-tap in a Fitts' Law setting. Results revealed that force-tap is significantly better at selecting smaller targets, while tap outperforms force-tap for larger targets. We then develop two new force-based keyboards to demonstrate the feasibility of force input in practical scenarios. These single-row alphabetical keyboards enable character-level text entry by performing slides and varying contact force. In a user study, these keyboards yield about 4 wpm with about a 2% error rate, demonstrating the viability of force input on smaller screens. The third part of the dissertation presents GeShort, a novel method for one-handed text editing and formatting on mobile devices. It uses simple rules to facilitate direct cursor positioning, gestural shortcuts inspired by keyboard hotkeys for editing and formatting, and a floating clipboard to enable delayed, repeated, and block editing. A comparison between GeShort and the default Google keyboard reveals that users perform editing and formatting tasks about 11% and 22% faster, respectively, with GeShort, achieved by significantly reducing selection time by 11% and action time by 17%. A second study comparing the clipboard features of the two methods revealed that users perform advanced editing tasks 34% faster with GeShort. The findings presented in this dissertation illustrate that numeric and character entry and editing can be significantly improved in terms of speed, accuracy, and user preference. This is achieved by employing careful interface and interaction design, utilizing simple rules, and applying language models, all while making use of the existing capabilities of mobile and wearable devices. Importantly, this enhancement does not require the addition of extra sensors, new hardware, or the computational power necessary for complex learning models.

As the affordability and availability of virtual reality hardware increases, its adoption is also rapidly growing. Yet, input and interaction in virtual reality remains a significant challenge. Existing interaction techniques often lack intuitiveness, precision, and the spatial feedback users are accustomed to in the real world. They can also be cumbersome and lack mobility, further limiting usability and immersion. Furthermore, virtual reality lacks effective and efficient methods for text input. These limitations restrict the widespread adoption of virtual reality, primarily confining it to entertainment and training simulations. Furthermore, since virtual reality is still in its infancy, there is a lack of design guidelines to help designers and developers, leading to decisions driven primarily by intuition. This dissertation investigates the use of wearable and haptic technologies to overcome these limitations and create more intuitive and efficient virtual reality experiences.

First, addressing the challenges of text input in virtual reality, this dissertation begins with an investigation into the impact of key shape and dimension on text entry performance and preference. The aim is to contribute to the standardization of design practices in virtual reality through empirical data. We compare three common key shapes: hexagonal, round, square, in both two-dimension (2D) and three-dimension (3D). The results indicate that the 3D square keys provide superior performance in terms of accuracy and user preference. This suggests that replicating familiar real-world elements can significantly enhance usability, especially when a technology is in its infancy.

Second, building on these findings, we investigate mid-air text input, which is a common scenario in virtual reality environments. To address the lack of spatial feedback resulting from the absence of a physical surface, we utilize an ultrasonic haptic feedback device. In addition to incorporating the design insights from the first study, we develop three different ultrasonic haptic feedback methods: feedback only on keypress, on both touch and keypress, and gradual feedback that increases in intensity as users push down a key. A pilot study revealed that the touch & press feedback performed significantly better, both quantitatively and qualitatively. We therefore compare a mid-air keyboard with and without touch & press feedback in a user study. Results revealed that haptic feedback improves entry speed by 16% and reduces the error rate by 26%. In addition, most participants feel that it enhances presence and spatial awareness in the virtual world by maintaining a higher consistency with the real world and significantly reduces mental demand, effort, and frustration.

Third, extending upon mid-air interaction, we investigate the effectiveness of different selection gestures augmented with ultrasonic haptic feedback. We compare four commonly used mid-air target selection methods: Push, Tap, Dwell, Pinch, with two types of ultrasonic haptic feedback: feedback upon selection only, and feedback on both hover and selection, in a Fitts' law experiment. Results reveal that Tap is the fastest, the most accurate, and one of the least physically and cognitively demanding selection methods. Pinch is relatively fast but error-prone and physically and cognitively demanding. Dwell is slowest by design, yet the most accurate and the least physically and cognitively demanding. Both haptic feedback methods improve selection performance by increasing users' spatial awareness. Participants perceive the selection methods as faster, more accurate, and more physically and cognitively comfortable with the haptic feedback methods. Based on these findings, we provide guidelines for choosing optimal mid-air selection gestures considering technological limitations and task requirements.

Fourth, we further extend selection gestures and input methods in virtual reality by developing a custom wearable device that does not occupy the hands, thereby leaving them free for other tasks. We introduce a novel finger-worn device for gesture typing in virtual reality, termed the "digital thimble", which users wear on their index finger. This thimble utilizes an optical sensor to track finger movement and a pressure sensor to detect touch and contact force. We also introduce Shapeshifter, a technique that enables text entry in virtual reality through gestures and varying contact force on any opaque, diffusely reflective surface, including the human body. A week-long in-the-wild pilot study shows that Shapeshifter yields, on average, 11 words per minute (wpm) on flat surfaces (e.g., a desk), 9 wpm on the lap when sitting, and 8 wpm on the palm and back of the hand while standing in text composition tasks. In a simulation study, Shapeshifter achieves 27 wpm for text transcription tasks, outperforming current gesture typing techniques in virtual reality.

Finally, we extend the functionality of the digital thimble further and explore its usability in the context of target selection, sorting, and teleportation. We start with a Fitts' law study that compares the digital thimble with a commercial wearable mouse (previously unexplored in virtual reality contexts) and a traditional controller, using two selection methods: press and touch-release. A second user study investigates the devices for sorting and teleportation tasks. While the finger mouse demonstrated superior throughput and task completion speed, the digital thimble showed greater accuracy and precision. Participants also favored the digital thimble for its enhanced comfort, convenience, and overall user-friendliness. These findings highlight the digital thimble's potential as a versatile and comfortable input device for virtual reality applications, offering valuable advantages over traditional alternatives.

This dissertation makes significant progress in addressing the core challenges associated with input and interaction in virtual reality, setting the foundation for more intuitive and natural interactions within virtual environments. The insights gained have the potential to enhance the accessibility and applicability of virtual reality across a broad spectrum of fields, such as education, training, collaboration, and healthcare, thereby broadening its impact and utility.

Introduction and hypothesis

Methods

Results

Conclusions