HMK+Pointing: Mixed-character text entry on HMDs via a multimodal head-mounted keyboard and pointing

Efficient text entry on Head-Mounted Displays (HMDs) remains challenging, especially when switching between alphabetic and non-alphabetic characters. We address mixed-character text entry in HMD environments, where users must frequently alternate between typing and pointing. We propose a multimodal technique that integrates a head-mounted split physical keyboard with head- or gaze-based pointing: users type alphabetic characters physically and select symbols or numerals from a virtual keyboard using pointing. This design keeps the user’s hands on the keyboard, supports fast text entry, and reduces switching overhead between typing and pointing operations inherent in mixed-character interaction. User studies show that the technique can match or exceed controller-based text entry in controlled mixed-character tasks and significantly reduces task time and switching cost compared with gaze-only input in combined typing–pointing tasks. We also present application examples such as hybrid game control, object-linked messaging, and quick memo creation, illustrating its potential for text-centric entertainment and everyday immersive use. Overall, the technique offers a practical approach for supporting mixed-character interaction in HMDs and provides a foundation for continuous, text-centric interaction in both everyday and entertainment contexts.