AI-ENHANCED HUMAN-MACHINE INTERFACES USING INTEGRATED MULTI-MODAL SENSING AND HAPTIC-AUGMENTED FUNCTIONS FOR DIGITAL TWIN AND METAVERSE

Abstract

This thesis reports the development of AI-enhanced human-machine interfaces using integrated multi-modal sensing and haptic-augmented functions for digital twin and metaverse. Firstly a smart soft gripper with a fully self-powered multifunctional perception system capable of monitoring self-deformation and pressure/temperature tactile stimuli simultaneously is developed based on triboelectric and pyroelectric sensors. With machine learning analytics, high-accuracy object recognition is achieved for the developed smart robot to implement automatic item management for digital-twin-based unmanned working space. To boost the perception capability, the ultrasonic sensor is further integrated to realize the auto-positioning function and enhance the performance/robustness of the robotic identification system via multimodal data fusion technologies. Besides, wearable manipulators with multimodal sensing and haptic-feedback functions are also essential for immersive experiences in human- machine/human-robot interactions under the digital-twin-based framework when integrated with metaverse technologies. Based on this, augmented rings integrated with self-powered tactile/temperature sensors and low-power vibro-/thermo-haptic feedback units are achieved.