Please use this identifier to cite or link to this item: https://doi.org/10.1038/s41467-021-25637-w
DC FieldValue
dc.titleAI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove
dc.contributor.authorWen, Feng
dc.contributor.authorZhang, Zixuan
dc.contributor.authorHe, Tianyiyi
dc.contributor.authorLee, Chengkuo
dc.date.accessioned2022-10-12T07:54:48Z
dc.date.available2022-10-12T07:54:48Z
dc.date.issued2021-09-10
dc.identifier.citationWen, Feng, Zhang, Zixuan, He, Tianyiyi, Lee, Chengkuo (2021-09-10). AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove. Nature Communications 12 (1) : 5378. ScholarBank@NUS Repository. https://doi.org/10.1038/s41467-021-25637-w
dc.identifier.issn2041-1723
dc.identifier.urihttps://scholarbank.nus.edu.sg/handle/10635/232300
dc.description.abstractSign language recognition, especially the sentence recognition, is of great significance for lowering the communication barrier between the hearing/speech impaired and the non-signers. The general glove solutions, which are employed to detect motions of our dexterous hands, only achieve recognizing discrete single gestures (i.e., numbers, letters, or words) instead of sentences, far from satisfying the meet of the signers’ daily communication. Here, we propose an artificial intelligence enabled sign language recognition and communication system comprising sensing gloves, deep learning block, and virtual reality interface. Non-segmentation and segmentation assisted deep learning model achieves the recognition of 50 words and 20 sentences. Significantly, the segmentation approach splits entire sentence signals into word units. Then the deep learning model recognizes all word elements and reversely reconstructs and recognizes sentences. Furthermore, new/never-seen sentences created by new-order word elements recombination can be recognized with an average correct rate of 86.67%. Finally, the sign language recognition results are projected into virtual space and translated into text and audio, allowing the remote and bidirectional communication between signers and non-signers. © 2021, The Author(s).
dc.publisherNature Research
dc.rightsAttribution 4.0 International
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/
dc.sourceScopus OA2021
dc.typeArticle
dc.contributor.departmentDEPT OF ELECTRICAL & COMPUTER ENGG
dc.contributor.departmentCOLLEGE OF DESIGN AND ENGINEERING
dc.description.doi10.1038/s41467-021-25637-w
dc.description.sourcetitleNature Communications
dc.description.volume12
dc.description.issue1
dc.description.page5378
Appears in Collections:Staff Publications
Elements

Show simple item record
Files in This Item:
File Description SizeFormatAccess SettingsVersion 
10_1038_s41467-021-25637-w.pdf10.29 MBAdobe PDF

OPEN

NoneView/Download

Google ScholarTM

Check

Altmetric


This item is licensed under a Creative Commons License Creative Commons