Hearing impaired people have the need, ability and right to enjoy music, however modern hearing aid technologies still unable to meet their needs for music perception and appreciation, which is not conducive to their social interaction and self-development. This paper firstly focuses on the needs and preferences of hearing impaired people in music activities, and emphasizes that the common way for deaf people to participate in music activities is to feel the vibration of sound waves through tactile channel. Secondly, it summarizes the advantages of tactile stimulation in representing auditory information, as well as the expressiveness competence, limitations and design methods of vibrotactile music, and reviews related works of wearable and static vibrotactile music devices. Tactile channel has advantages in several aspects such as cognitive load, temporal sequence, naturalness and immersiveness etc, not only it can compensate for hearing loss and enhance hearing perception, but also carry emotional communication. It also has limitations such as low resolution and difficulty in conveying high-frequency information. Finally, this paper proposes several research directions of UX design in the future, to provide reference for researchers in related fields.
[1] World Health Organization. World report on hearing[EB/OL]. (2021-03-03) [2022-03-25]. https://www. who. int/news-room/events/detail/2021/03/03/default-calendar/launch-of-the-world-report-on-hearing.
[2] Gugenheimer J, Plaumann K, Schaub F, et al. The impact of assistive technology on communication quality between deaf and hearing individuals[C]//Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing(CSCW'17), New York: ACM, 2017: 669-682.
[3] 陈四海, 魏姗 . 辅助技术的社会伦理挑战及应对[J]. 伦理学研究, 2020(2): 114-120.
[4] Evelyn Glennie[EB/OL]. [2022-03-25]. https://www. evelyn.co.uk.
[5] Mandy Harvey[EB/OL]. [2022-03-25]. https://mandyharveymusic.com.
[6] Troi Lee[EB/OL]. [2022-03-25]. https://www. deafrave.com.
[7] Sean Forbes[EB/OL]. [2022-03-25]. https://www.deafandloud.com/bio.
[8] Fulford R, Ginsborg J, Goldbart J. Learning not to listen: The experiences of musicians with hearing impairments [J]. Music Education Research, 2011(4): 447-464.
[9] Maler A. Music and deafness in the nineteenth-century U.S. imagination[J]. Journal of the Society for American Music, 2022(2): 184-205.
[10] 连赟 . 中国特殊音乐教育: 历史与现状研究[D]. 南京:南京艺术学院, 2010.
[11] 中华人民共和国教育部 . 聋校义务教育数学课程标准(2016年版)[S]. 北京: 北京师范大学, 2018.
[12] 杨晴 . 听障学生节奏感的训练与培养[J]. 现代特殊教育, 2019(5): 68-71.
[13] Fletcher M D. Can haptic stimulation enhance music perception in hearing-impaired listeners[J]. Frontiers in Neuroscience, 2021(15): e723877.
[14] Looi V, Rutledge K, Prvan T. Music appreciation of adult hearing aid users and the impact of different levels of hearing loss[J]. Ear Hear, 2019(3): 529-544.
[15] Jacobson D, Haptic or touch-based knowledge[J]. International Encyclopedia of Human Geography, 2020, doi: 10.1016/B978-008044910-4.00451-X.
[16] Lakatos P, Chen C M, O'connell M N, et al. Neuronal oscillations and multisensory interaction in primary auditory cortex[J]. Neuron, 2006(2): 279-292.
[17] Young G W, Murphy D, Weeter J, Haptics in music:The effects of vibrotactile stimulus in low frequency auditory difference detection tasks[J]. IEEE Transactions on Haptics, 2017(1): 135-139.
[18] Levänen S, Hamdorf D. Feeling vibrations: Enhanced tactile sensitivity in congenitally deaf humans[J]. Neuroscience Letters. 2001(1): 75-77.
[19] Haynes A C, Lawry J, Kent C, et al. Feel music: Enriching our emotive experience of music through audio-tactile mappings[J]. Multimodal Technologies and Interaction, 2021(6): 29.
[20] Good A, Reed M J, Russo F A. Compensatory plasticity in the deaf brain: Effects on perception of music[J]. Brain sciences, 2014(4): 560-574.
[21] 路璐, 田丰, 戴国忠, 等 . 融合触、听、视觉的多通道认知和交互模型[J]. 计算机辅助设计与图形学学报, 2014(4): 654-661.
[22] Birnbaum D, Wanderley M. A systematic approach to musical vibrotactile feedback[C]//Proceedings of the International Computer Music Conference, San Francisco:ICMC, 2007.
[23] Musical Vibration[EB/OL]. [2022-03-25]. http://www.musicalvibrations.com.
[24] Hopkins C, Maté -Cid S, Fulford R et al. Vibrotactile presentation of musical notes to the glabrous skin for adults with normal hearing or a hearing impairment: Thresholds, dynamic range and high-frequency perception[J]. PLoS One, 2016(5): e0155807.
[25] Mazzoni A, Bryan-Kinns N. How does it feel like? An exploratory study of a prototype system to convey emotion through haptic wearable devices[C]//Proceedings of the International Conference on Intelligent Technologies for Interactive Entertainment (INTETAIN). New York: ACM,2015: 64-68.
[26] Sharp A, Bacon B A, Champoux F. Enhanced tactile identification of musical emotion in the deaf[J]. Experimental Brain Research, 2020(5): 1229-1236.
[27] 熊兴福, 李姝瑶 . 感官代偿设计在产品中的应用[J]. 包装工程, 2009(10): 131-132, 139.
[28] Remache-Vinueza B, Trujillo-León A, Zapata M, et al. Audio-tactile rendering: A review on technology and methods to convey musical information through the sense of touch[J]. Sensors, 2021(19): 6575.
[29] Gunther E, Davenport G, O'Modhrain S. Cutaneous grooves: Composing for the sense of touch[C]//Proceedings of the 2002 conference on New interfaces for musical expression (NIME '02). New York: ACM, 2002: 1-6.
[30] Karam M, Russo F, Branje C, et al. Towards a model human cochlea: Sensory substitution for crossmodal audio-tactile displays[C]//Proceedings of Graphics Interface 2008 (GI '08). Canada: Canadian Information Processing Society, 2008: 267-274.
[31] Hattwick I, Franco I, Giordano M, et al. Composition techniques for the ilinx vibrotactile garment[C]//Proceedings of the International Computer Music Conference, Denton: ICMC, 2015: 420-423.
[32] Vest: A sensory substitution neuroscience project[EB/OL]. (2016-04-01)[2022-03-25]. https://www.kickstarter. com/projects/324375300/vest-a-sensory-substitution-neuroscience-project.
[33] Hashizume S, Sakamoto S, Suzuki K, et al. Livejacket: Wearable music experience device with multiple speakers[C]//Proceedings of the International Conference on Distributed, Ambient, and Pervasive Interactions, Cham, Switzerland: Springer, 2018: 359-371.
[34] SubPac M2[EB/OL]. [2022-03-25]. https://subpac. com/subpac-m2.
[35] Trivedi U, Alqasemi R, Dubey R V. Wearable musical haptic sleeves for people with hearing impairment[C]//Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments. New York: ACM, 2019: 146-151.
[36] La Versa B, Diamanti L, Peruzzi I, et al. Muvib: Music and vibration[C]//Proceedings of the 2014 ACM International Symposium on Wearable Computers Adjunct Program—ISWC'14 Adjunct. New York: ACM, 2014: 65-70.
[37] Petry B, Illandara T, Nanayakkara S. MuSS-bits: Sensordisplay blocks for deaf people to explore musical sounds [C]//Proceedings of the 28th Australian Conference on Computer-Human Interaction (OzCHI '16). New York: ACM, 2016: 72-80.
[38] Yamazaki Y, Mitake H, Oda R, et al. Hapbeat: Single DOF wide range wearable haptic display[C]//Proceedings of the SIGGRAPH Emerging Technologies (SIGGRAPH '17). New York: ACM, 2017: 1-2.
[39] M: NI(Vibrotextile™)[EB/OL]. [2022-03-25]. https://www.notimpossible.com/projects/music-not-impossible.
[40] 杜熙茹, 谷海玲, 王巧珍, 等 . 听障学生音乐节拍训练无线控制系统设计[J]. 电子设计工程, 2018(20): 178-182.
[41] Darrow A A. The effect of vibrotactile stimuli via the SOMATRONTM on the identification of pitch change by hearing impaired children[J]. Journal of Music Therapy. 1992(29): 103-112.
[42] Nanayakkara S, Taylor E, Wyse L, et al. An enhanced musical experience for the deaf: Design and evaluation of a music display and a haptic chair[C]//Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). New York: ACM, 2009: 337-346.
[43] Maria K, Carmen B, Gabe N, et al. The emoti-chair: An interactive tactile music exhibit[C]//Proceedings of the CHI Extended Abstracts on Human Factors in Computing Systems (CHI EA '10). New York: ACM, 2010: 3069-3074.
[44] Jack R, Mcpherson A, Stockman T. Designing tactile musical devices with and for deaf users: A case study[C]//Proceedings of the International Conference on the Multimedia Experience of Music. Sheffield: ICMEM, 2015:1-7.
[45] Hayes L. Skin music (2012): An audio-haptic composition for ears and body[C]//Proceedings of the SIGCHI Conference on Creativity and Cognition (C&C '15). New York: ACM, 2015: 359-360.
[46] Alves A F, Lima B F, Candido L S A, et al. Auris System: Providing vibrotactile feedback for hearing impaired population[J]. BioMed Research International.2017(1): 1-9.
[47] Fontana F, Camponogara I, Cesari P, et al. An Exploration on whole-body and foot-based vibrotactile sensitivity to melodic consonance[C]//Proceedings of the International Conference on Systems, Man, and Cybernetics, Hamburg, Germany: IEEE, 2016: 143-150.
[48] Eitan Z, Rothschild I. How music touches: Musical parameters and listeners' audio-tactile metaphorical mappings[J]. Psychology of Music, 2011, 39(4): 449-467.
