TOWARDS EMOTION RECOGNITION IN TEXTS – A SOUND-SYMBOLIC EXPERIMENT
The purpose of this study is to investigate the relationship between the phonetic content of prose texts in English and the emotion that the texts inspire, namely - the effect of vowel-consonant bi-phones on subjects’ evaluation of positive or negative emotional valence when reading. The methodology is based on data from an experiment where the participants, native speakers of three different languages, evaluated the valence invoked in them by one-page texts from English books. The sub-lexical level of the texts was obtained using phonetic transcriptions of the words and their further decomposition into vowel-consonant bi-phones. The statistical investigation relies on density-measures of the investigated bi-phones over each text as a whole. The result shows that there exists a correlation between the obtained sub-lexical representation and the valence perceived by the readers. Concerning the type of the consonants in the bi-phones (abrupt or sonorant), the influence of the abrupt bi-phones is stronger. However, sub-sets of both types of bi-phones showed relatedness with the emotional valence conveyed by the texts. In conclusion, the speech, expressed in written form, is laden with emotional valence even when the words’ lexicological meaning is not taken into consideration and the words are apprehended as mere phonetic constructs. This prompts hypothesizing that words’ semantics itself is partly underpinned by some mental emotion-related level of conceptualization, influenced by sounds. For practical purposes, the result suggests that based on the syllabic content of a text it should be possible to predict the valence that the text would inspire in its readers.
Abbassi, E., Kahlaoui, K., Wilson, M. A., & Joanette, Y. (2011). Processing the emotions in words: The complementary contributions of the left and right hemispheres. Cognitive, Affective, & Be-havioral Neuroscience, 11(3), 372-385. https://doi.org/10.3758/s13415-011-0034-1
Adelman, J. S., Estes, Z., & Cossu, M. (2018). Emotional sound symbolism: Languages rapidly signal valence via phonemes. Cognition, 175, 122-130. https://doi.org/10.1016/j.cognition.2018.02.007
Aryani, A., Conrad, M., & Jacobs, A. M. (2013). Extracting salient sublexical units from written texts:”Emophon,” a corpus-based approach to phonological iconicity. Frontiers in psychology, 4, 654. https://doi.org/10.3389/fpsyg.2013.00654
Aryani, A., Kraxenberger, M., Ullrich, S., Jacobs, A. M., & Conrad, M. (2016). Measuring the basic affective tone of poems via phonological saliency and iconicity. Psychology of Aesthetics, Creativity, and the Arts, 10(2), 191. https://doi.org/10.1037/aca0000033
Brandt, P. A., & Bennett, A. (2015). ‘It’s Five O’Clock’–Microprosody and Enunciation. Available at SSRN 2566228. Retrieved from: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2566228
Buechel, S., & Hahn, U. (2017, April). EMOBANK: Studying the impact of annotation perspective and representation format on dimensional emotion analysis. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers (pp. 578-585). https://doi.org/10.18653/v1/E17-2092
Burton, N. L. (2015). Heaven and hell: The psychology of the emotions. Kent, UK: Acheron Press.
D’Anselmo, A., Prete, G., Zdybek, P., Tommasi, L., & Brancucci, A. (2019). Guessing meaning from word sounds of unfamiliar languages: a cross-cultural sound symbolism study. Frontiers in Psychology, 10, 593. https://doi.org/10.3389/fpsyg.2019.00593
De Saussure, F. (1916/1983). Course in General Linguistics (trans. Roy Harris). London: Duckworth.
Edmiston, P., Perlman, M., & Lupyan, G. (2018). Repeated imitation makes human vocalizations more word-like. Proceedings of the Royal Society B: Biological Sciences, 285(1874), 20172709. https://doi.org/10.1098/rspb.2017.2709
Ekman, P. (1999). Basic emotions. Hand-book of cognition and emotion, 45-60. https://doi.org/10.1002/0470013494.ch3
Fantz, R. L., & Miranda, S. B. (1975). Newborn Infant Attention to Form of Contour. Child Development, 46(1), 224. https://doi.org/10.2307/1128853
Gökgöz, K. (2018). Syllables in TİD. Dilbilim Araştırmaları Dergisi, 29(1), 29-49. https://doi.org/10.18492/dad.374893
Hamann, S. (2012). Mapping discrete and dimensional emotions onto the brain: controversies and consensus. Trends in Cognitive Sciences, 16(9). http://doi.org/10.1016/j.tics.2012.07.006
Imai, M., & Kita, S. (2014). The sound symbolism bootstrapping hypothesis for language acquisition and language evolution. Philosophical transactions of the Royal Society B: Biological sciences, 369(1651), 20130298. https://doi.org/10.1098/rstb.2013.0298
Itô, J. (2018). Syllable theory in prosodic phonology. Routledge. https://doi.org/10.4324/9780429455124
Jakobson, R. (1960). Linguistics and poetics. In Style in language (pp. 350-377). MA: MIT Press.
Jones, J. M., & Vigliocco, G. (2017). Iconicity in Word Learning: What Can We Learn from Cross- Situational Learning Experiments?. Proceedings of the Annual Meeting of the Cognitive Society 2017. Retrieved from: https://pdfs.semanticscholar.org/ac61/7f1f53f8cc4df37b315edb4672df86294109.pdf
Kawahara, S., & Shinohara, K. (2012). A tripartite trans-modal relationship among sounds, shapes and emotions: A case of abrupt modulation. In Proceedings of the Annual Meeting of the Cognitive Science Society (p. Vol 34, No 34). Retrieved from https://pdfs.semanticscholar.org/3408/1ce795c89940e8fdd2a23b97a6aa834319dc.pdf
Kleinginna, P. R., & Kleinginna, A. M. (1981). A Categorized List of Emotion Definitions, with Suggestions for a Consensual Definition. Motivation and Emotion, 5(4), 981. https://doi.org/10.1007/BF00992553
Köhler, W. (1929). Gestalt Psychology. New York, NY: Liveright.
Leder, H., Tinio, P. P., & Bar, M. (2011). Emotional valence modulates the preference for curved objects. Perception, 40(6), 649-655. https://doi.org/10.1068/p6845
Macoir, J., Hudon, C., Tremblay M-P, Laforce R. Jr & Wilson M. A. (2019), The contribution of semantic memory to the recognition of basic emotions and emotional valence: Evidence from the semantic variant of primary progressive aphasia, Social Neuroscience, https://doi.org/10.1080/17470919.2019.1577295
Maddock, R. J., Garrett, A. S., & Buonocore, M. H. (2003). Posterior cingulate cortex activation by emotional words: fMRI evidence from a valence decision task. Human brain mapping, 18(1), 30-41. https://doi.org/10.1002/hbm.10075
Mandelker, A. (1983). Russian formalism and the objective analysis of sound in poetry. Slav. East Eur. J. 27, 327-338. https://doi.org/10.2307/307860
Marinkovic, K., Dhond, R. P., Dale, A. M., Glessner, M., Carr, V., & Halgren, E. (2003). Spatiotemporal dynamics of modality-specific and supramodal word processing. Neuron, 38(3), 487-497. https://doi.org/10.1016/S0896-6273(03)00197-1
Origlia, A., Cutugno, F., & Galatà, V. (2014). Continuous emotion recognition with phonetic syllables. Speech Communication, 57, 155-169. https://doi.org/10.1016/j.specom.2013.09.012
Perlman, M., Dale, R., & Lupyan, G. (2015). Iconicity can ground the creation of vocal symbols. Royal Society open science, 2(8), 150152. https://doi.org/10.1098/rsos.150152
Russell, J. A. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110(1), 145. http://doi.org/10.1037/0033-295X.110.1.145
Russell, J. A., & Mehrabian, A. (1977). Evidence for a Three-Factor Theory of Emotions. Journal of Research in Personality, 11, 273-294. http://doi.org/10.1016/0092-6566(77)90037-X
Saarimäki, H., Gotsopoulos, A., Jääskeläinen, I. P., Lampinen, J., Vuilleumier, P., Hari, R., ... & Nummenmaa, L. (2015). Discrete neural signatures of basic emotions. Cerebral cortex, 26(6), 2563-2573. https://doi.org/10.1093/cercor/bhv086
Scherer, K. R. (2000). A cross-cultural investigation of emotion inferences from voice and speech: Implications for speech technology. In Sixth International Conference on Spoken Language Processing. Retrieved from: https://www.isca-speech.org/archive/archive_papers/icslp_2000/i00_2379.pdf
Scherer, K. R., Banse, R., & Wallbott, H. G. (2001). Emotion inferences from vocal expression correlate across languages and cultures. Journal of Cross-cultural psychology, 32(1), 76-92. https://doi.org/10.1177/0022022101032001009
Shklovsky, V. (1990). Theory of Prose, transl. B. Sher. (Elmwood Park, IL: Dalkey Archive).
Sidhu, D. M., & Pexman, P. M. (2018). Five mechanisms of sound symbolic association. Psychonomic bulletin & review, 25(5), 1619-1643. https://doi.org/10.3758/s13423-017-1361-1
Trotsky, L. (1957). Literature and Revolution. New York, NY: Russell and Russell.
Ullrich, S., Aryani, A., Kraxenberger, M., Jacobs, A. M., & Conrad, M. (2017). On the relation between the general affective meaning and the basic sublexical, lexical, and inter-lexical features of poetic texts-a case study using 57 poems of HM Enzensberger. Frontiers in psychology, 7, 2073. https://doi.org/10.3389/fpsyg.2016.02073
Van Orden, G. C., Johnston, J. C., & Hale, B. L. (1988). Word identification in reading pro-ceeds from spelling to sound to meaning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 14(3), 371. https://doi.org/10.1037//0278-73220.127.116.111
Winter, B. (2016). Taste and smell words form an affectively loaded and emotionally flexible part of the English lexicon. Language, Cognition and Neuroscience, 31(8), 975-988. https://doi.org/10.1080/23273798.2016.1193619
Winter, B., Perlman, M., Perry, L. K., & Lupyan, G. (2017). Which words are most iconic?. Interaction Studies, 18(3), 443-464. https://doi.org/10.1075/is.18.3.07win
Whissell, C. (1999). Phonosymbolism and the Emotional Nature of Sounds: Evidence of the Preferential Use of Particular Phonemes in Texts of Differing Emotional Tone. Perceptual and Motor Skills, 89(1), 19-48. http://doi.org/10.2466/pms.1918.104.22.168
Copyright (c) 2019 International Journal of Cognitive Research in Science, Engineering and Education (IJCRSEE)
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.