Effects of integration of facial expression and emotional voice on inhibition of return

ZHANG Ming1,2 WANG Tingting1 WU Xiaogang3 ZHANG Yue’e1 WANG Aijun1

(1.Research Center for Psychology and Behavioral Sciences, Department of Psychology, Soochow University, Suzhou 215123)
(2. Graduate School of Interdisciplinary Science and Engineering in Health Systems, Okayama University , Okayama 700-8530)
(3.Department of Psychology, Suzhou University of Science and Technology, Suzhou 215009)

【Abstract】Both inhibition of return (IOR) and emotion have the characteristics of attentional bias and improving search efficiency. Previous studies mostly used a single modality presentation of emotional stimuli to investigate the relationship between the two, but the findings are inconsistent. Existing studies have shown that the congruent emotion of audiovisual dual modality can be integrated into the perceptual stage, which is the same as the processing stage of IOR. Therefore, the present study adopted the cue-target paradigm and used audiovisual dual modality to present emotional stimuli to further investigate the interaction between emotion and IOR. Experiment 1 was a three-factor within-subjects design. We manipulated the presentation of cue validity (cued vs. uncued), target modalities (visual vs. audiovisual), and emotion type (negative vs. neutral). The task of the participants was to identify the emotional stimuli of visual modality. Experiment 2 was similar to Experiment 1, but the emotional congruency was changed. The audiovisual dual modality presented incongruent emotional stimuli (visual negative face-auditory neutral sound; visual neutral face-auditory negative sound) to further investigate whether the impact of the audiovisual dual modality emotional stimulus on IOR was caused by the emotional stimulus of the auditory modality, that is, whether the emotional stimulus of the auditory modality was processed. In Experiment 1, the responses in the cued condition were slower than those in the uncued condition, which suggested that IOR occurred. More importantly, the interaction between emotion type and cue validity in the audiovisual dual modality condition showed that congruent negative emotion produces a smaller IOR effect (11 ms) than neutral emotion (25 ms). At the same time, the audiovisual dual modality condition produced a smaller IOR effect (18 ms) than the visual single modality condition (40 ms). We also found a larger multisensory response enhancement effect in the congruent negative emotion than in the neutral emotion. In Experiment 2, the results showed that there was no interaction between emotion and IOR under the condition of audiovisual dual modality, and there was no significant difference in IOR effect between single modality and audiovisual dual modality. This indicated that the IOR effect was not influenced by the presence of incongruent emotion in the audiovisual dual modality. In summary, the present study showed that the IOR effect was influenced only when the audiovisual dual modality presented the same emotion. Our findings revealed that IOR and audiovisual dual modality congruent emotion in the same processing stage had a mutual influence. Audiovisual dual modality congruent emotion weakened the IOR effect, and the differences between the negative emotion and the neutral emotion showed the adaptability of IOR. At the same time, this study further supports the perceptual inhibition theory of IOR.

【Keywords】 inhibition of return; emotion; audiovisual integration;


【Funds】 National Natural Science Foundation of China (31871092, 31700939) Humanities and Social Sciences Foundation of the Ministry of Education (17YJC190024) Basic Research Project of Japan Society for the Promotion of Science (20K04381)

Download this article


    Anderson, A. K., Christoff, K., Panitz, D., de Rosa, E., & Gabrieli, J. D. E. (2003). Neural correlates of the automatic processing of threat facial signals. The Journal of Neuroscience, 23(13), 5627–5633.

    Anderson, A. K., & Phelps, E. A. (2001). Lesions of the human amygdala impair enhanced perception of emotionally salient events. Nature, 411(6835), 305–309.

    Baijal, S., & Srinivasan, N. (2011). Emotional and hemispheric asymmetries in shifts of attention: An ERP study. Cognition and Emotion, 25(2), 280–294.

    Belin, P., Fillion-Bilodeau, S., & Gosselin, F. (2008). The Montreal Affective Voices: A validated set of nonverbal affect bursts for research on auditory affective processing. Behavior Research Methods, 40(2), 531–539.

    Berdica, E., Gerdes, A. B. M., & Alpers, G. W. (2017). A comprehensive look at phobic fear in inhibition of return: Phobia-related spiders as cues and targets. Journal of Behavior Therapy and Experimental Psychiatry, 54, 158–164.

    Chen, Q., Wei, P., & Zhou, X. (2006). Distinct neural correlates for resolving stroop conflict at inhibited and noninhibited locations in inhibition of return. Journal of Cognitive Neuroscience, 18(11), 1937– 1946.

    Collignon, O., Girard, S., Gosselin, F., Roy, S., Saint-Amour, D., Lassonde, M., & Lepore, F. (2008). Audiovisual integration of emotion expression. Brain Research, 1242, 126–135.

    de Gelder, B., & Vroomen, J. (2000). The perception of emotions by ear and by eye. Cognition and Emotion, 14(3), 289–311.

    Faul, F., Erdfelder, E., Buchner, A., & Lang, A. G. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41(4), 1149–1160.

    Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191.

    Fox, E., Russo, R., & Dutton, K. (2002). Attentional bias for threat: Evidence for delayed disengagement from emotional faces. Cognition and Emotion, 16(3), 355–379.

    Guan, J., Li, W. & Zhao, X. Journal of Psychological Science (心理科学), 41(6): 1353–1358 (2018).

    Hopfinger, J. B., & Mangun, G. R. (2001). Tracking the influence of reflexive attention on sensory and cognitive processing. Cognitive, Affective, & Behavioral Neuroscience, 1(1), 56–65.

    Jessen, S., & Kotz, S. A. (2011). The temporal dynamics of processing emotions from vocal, facial, and bodily expressions. NeuroImage, 58(2), 665–674.

    Jia, L., Wang, J., Zhang, K., Ma, H., & Sun, H. J. (2019). Do emotional faces affect inhibition of return? An ERP study. Frontiers in Psychology, 10(721), 1–8.

    Klasen, M., Kreifelts, B., Chen, Y. H., Seubert, J., & Mathiak, K. (2014). Neural processing of emotion in multimodal settings. Frontiers in Human Neuroscience, 8(822), 1–4.

    Klein, R. M. (2000). Inhibition of return. Trends in Cognitive Sciences, 4(4), 138–147.

    Kokinous, J., Kotz, S. A., Tavano, A., & Schröger, E. (2014). The role of emotion in dynamic audiovisual integration of faces and voices. Social Cognitive and Affective Neuroscience, 10(5), 713–720.

    Lang, P. J., Davis, M., & Öhman, A. (2000). Fear and anxiety: Animal models and human cognitive psychophysiology. Journal of Affective Disorders, 61(3), 137–159.

    Lange, W. G., Heuer, K., Reinecke, A., Becker, E. S., & Rinck, M. (2008). Inhibition of return is unimpressed by emotional cues. Cognition and Emotion, 22(8), 1433–1456.

    Li, P., Zhang, M., Li, S. et al. Advances in Psychological Science (心理科学进展), 27(7): 1205–1214 (2019).

    Lupiáñez, J., Klein, R. M., & Bartolomeo, P. (2006). Inhibition of return: Twenty years after. Cognitive Neuropsychology, 23(7), 1003– 1014.

    Miller, J. (1986). Timecourse of coactivation in bimodal divided attention. Perception & Psychophysics, 40(5), 331–343.

    Mineka, S., & Öhman, A. (2002). Phobias and preparedness: The selective, automatic, and encapsulated nature of fear. Biological Psychiatry, 52(10), 927–937.

    Müller, V. I., Habel, U., Derntl, B., Schneider, F., Zilles, K., Turetsky, B. I., & Eickhoff, S. B. (2011). Incongruence effects in crossmodal emotional integration. NeuroImage, 54(3), 2257–2266.

    Okon-Singer, H., Henik, A., & Gabay, S. (2020). Increased inhibition following negative cues: A possible role for enhanced processing. Cortex, 122, 131–139.

    Pan, F., Wu, X., Zhang, L., & Ou, Y. (2017). Inhibition of return is modulated by negative stimuli: Evidence from subliminal perception. Frontiers in Psychology, 8(1012), 1–8.

    Pérez-Dueñas, C., Acosta, A., & Lupiáñez, J. (2009). Attentional capture and trait anxiety: Evidence from inhibition of return. Journal of Anxiety Disorders, 23(6), 782–790.

    Pérez-Dueñas, C., Acosta, A., & Lupiáñez, J. (2014). Reduced habituation to angry faces: Increased attentional capture as to override inhibition of return. Psychological Research, 78(2), 196–208.

    Posner, M., & Cohen, Y. (1984). Components of visual orienting. Attention and Performance X: Control of Language Processes, 32, 531–556.

    Pourtois, G., de Gelder, B., Vroomen, J., Rossion, B., & Crommelinck, M. (2000). The time-course of intermodal binding between seeing and hearing affective information. NeuroReport, 11(6), 1329–1333.

    Pourtois, G., Schettino, A., & Vuilleumier, P. (2013). Brain mechanisms for emotional influences on perception and attention: What is magic and what is not. Biological Psychology, 92(3), 492–512.

    Prime, D. J., & Jolicœur, P. (2009). On the relationship between occipital cortex activity and inhibition of return. Psychophysiology, 46(6), 1278–1287.

    Prime, D. J., & Ward, L. M. (2006). Cortical expressions of inhibition of return. Brain Research, 1072(1), 161–174.

    Redden, R. S., MacInnes, W. J., & Klein, R. M. (2021). Inhibition of return: An information processing theory of its natures and significance. Cortex, 135, 30–48.

    Reuter-Lorenz, P. A., Jha, A. P., & Rosenquist, J. N. (1996). What is inhibited in inhibition of return. Journal of Experimental Psychology: Human Perception and Performance, 22(2), 367–378.

    Rutherford, H. J. V., & Raymond, J. E. (2010). Effects of spatial cues on locating emotional targets. Visual Cognition, 18(3), 389–412.

    Silvert, L., & Funes, M. J. (2016). When do fearful faces override inhibition of return? Acta Psychologica, 163, 124–134.

    Stein, B. E., & Stanford, T. R. (2008). Multisensory integration: Current issues from the perspective of the single neuron. Nature Reviews Neuroscience, 9(4), 255–266.

    Stoyanova, R. S., Pratt, J., & Anderson, A. K. (2007). Inhibition of return to social signals of fear. Emotion, 7(1), 49–56.

    Talsma, D., & Woldorff, M. G. (2005). Selective attention and multisensory integration: Multiple phases of effects on the evoked brain activity. Journal of Cognitive Neuroscience, 17(7), 1098–1114.

    Tang, X., Gao, Y., Yang, W., Ren, Y., Wu, J., Zhang, M., & Wu, Q. (2019). Bimodal-divided attention attenuates visually induced inhibition of return with audiovisual targets. Experimental Brain Research, 237(4), 1093–1107.

    Taylor, T. L., & Klein, R. M. (1998). On the causes and effects of inhibition of return. Psychonomic Bulletin & Review, 5(4), 625–643.

    Taylor, T. L., & Therrien, M. E. (2008). Inhibition of return for the discrimination of faces. Perception & Psychophysics, 70(2), 279–290.

    Tian, Y., & Yao, D. (2008). A study on the neural mechanism of inhibition of return by the event-related potential in the Go/Nogo task. Biological Psychology, 79(2), 171–178.

    Tottenham, N., Tanaka, J. W., Leon, A. C., McCarry, T., Nurse, M., Hare, T. A., Marcus, D. J., Westerlund, A., Casey, B. J., & Nelson, C. (2009). The NimStim set of facial expressions: Judgments from untrained research participants. Psychiatry Research, 168(3), 242–249.

    Troiani, V., Price, E. T., & Schultz, R. T. (2014). Unseen fearful faces promote amygdala guidance of attention. Social Cognitive and Affective Neuroscience, 9(2), 133–140.

    Ulrich, R., Miller, J., & Schröter, H. (2007). Testing the race model inequality: An algorithm and computer programs. Behavior Research Methods, 39(2), 291–302.

    van der Burg, E., Talsma, D., Olivers, C. N. L., Hickey, C., & Theeuwes, J. (2011). Early multisensory interactions affect the competition among multiple visual objects. NeuroImage, 55(3), 1208–1218.

    van der Stoep, N., van der Stigchel, S., Nijboer, T. C. W., & Spence, C. (2017). Visually induced inhibition of return affects the integration of auditory and visual information. Perception, 46(1), 6–17.

    Vizueta, N., Patrick, C. J., Jiang, Y., Thomas, K. M., & He, S. (2012). Dislocational fear, negative affectivity, and neuroimaging response to visually suppressed emotional faces. NeuroImage, 59(1), 761–771.

    Wang, J., Jia, L. Zhang, K. et al. Journal of Psychological Science (心理科学), 36(2): 335–339 (2013).

    Wang, Z., Zhang, K., & Klein, R. M. (2010). Inhibition of return in static but not necessarily in dynamic search. Attention, Perception, & Psychophysics, 72(1), 76–85.

    Zhao, X., Li, X., & Shi, W. (2017). Influence of inhibitory tagging (IT) on emotional and cognitive conflict processing: Evidence from event-related potentials. Neuroscience Letters, 657, 120−125.

This Article



Vol , No. 04, Pages 331-342

February 2022


Article Outline


  • 1 Introduction
  • 2 Experiment 1: the effect of audiovisual dual modality emotional consistency on IOR
  • 3 Experiment 2: the effect of audiovisual dual-modality emotional inconsistency on IOR
  • 4 Discussion
  • 5 Conclusion
  • References