The Effect of Incongruent Information on Object Recognition
¹Jenelle Hall, ²Simon Lacey, and ^1,2,3Krish Sathian1,2,3
¹Neuroscience and Behavioral Biology Program, Emory University;
²Department of Neurology, Emory University School of Medicine;
³Atlanta VMAC Rehabilitation Research and Development Center of Excellence;

Abstract

Cross-modal memory for objects is normally asymmetric: visual learning followed by haptic testing is superior to haptic learning followed by visual testing. We investigated whether this was because modality-specific information interferes with modality-independent shape information. We found, as expected, that introducing incongruent color information reduced within-modal visual recognition accuracy without affecting within-modal haptic or cross-modal recognition, and eliminated the usual asymmetry in cross-modal performance. We conclude that cross-modal memory relies on modality-independent representations of objects.

Introduction

An unexplained feature of visuo-haptic cross-modal object recognition is that visual encoding followed by haptic recognition normally leads to higher performance than vice versa [1]. This is unexpected if information about objects is encoded into a multisensory representation that is equally available to vision and touch [2]. One explanation may be that the multisensory hypothesis is correct but that modality-independent information (like shape which can be perceived by both vision and touch) can be poorly encoded if there is competition from more salient modality-specific information (like color which can only be perceived by vision) [3-4]. We manipulated the color and texture of objects while keeping shape information constant to examine the effects on visuo-haptic within- and cross-modal object recognition. We expected to find that introducing incongruent modality-specific information would induce errors in the relevant modality, have no effect in the irrelevant modality, and eliminate asymmetric performance in the cross-modal conditions.

Methods and Materials

16 participants were shown six sets of block structures that differed in shapes and color schemes. In each set they learned 2 objects (Figure 1A) either visually or haptically. At test (either in the same modality as learning or the opposite modality), 2 distractor objects were added that had the same shapes as the originals but exchanged color schemes with them (Figure 1B). The task was to correctly identify which objects had the same shape as the originals.



Figure 1: A Example of original blocks and color schemes; B Distractor objects with same shapes but exchanged color schemes A separate group of 16 participants performed the same task but this time the surface texture of the objects was varied. The textures were put on one surface so they could only be felt and not seen (Figure 2).



Figure 2: A Example of original blocks and texture schemes; B Distractor objects with same shapes but exchanged texture schemes

Results

We calculated the percentage change in performance due to the color and texture manipulations as: ((target performance-distracter performance)/target performance) x 100.



There was a main effect of modality in the color task (F(3,45)=3.05; p=.04). Post-hoc LSD tests showed that the visual-visual condition was less accurate than the other conditions (which did not differ significantly from each other) (Figure 3) (Error bars = s.e.m.)



There was no main effect of modality in the texture task (F(3,45)= .21; p=.99): none of the modality conditions significantly differed from another (Error bars = s.e.m).

Conclusions and Future Studies

The hypothesis that the color manipulation would lead to errors in within-modal visual but not haptic performance and symmetric performance in the cross-modal conditions was supported. This suggests that color is salient enough to disrupt shape encoding but that cross-modal object recognition is able to overcome this.

However, this result was not replicated in the texture task where there were small reductions in performance in all modalities. We expected that the texture manipulation would reduce within-modal haptic performance.

The inconclusive texture results may have been due to the textures being bound to the one side rather than the entire object. Alternatively, either the chosen textures may not have been sufficiently salient to disrupt haptic shape information, or texture is not as efficient a distracter to shape perception as color. Future experiments should address these questions.

We conclude that the color task supported the hypothesis that cross-modal asymmetry may be due to competition between modality-specific and modality- independent information but that the texture task was inconclusive on this point.

Resources

This research was supported in part by the Howard Hughes Medical Institute through the SURE program Grant no. 52005873. Thanks to the National Science Foundation for funding under FACES Award # 0450303 (Subaward # I-66-606-63 to Emory University) and grants to KS from the NSF and the National Institute of Health. I would also like to thank my mentors Simon Lacey, Ph.D. and Krish Sathian, MD, Ph.D.

References

1. Lacey, S., Campbell, C. (2006). Mental representation in visual/haptic crossmodal memory: Evidence from interference effects. The Quarterly Journal of Experimental Psychology, 59, 361-376.
2. Sathian, K. (2004). Modality, quo vadis? Behavioral and Brain Science, 27, 413-414.
3. Nicholson K.G., Humphrey, K.G. (2004). The effect of colour congruency on shape discriminations of novel objects. Perception, 33, 339-353.
4. Luo, Z., Imamiya, A. (2004). Do colors affect our recognition memory for haptic rough surfaces? Lecture Notes in Computer Science, 3038, 897-904.