1 Introduction
1.1 Statement of Research Question
The link between space, language and memory is one of the most intriguing topics is cognitive psychology. Decades of evidence has showed that language provides us with a framework to materialise and structure the relatively abstract notion of space (Bloom, Garrett, Nadel, & Peterson, 1996; Carlson-Radvansky & Logan, 1997; Majid, Bowerman, Kita, Haun, & Levinson, 2004; Miller & Johnson-Laird, 1976; Talmy, 1983). Thus, language influences how people perceive, think about and remember space (Levinson, 2003). The blooming fields of grounded-embodied cognition and the extended mind thesis have redefined the space-language-memory network and rekindled an interest about space with a novel perspective. Once seen as a central processing machine contained in the skull, the mind is now viewed as more of an interactive architecture extending onto the body and space. In accordance, burgeoning evidence suggests that space is not only a content but also a “medium” for language and memory (Mix, Smith, & Gasser, 2010). Memories can be indexed, and abstract thoughts can be grounded in space. Further, comprehending language can give rise to non-linguistic, spatial experiences. Despite the abundance of demonstrative experiments, how spatial perception and cognition influence language and memory operations is yet to be defined. To this end, the present thesis explores robust, systematic and often surprising ways that space is involved in memory for language. In particular, this thesis is an attempt to define and systematise different characteristics of spatial engagements during retrieval of words from memory. Thereby, it aims to contribute to the understanding of the effect of space on memory for language.
1.2 Definition of Concepts
The key concepts examined in the present work are defined in this section. Thereby, the scope of the thesis is outlined.
1.2.1 Spatial indexing
The human mind can anchor spatially-located information to external spatial locations. Simply put, the location of a visual item is encoded with the item itself. This behaviour is called as spatial indexing. Marr (1982) was among the first to argue that the visual system separates locations from visual features (what vs. where). He introduced the term place token to refer to representations of location. A place token indexes the locations of visual information at the early stages of visual processing. Pylyshyn (1989) operationalised the phenomenon of spatial indexing in an exhaustive model termed FINST. The model assumes that spatial indexing is a primitive, that is pre-attentive or a pre-cognitive mechanism, which precedes “higher” visual operations such as recognition of patterns. But importantly, spatial indices allow for stability of visual information by constructing stable representations of locations in a constantly changing visual world. Hence, an index keeps pointing to the same location even if the visual pattern moves across the retina. According to the model, therefore, spatial indexing is different from merely encoding the position of a feature because a spatial index makes it possible to locate the visual stimulus for further examination when the necessity arises (see also Ballard, Hayhoe, Pook, & Rao, 1997). In a similar vein, Coslett (1999) proposes a spatial registration hypothesis. This hypothesis assumes that “all stimuli, even if not relevant to the task at hand, are automatically marked with respect to spatial location in egocentric coordinate systems” (p. 703). Registering a location entails the creation of a marker that specifies the coordinate of an object in relation with the other objects in the environment. According to spatial registering hypothesis, spatial indexing is limited with the capacity of visual attention.
1.2.2 Looking at nothing
Consider the following situation: You are in the middle of a maths exam, trying to solve a problem. The problem you engage requires the application of a formula that you fail to remember at that point. However, you remember that the instructor has previously used the blackboard to explain the formula in one of the classes. You remember that she has written the formula on the top left corner of the board. Of course, the blackboard has already been cleaned and is totally blank now. Still, you raise your head in the hope of a sudden recall and look at the previous, but now-blank location of the formula. You have just looked at “nothing” (Spivey & Geng, 2000). Obviously, there is nothing on the board. What you are looking at is the spatial indice that represents the previous but now-blank location of the formula. In this respect, looking at nothing is a memory-guided behaviour. Your eyes have been oriented to the previous location of the formula because you have attempted to remember it. To be more precise, spatial indices tied to external visual and verbal information trigger eye movements when a mental representation is reactivated. Thus, when retrieving information from memory, people tend to exploit location-based indices and look at the seemingly uninformative, empty locations where the information originally occurred even if the location is irrelevant to the task. Looking at nothing follows spatial indexing in a typical memory task with encoding and retrieval stages. Individuals are expected to index the location of the information at encoding. Then, they are expected to look at the previous locations of the to-be-retrieved information during retrieval. A considerable number of empirical studies has documented looking at nothing (e.g., Altmann, 2004; Richardson & Kirkham, 2004; Richardson & Spivey, 2000; Spivey & Geng, 2000). The link between mental representations and looking behaviour (de Groot, Huettig, & Olivers, 2016; Ferreira, Apel, & Henderson, 2008; Martarelli, Chiquet, Laeng, & Mast, 2017; O’Regan, 1992; Renkewitz & Jahn, 2012; D. C. Richardson, Altmann, Spivey, & Hoover, 2009; D. C. Richardson & Spivey, 2000; Scholz, Mehlhorn, & Krems, 2011; Wantz, Martarelli, & Mast, 2015) and whether looks to blank locations improve memory (Johansson, Holsanova, Dewhurst, & Holmqvist, 2012; Johansson & Johansson, 2014; Scholz, Klichowicz, & Krems, 2018; Scholz, Mehlhorn, & Krems, 2016) have received much attention in looking at nothing research.
1.2.3 Simulation
In grounded-embodied cognition, a simulation is defined as a partial activation or reactivation of an original perceptual, motor, affective or introspective experience (Barsalou, 1999). A simulation can occur in the absence (offline) or upon the perception (online) of the original stimulus. A large body of neural and behavioural evidence indicates that major segments of cognition such as mental imagery, memory, language comprehension, consciousness, expertise and several cognitive performances such as facial mimicry, gesturing, reasoning and problem solving rely on simulations (see Dijkstra & Post, 2015; Hesslow, 2011; Körner, Topolinski, & Strack, 2016; Wilson, 2002 for reviews). The current thesis focuses on offline and online simulations of space in memory for language. Perceptual symbols lie at the heart of simulation mechanism as they make it possible the (re)activation of sensorimotor experiences (Barsalou, 1999). Perceptual symbols are mental representations that represent conceptual knowledge in the mind. Crucially, perceptual symbols are represented in the same system as the perceptual states that produced them (Barsalou, 1999, p. 578). As a result, they reflect physical and thus, perceptual characteristics of the referents they stand for, and represent continuous, rich and multimodal phenomenological experience. That said, perceptual symbols “partly” represent their referents rather than being similar to “high-resolution video-clips” or “high-fidelity sound clips” (Zwaan, Stanfield, & Yaxley, 2002). Based on these features, perceptual symbols are fundamentally different from physical symbols (Newell, 1980) asserted within the computational theories of mind (Fodor, 1975; Haugeland, 1985; Newell & Simon, 1976; Putnam, 1960). In contrast to perceptual symbols, physical symbols are amodal, abstract and discrete units. That is, they do not reflect the perceptual modality of or do not resemble to the physical entities that they refer to as in 1 and 0’s in a computation environment (Harnad, 1990; Pylyshyn, 1986).
1.2.3.1 Offline simulation
Consider that you need to remember a piece of conceptual information about cats (e.g., “What does a cat sound like?”). According to grounded-embodied cognition, in such a case, the human mind relies on the reactivation of perceptual, motor, affective or introspective experiences that are formed during previous interactions with a cat (e.g., how soft its fur is, how it smells, how you feel when you stroke it etc.). However, these experiences or states are not reinstated exactly on later occasions and “different contexts may distort activations of the original representations” (Barsalou, 1999, p. 584). Offline simulations have a situated character (Barsalou, 2003). For instance, they represent specific cats in specific situations rather than representing generic knowledge about cats. You might have noticed that remembering a piece of information about cats in the way described above bear similarities with remembering the location of a maths formula. In the latter case, the individual encodes conceptual information (the formula) along with a spatio-perceptual experience (perceiving the location of the formula on the blackboard). Later, when she needs to access the conceptual information through memory retrieval, she reactivates the perceptual experience associated with it. In other words, she “re-lives” the perceptual experience that she has during the encoding of the conceptual information. On this ground, looking at nothing can be understood as a reflection of spatial simulation that takes place in the absence of the original stimulus.
1.2.3.2 Online simulation
Perceptual, motor, affective or introspective experiences can be activated whenever an individual perceives a stimulus. For example, merely viewing a graspable object such as a cup, a knife or a frying pan simulates the potential act of grasping. In turn, brain regions associated with motor movements are activated (Chao & Martin, 2000; Tucker & Ellis, 1998). Similarly, remembering a Japanese kanji character stimulates motor activity in the areas that would be activated when actually writing the characters (Kato et al., 1999; Topolinski & Strack, 2009). Merely listening to words that involve strong tongue movements when pronounced such as “birra” (beer in Italian) or “ferro” (iron in Italian) activates tongue muscles (Fadiga, Craighero, Buccino, & Rizzolatti, 2002). Language comprehension gives rise to simulations in this manner (see Chapter 2.1.3). To summarise, the fundamental difference between an offline and an online simulation is the existence of an external stimulus at the time of the simulation. An offline simulation is a recreation of previous sensorimotor activations without any stimulus when the stimulus is re-accessed. An online simulation is a sensorimotor activation upon the perception of a stimulus (see Chapter 2.1 for further clarification with examples). There are other instances of a simulation mechanism in cognitive psychology. In social cognition, for instance, attributing mental states to others as in “mind reading” (i.e., theory of mind) (Gallese & Goldman, 1998; Premack & Woodruff, 1978) is thought to be a simulation based on mirror neurons that are activated merely by observing others (Caggiano et al., 1996; Gallese, Fadiga, Fogassi, & Rizzolatti, 1996). This aspect of simulations is beyond the scope of the current thesis despite the possible overlaps between theory of mind and mental simulations (Shanton & Goldman, 2010). Likewise, simulation within the context of this thesis does not refer to computer-based simulations.
1.2.4 Cognitive offloading
Cognitive agents can perform physical actions to outsource their cognitive work to the body and the world with the aim of reducing cognitive load (Wilson, 2002). This behaviour is called as cognitive offloading (Risko & Gilbert, 2016). Making a shopping list instead of keeping items to buy in your mind is a typical example of such behaviour. Cognitive offloading is treated as a fundamental mechanism of extended cognition, which posits that there is not a strict border between the mind and the world with the mind “leaking” into the world in surprising ways (A. Clark, 1998; A. Clark & Chalmers, 1998). Cognition can be extended to different body parts during several cognitive problems such as using hands in finger-counting (Butterworth, 2005) and co-speech gestures1 in communication (see Goldin-Meadow & Wagner, 2005; Pouw, de Nooijer, van Gog, Zwaan, & Paas, 2014 for reviews). Cognitive work can be offloaded directly onto the world as well. In particular, people can exploit their immediate space in an intelligent way to reduce their cognitive load. In such a case, space becomes a resource that must be managed, much like time or energy (Kirsh, 1995). For example, Kirsh and Maglio (1994) showed that in Tetris, participants solve perceptual and cognitive problems (e.g., judging whether a geometric piece can fit into a specific position) in the space by physically rotating the pieces on the screen rather than in their minds which would require mental rotation. Cognitive offloading with eye movements is a special case in which both body (via eye movements) and environment are used at the same time. In a seminal study, for instance, Hayhoe, Bensinger and Ballard (1998) asked participants to copy a pattern of coloured blocks on a computer screen. Participants had to use a mouse to drag scrambled blocks from the source window to an empty workspace window. The colour of blocks in the pattern was changed at different points during the task. Results showed that participants launched frequent fixations to the original pattern when they were copying it into the workspace window rather than keeping the original pattern in mind. In this respect, looking at nothing can be conceptualised as a type of cognitive offloading in which eyes are used to index certain locations in space for subsequent use in line with information-gathering goals (Ballard et al., 1997). There are two important aspects of cognitive offloading. First, cognitive offloading often results in betterment in performance. For example, children were found to be most accurate when they actively gesture to count (even when using a puppet to count for them) as compared to internal counting (Alibali & Dirusso, 1999). Second, exploitation of body and space to reduce cognitive load is correlated with the cognitive demands coming from the task and internal, “biologic” cognitive capacity. That is, higher cognitive load and/or lower cognitive capacity result in more frequent cognitive offloading. For example, in Risko and Dunn (2015), participants were asked to remember a mixture of random letters in a traditional short-term memory task. They either had to rely on internal memory only or had the option to write down the presented information and thus, to externalise memory work. Results showed that use of external storage became more frequent as the number of letters in a string increased. Further, individual short-term memory capacity predicted the likelihood of writing down the to-be-remembered information. The functional role of cognitive offloading and the link between cognitive load/capacity and the tendency of offloading indicate that there is a systematic trade-off between internal and external processing (Schönpflug, 1986). In the face of costs of cognitive operations and limited cognitive sources, a successful cognitive agent seeks to “get the job done” in the easiest way possible by integrating internal with external processes in complex environments (A. Clark, 1989).
1.3 Overview of Thesis
The current thesis investigates activations and re-activations of space in memory for language. It contains seven chapters (including this one). Chapter 2 includes the review of the literature. Chapter 2.1 focuses on the simulations in mental imagery, memory and language as the core mechanism of grounded-embodied cognition. Chapter 2.2 addresses the relation between eye movements and memory. Chapter 3 contains the first experimental study in the form of a paper. This study investigates whether individuals with better visuospatial memory relies more on space and simulation of word locations through looks at previous but now-blank locations when retrieving words from memory. Chapter 4 contains the second experimental study in the form of a paper. This study employs the experimental design developed in Chapter 3 to investigate whether words that are more difficult to maintain and retrieve from memory lead to more reliance on space and spatial simulations via looking at nothing. It also examines the contributions of word properties (e.g., word length, imageability, frequency etc.) to the looking behaviour and memory performance. Chapter 5 contains the third experimental study in the form of an article. This is a lab-based norming study in which participants were asked to read 1439 concrete and abstract words and associate them with horizontal and vertical locations on a two-dimensional coordinate system. Chapter 6 contains the forth experimental study in the form of a paper. This study uses spatially normed words from Chapter 5 to investigate how spatial locations suggested and simulated by word meanings affect recognition memory performance in relation with the physical locations of the words on the screen. Finally, Chapter 7 (General Discussion) summarises findings and conclusions of the empirical studies and discusses them in the context of the grounded-embodied and extended approaches to memory and language. Additionally, it includes suggestions for future work.
Note that cognitive offloading is only one suggested use of co-speech gestures. Co-speech gestures are assumed to play other roles as well (Goldin-Meadow, 1999).↩︎