NEIL H. SCHWARTZ, PH.D.SEMINAR IN COGNITIVE VISUALIZATION

Visual vs. Language-based Thinking

Introduction

In this lecture, we will deal with the following questions: When are images most effective? When are words and other formal symbols most

effective? If both images and words are used, how should they

be combined? For example, http://www.segmeasurement.com/

Learned Symbols

Natural language contains a set of nouns to denote concrete or abstract objects. Nouns are invented and arbitrary. As a language becomes more complex, arbitrariness increases.

Visual thinking is based on pattern perception; not learned symbols. Pattern perception is part evolution and part visual experience. Patterns convey meaning in ways that are not arbitrary and not

socially determined.Visual designs are almost always hybrids.

They have aspects that support visual thinking through pattern finding;

They have aspects that are conventional and processed via language.

Natural Language vs. Visual Thinking

Natural Languag

eVisual

Thinking

ArbitraryInventedNot

Socially Determin

ed

NotArbitrary

Based on nouns to denote objects; verbs to denote action

Based on pattern perception; not symbols

Pattern findingLanguage

Visual Designs

Symbols vs. Patterns

Arbitrary Symbols Visual & Pattern-BasedSymbolic & Pattern-Based

?

Summarizing Verbal Vs. Visual Thinking

Natural language: Incorporates a form of

logic that provides for abstract reasoning.

Uses socially-designed tools for communication.

Comprised of symbols and grammar known by all users.

Visual representation: Incorporates logic based

on pattern, object, and space.

Consists of spatially-based structural relationships.

Almost all designs are a combination of both…

Language & Visual Representation Combinations

Natural Language Visual Representation

Solar radiation can be converted to electrical energy. This is accomplished by allowing solar irradiance to connect with a module. The module sends the energy to a charge controller, which in turn, is sent to a battery. The battery will emit energy in the form of electricity as a direct current and add to the DC load. Or, it is also possible to send the energy to an inverter which converts the energy to an alternating current, which adds to the AC load.Nouns = radiation, energy, irradiance, module, controller, battery, electricity, current, load.Verbs = converted (converts), accomplished, allowing, connect, sends (sent), emit, add.

Deixis

Deixis is a concept that refers to the contextual-dependent reference of an utterance.

Words and phrases, when combined with gestures or symbols, provide deictic meaning. That is, they convey relational meaning by way of: person, place, or time.

Deixis is important to graphics because it is deixis which links words with symbols to convey meaning.

For example…

“Put that there.”

Mirror Neurons

Mirror neurons are neurons specialized for coding the actions of other people. [Origami] http://www.springerlink.com/content/j2455633g27ww835/

Mirror neurons are important for cognitive visualization because they are the basis for learning by imitation.

Most imitation is borne from watching.Mirror neurons have strong potential for

considering static vs. dynamic graphics….

http://www.scientificamerican.com/article.cfm?id=the-mirror-neuron-revolut

Check this out:

Mirror Neurons: van Gog, Paas, Marcus, Ayres & Sweller

“We have argued that the mirror neuron system may provide a neuroscience base for some educational procedures and hypotheses associated with the design of dynamic and static visualizations. The mixed findings concerning the effectiveness of instructional animations have been a puzzle for some time. At least a partial solution to that puzzle may be provided by the suggestion that the mirror neuron system assists in acquiring motor skills by observation, thus altering the effectiveness of dynamic compared to static visualizations. However, this remains an hypothesis to be tested, for which interdisciplinary research is required. Such research might also shed light on the relation between the mirror neuron system and working memory. We assume that visualizations depicting human movement may trigger an automatic and therefore effortless process of embodied simulation by the mirror neuron system. From a cognitive load perspective, this might benefit learning by leaving more working memory capacity available for processes such as elaboration or reflection on intentions of actions, compared to static visualizations. However, we do not know whether and how the mirror neuron system and working memory interact at a neural level.” (p. 27).

Educ Psychol Rev (2009) 21:21–30DOI 10.1007/s10648-008-9094-3

Visual Narrative

Visual narrative refers to capturing the cognitive thread of an audience. Cognitive thread is the sequence of concepts that are held

actively in visual and verbal working memory, together with the links between them.

Designing for narrative is very different than designing for information seeking.

Understanding a narrative and seeking information are two different cognitive goals. Information seeking establishes a cognitive thread moment-to-

moment in order to solve a problem. Understanding a narrative establishes a cognitive thread for

understanding a story grammar.

Building A Narrative Structure

Three Components to Narrative Structure

Establishing a

Structure

Elaborating a

ProblemResolving a Problem

Elements of Visual NarrativeQ & A P

atterns

• Question and answer pattern

• 1st two frames are unrelated to evoke tension and a question.

• Last frame provides the 1 to 2 link and resolution.

• Frames 1 – 3 lead to a new question; and so on.

Framing

• Frame captures and controls what the viewer looks at in a slide, shot, or animated graphic sequence.

• Transitions between shots adds to the frame. (based on content, perspective).

• Previous frames set the visual queries for the next frame.

• Transition from shot-shot must avoid disorientation.

• http://vimeo.com/10990804

Dire

cting Atte

ntion

• Attention is limited to 4 targets that can be tracked (FINST’s– fingers of instantiation.

• A fixation can target 1 object; 3 others can receive attention in the visual field.

• Depth of field can be used to direct attention in a frame.

Design Heuristics for Narrative Diagrams

A clear sequence of operations should be evident to maintain narrative sequence.

Components should be clearly visible and identifiable in the diagrams.

The spatial layout of components should be consistent from one frame to the next.

Actions should be illustrated along with connections between components.