DSCI 554 lecture 10

Patterns, memory, visual encodings, semiology and Gestalt.

Dr. Luciano Nocera

Outline

  • Pattern recognition, memory
  • Gestalt
  • Marks and encodings

Pattern recognition

Information from a stimulus ⟺ information from memory
  • Subconscious
  • Involves the What visual pathway
  • Top-down and bottom-up processing
Benjamin M. Schmidt portfolio

Apophenia

  • Perception of images or sounds in random stimuli
  • Priming increases likelihood of seeing the pattern
  • Likely evolutionary process from Type I (false positive) and Type II (false negatives) errors
Part of the Cydonia (Mars) region, taken by the Viking 1 orbiter and released by NASA/JPL on July 25, 1976

Priming

Effect in which exposure to one stimulus influences a response to a later stimulus. Works on VLTM.

Visual pathways

  • Where and what pathways of the two stream hypothesis
  • Works without visual input
  • Visual aids needed for visual thinking due to limited memory resources and interactions due to limited attention
Where / Dorsal
relative object location for motor tasks
What / Ventral
object identification and recognition

Visual cortex & features processed

Visual cortex areas V1, V2, V3, V4, V5/MT (middle temporal))

Visual cortex and other cortical structures involved in vision. Graphic design: P.A. based on Logothetis (1999) and Zeki (2003)
Simple features are detected in earlier visual areas, large patterns and shapes in higher visual areas

Visual cortex & information

Areas Features processed Example
Lower visual cortex V1, V2 Simpler features V1 neurons may fire to any vertical stimulus†
Higher visual cortex V4, MT, and IT Complex patterns IT neurons may fire only to a specific face‡
Lower visual cortex Higher visual cortex
Information Low High
Localization High Low
Specificity Low† High‡
Experience Universal Individual

V1 Neuronal tuning

  • Single V1 neurons are generally tuned to a particular characteristic
  • Results from convergence (group of cells form a receptive field for one neuron)
Some neurons of V1 are tuned to vertical lines, others to diagonal lines

Lower visual cortex

  • Typically takes $\sim$ 40ms (preattentive*)
  • Strong tuning to orientation, spatial frequency and color
  • Extremely sensitive tuning for horizontal and vertical lines
  • Feature hierarchy, e.g., corners generate more powerful responses than edges
Ware, Colin, Visual queries: The foundation of visual thinking, 2005. * Pre-attentive $\leq$ 200ms, in contrast saccades $\sim$ 200ms to initiate, last 20-200ms

Higher visual cortex

  • ≥ 100ms
  • Increased sensitivity to more global organization of the scene
  • Tuning to groups of patterns, motion patterns of large patterns
  • Specialized regions extract and represent generalized object structure, e.g., generalized contours are easily understood in sketches
    Ware, Colin, Visual queries: The foundation of visual thinking, 2005.
  • Tuning to motion patterns of large patterns
V4 response to motion of a large pattern.
Likely adaptation to tracking camouflaged objects.

Apprehendable chunk

Apprehendable chunks are:

  • Learnable composite pattern
  • Unlearned patterns that can be apprehended in one fixation
  • Consist of about three components
Ware, Colin, Visual queries: The foundation of visual thinking, 2005.

Selective attentional tuning

Can focus on a layer of a set of superposed layers. This property is used in thematic maps to display different data.

Disrupted when patterns are too similar. This is similar to a conjunction search in pre-attentive features.

Ware, Colin, Visual queries: The foundation of visual thinking, 2005.

Groups of patterns are robust to distortions

Neurons in higher visual cortex respond strongly despite distortions

Sketches

  • Easily understood complex patterns
  • Require less work to understand than full-color, textured images
Galilean moons. Drawing by Galileo.
Wong-Baker Faces Pain Rating Scale Happy-or-not Smiley Terminal™

Icons & spatial metaphors

Font Awesome icons
Ware, Colin, Visual queries: The foundation of visual thinking, 2005.

Isotype* [Otto & Marie Neurath - 1935]

Neurath's International picture language, 1936 *Isotype: International System Of TYpographic Picture Education: a symbolic representation
of qualitative and quantitative information via easily interpretable icons
O. Neurath, Modern Man in the Making, 1939. Home and Factory Weaving in England

Visual Memory

← Visual Persistance Information Persistance →
Iconic Memory Visual Short-term Memory (VSTM) Visual Long-term Memory (VLTM)
Unlimited capacity Limited capacity Large capacity
Retention: $\leq 1s$ Retention: $\leq 30s$ Retention: indefinite
  • High bandwidth
  • Works unconsciously
  • Provides temporal integration
  • Continuity during saccades
  • Buffer that stores temporary information
  • Constructs and manipulate visual images
  • Capacity increases over childhood, declines with old age.
  • Encodes information semantically for long term storage
  • Subject to fading, recalls help preserve it

Attentional bottleneck

Result of limited VSTM capacity

Anderson C., Van Essen D., and Olshausen A, Directed visual attention and the dynamic control of information flow, 2005.

Miller's law

The Magical Number Seven, Plus or Minus Two
  • Actual limit depends on the type information:
    • 5-9 items in a 1-D information judgment task [Miller, 1956*]
    • 4-5 items with characters [Sperling, 1960]
    • 3-4 items with basic visual features & interference task [Luck & Vogel, 1997]

Outline

  • Pattern recognition, memory
  • Gestalt
  • Marks and encodings

Gestalt theory of perception [1890]

  • An organized whole that is perceived as more than the sum of its parts
  • Gestalt means shape in German
  • Psychology theory to understand the design implications of how we perceive patterns

Bahaus logo of Berlin art school

Gestalt principles

Emergence
We perceive images as a whole 		Reification
We perceive more than the stimulus contains
Multi-stability
Some stimuli are perceived as changing between two or more interpretations 		Invariance
Simple objects are recognized independent of pose, deformations, lighting, and features

Gestalt laws

Pithiness (Prägnanz)
We order our experience in a manner that is regular, orderly, symmetric, and simple 		Figure and ground
We tend to separate an object from its background 		Parallelism
Parallel elements are seen as more related than elements not parallel
Symmetry
$[\ \ \ \ ]\ \ \ \{\ \ \ \ \}$ We perceive objects as being symmetrical and forming around a center point 		Focal points
Elements with a point of interest, emphasis or difference will capture and hold attention 		Past experience
Elements are perceived according to past experience

Gestalt laws of grouping

Proximity
Elements close together are perceived as grouped 		Similarity
Objects with similar appearance are perceived as grouped 		Closure
Parts of an object tend to be grouped together and we perceive the whole figure 		Continuity
We perceive the pieces to form a continuation as parts of a whole object
Common fate
Objects moving in the same direction are perceived as grouped 		Connection
Objects that are connected are perceived as a group 		Common region
Objects enclosed by a boundary are perceived as a group
A: common region
B: focal point, similarity, proximity
C: principle of reification, closure
D: principle of multi-stability, figure and ground
E: principle of invariance, proximity, similarity
F: principle of reification, closure
G: principle of multi-stability, figure and ground,
H: similarity, proximity
I: principle of reification, closure
J: principle of multi-stability
K: similarity, proximity
L: common region, proximity, continuity

Outline

  • Pattern recognition, memory
  • Gestalt
  • Marks and encodings

Semiology of Graphics [Bertin 1967]

Jaques Bertin, French cartographer and theorist
  • Visual language is a sign language
  • Sender encodes information in signs, receiver decodes information from signs
  • Semiotics (semiology) is the study of signs and symbols and their use or interpretation

Marks (visual variables) and channels (encodings)

Semiology of Graphics, J. Bertin, 1967

Accuracy of perceptual tasks [McKinley 1986]

Higher tasks are accomplished more accurately than lower tasks.
Mackinlay, J., Automating the design of graphical presentations of relational information. ACM Transactions On Graphics, 1986.

Accuracy of perceptual tasks by data type [McKinley 1986]

Ranking of perceptual tasks. Tasks in gray boxes are not relevant to these types of data.
Mackinlay, J., Automating the design of graphical presentations of relational information. ACM Transactions On Graphics, 1986.