its about time: analyzing temporal microlevel behavioral patterns
DESCRIPTION
TRANSCRIPT
![Page 1: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/1.jpg)
It’s About Time – Analyzing Temporal Micro‐It s About Time Analyzing Temporal MicroLevel Behavioral Patterns
Chen YuIndiana University
![Page 2: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/2.jpg)
Temporal Bands of Human Computation (B ll d H h P k & R 1997)(Ballard, Hayhoe, Pook, & Rao, 1997)
micro level
macro-level behavior
tern
alna
l
micro-level behavior ex
tin
tern
![Page 3: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/3.jpg)
Dual Eye Tracking in Child-Parent Interaction (with Linda Smith, Damian Fricker, & Linger Xu) ( , , g )
child’s first person view parent’s first person view
from eye camera
from head camera
child eye tracker
from head camera
adult eye trackertracker
![Page 4: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/4.jpg)
Dual Eye Tracking in Child-Parent InteractionInteraction
![Page 5: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/5.jpg)
Multimodal data• Vision: 720*480, 30 frames/second, 3 cameras, 18,000
frames per dyads.
• Motion tracking: 250 Hz/second, 6 sensors with six dimensions (x,y,z,h,p,r) on each.900 000 data points per dyads900,000 data points per dyads.
Eye tracking: 30Hz
• Speech: 44.1Hz
• Eye tracking: 30Hz
• 12GB per participant
![Page 6: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/6.jpg)
Dealing with data
• Synchronization of multiple data streams
• Data annotation (automatically or manually)
• Data management
• Data Mining and Knowledge Discovery
![Page 7: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/7.jpg)
Multi‐Streaming Multimodal Data
speech
0 500 1000 1500 2000 2500 3000 350
5
1
5
2
5x 104
x 104
0 500 1000 1500 2000 2500 3000 30
2
4
6
8x 10
x 104
vision
…0 500 1000 1500 2000 2500 3000
…
motion
![Page 8: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/8.jpg)
Data Mining• Our goal of analyzing gaze data is to find new patterns• Our goal of analyzing gaze data is to find new patterns
and gain new knowledge from such data.
• With micro level data even if we have some predictions• With micro-level data, even if we have some predictions from our experimental designs, we nonetheless lack precise predictions about the structure and patterns of
• But how can we discover new and meaningful patterns if d k h l k f
data at the micro-level.
we do not know what we are looking for?
• Discovering new knowledge requires the ability to detect unknown, surprising, novel, and unexpected patterns.
• A particular challenge is not just from the amount of data but p g jfrom how to extract, select and interpret meaningful patterns from a sea of complex data.
![Page 9: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/9.jpg)
childchildgaze
parentparentgaze
![Page 10: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/10.jpg)
Interactive Data Analysis
top‐down knowledge
data
knowledge
patterndata visualization
pattern extraction
• This solution relies on both computational techniques and human domain knowledge. Data visualization and pattern extraction techniques provide candidate patterns through a bottom-up waycandidate patterns through a bottom up way.
• Compared with “blind” data mining, what we suggest is that researchers with top-down theoretical knowledge need to be in the loop of data mining andtop down theoretical knowledge need to be in the loop of data mining and data analysis.
![Page 11: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/11.jpg)
childchildgaze
parentparentgaze
![Page 12: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/12.jpg)
face three objects
hild
red blue red green red blue
childgaze
parentparentgaze
face gaze and mutual gaze joint attention
![Page 13: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/13.jpg)
Joint Attention StateJoint Attention State60%
40%
50%
20%
30%
0%
10%
t l hild f / t bj t t f / hild bj t bj t diff t bj t
‐10%
mutual gaze child‐face/parent‐object parent‐face/child‐object same object different objects
![Page 14: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/14.jpg)
child followingchild followingface three objects
childgaze
parentgaze
parent followingchildgaze
parent following
g
parentgaze
![Page 15: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/15.jpg)
What is joint attention made of?
hild t
when baby looks mom’s face, mom looks at baby’s face 54% of the time; when baby looks at an object mom
looks at 32% of time
when MOM looks baby’s face, baby looks at mom’s face 13% of the
time; when mom looks at an object baby looks at 14% of timechild parent
54%(face)32%(object)
13%(face)
baby looks at 14% of time
gaze gaze
13%(face)14%(object)
56%55%
handhand
17%
21%
probabilities of following an attended object/face
![Page 16: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/16.jpg)
Sequential Patterns From Multimodal Data Stream
face-to-active
Multimodal Data Stream
1 1learner’s gaze 1
facejoint attention following
1 1gaze
teacher’s gaze
1
3 33 222
4 4
teacher’s hand action 4
face three objects
![Page 17: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/17.jpg)
Segmentation and AlignmentSegmentation and Alignment
![Page 18: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/18.jpg)
Discovering Statistically Reliable Sequential Patterns
(Fricker, Zhang, & Yu, 2011)
multiple instancesmultiple instancesnaming utterance
describing utterance
face look
sequential prototype
durations and timings within and across multiple events
![Page 19: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/19.jpg)
More complex patterns from lti l d t tmultiple data streams
![Page 20: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/20.jpg)
![Page 21: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/21.jpg)
From Data To Patterns
![Page 22: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/22.jpg)
From Patterns To Knowledge
……
![Page 23: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/23.jpg)
Interactive Data Mining
top‐down knowledge
data visualization
pattern extractionvisualization extraction
• Human-in-the-Loop data analysis involves active and iterative examination and exploration of visually displayed patterns to select useful information and guide further knowledge discoveryselect useful information and guide further knowledge discovery.
• “Discovery takes place between the ears” – Ben Shneiderman
![Page 24: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/24.jpg)
Visual Data Mining (Yu, Zhong, Smith, Park, & Huang, 2008, 2009)
![Page 25: Its About Time: Analyzing Temporal MicroLevel Behavioral Patterns](https://reader036.vdocuments.mx/reader036/viewer/2022081719/54578c32af7959795d8b5041/html5/thumbnails/25.jpg)
Acknowledgement
Data collection and pre-processing : Damian Fricker, Amanda p p g ,Favata, Char Wozniak, Melissa Elston.
Data Processing: Tian Xu Damian Fricker Thomas Smith HenryData Processing: Tian Xu, Damian Fricker, Thomas Smith, Henry Shen.
This esea ch as s ppo ted b NSF BCS 0924248 and NIH T32This research was supported by NSF BCS 0924248 and NIH T32 HD 07475
Thanks!