Is Your User Hunting or Gathering Insights? Identifying Insight Drivers Across Domains

Michael Smuc, Eva Mayr, Hanna Risku

Motivation

Goal: Evaluate a visualization tool for temporal pattern analysis

Expert-User DilemmaReal domain experts are rare, hard to find (and to motivate)Sometimes they don‘t even exist

Across-domain developmentWhat if you wanted to develop a tool suitable for different domains?

Solutions

Use heuristics instead of insight analysisIn case you find suitable ones for your tool, you can‘t keep in touch with the users

Educate non-experts time consuming, „they will tell you what you taught them to tell” even more time-consuming for multiple domains

=> we would like to propose another solution

Our approach

► Across-domain testing: Experts have to solve tasks in data visualizations of their own domain and from other domains

► Additional aid story about the data

► Research questions How do experts differ in different domains? More insights?Results only useful when experts work in own domain?

… and what makes an domain expert?

Specific knowledge of domain experts

Common ground of domain experts in our case „temporal data explorers“

Insight Study

► 9 experts in temporal data analysis from 4 different domains

► think-aloud

► Insights: quantitative & qualitative analysis

Q: Effect of domain expertise?

05

101520

25303540

time on task(min)

sec/datainsight

number ofoverview

data insights

number ofdetail data

insights

domain expertise no domain expertise

t = -0.29, df = 24,

p > .05

=> Across domain testing works!

Effect of domain expertise on insights

Different kinds of insights in a catering business dataset

Type 1: "At noon there is a red belt." Type 2: "There is quite some breakfast business."

Type 1

Type 2

Typology

► Type 1: insight-gatherers: simple description

► Type 2: insight-hunters: active search for insights, driven by prior knowledge

“…. even the smallest domain information is used to create novel interpretations and make as much sense of the data as possible”

no sign. differences for the number of insights but significant differences for use of prior

knowledge & hypotheses

Is insight hunting driven by domain-expertise?

NO, 50% of the domain experts, but 25% of the non-domain experts showed this behavior (no sign., but small dataset)

only every second domain expert hunted for insights

Discussion

► Across-domain-testing works, but domain expertise needs a differentiated approach? (redefinition?)

► experts common ground & the story are insight-drivers

► Even shallow insights are useful, insights interrelate

► Typology iHunters | iGatherers

allows selective sampling

“hunters’ insights are the best argument to sell“

sampling is easier

“sometimes it is sufficient to gather the second choice experts”

Questions?

what is expertise?

across domain testing

shallow insightsRelational Insight Organizer RIO

what is a domain?insight hunting

insight gathering

experimental setting

story about the data

sampling

what makes an expert?prior knowledge

compensation by experts common ground

applicability

future research

generalizability

additional

0510152025

data insights

hypotheses

prior knowledge

data insights

hypotheses

prior knowledge

ed

uca

tio

nfi

na

nce

insight hunters insight gatherers

t = 0.16, df = 14, p > .05

t = -3.80, df = 7, p < .01

t = -4.87, df = 14, p < .001

Definition of insights

“… the understanding gained by an individual using a visualization tool (or parts thereof) for the purpose of data analysis, which is a gradual process towards discovering new knowledge“