game accessibility at hanze hogeschool
Post on 22-Sep-2014
447 views
DESCRIPTION
TRANSCRIPT
Human Computer Interaction Lab
University of Nevada, Reno
Playing Games without Visual Feedback
Eelke Folmer, Associate Professor
Human Computer Interaction Lab
University of Nevada, Reno
reno
Player-Game Interaction Research
University of Nevada, Reno
Research Interests
Accessible Computing Visual Impairments Spatial Perception Video Games Augmented Reality 3D interfaces Wearable computing Natural Interfaces Sonification Haptics Human Navigation CrowdSourcing Sensory Substitution exergaming Spatial computing
Automotive Interfaces Virtual Reality Gestures Mobile computing
Player-Game Interaction Research
University of Nevada, Reno
Overview of Talk
What are some of the barriers that blind people face when playing video games?
Why should video games be accessible to users who are blind?
How can virtual world & video games be made accessible?
Player-Game Interaction Research
University of Nevada, Reno
How do we play games?
Player-Game Interaction Research
University of Nevada, Reno
How do we play games?
Player-Game Interaction Research
University of Nevada, Reno
How do we play games?
Abstraction
1. feedback2. fire gun3. pull trigger
1. feedback2. kick & punch3. move feet & arm
1. feedback2. aim for point3. drag with finger
Player-Game Interaction Research
University of Nevada, Reno
Interaction Model
Player-Game Interaction Research
University of Nevada, Reno
Gaming with Visual Impairment
retintis pigmentosa color blindness
macular degeneration blindness
Player-Game Interaction Research
University of Nevada, Reno
Interaction Model
Player-Game Interaction Research
University of Nevada, Reno
Sensory Substitution Research
Visual
AudioHaptic
information
closed captioning
braille
speech synthesis
Player-Game Interaction Research
University of Nevada, Reno
Research Challenges
Visual
AudioHaptic
information
informationinformation
Why should people with Visual Impairments be able to play video games?
Player-Game Interaction Research
University of Nevada, Reno
Benefits
Social
HealthEmployment
Education
Player-Game Interaction Research
University of Nevada, Reno
Visual Impairment
1.3 million people who are blind (60k kids) [US]6.8 million who have a visual impairment [US]Population of VI expected to double [WHO]Obesity --> cataracts --> blindness
Player-Game Interaction Research
University of Nevada, Reno
“Extreme Interaction” Design Design Interfaces for “extreme” playersGames are complex real time simulationsDisability as a driver of innovation
speech synthesis
screen readers
speech recognition
automatic captions
Player-Game Interaction Research
University of Nevada, Reno
Innovation by solving extreme problems
? ?
Human Computer Interaction Research
University of Nevada, Reno
Example accessible interfaces
Natural Language Virtual World Interfaces»Text SL »Syntherella
Non-Visual Natural User Interfaces (Exergames)»VI Tennis»VI Bowling
»Seek-n-tag
»Pet-n-Punch»RTSS
Natural Language Virtual World Interface
HCC-Small: TextSL: A Virtual World Interface for Visually Impaired, National Science Foundation Eelke Folmer (PI), George Bebis (Co-Pi). Amount: $499,332. SGER: Developing an Accessible Client for Second Life, National Science Foundation Eelke Folmer (PI), Amount: $90,488.
TextSL
Player Game Interaction Research
University of Nevada, Reno
Virtual WorldsEducation Museums Communities
Player Game Interaction Research
University of Nevada, Reno
Web Accessibility
“ The eiffel tower was built in ...”“alttag:picture of the eiffel tower”
Screen reader
text/images
Player Game Interaction Research
University of Nevada, Reno
Virtual World Accessibility
“.............................”
Screen reader
pixels
Human Computer Interaction Research
University of Nevada, Reno
Inspiration
Command Line Interface»Text only»Iterative »Natural language»Screen reader accessible
Zork
web based interface»screenreader /
speechbasic commands
» navigation
» exploration
Text SL
» communication
» interaction
Human Computer Interaction Research
University of Nevada, Reno
Natural Language Interpreter
Avoid learning specific commandsIntuitive & Easy to Learn
walk
go
move
move
natural language prepositions
move
tochai
rthe
adjectives
give my
flower to jane
Human Computer Interaction Research
University of Nevada, Reno
SL Content Problems
Densely populated with objects »Overwhelm users with feedback»Difficult to navigate collision free
Lack of meta data (40%)»underwhelm users with feedback
tree
?fire
bike
dogtree
moe
jack
this object has a really long name
wall
chair
tree
table
jill
curly
carchest
larrybike bike
bike
tree
tree
?
?
?
?
?
?
bike
car
car
Human Computer Interaction Research
University of Nevada, Reno
Navigation
Basic Obstacle avoidanceTeleport when stuckMake avatar appear as normal as possible
>move north 10>move to name (object / avatar)>teleport to Help Island
Human Computer Interaction Research
University of Nevada, Reno
Communication>say “Hello” >whisper to Jack “bla”>mute Jack
WTF?
>sit on chair
Human Computer Interaction Research
University of Nevada, Reno
Interaction>touch billboard
- no textual output (yet)
Human Computer Interaction Research
University of Nevada, Reno
Iterative query mechanisms
chair
fire
dog
jill
>describe“you see avatar jill, and the objects chair, a fire and a dog”>describe dog“the dog is a golden retriever and is for Sale for $10”>where dog“the dog is 10 meters to your right”
Human Computer Interaction Research
University of Nevada, Reno
Syntherella: Scene Synthesizer
tree
fire
bike
dogtree
jack
wall
chair
tree
table
curly
carchest
larrybike
bike
tree
?
?
?? bike
object with a really long name
moejill
bike
tree
?
?
?
car
tree
larrybike?bike
dog
dog
dog
chest
chest
- Content within range - Remove non descriptive objects
- Based on #worldlimitif #words > #wl {abstraction}else {detailing}
Player Game Interaction Research
University of Nevada, Reno
Abstraction
>describe“there are 3 animals and 3 vehicles
bikecatdogtruc
kcarpig
vehiclesanimals
object taxonomy
>describe“there is a cat a dog, a car, a bike a car, a truck and a pig
bike
cat
dog
truck
car
car
pig
Player Game Interaction Research
University of Nevada, Reno
Detailing
>describe“there is a cat and a dog and a car
>describe“to your left is a black cat and a brown dog who is barking. In front of you is a red race car.
Player Game Interaction Research
University of Nevada, Reno
User Studies
slower
slower
slower
same
Player Game Interaction Research
University of Nevada, Reno
Lack of Meta data
?
?
?
?
?
?
?
>describe“there is an object an object an object an object an object etc...
Player Game Interaction Research
University of Nevada, Reno
Labeling images vs 3D objects
- Defined in Isolation- Solid bodies (prims)- Efficient discrimination
- Segmentation- 2D info
Player Game Interaction Research
University of Nevada, Reno
Construct Classifier
object categories
shape descriptors
unknown object
Player Game Interaction Research
University of Nevada, Reno
how to derive object categories?
dog?
Player Game Interaction Research
University of Nevada, Reno
Use Human Computation
find a cat
label
label
label
Player Game Interaction Research
University of Nevada, Reno
Seek-n-TagGoal:
»Player participates in Scavenger hunt
Rules:»30 seconds to “tag” an object»Score 50 points per tagged object»Initially start with 900 seconds»Game over when time =0»Remaining time is added to clock.
Competition»Leaderboard
Find a Cat
Find a Tree
Find a Dog
Find a Shoe
✔ + 5s
✔ + 15s
✘ - 30s
✔ + 10s
Player Game Interaction Research
University of Nevada, Reno
User study
Manual labeling versus labeling with a game
Seek-n-tag faster Seek-n-tag takes fewer labeling attempts
to reach consensus on object name.
object
object objec
t
object
object
object
object
object objec
t
Player Game Interaction Research
University of Nevada, Reno
Manual labeling using AMT
90%
1000
500
Currently Researching
Human Computer Interaction Research
University of Nevada, Reno
Interaction with interactive Objects
>touch billboard>touch horse
Collect descriptions using AMTSemantic analysis of descriptions to understand
what properties of objects play a roleSee if properties can be parsed from state
changes in object
>create green cube “block1”
Human Computer Interaction Research
University of Nevada, Reno
Content Creation
>create green cube “block1”>link block1 block2
>create brown d0g
Human Computer Interaction Research
University of Nevada, Reno
Publications / More info
ear.textsl.org
• Bugra Oktay, Eelke Folmer. Syntherella: A Feedback Synthesizer for Efficient Exploration of Virtual Worlds using a Screen Reader,Proceedings of Graphics Interface 2011. Pages 65-70, St John, Canada, May 2011.
• Bei Yuan, Manjari Sapre, Eelke Folmer. Seek-n-Tag: A Game for Labeling and Classifying Virtual World Objects, Proceedings of Graphics Interface (GI). Pages 201-208, Ottawa, Ontario, June 2010.
• Eelke Folmer, Bei Yuan, Dave Carr, Manjari Sapre. TextSL: A Command-Based Virtual World Interface for the Visually Impaired, In Proceedings of the 11th international ACM SIGACCESS conference on Computers and Accessibility, Pages 59-66, Pittsburgh, Pennsylvania, October 2009.
code.google.com/p/textsl
Try ?
Contribute?
Non Visual Natural User Interfaces
HCC: Small: Proprioceptive Displays to Engage Blind Users into Healthy Whole Body Interaction, National Science Foundation Eelke Folmer (PI), Amount: $420,320.
John Folycollaborators:Lauren Liebermann
VI Fit
Player-Game Interaction Research
University of Nevada, Reno
Gesture Based Interaction
Player-Game Interaction Research
University of Nevada, Reno
Reliance on Visual Cues
dodge
♫♫
Player-Game Interaction Research
University of Nevada, Reno
visual impairment & obesity
60,000 Blind Children in US More obese
Player-Game Interaction Research
University of Nevada, Reno
Barriers to physical activity
Rely on sighted guide Safety
Player-Game Interaction Research
University of Nevada, Reno
Exercise games
★ Moderate to Vigorous PA (MVPA)★60 min of MVPA daily (CDC)
★Independently★Safer
Player-Game Interaction Research
University of Nevada, Reno
Research question
Player-Game Interaction Research
University of Nevada, Reno
Gestures w/o visual cues?
?
Player-Game Interaction Research
University of Nevada, Reno
Physical Activity = Spatial-Temporalwhere
?
when?
Target Acquisition
Player-Game Interaction Research
University of Nevada, Reno
Exergames are temporal-spatial
Kinect Sports Eye Toy Kinetic
★Temporal/Spatial: Jump when hurlde is close
★ Spatial: Punch/ Kick Targets★ Temporal: Dodge target
Player-Game Interaction Research
University of Nevada, Reno
Sensory Substitution Research
Visual
AudioHaptic
information
target acquisition ??(directed) gestures
Player-Game Interaction Research
University of Nevada, Reno
Constraints with regard to SS
Social Contexts Music based
cool!!
Audio Haptic?
Player-Game Interaction Research
University of Nevada, Reno
Temporal Challenge
serve
Wii Sports
Buzz
♫
♫bounce
return
VI Tennis
Human Computer Interaction Research
University of Nevada, Reno
Sensory Substitution
VI Tennis
WII Tennis
Human Computer Interaction Research
University of Nevada, Reno
Wii Bowling VI Bowling
Spatial Challenge
VI Bowling
Wii Bowling
Player Game Interaction Research
University of Nevada, Reno
Studies @ Camp Abilities
Vi Tennis/Bowling yield Active energy expenditure that is considered moderate physical activity
Kids need 20 minutes of moderate to vigorous PA
Player-Game Interaction Research
University of Nevada, Reno
User studies
light
moderate
vigorous
VI Tennis
Upper Body
Whole Body
Player-Game Interaction Research
University of Nevada, Reno
Pet-n-Punch
Player-Game Interaction Research
University of Nevada, Reno
Instrumentation
Player-Game Interaction Research
University of Nevada, Reno
Real time Video Analysis<SECTION> <XMIN>100</XMIN> <XMAX>300</XMAX> <YMIN>150</YMIN> <YMAX>300</YMAX> <MINR>230</MINR> <MAXR>999</MAXR> <MING>230</MING> <MAXG>999</MAXG> <MINB>0</MINB> <MAXB>100</MAXB> <ACTION <WIIMOTE> <ID>0</ID> <DURATION>-1</DURATION> </WIIMOTE> <AUDIO> <ID>alert.wav</ID> <LOOP>0</LOOP> </AUDIO> </ACTION></SECTION>
QuickTime™ and a decompressor
are needed to see this picture.
Player-Game Interaction Research
University of Nevada, Reno
VI Ski
Related Projects
Human Computer Interaction Research
University of Nevada, Reno
Publications / More info
vifit.orgcode.google.com/p/vifit
Try ?
Contribute?
•Tony Morelli, Eelke Folmer. Real-time Sensory Substitution to Enable Players who are Blind to Play Gesture based Videogames, In Proceedings of Foundations of Digital Interactive Games, to appear, Bordeaux France, June 2011•Tony Morelli, John Foley, Lauren Lieberman, Eelke Folmer. Pet-N-Punch: Upper Body Tactile/Audio Exergame to Engage Children with Visual Impairments into Physical Activity, Proceedings of Graphics Interface (GI) •Tony Morelli, John Foley, Eelke Folmer VI Bowling: A Tactile Spatial Exergame for Individuals with Visual Impairments , In Proceedings of ASSETS, Pages 179-186, Orlando, Florida, October 2010. •Tony Morelli, John Foley, Luis Columna, Lauren Lieberman, Eelke Folmer. VI-Tennis: a Vibrotactile/Audio Exergame for Players who are Visually Impaired, Proceedings of FDG, Pages 147-154, Monterey, California, June 2011