bringing characters to life for immersive storytelling - dioselin gonzalez

Bringing characters to life for immersive storytelling

Dioselin GonzalezLead VR Engineer, Unity Labs

VR With The Best - March 19, 2016

Relevant background

Since Oct/2016

VR authoring tools

labs.unity.com

Character animation&

Partially autonomous character system

Networked VR simulation for dismounted soldiers

CAVE™Omnidirectional treadmill

M.S. Computer Graphics Technology

Collaborative virtual reality

InterPlay: Loose Minds in a Box (2005) - anotherlanguage.org/interplay/003_lmib/

Scenario

©2014-2016 AlloyRabbit - http://alloyrabbit.deviantart.com/

Sense-Think-Act-Communicate

M. Siegel, "The sense-think-act paradigm revisited," Robotic Sensing, 2003. ROSE' 03. 1st International Workshop on, 2003, pp. 5 pp.-.doi: 10.1109/ROSE.2003.1218700

Facial recognition

Action Units and the Facial Action Coding System

Shichuan Du et al. Compound facial expressions of emotion. PNAS 2014;111:E1454-E1462. doi: 10.1073/pnas.1322355111

Shichuan Du et al. Compound facial expressions of emotion. PNAS 2014;111:E1454-E1462. doi: 10.1073/pnas.1322355111

MPEG-4 Face andBody Animation (MPEG-4 FBA) [ISO14496] International Standard

Facial Feature Points (FP)

https://visagetechnologies.com/uploads/2012/08/MPEG-4FBAOverview.pdf

Shichuan Du et al. Compound facial expressions of emotion. PNAS 2014;111:E1454-E1462. doi: 10.1073/pnas.1322355111

Facial Feature Points (FP)

Fully Automatic Facial Feature Point Detection Using Gabor Feature Based Boosted Classifiers

Danijela Vukadinovic and Maja PanticDelft University of Technology

2005 IEEE International Conference on Systems, Man and Cybernetics

http://ibug.doc.ic.ac.uk/media/uploads/documents/VukadinovicPantic-SMC05-FINAL.pdf

Open Source Computer Vision Library

http://docs.opencv.org/3.1.0/d7/d8b/tutorial_py_face_detection.html#gsc.tab=0

OpenCV face Detection using Haar Cascades

http://docs.opencv.org/3.1.0/d7/d8b/tutorial_py_face_detection.html#gsc.tab=0

visagetechnologies.com

visage|SDK 8.0

www.ulsee.com

https://facerig.com/

Other facial features

Using Cr-Y Components to Detect Tongue Protrusion Gesturess

Crawford, C.; Bailey, S.; Badea, C.; Gilbert, J.

CHI EA '15 Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems

http://dl.acm.org/citation.cfm?id=2732916

Speech recognition

CMUSphinx toolkitPocketsphinx — lightweight recognizer library written in C.

Sphinxbase — support library required by Pocketsphinx

Sphinx4 — adjustable, modifiable recognizer written in Java

Sphinxtrain — acoustic model training tools

http://cmusphinx.sourceforge.net/

OpenEars®

Offline iPhone Voice Recognition and Text-To-Speech

http://www.politepix.com/openears/

#import <OpenEars/OELanguageModelGenerator.h>#import <OpenEars/OEAcousticModel.h>

- (void)viewDidLoad{

OELanguageModelGenerator *lmGenerator = [[OELanguageModelGenerator alloc] init];NSArray *words = [NSArray arrayWithObjects:@"WORD", @"STATEMENT", @"OTHER WORD", @"A

PHRASE", nil];NSString *name = @"NameIWantForMyLanguageModelFiles";NSError *err = [lmGenerator generateLanguageModelFromArray:words withFilesNamed:name

forAcousticModelAtPath:[OEAcousticModel pathToModel:@"AcousticModelEnglish"]]; NSString *lmPath = nil;NSString *dicPath = nil;if(err == nil) {

lmPath = [lmGenerator pathToSuccessfullyGeneratedLanguageModelWithRequestedName:@"NameIWantForMyLanguageModelFiles"];

dicPath = [lmGenerator pathToSuccessfullyGeneratedDictionaryWithRequestedName:@"NameIWantForMyLanguageModelFiles"];

} else {NSLog(@"Error: %@",[err localizedDescription]);

}}

OpenEars language model

The characters’ “brain”

Finite state machines

http://gamedevelopment.tutsplus.com/tutorials/finite-state-machines-squad-pattern-using-steering-behaviors--gamedev-13638

boost::statechart

http://www.boost.org/doc/libs/release/libs/statechart/doc/tutorial.html

http://www.boost.org/doc/libs/release/libs/statechart/doc/tutorial.html

namespace sc = boost::statechart;

struct EvShutterHalf : sc::event< EvShutterHalf > {};struct EvShutterFull : sc::event< EvShutterFull > {};struct EvShutterRelease : sc::event< EvShutterRelease > {};struct EvConfig : sc::event< EvConfig > {};

struct NotShooting;struct Camera : sc::state_machine< Camera, NotShooting >{ /* ... */};

struct Idle;struct NotShooting : sc::simple_state<NotShooting, Camera, Idle >{ typedef sc::custom_reaction< EvShutterHalf > reactions; sc::result react( const EvShutterHalf & );};

struct Idle : sc::simple_state< Idle, NotShooting >{ typedef sc::custom_reaction< EvConfig > reactions;

// ... sc::result react( const EvConfig & );};

boost::msm

http://www.boost.org/doc/libs/release/libs/msm/doc/HTML/ch03s02.html

http://www.boost.org/doc/libs/release/libs/msm/doc/HTML/ch03s02.html

struct transition_table : mpl::vector<// Start Event Target Action Guard // +---------+------------+-----------+---------------------------+----------------------------+ a_row< Stopped , play , Playing , &player_::start_playback >,a_row< Stopped , open_close , Open , &player_::open_drawer >, _row< Stopped , stop , Stopped >,// +---------+------------+-----------+---------------------------+----------------------------+ a_row< Open , open_close , Empty , &player_::close_drawer >,// +---------+------------+-----------+---------------------------+----------------------------+ a_row< Empty , open_close , Open , &player_::open_drawer >, row< Empty , cd_detected, Stopped , &player_::store_cd_info , &player_::good_disk_format >, row< Empty , cd_detected, Playing , &player_::store_cd_info , &player_::auto_start >,// +---------+------------+-----------+---------------------------+----------------------------+ a_row< Playing , stop , Stopped , &player_::stop_playback >,a_row< Playing , pause , Paused , &player_::pause_playback >,a_row< Playing , open_close , Open , &player_::stop_and_open >,// +---------+------------+-----------+---------------------------+----------------------------+ a_row< Paused , end_pause , Playing , &player_::resume_playback >,a_row< Paused , stop , Stopped , &player_::stop_playback >,a_row< Paused , open_close , Open , &player_::stop_and_open >// +---------+------------+-----------+---------------------------+----------------------------+ > {};

Behavior trees

Image from http://charliepark.tumblr.com/post/97667486/a-sample-behavior-tree-from-spore-from-chris

Behavior trees

Behavior trees

HALO 2 (2004)

GDC 2005 - Handling Complexity in the Halo 2 AI

Image from http://en.wikipedia.org

Behavior trees

Façade: a one-act interactive drama (2005)

GDC 2005 - AI and Interactive Storytelling: How We Can Help Each Other

Image from http://www.macobserver.com/

Behavior trees

SPORE

GDC 2010 - Behavior Trees: Three Ways of Cultivating Strong AI

Image from http://en.wikipedia.org

BT_root

decider

decider

decider

decider

decider

behavior(s)

behavior(s)

behavior(s)

behavior(s)decider

behavior(s)decider

Group decider

Behavior decider

BT_root

decider

decider

decider

decider

decider

behavior(s)

behavior(s)

behavior(s)

behavior(s)decider

behavior(s)decider

id type data

... ... ...

Blackboard

BT_root

decider

decider

decider

decider

decider

behavior(s)

behavior(s)

behavior(s)

behavior(s)decider

behavior(s)decider

bool GroupDecider::decide(Blackboard &);

BT_root

decider

decider

decider

decider

decider

behavior(s)

behavior(s)

behavior(s)

behavior(s)decider

behavior(s)decider

enum Status {RUNNING, SUCCESS, FAILURE};

bool BehaviorDecider::activate(Blackboard &);bool BehaviorDecider::deactivate(Blackboard &);Status BehaviorDecider::tick(Blackboard &);

BT_root

decider

decider

decider

decider

decider

behavior(s)

behavior(s)

behavior(s)

behavior(s)decider

behavior(s)decider

Group decider types - Priority List

BT_root

Flee

Feed_self

behavior(s)

behavior(s)

behavior(s)

eat_food

Find_food

Idle behavior(s)

Group decider types - Priority List

BT_root

Flee

Feed_self

behavior(s)

behavior(s)

behavior(s)

eat_food

Find_food

Idle behavior(s)

? ?

Group decider types - Priority List

BT_root

Flee

Feed_self

behavior(s)

behavior(s)

behavior(s)

eat_food

Find_food

Idle behavior(s)

12:00 pm

? ?

Group decider types - Priority List

BT_root

Flee

Feed_self

behavior(s)

behavior(s)

behavior(s)

eat_food

Find_food

Idle behavior(s)

12:00 pm

? ?

Group decider types - Priority List

BT_root

Flee

Feed_self

behavior(s)

behavior(s)

behavior(s)

eat_food

Find_food

Idle behavior(s)

? ?

Group decider types - Priority List

BT_root

Flee

Feed_self

behavior(s)

behavior(s)

behavior(s)

eat_food

Find_food

Idle behavior(s)

? ?

Group decider types - Priority List

BT_root

Flee

Feed_self

behavior(s)

behavior(s)

behavior(s)

eat_food

Find_food

Idle behavior(s)

? ?

Group decider types - Sequence

BT_root

Mix_ingredients

Bake

behavior(s)

behavior(s)

behavior(s)

Put_food_in

Preheat_oven

Decorate behavior(s)

→ →

Group decider types - Parallel

BT_root

Sing

Dance

behavior(s)

behavior(s)

behavior(s)

Jump

Wave_arms

Idle behavior(s)

→→

?

→→

More group decider types (and decorators)

randomrepeat <x>

timesloop_seq

Behavior Designer @ Unity Asset Store

http://www.opsive.com/assets/BehaviorDesigner/

Concurrency: Resource allocators

http://aigamedev.com/open/article/popular-behavior-tree-design/

Aborting/Enabling subtrees

http://aigamedev.com/open/article/popular-behavior-tree-design/

Hybrid architectures

Horizontal layering

Sensor input

Action output

http://www.inf.ed.ac.uk/teaching/courses/abs/slides/abs05-reactivehybrid.pdf

Hybrid architectures

Horizontal layering Vertical layering

Sensor input

Action output

Sensor input

Action outputOne-pass control

http://www.inf.ed.ac.uk/teaching/courses/abs/slides/abs05-reactivehybrid.pdf

Hybrid architectures

Horizontal layering Vertical layering

Sensor input

Action output

Sensor input

Action output

Sensor input Action output

One-pass control Two-pass control

http://www.inf.ed.ac.uk/teaching/courses/abs/slides/abs05-reactivehybrid.pdf

InteRRaP

http://www.sharprobotica.com/2011/01/interrap-hybrid-architecture-for-robotic-multi-agent-systems/

Behavior realization language(i.e. messaging protocol)

(some) Issues

AI Sound

Animation

Physics

(some) Issues

AI Sound

Animation

Physics

Behavior expression

(some) Issues

AI Sound

Animation

Physics

Timing

(some) Issues

AI Sound

Animation

Physics

Synchronization

(some) Issues

AI Sound

Animation

Physics

Interruption

Behavior Markup Language (BML)

Intent planning Behavior planning Behavior realization

SAIBA framework (Situation, Agent, Intention, Behavior, Animation)

FML BML

Towards a Common Framework for Multimodal Generation: The Behavior Markup Language http://xenia.media.mit.edu/~kris/ftp/BML-IVA-06-KoppEtAl.pdf

Behavior Markup Language (BML)

http://www.mindmakers.org/projects/bml-1-0/wiki

BML: synchronization

<bml id="bml1" xmlns="http://www.bml-initiative.org/bml/bml-1.0" character="Alice">

<pointing id="behavior1" target="blueBox" mode="RIGHT_HAND" start="speech1:start"/>

<speech id="speech1"><text>Look there!</text></speech>

...

<constraint>

<synchronize>

<sync ref="speech1:sync4"/>

<sync ref="beat1:stroke:2"/>

<sync ref="nod1:stroke"/>

</synchronize>

</constraint>

http://www.mindmakers.org/projects/bml-1-0/wiki

BML: behavior elements

Towards a Common Framework for Multimodal Generation: The Behavior Markup Language http://xenia.media.mit.edu/~kris/ftp/BML-IVA-06-KoppEtAl.pdf

What is interactive storytelling anyway?

What is interactive storytelling anyway?

Image from http://ragreynolds.net/review-the-last-of-us-remastered/

More referencesGDC 2005 Handling Complexity in the Halo 2 AI - Gamasutra, GDC Vault.

AiGameDev.com Behavior Trees for Next-Gen Game AI - Part 1, Part 2, Part 3.

University of Denver Video Game AI: Behavior Trees & Blackboards

GDC AI Summit 2010 Behavior Trees: Three Ways of Cultivating Strong AI - GDC Vault, AiGameDev.com

Chatting Up Façade’s AI: 23 Ideas to Talk Your Game Into - AiGameDev.com

Popular Approaches to Behavior Tree Design - AiGameDev.com