underlying technologies part two: software mark green school of creative media
Post on 22-Dec-2015
219 views
TRANSCRIPT
Underlying TechnologiesUnderlying TechnologiesPart Two: SoftwarePart Two: Software
Mark GreenMark Green
School of Creative MediaSchool of Creative Media
IntroductionIntroduction
Software not as easy as hardware:Software not as easy as hardware: wide range of software techniques, hard to wide range of software techniques, hard to
classify like hardwareclassify like hardware several components that need to work several components that need to work
together, hard to know where to starttogether, hard to know where to start wide range of hardware configurations, not as wide range of hardware configurations, not as
simple as 2D softwaresimple as 2D software
Hardware ConfigurationsHardware Configurations
In 2D have a standard hardware In 2D have a standard hardware configuration:configuration: input: keyboard and mouseinput: keyboard and mouse output: single 2D displayoutput: single 2D display
with 3D can have many configurations:with 3D can have many configurations: HMDHMD projectionprojection single screensingle screen
Hardware ConfigurationsHardware Configurations
Want to produce an application once, not Want to produce an application once, not once for every possible hardware once for every possible hardware configurationconfiguration
software needs to be more adaptable, software needs to be more adaptable, change based on hardware configurationchange based on hardware configuration
complicates the development of support complicates the development of support softwaresoftware
Range of Software TechniquesRange of Software Techniques
Want our software to be very efficient: Want our software to be very efficient: reduce latency, high update ratesreduce latency, high update rates
some applications can be quite large, some applications can be quite large, need to efficiently organize dataneed to efficiently organize data
all of this complicates VR software, too all of this complicates VR software, too many things to consider, hard to know many things to consider, hard to know where to startwhere to start
ComponentsComponents
What are the main components of a VR What are the main components of a VR application?application? 3D Objects: geometry and appearance, but 3D Objects: geometry and appearance, but
may also want sound and forcemay also want sound and force Behavior: the objects need to be able to do Behavior: the objects need to be able to do
things, move and reactthings, move and react Interaction: users want to interact with the Interaction: users want to interact with the
application, manipulate the objectsapplication, manipulate the objects
3D Objects3D Objects
Need object geometry, object’s shape, Need object geometry, object’s shape, basis for everything else, called modelbasis for everything else, called model
polygons used for geometry, sometimes polygons used for geometry, sometimes restricted to trianglesrestricted to triangles
different from animation, free form different from animation, free form surfaces based on sophisticated mathsurfaces based on sophisticated math
need speed, so restricted to polygonsneed speed, so restricted to polygons
3D Objects3D Objects
Where does geometry come from?Where does geometry come from? Really depends on the applicationReally depends on the application Could use a text editor to enter all the Could use a text editor to enter all the
polygon vertices, some people actually do polygon vertices, some people actually do this!this!
Could use a program, for example Could use a program, for example OpenGL, works for small modelsOpenGL, works for small models
3D Objects3D Objects
Use a 3D modeling or animation programUse a 3D modeling or animation program for non-programmers this is the easiest for non-programmers this is the easiest
way, but it takes time to develop modeling way, but it takes time to develop modeling skillsskills
also many different program and file also many different program and file formatsformats
want a modeler that does a good job of want a modeler that does a good job of polygons, not all modelers are good at thispolygons, not all modelers are good at this
3D Objects3D Objects
Another source of objects is scientific and Another source of objects is scientific and engineering computationsengineering computations
can be easy to convert to polygons, can be easy to convert to polygons, already have position dataalready have position data
other types of data can also be converted other types of data can also be converted into geometry, but this can be more into geometry, but this can be more difficultdifficult
3D Objects3D Objects
Also need to consider appearance:Also need to consider appearance: colour of the objectcolour of the object how it reflects lighthow it reflects light transparencytransparency texturetexture
can be done with modeler, or later in the can be done with modeler, or later in the VR programVR program
BehaviorBehavior
How should objects behave?How should objects behave? What happens when the user hits an object?What happens when the user hits an object? What happens when an object hits another What happens when an object hits another
object?object? Can objects move around the environment?Can objects move around the environment?
Each object could have a range of Each object could have a range of behaviors, react differently to different behaviors, react differently to different events in the environmentevents in the environment
BehaviorBehavior
Behavior is harder than modelingBehavior is harder than modeling animation programs can be useful, but not animation programs can be useful, but not
alwaysalways animation is quite different:animation is quite different:
animator is in complete control, knows what’s animator is in complete control, knows what’s happening all of the timehappening all of the time
in VR the user is in control, can interrupt or in VR the user is in control, can interrupt or mess up any animationmess up any animation
BehaviorBehavior
Short animations (less than 5 seconds) Short animations (less than 5 seconds) can be useful, basic motion unitscan be useful, basic motion units
other types of behaviors must be other types of behaviors must be programmed or scriptedprogrammed or scripted
more flexible, can respond to the events more flexible, can respond to the events that occur in the environmentthat occur in the environment
easier to combine, objects can do two easier to combine, objects can do two things at same timethings at same time
InteractionInteraction
Users want to interact with the environmentUsers want to interact with the environment pick up objects and move them aroundpick up objects and move them around very different from 2D interactionvery different from 2D interaction much more freedom, more direct interactionmuch more freedom, more direct interaction still exploring the design space, not stable still exploring the design space, not stable
like 2D interactionlike 2D interaction still working on standard techniquesstill working on standard techniques
Application StructureApplication Structure
look at application structurelook at application structure provides a framework for discussing provides a framework for discussing
various software technologiesvarious software technologies divide an application into various divide an application into various
components, and then look at the components, and then look at the components individuallycomponents individually
Application StructureApplication Structure
Model
ApplicationProcessing
Output Devices
InputDevices
Model Traversal
InputProcessing
Application StructureApplication Structure
Model: representation of objects in the Model: representation of objects in the environment, geometry and behaviorenvironment, geometry and behavior
Traversal: convert the model into Traversal: convert the model into graphical, sound, force, etc outputgraphical, sound, force, etc output
Input Processing: determine user’s Input Processing: determine user’s intentions, act on other part of applicationintentions, act on other part of application
application processing: non-VR parts of application processing: non-VR parts of the applicationthe application
Interaction LoopInteraction Loop
Logically the program consists of a loop Logically the program consists of a loop that samples the user, performs that samples the user, performs computations and traverses the modelcomputations and traverses the model
Input processing
Computation
Model Traversal
ModelModel
Contains the information required to Contains the information required to display the environment:display the environment: geometry, sound, forcegeometry, sound, force behaviorbehavior
the graphical part is the most developed, the graphical part is the most developed, so concentrate on itso concentrate on it
try to position sound and force within this try to position sound and force within this modelmodel
GeometryGeometry
This is what we know the bestThis is what we know the best need to have a graphical representation of need to have a graphical representation of
objects in the environment:objects in the environment: accurate shape representationaccurate shape representation ease of modelingease of modeling efficient displayefficient display integrates with behaviorintegrates with behavior
Scene GraphScene Graph
Main technique for structuring the modelMain technique for structuring the model based on hierarchical structure, divide the based on hierarchical structure, divide the
object into parts or componentsobject into parts or components simplifies the modeling task, work on one simplifies the modeling task, work on one
part at a timepart at a time easy to modify the individual partseasy to modify the individual parts add behaviors, sound, force, etc to the add behaviors, sound, force, etc to the
modelmodel
Scene GraphScene Graph
car
Wheel Wheel Wheel Wheel Body
Scene GraphScene Graph
Individual units are called nodes:Individual units are called nodes: shapes: polygons, meshes, cubes, etcshapes: polygons, meshes, cubes, etc transformations: position the nodes in spacetransformations: position the nodes in space material: colour and texture of objectsmaterial: colour and texture of objects grouping: collecting nodes together as a grouping: collecting nodes together as a
single objectsingle object soundssounds behaviorbehavior
Scene GraphScene Graph
Many different scene graph architectures, Many different scene graph architectures, will look at one in more detail laterwill look at one in more detail later
differences:differences: scene graph for whole VE Vs. one per objectscene graph for whole VE Vs. one per object types of nodes in the scene graphtypes of nodes in the scene graph ease of modification, static Vs dynamicease of modification, static Vs dynamic
BehaviorBehavior
Harder to deal with than geometryHarder to deal with than geometry simple motions aren’t too bad, but much simple motions aren’t too bad, but much
harder to get sophisticated behaviorharder to get sophisticated behavior the general solution now is to write code, the general solution now is to write code,
okay for programmersokay for programmers would like to have a higher level approach would like to have a higher level approach
for non-programmersfor non-programmers
BehaviorBehavior
Problem: want objects to respond to Problem: want objects to respond to events in the environmentevents in the environment
can have some motions that are simple can have some motions that are simple animations, but most of the motions need animations, but most of the motions need some knowledge of the environmentsome knowledge of the environment
example: an object moving through the example: an object moving through the environment must be aware of other environment must be aware of other objects so it doesn’t walk through themobjects so it doesn’t walk through them
BehaviorBehavior
Some simple motions produced by Some simple motions produced by animating transformation nodesanimating transformation nodes
animation variables used to control animation variables used to control transformation parameters, example: transformation parameters, example: rotation or translationrotation or translation
could import animations, use some form of could import animations, use some form of keyframing package to produce the motionkeyframing package to produce the motion
BehaviorBehavior
Simple motions could be triggered by Simple motions could be triggered by events in the environmentevents in the environment
example: collision detection, if an object is example: collision detection, if an object is moving through the environment and a moving through the environment and a collision detected it changes directioncollision detected it changes direction
hard to come up with good trigger hard to come up with good trigger conditions, a few obvious ones, but not conditions, a few obvious ones, but not much after thatmuch after that
BehaviorBehavior
Another approach is to use a general Another approach is to use a general motion modelmotion model
best example of this is physics, try to best example of this is physics, try to simulate real physics in the environmentsimulate real physics in the environment
this gives a number of natural motions, this gives a number of natural motions, and objects respond to the environmentand objects respond to the environment
works well in some environment, but has works well in some environment, but has some problemssome problems
BehaviorBehavior
One problem is the complexity of the One problem is the complexity of the mathematics, often need to simplifymathematics, often need to simplify
computations can be a problem, computations can be a problem, particularly for complex objectsparticularly for complex objects
hard to control, need to know forces and hard to control, need to know forces and torque's that produce the desired motions, torque's that produce the desired motions, can be very hard to determinecan be very hard to determine
BehaviorBehavior
Some attempts to produce general motion Some attempts to produce general motion controllerscontrollers
maybe the eventual solution, but nothing maybe the eventual solution, but nothing much nowmuch now
use of scripting languages, can add some use of scripting languages, can add some program control to the scene graph, but program control to the scene graph, but not full programmingnot full programming
Model TraversalModel Traversal
The process of going through the model The process of going through the model and generating the information to be and generating the information to be displayeddisplayed
this is part software and part hardware, this is part software and part hardware, look through the entire processlook through the entire process
hardware parts have implications for how hardware parts have implications for how we build models and the graphics we build models and the graphics techniques usedtechniques used
A Simple ModelA Simple Model
A simplified model of the display process, A simplified model of the display process, explains hardware performanceexplains hardware performance
traverse geometry Pixel
Model Screen
TraverseTraverse
Traverse the model, determine objects to Traverse the model, determine objects to be drawn, send to graphics hardwarebe drawn, send to graphics hardware
usually combination software/hardware, usually combination software/hardware, depends on CPU and bus speeddepends on CPU and bus speed
early systems were hardware, didn’t scale early systems were hardware, didn’t scale wellwell
many software techniques for culling many software techniques for culling modelsmodels
GeometryGeometry
Geometrical computations on polygons: Geometrical computations on polygons: transformations and lightingtransformations and lighting
floating point intensivefloating point intensive divide polygons into fragments, screen divide polygons into fragments, screen
aligned trapezoidaligned trapezoid time proportional to number of polygons time proportional to number of polygons
and verticesand vertices
PixelPixel
Fill fragments, colour interpolation, texture Fill fragments, colour interpolation, texture mapping, transparency, hidden surfacemapping, transparency, hidden surface
all the per pixel computations, time all the per pixel computations, time depends on number of pixels, also colour depends on number of pixels, also colour depth on low end displaysdepth on low end displays
scalable operations, can add more scalable operations, can add more processors for more speedprocessors for more speed
Design ConsiderationsDesign Considerations
Any of the stages could block, depend on Any of the stages could block, depend on display mixdisplay mix
lots of small polygons cause traversal and lots of small polygons cause traversal and geometry stages to blockgeometry stages to block
large polygons cause pixel stage to blocklarge polygons cause pixel stage to block can use buffers to reduce blockingcan use buffers to reduce blocking a careful balancing processa careful balancing process
Design ConsiderationsDesign Considerations
CPU/Image Generator trade-offCPU/Image Generator trade-off cheap boards just do pixel stage, use CPU cheap boards just do pixel stage, use CPU
for everything else:for everything else: scales with CPU speedscales with CPU speed large polygons and texture mappinglarge polygons and texture mapping
moving geometry onto board increases moving geometry onto board increases performance, trend in low cost displaysperformance, trend in low cost displays
PC Hardware EvolutionPC Hardware Evolution
Start with CPU doing most of the workStart with CPU doing most of the work Graphics board:Graphics board:
image memoryimage memory fill and hidden surfacefill and hidden surface texture mappingtexture mapping
graphics speed determined by CPU, graphics speed determined by CPU, limited assistance from graphics cardlimited assistance from graphics card
Graphics Card MemoryGraphics Card Memory
Memory used for three things:Memory used for three things: image storeimage store hidden surface (z buffer)hidden surface (z buffer) texture mapstexture maps
texture can be stored in main memory with texture can be stored in main memory with AGP, but this isn’t most efficientAGP, but this isn’t most efficient
better to have texture memory on boardbetter to have texture memory on board
Image MemoryImage Memory
Amount depends on image sizeAmount depends on image size double buffer, two copies of image double buffer, two copies of image
memorymemory front buffer: image displayed on screenfront buffer: image displayed on screen back buffer: where the next image is back buffer: where the next image is
constructedconstructed can construct next image while the current can construct next image while the current
image is displayed, better image quality image is displayed, better image quality and faster displayand faster display
Z BufferZ Buffer
Used for hidden surface removalUsed for hidden surface removal z buffer: one value for each pixel, distance z buffer: one value for each pixel, distance
from eye to object drawn at that pixelfrom eye to object drawn at that pixel when drawing a pixel, compare depth of when drawing a pixel, compare depth of
pixel to z bufferpixel to z buffer if closer draw pixel and update z bufferif closer draw pixel and update z buffer otherwise, ignore the pixelotherwise, ignore the pixel
Graphics AccelerationGraphics Acceleration
Next step: move pixel operations to Next step: move pixel operations to graphics cardgraphics card
fill and z buffer 3D trianglesfill and z buffer 3D triangles add smooth shading and texture mappingadd smooth shading and texture mapping CPU does traversal and geometry CPU does traversal and geometry
processingprocessing
Graphics AccelerationGraphics Acceleration
Next step: move geometry processing to Next step: move geometry processing to graphics cardgraphics card
CPU traverses model, send graphics CPU traverses model, send graphics primitives to display cardprimitives to display card
all transformations and lighting done on all transformations and lighting done on graphics cardgraphics card
less dependence on CPUless dependence on CPU
Current TrendsCurrent Trends
Pixel processing (Geforce 2): a program Pixel processing (Geforce 2): a program that processes each pixel, control lighting that processes each pixel, control lighting and other effectsand other effects
support for multiple textures, etcsupport for multiple textures, etc Vertex processing (Geforce 3): a program Vertex processing (Geforce 3): a program
processes each vertex, can change processes each vertex, can change geometry at display timegeometry at display time
real-time deformations and IKAreal-time deformations and IKA
Current TrendsCurrent Trends
Move to programming all aspects of the Move to programming all aspects of the graphics card (3DLabs VP series)graphics card (3DLabs VP series)
Also making programming more Also making programming more sophisticated, closer to CPUsophisticated, closer to CPU
Floating point textures and image memory Floating point textures and image memory (ATI and 3DLabs VP series)(ATI and 3DLabs VP series)
Higher dynamic range -> better image Higher dynamic range -> better image quality, better for programmingquality, better for programming
Input ProcessingInput Processing
Users need to interact with the Users need to interact with the environmentenvironment
they have a set of input devices, have they have a set of input devices, have position and orientation informationposition and orientation information
need to translate this into their intentionsneed to translate this into their intentions Interaction Technique (IT): basic unit of Interaction Technique (IT): basic unit of
interaction, converts user input into interaction, converts user input into something the application understandssomething the application understands
Input ProcessingInput Processing
Each IT address a particular interaction Each IT address a particular interaction task, something that the user wants to dotask, something that the user wants to do
look at interaction tasks first, then talk a look at interaction tasks first, then talk a little bit about ITs for themlittle bit about ITs for them
interaction tasks divide into two groups:interaction tasks divide into two groups: application independent: required by many application independent: required by many
different applicationsdifferent applications application dependentapplication dependent
Interaction TasksInteraction Tasks
Mainly look at application independent Mainly look at application independent interaction tasksinteraction tasks
the main ones are:the main ones are: navigationnavigation selectionselection manipulationmanipulation combinationcombination
NavigationNavigation
Need to get from one part of the Need to get from one part of the environment to anotherenvironment to another
two types:two types: locallocal globalglobal
with local navigation the destination is with local navigation the destination is within view, move on continuous path from within view, move on continuous path from current location to destinationcurrent location to destination
NavigationNavigation
In global navigation the destination is In global navigation the destination is remote, can’t move directly to itremote, can’t move directly to it
need some way of locating destination, need some way of locating destination, and then some way of jumping to itand then some way of jumping to it
variation: browsing / exploration don’t have variation: browsing / exploration don’t have a destination, exploring the environment or a destination, exploring the environment or looking for particular objectslooking for particular objects
SelectionSelection
The selection tasks involves selecting The selection tasks involves selecting somethingsomething
there are several variations, depending there are several variations, depending upon what’s being selected:upon what’s being selected: list or command selectionlist or command selection object selectionobject selection location selectionlocation selection
SelectionSelection
List selection: a pre-defined list of things to List selection: a pre-defined list of things to select fromselect from
example: the commands on a menuexample: the commands on a menu need to present the list, and the user need to present the list, and the user
selects one item from the listselects one item from the list object selection: number of objects not object selection: number of objects not
pre-defined, created by the user, changes pre-defined, created by the user, changes in size as the program runsin size as the program runs
SelectionSelection
For object selection can’t just present a list For object selection can’t just present a list of objects to be selected fromof objects to be selected from
location selection: selecting a point in location selection: selecting a point in space, may be used as location of object, space, may be used as location of object, or as part of an object’s shapeor as part of an object’s shape
can’t see a point in empty space, so this is can’t see a point in empty space, so this is harder than the previous twoharder than the previous two
ManipulationManipulation
Standard set of object manipulations, Standard set of object manipulations, change position, size and orientationchange position, size and orientation
grab the object and move itgrab the object and move it could also have deformations that change could also have deformations that change
the object’s shapethe object’s shape hard to get general techniques beyond the hard to get general techniques beyond the
few standard onesfew standard ones
CombinationCombination
Take two or more objects and put them Take two or more objects and put them together to form a new objecttogether to form a new object
need to match up the shapes exactly, so need to match up the shapes exactly, so they join in the right waythey join in the right way
difficult to do unaided, usually need some difficult to do unaided, usually need some form of constraint to simplify the processform of constraint to simplify the process
Application Dependent TasksApplication Dependent Tasks
Usually involve the application dataUsually involve the application data ways of controlling the view of the dataways of controlling the view of the data ways of manipulating the dataways of manipulating the data example: a CAD or animation program will example: a CAD or animation program will
have a different set of manipulations than have a different set of manipulations than a network visualization programa network visualization program
Interaction TechniquesInteraction Techniques
Not a well established set of techniques, Not a well established set of techniques, yetyet
depend on input devices and styledepend on input devices and style example: a fixed range device (tracker) example: a fixed range device (tracker)
sometimes works best with pointing at sometimes works best with pointing at objects, while a puck or joystick might objects, while a puck or joystick might work better with grabbingwork better with grabbing
need to try different combinationsneed to try different combinations
Interaction TechniquesInteraction Techniques
Some problems encountered:Some problems encountered: distance: objects too far away to grabdistance: objects too far away to grab feedback: how do you highlight the object that feedback: how do you highlight the object that
has been selected?has been selected? Object to be selected may be hidden by other Object to be selected may be hidden by other
objectsobjects object density may make selection and object density may make selection and
manipulation difficultmanipulation difficult
Application ProcessingApplication Processing
Not much to say hereNot much to say here some applications have a considerable some applications have a considerable
amount of processing, computation based amount of processing, computation based on user inputon user input
don’t want this to effect application latencydon’t want this to effect application latency need to control resources devoted to need to control resources devoted to
computation, use other processorscomputation, use other processors
Making it run rightMaking it run right
Now that we have an idea of what’s Now that we have an idea of what’s involved, how do we put it all togetherinvolved, how do we put it all together
want to have low system latency, get want to have low system latency, get images on the screen as fast as possibleimages on the screen as fast as possible
don’t want to wait for anythingdon’t want to wait for anything divide the application into components that divide the application into components that
execute separatelyexecute separately
Decoupled Simulation ModelDecoupled Simulation Model
Separate process for application Separate process for application computations, this is easycomputations, this is easy
separate process for expensive input separate process for expensive input devices, trackers that need lots of devices, trackers that need lots of computation or have latencycomputation or have latency
a separate process for input processing a separate process for input processing and displayand display
maybe a separate process for modelmaybe a separate process for model
Application StructureApplication Structure
Model
ApplicationProcessing
Output Devices
InputDevices
Model Traversal
InputProcessing
Decoupled Simulation ModelDecoupled Simulation Model
Each process can run at its own rateEach process can run at its own rate display process runs as fast as possible, display process runs as fast as possible,
doesn’t wait for other processesdoesn’t wait for other processes uses most recent value from input devices uses most recent value from input devices
and application computationand application computation reduces system latencyreduces system latency