Making tele-presence workMaking tele-presence workoror

Making the AG work Making the AG work betterbetter

Markus.Buchhorn@anu.edu.auMarkus.Buchhorn@anu.edu.au

Head, ANU Internet FuturesHead, ANU Internet Futures

It’s all about tele-presenceIt’s all about tele-presence The feeling of being there with somebody or The feeling of being there with somebody or

somethingsomething

Requires sufficient “quality” to make the Requires sufficient “quality” to make the technology invisibletechnology invisible

Quality:Quality:• Image resolution, colour, framerate, …Image resolution, colour, framerate, …• Audio realismAudio realism• Latency, jitter, errors, …Latency, jitter, errors, …• ““Feel” for remote site(s)Feel” for remote site(s)• Minimise cognitive distractionsMinimise cognitive distractions• Avoid loss of functionalityAvoid loss of functionality

‘‘Sorry, you can’t do that with this system’Sorry, you can’t do that with this system’

www.accessgrid.orgwww.accessgrid.org (open-source ANL project)(open-source ANL project) Multicast-based Multicast-based (mostly)(mostly) Many-2-many interaction Many-2-many interaction (in principle)(in principle) Multiple A/V streams per site Multiple A/V streams per site (usually)(usually) Some Application sharing Some Application sharing (sort of)(sort of)

• PPT, WWWPPT, WWW• Text Edit, Whiteboard, InstrumentsText Edit, Whiteboard, Instruments

No bandwidth constraints No bandwidth constraints (ideally)(ideally)• Today (h.261 x [2 - 4] + audio) ~ 2Mb/s per siteToday (h.261 x [2 - 4] + audio) ~ 2Mb/s per site

Can scale down to desktop and PDACan scale down to desktop and PDA

Access Grid usageAccess Grid usage

Formal multisite eventsFormal multisite events• Seminars, distributed classes, virtual Seminars, distributed classes, virtual

conferencesconferences• Board/committee meetingsBoard/committee meetings

Casual usageCasual usage• Walk-in discussions, ad-hoc meetingsWalk-in discussions, ad-hoc meetings• Desktop monitoringDesktop monitoring

Wrapper to larger applicationsWrapper to larger applications• Visualisation/Virtual Reality, Visualisation/Virtual Reality, • Computational SteeringComputational Steering• Data browsingData browsing

Who, Where?Who, Where? 400+ sites today 400+ sites today

150+ U.S., 40+ Europe, 30+ Asia150+ U.S., 40+ Europe, 30+ Asia 30 live in Oz, 15+ in plan/build (4+ at ANU)30 live in Oz, 15+ in plan/build (4+ at ANU) Mostly at Universities, so farMostly at Universities, so far

Developers:Developers: 20+ groups on shared applications20+ groups on shared applications 10+ groups on visualisation/VR10+ groups on visualisation/VR 10+ groups on high-quality video10+ groups on high-quality video 3+ groups on user interfaces3+ groups on user interfaces

An AG An AG (room sized)(room sized) Node Node

Screen

Proj Proj ProjCam

Cam Cam

Mike Mike

Spkr

Display PC

VideoCap PC

Audio PC

Control PC

Now: Now: Making the AG work Making the AG work betterbetter

The power of the AG is its flexibilityThe power of the AG is its flexibility

The problem with the AG is too much The problem with the AG is too much flexibilityflexibility• Lots of effort on new functionalityLots of effort on new functionality

Often in domain-specific areasOften in domain-specific areas

• Limited effort on Limited effort on Robust-ifying existing functionalityRobust-ifying existing functionality Enhancing the common functionalityEnhancing the common functionality

The problemThe problem 4+ AG (room) Nodes at ANU4+ AG (room) Nodes at ANU

• 3 almost complete 3 almost complete Computer Science, HPC, ObservatoryComputer Science, HPC, Observatory

• 1 under way 1 under way Medical SchoolMedical School

• 4 more being discussed4 more being discussed Including a 1500 seat theatre!Including a 1500 seat theatre!

• Plus various desktops and test systemsPlus various desktops and test systems

Pressure to make the AG nodes Pressure to make the AG nodes • Production quality (robust)Production quality (robust)• Cheaper to operate (less qualified operators)Cheaper to operate (less qualified operators)• Remotely operableRemotely operable• USER FRIENDLY !!!!!!USER FRIENDLY !!!!!!

Maximise the feeling of tele-presenceMaximise the feeling of tele-presence

AG problemsAG problems

1.1. Performance, benchmarkingPerformance, benchmarking2.2. MulticastMulticast3.3. User InterfaceUser Interface4.4. Capture, codecsCapture, codecs5.5. Integration with “standard” videoconf.Integration with “standard” videoconf.6.6. Recording, recoding and playbackRecording, recoding and playback7.7. Hardware add-ons, control, and Hardware add-ons, control, and

applicationsapplications

1-Performance measurement1-Performance measurement

RTPreplicateRTPreplicate Stress your display/audio machinesStress your display/audio machines One stream goes in, N come outOne stream goes in, N come out Use feedback from rtcp to benchmarkUse feedback from rtcp to benchmark

Derive a meaningful numberDerive a meaningful number

Full throughput testFull throughput test network, interfaces, CPU/RAM/graphics busnetwork, interfaces, CPU/RAM/graphics bus

2-Multicast2-Multicast

Multicast isn’t everywhereMulticast isn’t everywhere Bandwidth isn’t everywhereBandwidth isn’t everywhere Multicast/unicast bridges Multicast/unicast bridges

don’t scale well in don’t scale well in managementmanagement

Host must reconfigure all the Host must reconfigure all the timetime

RCbridgeRCbridge Web interface, per request bridge Web interface, per request bridge

set up and configured on set up and configured on demand, any number of demand, any number of clients/sessionsclients/sessions

rtp stream selection (home use, rtp stream selection (home use, wireless use, …)wireless use, …)

RCBridgeRCBridge

•http is scriptable – command line clients

•Can build a P2P network of unicast fowarders, e.g. for failover, or application-layer multicast – but prefer multicast to be FIXED!!!

3-User Interface3-User Interface Vic (video) and rat (audio)Vic (video) and rat (audio)

old, unfriendlyold, unfriendly Few new codecs (DV, MPEG-x, MJPEG, HDTV, Few new codecs (DV, MPEG-x, MJPEG, HDTV,

surround/hi-fi sound, …)surround/hi-fi sound, …) And don’t scale all that wellAnd don’t scale all that well

• No lip-synch No lip-synch • No audio-localisation, to match displayNo audio-localisation, to match display• Manageability is a problemManageability is a problem

Major effort of node operatorMajor effort of node operator

• No ability to scale hardware further, No ability to scale hardware further, across clustersacross clusters

Performance and reliability advantagesPerformance and reliability advantages

VP site-tiles and layout managersVP site-tiles and layout managers

VP – a new startVP – a new start

Site-tiles Site-tiles (grouping and prioritising)(grouping and prioritising) Display manager Display manager (layout of site tiles)(layout of site tiles) Use of OpenGL capabilitiesUse of OpenGL capabilities

• Smooth (not just quantised) scaling, done in hardwareSmooth (not just quantised) scaling, done in hardware

• Much more flexible user interfaces possibleMuch more flexible user interfaces possible

Other metadataOther metadata• Stream layout Stream layout (e.g. panorama)(e.g. panorama)

• Local time Local time (why is that person asleep??)(why is that person asleep??)

• Network issues Network issues (have we lost multicast again??)(have we lost multicast again??)

• Flags, icons Flags, icons (graphics to identify a site)(graphics to identify a site)

VP – a new startVP – a new start

Remote OperationRemote Operation• ““VP-master” controls the “VP-client”, over the networkVP-master” controls the “VP-client”, over the network

Cluster display: Cluster display: • MULTIPLE VP clientsMULTIPLE VP clients, still single VP-master, still single VP-master

• Works across different operating systems at the same time!Works across different operating systems at the same time!

Audio Audio (via existing audio tool for now)(via existing audio tool for now) • lip-synchronisation, lip-synchronisation,

• Visual highlightsVisual highlights

• Virtual audio localisation, Virtual audio localisation,

Runs on Windows, Linux; FreeBSD and MacOSXRuns on Windows, Linux; FreeBSD and MacOSX

VP – with DVVP – with DV

and MPEG4 and H263 in the next release

4 – Capture, Encoding 4 – Capture, Encoding

““VC” – new project in 2004VC” – new project in 2004 Supported by AUDF; Mac first!Supported by AUDF; Mac first!

Any number of input devicesAny number of input devices Various filtersVarious filters

WatermarksWatermarks Captions, bannersCaptions, banners ChromakeyChromakey

MetadataMetadata Layout, site information, priorities, …Layout, site information, priorities, …

Various codecsVarious codecs H.261, m(j)peg-X?, DV?, H.263, H.264H.261, m(j)peg-X?, DV?, H.263, H.264

5 - Integration with H.323, …5 - Integration with H.323, … H.323 is wide-spread, AG H.323 is wide-spread, AG

is not (yet)is not (yet) Bridging between two Bridging between two

worldsworlds• Analogue Analogue • VRVSVRVS

Silver323Silver323• Open source, software Open source, software

gatewaygateway• Some MCU-like functionalitySome MCU-like functionality

• Hardware constrained, but Hardware constrained, but cluster-ablecluster-able

• http managed, very flexible http managed, very flexible and “automate-able” and “automate-able” stream selectionstream selection

• Provides an easy path into Provides an easy path into VoIP and PSTNVoIP and PSTN

• SIP, streaming easy to addSIP, streaming easy to add

6 – Recording, recoding, playback6 – Recording, recoding, playback

Current practice: rtpdump, rtpplayCurrent practice: rtpdump, rtpplay• Poor design, does not scale, Poor design, does not scale, • no remote controlno remote control

New toolkit New toolkit • Recording, playout daemons, remote controlRecording, playout daemons, remote control• Scalable file formatScalable file format• Additional media typesAdditional media types

Whiteboard, shared-apps, annotations, …Whiteboard, shared-apps, annotations, …

• Tools to extract subset of streamsTools to extract subset of streams• Tools to convert to e.g. DVD formatsTools to convert to e.g. DVD formats

7 – Hardware, Applications7 – Hardware, Applications Integrate electronic pen-tracking whiteboardIntegrate electronic pen-tracking whiteboard

Astronomers want blackboard/chalk integration!Astronomers want blackboard/chalk integration! E-beam and Mimio are excellentE-beam and Mimio are excellent Existing whiteboard apps are mouse driven, and oldExisting whiteboard apps are mouse driven, and old

Remote control of nodesRemote control of nodes Design nodes to be remotely managed, user managed Design nodes to be remotely managed, user managed

(cameras, projectors, vcr/dvd, …)(cameras, projectors, vcr/dvd, …) Maintain rapid user support in rooms (operator-cam, with Maintain rapid user support in rooms (operator-cam, with

dedicated audio)dedicated audio)

More AG-literate applicationsMore AG-literate applications Shared Acrobat Shared Acrobat Library for generic apps (cross-platform) to code toLibrary for generic apps (cross-platform) to code to Most generic: VGA capture!Most generic: VGA capture!

Conclusion: AG developmentConclusion: AG development

1.1. Performance, benchmarkingPerformance, benchmarking2.2. MulticastMulticast3.3. User InterfaceUser Interface4.4. Capture, codecsCapture, codecs5.5. Integration with “standard” videoconferencingIntegration with “standard” videoconferencing6.6. Recording, recoding and playbackRecording, recoding and playback7.7. Hardware add-ons, control, applicationsHardware add-ons, control, applications

But we need your help, your code, But we need your help, your code, your ideas, and your advice – you, the users!your ideas, and your advice – you, the users!

http://if.anu.edu.auhttp://if.anu.edu.au