Multi-Touch on Microsoft Surface and

Windows 7 for .NET Developers

CL27

Robert LevyProgram ManagerSurface

Anson TsaoLead Program ManagerWPF

Win one of thirty $50 gift card!!!

Complete a quick survey atwww.WontTakeLong.com

Agenda> Introduction> Multi-touch in WPF 4 & SL 3

> Raw Input APIs> Manipulation/Inertia APIs> Controls

> Microsoft Surface: Beyond Touch> Controls> Unique capabilities

> Roadmap for platform alignment> Obligatory slides> Q&A

Multi-Touch…> …creates new opportunities.

> Software that is intuitive for users to walk up & master> Software that is collaborative with multiple users

> …is not new.> Researchers have been exploring this stuff for decades

> …is now going mainstream.1. Hardware: Robust touch-capable hardware from Microsoft

and OEMs 1. Dozens of Win7 touch devices now available

2. OS: Windows 7 has great touch support baked in3. SDKs: Microsoft is making touch easy to leverage4. You: Developers & designers creating innovative apps that

take advantage of the technology

Microsoft Surface & WPF> With Surface, Microsoft became a

thought leader in the multi-touch arena

> Surface has been shipping a multi-touch dev platform for several years> Built as a set of extensions to WPF 3.5

> Touch APIs from Surface are being baked into WPF 4 for cross-platform use

WPF 4 Touch Capabilities

Anson Tsao

WPF 4 Multi-Touch APIs Overview

WPF Controls Styles with Panning Enabled

ScrollViewer Panning Support

Manipulation Events

Touch Events

Extensible Touch Device

Win7 Touch Device

Surface Touch Device

Beta 1

Beta 2

Release Candidat

e

Surface V2

Raw Multi-Touch Events

demo

Multi-Touch on the Web with Silverlight

> Subset of the WPF4 touch input APIs> Reports groups of touch events in

“frames”> Apps are responsible for hit testing and

event routing

> Manipulation & Inertia Processor API will be available on surface.com> Same algorithms used by Surface, WPF, &

Win7

Silverlight 3 Touch APIs - Listeningpublic partial class MainPage : UserControl { public MainPage() { ... // listen to touch events from the system Touch.FrameReported += new TouchFrameEventHandler(OnFrame); }

void OnFrame(object sender, TouchFrameEventArgs e) { // enumerate and respond to touch events }}

Silverlight 3 Touch APIs - Processingvoid Touch_FrameReported(object sender, TouchFrameEventArgs e) { TouchPointCollection touchPoints = e.GetTouchPoints(LayoutRoot);

foreach (TouchPoint tp in touchPoints) { if (tp.Action == TouchAction.Down) { // a new touch has come down } if (tp.Action == TouchAction.Move) { // a previously down touch has moved } if (tp.Action == TouchAction.Up) { // a touch has been removed } }}

Multi-Touch Manipulations

demo

Manipulation ContainerCanvas

Manipulation Container

IsManipulationEnabled=true

IsManipulationEnabled=true

Update RenderTranform

Handle ManipulationDelta

Event• Coordinates relative

to Manipulation Container

Handle ManipulationStarting

Event• Set Manipulation

Container

Manipulation Events

Starting Started Delta Inertia Starting Delta Complet

ed

StartInertia() StartInertia()

Touch Down

(Initial)Touch Move

Touch Up (All)

Initialize:• Mode• Container• Pivot

Completed()

Manipulation Inertia

demo

Inertia BehaviorsManipulationInertiaStartingEventArgs

Properties Units

TranslationBehavior InitialVelocity 1/96th DIP per millisecDesiredDeceleration 1/96th DIP per millisec^2DesiredDisplacement 1/96th DIP

RotationBehavior InitialVelocity Degrees per millisecDesiredDeceleration Decrees per millisec^2DesiredRotation Degrees

ExpansionBehavior InitialVelocity 1/96th DIP per millisecInitialRadius 1/96th DIPDesiredDeceleration 1/96th DIP per millisec^2

DesiredExpansion 1/96th DIPS

• Deceleration and Displacement/Rotation/Expansion mutually exclusive• Deceleration – useful to simulate friction• Displacement – useful on page flips• Default – no inertial movements

Touch Panning Demo

demo

>>FUTURETouch Panning Support

> In WPF4 Release Candidate Builds (not Beta2)

> Cancel() method on manipulation events> Cancelled manipulation promoted to mouse

> ScrollViewer handles manipulations> PanningMode

> None, Both, Horizontal/VerticalOnly, Horizontal/VerticalFirst> PanningDeceleration> PanningRatio

> Styling changes to intrinsic controls

Microsoft Surface’s Extensions of WPF

Robert Levy

Surface WPF controls make touch UI easy

> Designed for multi-touch

> Designed for simultaneous use

> Designed for manipulations & inertia

Drawing with Surface Controls

<s:SurfaceInkCanvas/>

Manipulating Photos with Surface Controls<s:ScatterView Name=“scatter"> <s:ScatterView.ItemTemplate> <DataTemplate> <Image Source="{Binding}"/> </DataTemplate> </s:ScatterView.ItemTemplate></s:ScatterView>

scatter.ItemsSource = Directory.GetFiles(…);

Inking on Photos with Surface Controls

<s:SurfaceScrollViewer> <s:ScatterView Name=“scatter"> <s:ScatterView.ItemTemplate> <DataTemplate> <Grid> <Image Source="{Binding}"/> <Viewbox Margin="50"> <s:SurfaceInkCanvas/> </Viewbox> </Grid> </DataTemplate> </s:ScatterView.ItemTemplate> </s:ScatterView></s:SurfaceScrollViewer>

scatter.ItemsSource = Directory.GetFiles(…);

Inkable Photos on Surface

demo

Surface Toolkit for Windows Touch

Controls, samples, templates, and docs

Coming shortly after the WPF4 launch

announcing

Surface SDK Features for Multi-Touch> Common controls optimized for Multi-Touch

> Controls design primarily for Multi-Touch

> Essential Multi-Touch UX Functionality

SurfaceWindow SurfaceButton SurfaceSlider SurfaceScrollViewer SurfaceInkCanvas

SurfaceCheckBox SurfaceRadioButton SurfaceListBox SurfaceListBoxItem

SurfaceThumb SurfaceScrollBar SurfaceRepeatButton SurfaceToggleButton

ScatterView ScatterViewItem

LibraryContainer LibraryBar LibraryStack

LibraryBarItemLibraryStackItem

Rich async drag & drop Input visualizations

Inkable Photos on Windows 7

demo

Surface is not just multi-touch> Windows 7 & WPF 4 are a great baseline for

touch

> Surface creates unique opportunities with specialized HW> Robust top> Horizontal form factor> Infrared camera array

Surface: Beyond Multi-Touch> Massive multi-touch

> Multi-user experiences

> Object recognition

> Optical engineering

Surface: Multi-User> Trend: Software continues to reduce the

need for face-to-face human interactions

> Problem: Happiness and creativity depend on social interactions

> Surface brings people together> Industrial design encourages users to gather> Input capabilities allow simultaneous use> Apps facilitate collaboration / competition

Surface: Massive Multi-Touch

> Surface can efficiently track 50+ simultaneous fingers and objects

Surface: Objects> Uniquely identify & react to billions of tagged

objects

Byte Tags256 unique values

Identity Tags>340,282,366,920,938,000,000,000,000,000,000,000,000

unique values

> Tags include orientation indictors dots

Surface: Optics> Determine the size and shape of every

input> See into and project onto physical

objects

Optical Magic with Surface

demosurface.com/monster

Optical Magic with Surface

demoBehind the scenes video at: surface.com/monster

Surface Application Showcase

video

Surface Application Showcase

video

>>FUTURESurface vNext Integration & Extension of

WPF 4> All input is routed through the WPF InputManager

> Surface input provider can send to multiple windows

> Surface extension methods provide access to additional input data (tag values, orientation, shape, etc)

WPF InputManagerSurface

input provider

Windows Touch input

provider Apps&

Controls

Manipulation & Touch Events

Surface extension methods

Getting Surface Data from WPF Extensions

void OnTouchDown( object sender, TouchEventArgs e ){ // built in to WPF point p = e.Touch.GetPosition(this);

// calling Surface extension directly TagData t1 = Microsoft.Surface.TouchExtensions.GetTagData(e.Touch);

// calling Surface extension w/ C# magic TagData t2 = e.Touch.GetTagData();

// some other Surface extensions double o = e.Touch.GetOrientation(this); Ellipse e = e.Touch.GetEllipse(this);}

Surface HardwareWindows

VistaWindows 7

NativeWin32

Application

Managed Wrapper and Interop

Touch Development Roadmap

WinForms Application

Windows 7 (2009)

Multi-Touch Controls

Multi-Touch API

WPF 3.5 SP1

WPF Application

Surface SDK1.0

Multi-Touch Controls &

API

Surface Application

>>FUTURE

Surface HardwareWindows

VistaWindows 7

NativeWin32

Application

Touch Development Roadmap

WPF 3.5 SP1

Surface SDK1.0

Managed Wrapper and

Interop

WinForms Application

.NET 4.0 & Surface Toolkit (Q1 2010)

WPF 4.0 w/ Multi Touch APIs

Multi-Touch ControlsSurface

Multi-Touch Controls

&API

Multi-Touch API

Surface Toolkit for Windows

Touch

Multi-Touch Controls &

API

WPF Application

Surface Application

>>FUTURE

Surface HardwareWindows 7

Windows 7

NativeWin32

Application

Touch Development Roadmap

Managed Wrapper and

Interop WPF 4.0 w/ Multi Touch APIs

Surface Toolkit for Windows Touch

Surface SDKvNext

WinForms Application

Surface vNext (Future)

Multi-Touch ControlsSurface

Multi-Touch Controls

&API

Multi-Touch API

Surface-specific

Controls & API

WPF Application

Surface Application

>>FUTUREWhat’s next for Surface hardware?

>Things we’re exploring:>Cheaper >Thinner>Vertical

>Stay tuned…

Call to Action> Windows 7

> Download Silverlight 3 RTM or Silverlight 4 Beta 1> Download Visual Studio 2010 Beta 2 w/ WPF 4.0

> Surface

> By Invitation Only: http://community.surface.com

1. Get the Surface SDK & Simulator2. Check out the samples, docs, and templates3. Browse and participate in the online forums

Purchase Surface hardware from Surface.com (starting at $12,500)

SDK & resources now available free to everyone! Integrated with TechNet & MSDN for downloads, online docs, and forums.

Start at our portal: http://surface.com/developerNEW

!

Related Resources> Windows Touch Deep Dive (Session CL17)

> WPF 4 Resources: http://connect.microsoft.com/wpf

> Surface Resources: http://surface.com

> Infragistics.com – free controls for Surface v1!> Map, Chart, and Carousel > CTP now available

> NSquaredSolutions.com/SurfaceBook> Coming soon!

Win one of thirty $50 gift card!!!

www.WontTakeLong.com

YOUR FEEDBACK IS IMPORTANT TO US! Please fill out session evaluation

forms online atMicrosoftPDC.com

Learn More On Channel 9> Expand your PDC experience through

Channel 9

> Explore videos, hands-on labs, sample code and demos through the new Channel 9 training courses

channel9.msdn.com/learnBuilt by Developers for Developers….

© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.