t4g the intelligent application of technology tm natural user interfaces & microsoft surface...

T4G The Intelligent Application of TechnologyTM

Natural User Interfaces &

Microsoft Surface Development


Why are we here?

• Obligatory corporate plug• Obligatory personal plug• The real content:

– What is a Natural User Interface (NUI)?– What is the Microsoft Surface?– MS Surface development environment– Demos and code:

• Hello, Surface!• Raw Images in WPF• Contact Data Visualization• Surface Controls• Tag Visualizations

– The Future: Windows Touch and Surface 2.0


Our Company, Culture and Success• Based on our philosophy of The Intelligent

Application of TechnologyTM, we design, develop, and implement technology solutions that help our customers run their businesses better.


T4G at a glance

1996Year T4G was

established

260+# of employees

in 7 locations across North

America

80% of customers

in 2010 who invited us back for

multiple projects

3# of consecutive years on the Great Place to Work® list


OUR VALUES Integrity Passion Communication Teamwork

Adaptive Innovation Entrepreneurialism Value Conscious Social Responsibility


Delivering leadership and expertise

Travel and Hospitality

Retail

Energy

Financial Services

Professional Services

Healthcare

Government

Communications

Our Vertical ExpertiseOur Supporting Services

Business Intelligence

Infrastructure & Security

Software Development & Integration

Content Delivery

Mobile Solutions

Social Media Strategy & Measurement

Strategic Planning

Hosting & Application Management

Portals & Collaboration

User Experience & Design


Who am I?

• Principal Consultant at T4G• Focused on Portal Technologies (SharePoint)• Also work on NUIs and Multi-touch technologies, among

other things• 25+ years on software and various technology things


What is a Natural User Interface?

A natural user interface is a user interface designed to use natural human behaviours for interacting

directly with content.


NUI is much more than Touch

A New Way of Interacting

Augmented Reality

Speech

Object Recognition

Gestures

Eye Tracking

User eXperie

nce

Natural Learning

Social Interaction


A New Way of Interacting

SocialImmersive

“The content is the application”


Implications

Pervasive UIsDon’t replace human interaction, enhance it

Extend the real world, blur the lines between UI and real life


What is the Surface?

“Microsoft Surface is a revolutionary multi-touch computer that responds to natural hand

gestures and real-world objects, helping people interact with digital content in a simple and

intuitive way.”


Surface is not just multi-touch

• Windows 7 & WPF 4 are a great baseline for touch

• Surface creates unique opportunities with specialized HW– Robust top– Horizontal form factor– Infrared camera array


Surface: Beyond Multi-Touch

• Massive multi-touch

• Multi-user experiences

• Object recognition

• Optical engineering


Surface: Multi-User

• Trend: Software continues to reduce the need for face-to-face human interactions

• Problem: Happiness and creativity depend on social interactions

• Surface brings people together– Industrial design encourages users to gather– Input capabilities allow simultaneous use– Apps facilitate collaboration / competition


Surface: Massive Multi-Touch

• Surface can efficiently track 50+ simultaneous fingers and objects


Surface: Objects

• Uniquely identify & react to billions of tagged objects

Byte Tags256 unique values

Identity Tags>340,282,366,920,938,000,000,000,000,000,000,000,000

unique values

• Tags include orientation indictors dots


Surface: Optics

• Determine the size and shape of every input• See into and project onto physical objects


What makes a GOOD Surface Application?

• The app must be right for the Surface paradigm• It should be social• Should not involve large amounts of “traditional” content• Needs to be implemented thinking about touch


Who is using it?

• Retailers (Microsoft, Wind Mobile, etc.)• Hospitality (Hard Rock Café, Rio iBar, Sheraton, etc.)• Museums (Smithsonian)• Media (MSNBC)• Healthcare, Education, Others


Development Environment

Architecture of the Surface Development Platform



• Windows Vista 32-bit• .NET 3.5• WPF or XNA

All stuff you know…



On a unit, you can use the Microsoft Surface SDK to develop and test Microsoft Surface applications directly. You can run the unit in two modes. In administrator mode, you can access all of the Windows Vista functionality. User mode demonstrates how the unit appears in a commercial venue. That is, the unit suppresses the Windows Vista user interface, automatically starts required applications, and makes sure that those applications are restarted if they need to. Typically, you develop applications in administrator mode.

Get your own Surface…



On a separate workstation, you can install the Microsoft Surface SDK and use the Surface Simulator tool to simulate a Microsoft Surface test environment.

– Runs the same Surface Shell as the Surface does– Use 1 or more mice to simulate multi-touch– Tools for simulating Tags or Blobs– Record and Replay interactions to support testing and debugging

Limitations:– No cameras (obviously) so if you use raw image data, cannot test it on simulator– Should have a pretty good sized screen– The SDK is only supported on Windows Vista Business, 32-bit! But, there are

ways around that! (http://www.brianpeek.com/blog/archive/2009/03/10/install-the-surface-sdk-on-windows-7-and-or-x64.aspx)

If you do not have a Surface or you share one)…

http://www.brianpeek.com/blog/archive/2009/03/10/install-the-surface-sdk-on-windows-7-and-or-x64.aspx




Surface SDK Features for Multi-Touch

• Common controls optimized for Multi-Touch

• Controls design primarily for Multi-Touch

• Essential Multi-Touch UX Functionality

SurfaceWindow SurfaceButton SurfaceSlider SurfaceScrollViewer SurfaceInkCanvas

SurfaceCheckBox SurfaceRadioButton SurfaceListBox SurfaceListBoxItem

SurfaceThumb SurfaceScrollBar SurfaceRepeatButton SurfaceToggleButton

ScatterView ScatterViewItem

LibraryContainer LibraryBar LibraryStack

LibraryBarItemLibraryStackItem

Rich async drag & drop Input visualizations


Lessons Learned

• There is a lot to learn about NUIs• There is just as much to unlearn• Forget most of what you know about what makes a

“good” UI• Visuals/graphics/interaction design more important

than code• Less is more• Development is easier on a workstation


Hello, World!

Plain Old WPF XAML

<Window x:Class="WpfApplication1.MainWindow“xmlns=http://schemas.microsoft.com/winfx/2006/xaml/presentationxmlns:x=http://schemas.microsoft.com/winfx/2006/xamlTitle="MainWindow" Height="350" Width="525">

<Grid>

</Grid></Window>

http://schemas.microsoft.com/winfx/2006/xaml/presentation



http://schemas.microsoft.com/winfx/2006/xaml




Hello, World!

Surface XAML

<s:SurfaceWindow x:Class="HelloSurface.SurfaceWindow1“xmlns=http://schemas.microsoft.com/winfx/2006/xaml/presentationxmlns:x=http://schemas.microsoft.com/winfx/2006/xamlxmlns:s=http://schemas.microsoft.com/surface/2008Title="Hello Surface">

<Grid>

</Grid></s:SurfaceWindow>







http://schemas.microsoft.com/surface/2008




Code Sample 1

• Basic Surface Application• Responding to Touch Events• Responding to multiple Touch Events• Handling orientation information

Hello, Surface!


Code Sample 2

• Types of Contact events Surface sees (fingers, tags, and blobs)• Information about contacts

Contact Data Visualization


Code Sample 3

Surface Controls• Common controls optimized for Multi-Touch









Code Sample 3Surface Controls

• Common controls optimized for Multi-Touch









Code Sample 3

• ScatterView Control• Adding ScatterViewItems in XAML• Adding ScatterViewItems in code (in response to Contact

Events)• ScatterViewItems through Binding (with simple item

template)• Handling contact events when ScatterViewItem content is

“active”

Surface Controls


Surface: Objects

• Uniquely identify & react to billions of tagged objects

Byte Tags256 unique values

Identity Tags>340,282,366,920,938,000,000,000,000,000,000,000,000

unique values

• Tags include orientation indictors dots


Code Sample 4

• Adding a Tag Visualizer to recognize a tag• Extracting the information in the tag• Displaying a Tag Visualization

Tag Visualizations


Surface Hardware

Windows VistaWindows 7

NativeWin32

Application

Managed Wrapper and Interop

Touch Development Roadmap

WinForms Application

Windows 7 (2009)

Multi-Touch Controls

Multi-Touch API

WPF 3.5 SP1

WPF Application

Surface SDK1.0

Multi-Touch Controls & API

Surface Application


Surface Hardware

Windows VistaWindows 7

NativeWin32

Application


WPF 3.5 SP1

Surface SDK1.0



.NET 4.0 & Surface Toolkit (Q1 2010)

WPF 4.0 w/ Multi Touch APIs

Multi-Touch ControlsSurface

Multi-Touch Controls &API

Multi-Touch API

Surface Toolkit for Windows Touch

Multi-Touch Controls & API

WPF Application

Surface Application


Surface Hardware

Windows 7Windows 7

NativeWin32

Application



WPF 4.0 w/ Multi Touch APIs

Surface Toolkit for Windows Touch

Surface SDKvNext


Surface 2.0

Multi-Touch ControlsSurface

Multi-Touch Controls &API

Multi-Touch API

Surface-specificControls & API

WPF Application

Surface Application


Surface 2.0 Integration & Extension of WPF 4

• All input is routed through the WPF InputManager

• Surface input provider can send to multiple windows

• Surface extension methods provide access to additional input data (tag values, orientation, shape, etc)

WPF InputManager

Surface input provider

Windows Touch input

providerApps

& Controls

Manipulation & Touch Events

Surface extension methods


Surface 2.0

• Announced at CES 2011• Available later this year• SDK Available April 2011 (?)• Embedded Windows 7 Professional 64-bit • .NET Framework 4.0 • Windows Presentation Foundation (WPF) 4.0 • Microsoft XNA® Framework 4.0 • Windows PowerShell and DMTF DASH support, and enhanced

administrator tools


Resources

On the web:Surface Web Site (http://www.surface.com)Deconstructing the NUI (Joshua Blake, Surface MVP)My Blog (http://fyeomans.com)

On Twitter:@Surface – the official MS Surface twitter identity#Surface@fyeomans

http://www.surface.com/



http://nui.joshland.org/

http://fyeomans.com/


Questions?

t4g the intelligent application of technology tm natural user interfaces & microsoft surface...

Documents

application slide

technology solutions

year t4g

list slide

various technology things

real life slide

touch surface creat

natural user interface