1. Jeffrey Funk National University of Singapore Karthik Nandakumar AStar The Future of Human-Computer Interfaces For information on other technologies, see http://www.slideshare.net/Funk98/presentations

2. What do the Following Trends Mean for Human-Computer Interfaces? Moores Law Better cameras Higher resolution and sensitivity Better displays Cheaper, flexible and more durable More sensitive touch New forms of touch Better combinations of these interfaces Better neural interfaces 3. A New Generation of Input Interfaces Neural Interfaces Speech Touch Gesture Augmented Reality 4. That Can Change the Way We Live and Work Not just consumer applications Assemblers can see drawings Construction workers can see through walls, wires, and pipes Prospectors can see through the ground Architects can see entire 3D image of building Students can see 3D representation of anatomy, materials, universe Similar systems could be useful for tourists, shopping, soldiers, and artists 5. HCI Human-Computer Interface (HCI) HCI is the technology that connects man and machine Robust HCIs are needed to enable ubiquitous computing We primarily focus on input interfaces Human Computer Thoughts Action Input Interface Action Recognition Task Execution Understanding Output Interface Sensory Perception Rendering 6. Traditional Input Interfaces are Disappearing.. Command Line Interfaces (CLI), i.e., keyboard Batch Interfaces Graphical User Interfaces (GUI) 7. Even God is Interested! 8. Session Technology 1 Objectives and overview of course 2 When do new technologies become economically feasible? 3 Two types of improvements: 1) Creating materials that better exploit physical phenomena; 2) Geometrical scaling 4 Semiconductors, ICs, electronic systems, big data analytics 5 MEMS and Bio-electronics 6 Lighting, Lasers, and Displays 7 Information Technology and Land Transportation 8 Human-Computer Interfaces, Biometrics 9 Superconductivity and Solar Cells 10 Nanotechnology and DNA sequencing This is the Eighth Session of MT5009 9. Outline Overview Speech Interfaces Touch Interfaces Gesture Interfaces Augmented Reality (various combinations) Neural Interfaces 10. Performance of Input Interfaces Accuracy: Precision in recognizing the action Throughput: Information that can be processed per unit time Affordability: Inversely proportional to the cost Ease of use: Ease in learning to use the interface Sociability: Multi-person interactivity Mobility: Size and mass of device, power consumption User experience: Subjective perceptions of utility & efficiency 11. Comparison of Input Interfaces 12. Key Components of Input Interfaces Human-Computer Input Interfaces Neural Interfaces Natural UI Speech Micro phone Neural electrodes/ sensors GestureTouch 3D Camera Touch sensor Tracking & Recognition Software Materials/ Nanotechnology Signal Processing Hardware (Semicon- ductors) 13. Outline Overview Speech Interfaces Touch Interfaces Gesture Interfaces Augmented Reality (various combinations) Neural Interfaces 14. Speech Interfaces Key Components Microphone Automated Speech Recognition (ASR) and Natural Language Understanding (NLU) Software Possible methods of improvement are Increase Signal to Noise Ratio (SNR) from microphone Achieve human-level performance on ASR/NLU tasks Key dimension that needs improvement is Accuracy Video on Siri (iPhone 4S) http://www.youtube.com/ watch?gl=SG&hl=en- GB&v=L4D4kRbEdJw 15. Evolution of Microphone Technology Electret Condenser Microphone (ECM) Signal to Noise Ratio (SNR): 55-58 dB MEMS Digital Microphone SNR: 61 dB Flatter frequency response Smaller size (CMOS fabrication) Can be reflow soldered Analog Devices, MEMS Microphone Technology, October 2010 16. MEMS Microphone Technology Signal to Noise Ratio (SNR) is directly proportional to Area of back-plate Air gap spacing Applied bias voltage Mechanical compliance factor (inversely proportional to stiffness of the diaphragm) Air gap spacing Sound wave Elko et al., Capacitive MEMS Microphones, Bell Labs Technical Journal, 2005 17. Can MEMS Microphones be Improved? Size of microphone depends on area of backplate & gap spacing Reducing size will also reduce the quality (SNR), unless it is compensated by other factors Wavelength of audible sound waves is > 17 mm, while size of a MEMS microphone is only a few mm; noise level in a microphone is already close to the thermal noise limit Dramatic reductions in size or increase in SNR of single MEMS microphones does not appear to be possible S. Beus, MEMS mic enables thinner phone designs, 2005 18. Microphone Array Microphone array can mitigate background noise & interference Courtesy: Audience Inc. 19. Accuracy based on Microphone Array 1. LOUD project from MIT Computer Science and Artificial Intelligence Laboratory, 2005 2. Microphone Array project in MSR: Approach and Results, Microsoft Research, June 2004 Noise suppression algorithms can increase SNR by 18dB with just 4 microphones in an array1,2 Improvement in SNR Corresponding decrease in Word Error Rate 20. Automated Speech Recognition (ASR) Increase in vocabulary sizes needs exponential increase in computing power due to potential combinatorial explosions L. Rabiner, Challenges in Speech Recognition, NSF Symp. on Next Gen. ASR, 2003 21. Automated Speech Recognition (ASR) Increase in vocabulary sizes needs exponential increase in computing power due to potential combinatorial explosions L. Rabiner, Challenges in Speech Recognition, NSF Symp. on Next Gen. ASR, 2003 22. Only Acceptable in Some Niches Worderrorrate 23. ASR Accuracy in Text Dictation Stand-alone speech interfaces may be useful for tasks like dictation Speech as an important modality in multimodal user interfaces (e.g., Microsoft Kinect) may be the future * http://blogs.msdn.com/b/sprague/archive/2004/10/22/246506.aspx * ? 24. For Online Translations Googles service translates documents with inference models that were developed from analysis of a trillion words or 95 billion English sentences. By 2010, its dataset covered more than 60 languages and could accept voice input in 14 languages and English is sometimes used as a bridge between two different languages when direct translations dont exist 25. Outline Overview Speech Interfaces Touch Interfaces Gesture Interfaces Augmented Reality (various combinations) Neural Interfaces 26. Touch Screens Many kinds But most are variations of either Resistive Capacitive (iPhone) Depend on new materials that are deposited on top of an LCD display Processors interpret the datahttp://www.youtube.com/watch?v=FyCE2h_yjxI&src_vid=5fOI -EQCOOQ&feature=iv&annotation_id=annotation_558874 27. What are the limitations with these touch screens? Which technologies might contribute towards overcoming these limitations? 28. Source for next 15 slides: Group presentation in Fall 2014 29. Challenges Sensitivity y = 0.1383x-1.082 0.00% 20.00% 40.00% 60.00% 80.00% 100.00% 120.00% 140.00% 160.00% 180.00% 200.00% 0 1 2 3 4 5 6 Percentage Change in Raw Count Overlay Thickness (mm) Sensitivity Level vs Overlay Thickness Ratio trendline *Note: Microprocessor doesnt recognize capacitance domain but rather it register the change in raw count Source: FYP Industrial Collaboration (Fischer-Tech and NUS) Project Report 2013 An Empirical Approach towards Capacitive Touch-Sensing in Functional Plastics APPLE IPHONE Model Type Overlay Thickness 3GS, 4, 4S Gorilla Glass 1 1.0mm 5, 5S Gorilla Glass 2 0.8mm SAMSUNG GALAXY Model Type Overlay Thickness S1, S2 Gorilla Glass 1 1.0mm S3 Gorilla Glass 2 0.8mm S4, S5, Note 3 Gorilla Glass 3 0.4mm 1.0mm 0.8mm 0.4mm 30. Thinner Glass Increases Sensitivity and Use of Smart Phones in Cold Countries Source: [Spec sheet download for glass 1, 2 and 3] http://www.corninggorillaglass.com and http://www.corning.com/WorkArea/showcontent.aspx?id=63819 31. Low Damage Resistance Low Bending Strength Low Critical Load Bearing 32. New Generations of Gorilla Glass: Higher Loads Through IOX: ion-exchanged glass and new process https://www.youtube.com/watch?v=q4ZU7zUxdM8 Source: [Spec sheet download for glass 1, 2 and 3] http://www.corninggorillaglass.com *Note: Critical Load is the min. amount of load before radial cracks start to form and propagate 4500 15000 7500 33. New Generations of Gorilla Glass Have Tighter Strength Distributions (low probabilities have been eliminated) Source: [Spec sheet download for glass 1, 2 and 3] http://www.corninggorillaglass.com Gorilla Glass 1 Gorilla Glass 2 Gorilla Glass 3 4 MPa 6.25 MPa 6.75 MPa 34. Greater Strength at all Thicknesses Source: [Spec sheet download for glass 1, 2 and 3] http://www.corninggorillaglass.com 65 75 85 Gorilla Glass 1 Gorilla Glass 2 Gorilla Glass 3 35. Achieving Flexibility and Conformity OLEDs are more flexible than are LCDs But other parts of the display are not flexible Transparent conductor Existing Glass Indium-tin oxide is inflexible and must be replaced with new material Silver Nano wires? Carbon nanotubes, graphene? Glass need thinner glass, which is becoming feasible 36. Carbon Nano Tubes are More Flexible than Indium-Tin Oxide Source: http://iopscience.iop.org/1347-4065/53/5S1/05FD04/article 37. Thinner Glass Leads to Greater Flexibility Source: www.corning.com/WorkArea/downloadasset.aspx?id=48957 Willow Glass 38. Thinner Glass Leads to Lower Bend Stress and Failure Probability Source: www.corning.com/WorkArea/downloadasset.aspx?id=48957 and http://www.corning.com/WorkArea/showcontent.aspx?id=63819 39. Read and Input on Both Sides Games Surgical Discussions through Glass 40. LG announced a transparent television (30%) Do we need one? Or is that an oxymoron? http://www.extremetech.com/computing/18 6241-lgs-flexible-and-transparent-oled- displays-are-the-beginning-of-the-e-paper- revolution 41. Need New Forms of Transparent Conductors Source: http://iopscience.iop.org/1347-4065/53/5S1/05FD04/article , http://www.beilstein-journals.org/bjnano/single/articleFullText.htm?publicId=2190-4286-4-12 ITO is expensive and inflexible Can we use: Silver Nanowires? CNTs? Graphene? 42. Are Silver Nano-Wires the Best? At 100 Ohm per Area, Light Transmission of: ITO 91% Cambrios ClearOhm (Ag Nanowire) 98% 43. OTHER CHALLENGES Existing touch screens require one to look carefully at screen while touching a specific place Fingers can easily touch wrong places Tactus offers an overlay to existing touch screens that facilitates proper location of finger Bubbles rise out of the display when fingers touch the display thus helping fingers find the right spot These bubbles are formed using MEMS (micro- electronic mechanical systems) Studies have found that faster and more accurate typing are achieved with the Tactus overlay 44. How the Tactus System Works Micro-channels are filled with fluid whose refractive index matches that of top polymer layer. Thus, transparency is even across surface. http://www.youtube.com/watch?v=wrSKbTzc4BI 0:40-1:30 45. Texture Touch Displays Sensation of texture can provide more information for users This can be done using changes in vibration with small motors or transparent electrodes (Senseg) that provide information about texture, etc. www.youtube.com/watch?v=FiCqlYKRlAA (from 0:30-2:00 minute mark) Another one from Disney: www.washingtonpost.com/ blogs/the-switch/wp/2013/10/08/disney-invents- touchscreen-that-lets-you-feel-textures/ Early applications: 3D modeling or remote 46. Applications for Touch Screens Smart phones for purchasing? http://www.youtube.com/watch?v=Gg3tmZrwbDs Smart displays in laboratories Also Advertising displays at bus stops or MRT stations Mall information displays Self checkout in stores Information counter in stores For example, Sonys AtracTable is being developed for these applications 47. Outline Overview Speech Interfaces Touch Interfaces Gesture Interfaces Augmented Reality (various combinations) Neural Interfaces 48. Components of Gesture Interfaces Key Components 2D/3D Camera (image sensor) Tracking, Recognition & Gesture Understanding Software Key dimensions that need improvement are Accuracy, Throughput and Affordability 49. Working of a 2D Image Sensor http://www.cameratechnica.com/2011/11/30/five-reasons-you-may-soon-be-shooting-at-iso-50000/ 50. Image Sensor Characteristics Spatial resolution: Number of pixels Temporal resolution: Frames per second Image sensor area: Size of the image sensor area is proportional to no. of pixels and pixel size Photometric exposure: light gathering ability of the sensor depends on the properties of the lens Light available per pixel: No. of photons incident on a pixel proportional to photometric exposure and pixel size Pixel sensitivity: is proportional to light available per pixel, quantum efficiency of photodiodes, and optical efficiency Dark Limit & Dynamic range: Ability to detect dim details & bright details in one image depends on pixel sensitivity and capacity 51. Improvements in Image Sensors Accuracy Higher spatial resolution (no. of pixels) Robustness to lighting changes (high pixel sensitivity, low dark limit, and high dynamic range) More accurate depth sensing (lower depth error) Throughput Higher frame rate Affordability Smaller pixel size reduces price per pixel 52. Improvements in Spatial Resolution T. Suzuki, Challenges of Image-Sensor Development, ISSCC, 2010 Number of pixels (resolution) has increased, but image sensor size has not increased because of reduction in pixel size Year 53. Pixel Size vs. Sensitivity Tradeoff CMOS-based image sensors are also expected to follow Moores Law in size and cost scaling T. Suzuki, Challenges of Image-Sensor Development, ISSCC, 2010 As pixel size decreases, light available per pixel will become less, so sensitivity decreases Back illuminated CMOS technology provides better trade-off between pixel size and sensitivity than traditional charge coupled device (CCD)- based image sensors Lightavailableperpixel(arb.unit) 54. Camera Technology Improvements T. Suzuki, Challenges of Image-Sensor Development, ISSCC, 2010 Reducing pixel-size (green square) and improving sensitivity (Yellow circle ) miniaturized cameras without reducing quality 55. Image Sensors vs. Human Eye Number of frames per second Spatialresolution(cyclesperdegree) Human Eye Better than Human Eye Modern cameras are close to human eye in terms of resolution Skorka & Joseph, Toward a digital camera to rival the human eye, J of Electronic Imaging, 2011 56. Image Sensors vs. Human Eye Dynamic Range (dB) DarkLimit(cd/sq.m) Human Eye Better than Human Eye But improvement can be achieved in terms of sensitivity Pixel sensitivity determines the dark limit and dynamic range of an image sensor 57. Further Improvements in Sensitivity Increase quantum efficiency through the design of better photosensitive materials using nanotechnology (e.g., Single Carrier Modulation Photo Detector) Increase optical efficiency through a vertically integrated layered arrangement as in the human retina 58. http://www.future-fab.com/documents.asp?d_ID=4926 Cost per pixel of Camera Chips has fallen dramatically 59. 3D Depth Sensing Technologies * R. Lange, 3D Time-of-flight distance measurement with custom solid-state image sensors in CMOS/CCD-technology, PhD Thesis, 2000 60. Comparison of 3D Sensing Technologies Application Range (m) DepthResolution(m) Usable Range for Gesture Interfaces Microsoft Kinect Cost-effective 3D image sensors are now becoming available (e.g., Microsoft Kinect ~ 150 USD) Such cameras will further improve the accuracy of gesture UIs 61. 3D Depth Sensing: Interferometry Most accurate depth sensing technology (accuracy depends only the wavelength of light) Low miniaturization potential and very limited range 62. 3D Depth Sensing: Time of Flight Time of flight (ToF) cameras requires processors with high clock speed (3 GHz speed can provide only 4.5 cm depth resolution) High miniaturization potential and large range Improvements in CMOS technology are likely to very beneficial 63. 3D Depth Sensing: Triangulation Passive Triangulation Active Triangulation Limited range Low miniaturization potential Depth accuracy decreases with square of the distance 64. Leap has Generated Excitement http://www.youtube.com/watch?v=_d6KuiuteIA Leap uses multiple camera sensors to recognize gestures Workspace is about 3 cubic meters. Better sensors will enable larger work spaces. $70 control system that plugs into any computer. MITs Technology Review calls Leap, The most important new technology since the smart phone How about Microsofts Kinect? http://www.youtube.com/watch?v=o4U1pzVf9hY Or wearable ring (each position represents different number)? http://www.youtube.com/watch?v=Gx3zWHS8amA 65. Another Approach Use cameras to track eye movements Monitor drivers or other operators of machines Help paralyzed people use computers As cost of cameras fall Eye tracking might become user interface for non-paralyzed Eye tracking can also be used with Google Glasses (see below)Source: http://www.economist.com/news/technology-quarterly/21567195-computer-interfaces-ability-determine-location-persons-gaze 66. Replace Cameras with MEMS-based wrist band from Thalmic Labs, called MYO Gestures are recognized before movement Muscle activity is monitored with 9-axis inertial measurement unit (MEMS) 67. Outline Overview Speech Interfaces Touch Interfaces Gesture Interfaces Augmented Reality (various combinations) Neural Interfaces 68. Types of Augmented Reality Glasses Phones 69. How Different from Virtual Reality? Augmented Reality Supports our understanding of the real world while virtual reality immerses us in a new type of world Recently Oculus VR was acquired by Facebook Improvements in 3D displays, accelerometers, gyroscopes, compasses (MEMS) and graphic processors are enabling rapid improvements in VR How might Oculus VR help Facebook? 70. Returning to Augmented Reality 71. What do you see through the glasses or Lens? 72. Handheld devices may be sufficient, particularly if the images are easily integrated with your surroundings 73. What about a or a virtual one Supermarket? at a subway station? http://www.youtube.com/watch?v=yKNSOwLcrkE 74. What about superimposing the images on a cars windshield? 75. Videos of augmented reality http://www.youtube.com/watch?v=CqxF2Vdkyxw (0-60 seconds) http://www.youtube.com/watch?v=t-m6YL64lkU 76. Googles Project Glass Image and information are displayed on the glasses Users choose which information to display on the glasses Choice controlled by voice, remote control, or maybe thoughts in future http://www.youtube.com/watch?v=9c6W4CCU9M 4 Improvements in ICs, displays, other components are leading to better performance and cost of glasses But maybe they wont help men find a girlfriend 77. Outline Overview Speech Interfaces Touch Interfaces Gesture Interfaces Augmented Reality (various combinations) Neural Interfaces 78. Neural Interfaces Key Component Brain scanning device Key dimensions that need improvement are Accuracy, Throughput and Affordability Required improvements in brain scanning technology Accuracy & Throughput Higher spatial and temporal resolution Affordability Size and better materials 79. Current State of Art Best systems enable a person to control a robot or cursor or type one letter a minute with their mind For paralyzed, very expensive http://www.youtube.com/watch?v=C7H_M8-dBHc (0-1 minute) http://www.youtube.com/watch?v=cDiWFcA0gaw&playn ext=1&list=PL7FD931F8953A0F87&feature=results_mai n (0-2:45 minute) Accessory for your iPhone $99 device measures your brain waves. App displays data on phone http://singularityhub.com/2011/01/07/iphone-accessory-from- xwave-channels-your-brain-waves-to-the-iphone/ 80. SPECTEEG 1936 1950 1972 19751968 CT Scan 1983 MEG 1991 fMRINIRS 1973 MRI PET US$2.9M US$1M-1.5M US$250K US$2.4M US$0.5M-3M US$180K- 250K>US$30K Non-Invasive Brain Scanning Electro Encephalo Graphy Magneto Encephalo Graphy Near- Infra Red Spectros copy functional Magnetic Resonance Imaging EEG & MEG directly measure neuronal activity, NIRS & fMRI measure blood activity 81. What do EEG & MEG Measure? 82. Localization of EEG vs. MEG EEGMEG 83. Where we Are for Resolution (1) Ideally, a non-invasive technology with high spatial resolution and high temporal resolution is required Additionally, the technology must be affordable and portable in order to be useful in HCI applications Gerven, M. v., et al., The Brain-Computer Interface Cycle, J. Neural Eng, 2009 Non-invasive Neuron can fire ~0.1mm (spatial) & ~10 ms (temporal) Invasive 84. Where we are, and where we want to be (2) (in 60 years?) http://singularityhub.com/2011/01/07/iphone-accessory-from-xwave-channels-your-brain-waves-to-the-iphone/ Invasive techniques 85. Spatial Resolution Improvement While spatial resolution is important for accuracy, high temporal resolution is also critical for user interfaces R. Kurzweil, The Singularity is Near, 2005 fMRI 86. EEG Challenges Key limitation: Poor spatial resolution Increasing number of EEG electrodes may provide limited improvement in spatial resolution and higher SNR J. Malmivuo, Comparison of the Properties of EEG and MEG, Intl J of Bioelectromagnetism , 6 (1), 2004 87. MEG Challenges Baranga, A. B.-A. (2010). "Brain's Magnetic Field: a Narrow Window to Brain's Activity". Electromagnetic field and the human body workshop, (pp. 12). fT 88. MEG: Improvements in Millimeter- scale Atomic Magnetometer Target: ~100fT and