istd 2003, sensing & sensors interactive systems technical design seminar work: sensing &...

ISTD 2003, Sensing & Sensors

Interactive Systems Technical Design

Seminar work: Sensing & Sensors

Hannu KaskiJukka-Pekka Laitinen

Miika Vahtola


Introduction 1/4

Sensing• Webster: “To perceive by the senses, to detect automatically especially in a

response to physical stimulus”

• Way to achieve knowledge of the world (temperature, force, acceleration…)

• Human senses: sight, hearing, touch, smell and taste • Vision usually seen as the primary sense and hearing

secondary • Exceptions to general rules like blindness or deafness• Touch can also be important when interacting with systems

• Haptic systems - systems that use touch (haptic feedback) e.g. force feedback joysticks

• Smell and taste generally ignored within computer interfaces


Introduction 2/4

• Human sensing capability in active touch

• Differences• Length and velocity 10%• Acceleration 20%• Force 7%• Mass 21%• Viscosity 14%

• Resolution• Pressure 0,03 N/cm2

• Transient temperature 0,05 ºC• Skin displacement 20 micro meters• Surface texture 0,1 micro meters


Introduction 3/4

• Tactile user interface is one type of sensing oriented UIs (Webster defines tactile: “of or relating the sense of touch” )

• Interface which can be controlled by touching and may give tactile output

• Input• Handheld / tablet computers• Computer input devices• Information kiosks (touch screens)

• Output• Vibrator alarms (cell phones, pagers)• Force Feedback (Entertainment applications e.g. game

controllers, robotic surgery)


Introduction 4/4

Sensors• Webster: “A device that responds to a physical stimulus (as heat, light, sound, pressure,

magnetism, or a particular motion) and transmits a resulting impulse (as for measurements or operating a control)”

• Comprised of two basic parts - a sensing element and a transducer

• Contact/contactless sensing• Sensors + signal processing and logic (AI) enable

“sensing” in machine domain• Nowadays sensors have integrated microcontrollers• Sensor technologies are rapidly evolving

• Drivers: miniaturization, cost, processing power, power consumption factors

• In particular the fact that sensors are vital technology enablers for new applications


Problem Complexity: Human vs. Machine

HUMAN

MA

CH

INE

EASY HARD

EASY

HARD Maximum

Potential Benefit

• Object recognition• Linguistics• Extraction of Relevant

Features from Sensor Arrays

• Arithmetic• Logic

• Thresholding• Tallying

• Judging

Human vs. Machine Characteristics related to sensing

© Thad Roppel from Auburn University


Motivation

• Sensors are vital technology enablers for new applications• When applied in a right way they will probably ease your

everyday life (e.g. intelligent environments)

• Context-aware computing• “Perception without the context of action is meaningless”

Sensors enable context-awareness (sensor fusion important)

• Ubiquitous and pervasive computing

• Usability of those devices that can “sense” may be better, because sensors enable a more sophisticated user interaction Possibly better user experience


Implementation

• Humans and computers “sense” differentlyMachines can only emulate real sensing (i.e. human) with

the help of different kind of sensors, signal processing, microcontrollers and logic

• OBJECTIVE: To intelligently integrate multiple sensors and multiple sensor modalities (i.e. sensor fusion) to serve the needs of Human-Computer Interaction More natural and intuitive interaction between humans and

computer “Smart interaction” usually requires a network of sensors

working in concert Remember to keep in mind that systems should be build for

people not vice versa Natural interaction as a design paradigm when possible


Sensor selection 1 of 2• According to www.sensors-ez.com there are 1022

sensor manufacturers and tens of sensor categories• Excellent source of sensor related information:

www.sensorsmag.com

• Capacitance sensors (based on charge sensing)• Cheap, simple, no calibration• Enables touch (position) and proximity sensing• Some issues that should be noticed when

implementing: Good quality ground referenceLow impedance connectionsKeep connections short and of low inductanceStop ringing by adding a series resistor

• Photoelectric sensors (color sensing, lasers for distance sensing)• Reliable, versatile• Able to sense objects of almost any material, size and

shape

http://www.sensors-ez.com/

http://www.sensorsmag.com/


Sensor selection 2 of 2

• Other sensor categories• Acceleration & speed• Acoustic (e.g. ultrasonic sensors) • Displacement & motion• Force, pressure & tension• Light (e.g. IR)• Position & tilt• Presense & proximity• RF• Temperature & humidity • Torque & vibration• Optical imaging based sensors (e.g. cameras)


transduce perception to electrical signal

measure involts, amps, ohms,henrys, farads, etc.

convert fromsignal to symbol

compute control actiontransduce signal toheat, displacement,

illumination, etcconvert from

symbol to signal

SENSOR

ACTUATOR

ADC

DAC

measurand

e n v i r o n m e n t

How to ’sense’ using sensors:Sense-Model/Think-Act Loop

© Mel Siegel from CMU


How to implement?

• STEPS to systematize the sensing process:1. Decomposition of relevant context information

acquired by sensors Model of discrete facts and quantitative

measurements

2. Build a system based on some sensor fusion system architecture (below is one example)

© Mel Siegel from CMU


Usual requirements for an implementation

• Small & lightweight -> miniaturization (HDP/ASIC/MEMS)

• Reliable• Information security• Biocompatibility• Low power consumption• Shock proof• Low cost


Application

Proactive Furniture AssemblyBy Stavros Antifakos, Florian Michahelles and Bernt Schiele

from ETH Zurich

http://www.vision.ethz.ch/projects/A subproject of the Smart-Its Project that is funded in part by

the Commission of the European Union and the Swiss Federal Office for Education and Science

VIDEO:http://www.vision.ethz.ch/publ/ubicomp02.mov

http://www.vision.ethz.ch/projects/

http://www.smart-its.org/

http://www.vision.ethz.ch/publ/ubicomp02.mov


Application Introduction

• an experimental case study with the IKEA PAX wardrobe

• PROBLEM: The presentation of plans by today's instructions is neither sufficient nor satisfying

• 3 usage modes were identified: Full-walk-through, Assistance-on-demand and Rescue-from-trap

• OBJECTIVE: To develop Proactive Instructions for Furniture Assembly-> better usability of instructions

• Chosen approach was to immerse instructions into the objects of interest (i.e. parts of a wardrobe)


IKEA’s assembly instructions


Different ways to assembly the IKEA PAX wardrobe


Assembly actions and possible sensor configurations to perceive the action


Detection of actions

• Simple Markov chains were designed for each action

• States and state transition probabilities were modeled by hand -> investigations to use Hidden Markov Models in order to train those probabilities automatically are currently ongoing screwdriver (gyroscope)

force sensor

accelerometer


Challenges

• Precision vs. cost (sensors aren’t free)• Cheapest and most unobtrusive sensor

configuration enabling a high recognition precision should be the goal

• How to inform the user (assembler) about the next steps to be taken?• Parts giving notice (flashing leds, beeping)• Guidance through a PDA/wearable

computer/smart phone (should be avoided)

• Closed world assumption narrows down the possible applications • we have to be able to fully model all tasks


Other applications 1 of 4

• Applications needing• Proximity sensing• Presense detection• Position sensing• New control interfaces etc.

• Automotive• Controls and lighting• Safety -> Electronic Stability Program, Acceleration Skid

Control, Brake Assistant, Anti-lock Braking system• Alarms and entry access controls

• Computers• Peripheral, mouse and joystick controls• Tactile input/output devices (force feedback, in-keyboard

‘mouse’)

• Handheld devices (PDAs, phones etc.)



• Biomedical/Biometrics• Health care, personal fitness

• Wearable, personal health systems like AMON• bio-sensors (pulse, blood pressure, blood oxygen saturation, body temperature,

skin perspiration, ECG)

• Robotic surgery (with PHANTOM™-like products)



• Smart environments (e.g. home, office) • Access controls• Room light switches, remote controllers (no push

buttons)• Appliance controls (A/V & kitchen)• Hidden controls and alarms (in walls, furniture) • Object sensing (e.g. sense when somebody

touches something they shouldn't)• Human presence sensing (e.g. automated lights

and doors)• Hand-wave controls -> Make objects sense (e.g.

automatic faucet, power-ups)

• Wearable computing



• Disability/elderly Aids• electronic assistance devices

reduce need for pressure or pull strength

• Safety• Tool auto-shutoff (dead-man switches)• Child detection in unsafe areas• Intrusion detection

• Security• 'Smart Objects' - arbitrary objects

as 'smart cards' (e.g. RFID)

• Toys• Dolls, SONY’s Aibo, LEGO MindStorms


Strengths / Advantages 1 of 2

• More natural interaction, unobtrusiveness and zero activation forceFlexible form factorsBetter user experience and usability

• More intuitive usage Faster and easier to learn

• Sensors can provide/acquire information not possible to perceive by human senses HC interaction may work better than human-human

interaction in some aspects (e.g. machines try to serve you proactively)

People can acquire additional information (e.g. health state)


Strengths / Advantages 2 of 2

• Eases the life of people with disabilities

• When deployed well, will make life easier, more comfortable and safer


Limitations / Weaknesses 1 of 2

• Context-understanding is challenging• Integration of sensors is demanding because sensed

information may have overlaps or even conflictsSensor fusion techniques (AI algorithms)

• Decrease in user’s intentional controlNeed for profiles

• Increase in SW inferential burden

• Fail decisionsEffect on user acceptance

• Sensors don’t work in all conditions• Temperature, humidity, EMC and calibration issues


Limitations / Weaknesses 2 of 2

• Accuracy vs. CostMEMS technology enables SoC implementations that are

cheaper

• Noise and bandwidthLocal processing of sensor data decreases bandwidth

requirementsBetter noise filtering techniques

• Limited power supply• Processing of sensor data needs power


Selected Industrial Players

• Microsoft Corp. – Wireless IntelliMouse Explorer

• Quantum Research Group – QTouch™& QMatrix™

• SensAble Technologies Inc. – PHANTOM™

• Sony Electronics Inc. – AIBO product family

• The LEGO Group – MindStorms™ product family

• VTI Technologies Oy – SCA620 series z-axis accelerometer family


Selected International Research Groups and Projects 1 of 3

• Carnegie Mellon University HCI Institute www.hcii.cmu.edu- GM/CMU Project: Driver-Vehicle Interface- Manipulation in a Virtual Haptic Environment Based on Magnetic Levitation- Robotic Assistants for the Elderly

• ETH Zurich www.ethz.ch• Perceptual Computing and Computer Vision Group

- Smart-Its [with Lancaster University (UK), University of Karlsruhe (GER), Interactive Institute (SWE) and VTT (FIN)]

• Wearable Computing Laboratory - Wearable Microsensor Network- Advanced care and alert portable telemedical MONitor (AMON)

• Max Planck Institute for Biological Cybernetics www.kyb.tuebingen.mpg.de/bu- HapSys - High-Definition Haptic Systems- CogVis - Cognitive Vision Systems- ECVision

http://www.hcii.cmu.edu/

http://www.ethz.ch/

http://www.vision.ethz.ch/pccv/

http://www.wearable.ethz.ch/

http://www.kyb.tuebingen.mpg.de/bu



• MIT Media Lab www.media.mit.edu/research

• Context-Aware ComputingChrysler 300M IT Edition, Context-Aware Tables,

Disruptive Interruptions, Electronic Necklace• Human Design

Learning Humans, MIThril, Project Zaurus, Shortcuts• Nanoscale Sensing

High-Resolution Interferometric Accelerometer• Object-Based Media

Smart Architectural Surfaces• Responsive Environments

Design Principles for Efficient Smart Sensor System, Functional Integration for Embedded Intelligence,

Modular Platform for High Density Wireless Sensing, Wearable Badge

• Robotic LifeSensate Skin, Sociable Robots

http://www.media.mit.edu/research

http://www.media.mit.edu/research

http://www.media.mit.edu/nanoscale/

http://www.media.mit.edu/nanoscale/



• Tangible Media

Door Collision Avoidance Sensor, Tangible Bits

• Harvard BioRobotics Laboratory www.biorobotics.harvard.edu

- Remote Palpation Instruments for Minimally Invasive Surgery- Vibrotactile Sensing and Display- Force Feedback in Surgery: An Analysis of Blunt Dissection


Selected Finnish Research Groups and Projects

• Tampere Unit for Computer-Human Interaction, University of Tampere • Multimodal Interaction Group

www.cs.uta.fi/hci/mmig/projects.htm• Tactile User Interfaces• Multimodal Interface for Persons with Low Vision and/or

Hearing Impairment• Recognition and Synthesis of Faces, Gestures, and Actions

• Tampere University of Technology• Personal Electronics group

www.ele.tut.fi/research/personalelectronics• Smart Home• Wearable Computing• Smart Clothing


Companies and Research Groups in Oulu

• VTT Electronics• Advanced Interactive Systems www.vtt.fi/ele/research/ais/

• Interactive Intelligent Electronics (IIE) www.vtt.fi/ele/projects/iie/

• University of Oulu/Department of Electrical and Information Engineering http://www.ee.oulu.fi• Machine Vision and Media Processing Unit• Optoelectronics and Measurement Techniques Laboratory

• Polar Electro Oy• http://www.polar.fi

• Idesco Oy• http://www.idesco.fi

Proximity/focus sensing Smart Phone interfaces:• J-P Metsävainio Design Oy http://www.jpmdesign.fi• MyOrigo Oy http://www.myorigo.com


Future Developments

• In near term we will see “sensing” slowly become a mainstream feature in man-machine interfaces

• Nanotechnology will offer new possibilities because then sensors are so unnoticeable• We won’t know if we have drunk or eaten a

sensor• People’s acceptance?


Thank you!

• Any questions?

istd 2003, sensing & sensors interactive systems technical design seminar work: sensing &...

Documents

sensing sensors introduction

sensing webster

sensors webster

sensing element

meaningless sensors

human sensing capability

sensing thad roppel

sense of touch interface