Overview of interesting problemsin

multimedia signal processing

Multimedia Signal Processing

• Deals mostly with information related to the

human sensory system• We do not have general methods to solve

information processing problems which

human information processing system is

solving very well• Multimedia signal processing is very

currently very important

Overview of current problems

• In this final lecture we present some current

problems which are very important for the

industry and industry is working on solving

them. Very often, the work in industry is secret

but in this case problems are so hard that industry

is encouraging work by publishing them and even

offering prizes for solution

DEMOLA PROBLEMS

• Demola is an activity in Tampere: Demola is an opportunity for students to contribute real-life innovations with end-users and globally connected organisations.

WWW.DEMOLA.FI• For students Demola provides• Collaboration with Finland’s top companies • Training and guidance from top professionals • Real world project experience • Credit points and an opportunity to do a thesis on learning by

doing • Multidisciplinary and international teamwork • IPR and business opportunities • Enriching interaction in Demola’s premises in Finlayson • A reward from the job well done

DEMOLA CURRENT PROGRAM

You will work for a real company as part of a project team.

You will develop something that you have always wanted and get paid for the job and learn something totally new.

R&D specialists from the enabling companies will guide your work.

You can get credit for Multimedia Project 5 cpYou can make your Master Thesis related to the project

Time periodThe project constitutes full-time work for the project team between 24.5 - 31.8.2010.

Project areas & topics offered

• Sensor Assisted Image Tagging• Wearable Computing• Music Space• Haptic Paintball• Killer Mixed Reality Application• One-eyed Wonder

We make an overview of these topics – we may think

about solutions

Project 1: Sensor Assisted Image Tagging

• Background and Motivation

Current camera applications on mobile phones, allow adding textual tags to images after taking a picture. In addition, users may be able to add geotags which describe the location. This project will look beyond the conventional textual tags. In the future, it is possible that some information of your environment and activity can be obtained automatically from sensory data collected by the mobile phone.

Sensor Assisted Image Tagging

• What this project means?

We mentioned that cameras can use information

about time, place (GPS) when the picture was taken

In this project the goal is to use other types of

information like camera position, acceleration,

sound and content of the picture (e.g. place, type of

scene)

• Project GoalThis project will implement a camera application on Nokia N900 which tags images with rich context information obtained from a context recognition engine. The team's goal is to implement a mobile camera application which suggests tags based on rich context information collected from mobile phone sensors. And optionally carry out user trials on the application. Development tools, environments and standardsThe company partner Nokia Research Center will provide a context recognition engine software package that detects the user's environment (restaurant, street, nature) and activity (idle, walking, in a train, in a car). The software is implemented on the Nokia N900 using e.g. C language and Qt. Development phones will also be provided.

Potential solution approach

• We were showing example of ambulatory video

but this is something else, only pictures are taken

In this project context recognition engine is available, the question is what are its capabilities.

One could also look into some kind of training system for identifying scenes. Literature review can be made for this.

The system should be simple but work relatively well. How to make it….think about it!

Project 2: Wearable Computing

• Background and Motivation

Wearable computing is one of the future trends, where development in the areas such as smart materials, miniaturization, sensor systems and flexible displays will open new opportunities. While these technologies are still emerging, we can now look at creative ways to use the currently available components. This project is an exploratory hands-on project which aims to demonstrate what we can do now with creativity and existing, everyday technology.

• Project Goal

The goal of the project is to innovate creative wearable computing items (e.g. integrated on t-shirts, hats, sunglasses, shoes), and build them with existing technical components. In addition to utilizing electronics kits, the material can be extracted from toys, legos, Christmas lights, game boards, etc. – what ever you may find useful. The resulting demonstrators can be useful but they can be also be purely fun – let your creativity rule.

• Desired skills/competenciesThe students should have creative minds, positive can-do spirit, and earlier hands-on experience on building things – whether it is hacking electronics or playing with Mindstorm kits.

• Development tools, environments and standards

The students can decide what kits and components to use within the given budget frame (estimated to be a few hundreds euro).

Potential solution approach

• This may be something funny like here• e-fashion. The jewelry blinks when• somebody calls or message is send • via Bluetooth? •

A company www.ifmachines.com

makes Electropuff – lamp

dimmer for kids

Potential solution approach

• It is impossible to give and idea – which is original

so one has to think about this, but important

consideration may be power supply and water resistance. Creative thinking helps:

Swiss Army knife ??? Sunglasses ??? Gloves??? Makeup???

Something new is needed!

Project 3: Music Space

• Background and Motivation

New service offerings such as Comes with Music have enabled users to store a huge selection music on their mobile devices. Selecting and finding a desired item is getting more difficult when the user needs to browse through long lists of items. The conventional access to music catalog using indexed in hierarchic lists based on the name of the artist, album, composer and genre should be replaced by more intuitive search methods.

Music Space

• Project GoalDesign a media player user interface using touch screen, available sensors, 3D graphics and 3D audio rendering. Let the user navigate in audio visual 3D space filled with music

Development tools, environments and standardsQT environment Gstreamer media

Potential solution approach

• It looks that instead of the list, the use would use

3D graphics model with navigation by touch. The problem is how to map music pieces to the model.

One could imagine e.g. a street with different clubs ’Jazz’, ’Pop’, ’ Techno’…. with user navigating them.

The model can not be too complicated, very clear,

it should associate graphics with music style

Project 4: Haptic Paintball• Background and Motivation

Location based applications are one of the rapid growing areas for mobile devices. Typically the location sensing is based on GPS satellites for outdoors and WiFi networks for indoors.  At the same time mixed reality applications are gaining speed as well. Augmentation of reality is typically based on sophisticated video capturing and rendering technologies, however new eyes-free haptic and/or audio methods are entering the field currently.

NRC Helsinki has demonstrated their indoor positioning system, which could be combined with NRC Tampere's mixed reality expertise to create multi-player “paintball” game. Alternatively GPS could be used in outdoors settings. NRC Tampere has experience in using haptic pointing devices which could use orientation and simple gestures for pointing at each other and shooting. Tactile feedback representing capturing the target could be delivered by the same hand-held pointers. Additional audio feedback could be used for enhancing the experience.

Project Goal

• Explore the possibilities for such mixed reality multi-player game in both indoors and outdoors environments and implement (at least) one of them. Creativity in interaction design using the specified multiple modalities and finding suitable metaphors are essential part of the main outcome.

• Development tools, environments and standardsExisting technology descriptions (keywords: augmented reality, haptics, tactile feedback, wearable computing)

Potential Solution Method

• It is not very clear what is the role of positioning

in this type of hand paintball

Maybe there would be some devices (robots?) shooting when they know positions of players?

Or part of the game is played on computers?

Or there is ”intelligence” where other players are?

Seems the goal is to invent attractive scenario

Project 5: Killer Mixed Reality Application

• Background and Motivation

Mobile Mixed Reality promises to “fuse” the real world with digital information, creating the real-world-web. This is possible as mobile phones can sense the real world environment (via camera and other sensors), and on top of that overlay information (Augmented Reality) that has been downloaded from the Internet.

Project Goal

• We would like you to explore, concept, design and prototype the next killer mixed reality application building on top of our many existing components (mobile mixed reality browsers, mash-up APIs, unique geo-content). If you had all the needed technology components, and could mash-up content from anywhere, what would you make to change the everyday life of millions of people?

Killer Mixed Reality Application

• Development tools, environments and standards

We will provide access to the Mixed Reality Solutions Web service platform that allows you to easily build mixed reality services and also  access to unique Navteq geo-content like maps, street-view panoramas, POIs and 3D building models through a ReSTful API.

• Relevant technologies include HTTP, ReSTful Web services, XML/JSON, MySQL, Qt, etc.

Potential Solution Method

• The project aims for extending current multiplayer games and virtual environments like Second Life by data from real world: maps, GPS location etc.

• How to do this is big question – Maybe building

world model and having own information put on

it?

For example, getting contacts based on

geolocation?

Project 6:One-eyed Wonder

• Background and Motivation

Imagine controlling your home media and devices by pointing your smart phone (camera) at meaningful objects like the TV, or turning your coffee mug into a magic wand control. And then performing complex combinations of content acrobatics with ease: playing out photos, music and video around your lounge and home. This would change the way people interact with their home electronics and digital content. And this is what we want you to do by combining networked smart devices and computer vision.

One-eyed Wonder

• Project GoalWe propose three progressively more involved challenges:

• Create a simple N900 Maemo app to understand what  device (or object) a user points their phone camera at (the user focus), and initiate basic multimedia apps – like playing music from a phone to a networked stereo.

• Create an "eye in the sky" app (on Maemo or Ubuntu connected to a webcam "embedded in the environment) to recognise when a user is pointing at an object in a room by: recognising pointing and target objects, recognising pointer activation (a user taking it in hand) and recognise user focus (the line/cone/plane the user is pointing at, to see what target objects are in "the line of focus"). Add this to the multimedia app so that it no longer needs an active networked electronic device in hand to work.

• Innovate a way to also identify which user is performing which pointing (e.g. for multiple people in the same room, and for personalised effects to the multimedia app).

One-eyed Wonder

• Development tools, environments and standards

Probably: Python, OpenCV, Maemo5/N900, Ubuntu 9.10, pen and paper.Possibly: Qt Designer, USBwebcams, Gstreamer and web services. Possibly Eclipse)Also: support from the NRC Tampere 3D Platform team with distributed/cloud and proximity/smart space access to user content and networking aspects.And: anything that comes with a good reason why it should be used.

Potential solution approach

• We have seen in this course devices like Nintendo Wii

Wii by Nintendo

Contollers have motion sensors

In this case the goal is to use mobile device instead of controllerand mobile device has the same sensor. But the device should also select specific action for example related to TV so the systemshould know if the action is related to TV. This can be discovered by a camera installed on the ceiling but that is not easy. Maybe some kind of positioning system can be used?

International Projects

• These are available at

www.Multimediagrandchallenge.com

They are offered by companies looking for solutions

maybe more to researchers than to students

Nokia Challenge Where was this Photo Taken, and How?

• The problem can be stated simply: try to derive exact camera poses (location and orientation) of given photos that are lacking location annotation. This kind of technology could potentially be used to add metadata to existing or newly captured photos.

• Assumptions: You can assume the availability of nearby photos/video with known location that can be used to derive unknown camera poses; other ideas that do not require existing content will be welcome. While a “clean” solution is ideal, other models that help could be used, for example, exploiting inertia sensor data, properties of personal collections, or the presence of textual

Google Challenge Robust, As-Accurate-As-Human Genre Classification for

Video

• Having videos classified into a pre-existing hierarchy of genres is one way to make the browsing task easier. The goal of this task would be to take user generated videos (along with their sparse and noisy metadata) and automatically classify them into genres

Google Challenge Indexing and Fast Interactive Searching in Personal Diaries

• Diaries can be any combination of audio, video, geographic location, photos, phone logs, and whatever other multimedia data the user generates or accesses. To make the data accessible, it needs to be parsed into indexable, browsable, and searchable structures such as places, environments, episodes, actions, and events of various sorts, and clustered and tagged with categories, identities, and tags of whatever sort the user proposes. The challenge is to develop good schema, algorithms, UI, etc., that will be useful for diaries from audio-only through full-featured multimedia.

3DLife Challenge Sports Activity Analysis in Camera Networks

• This challenge focuses on exploring the limits of what is possible in terms of 2D and 3D data extraction from a low-cost camera network for sports. Tennis is chosen as a case study as it is a sporting environment that is relatively easy to instrument with cheap cameras and features a small number of actors (players) who exhibit explosive and rapid sophisticated motion.

• The goal is to facilitate coaches and mentors to provide better feedback to athletes based on recorded competitive training matches, training drills or any prescribed set of activities.

Radvision Challenge Video Conferencing To Surpass “In-Person” Meeting

Experience

• This challenge focuses on developing new technologies and ideas to surpass the “in-person” meeting experience. In the process a set of subjective and objective measures to evaluate “meeting” experience will be developed. With these measures, alternative solutions could be compared to each other and to in-person meetings, and optimized accordingly.

• It is assumed that when meeting experience will be good enough, or even better, the technology could potentially minimize the need for “physical” meetings (at least for business purposes).

Conclusions

• Multimedia signal processing is very important

in many practical applications

Problems in this area are difficult to solve, many of them are already done by biological systems

We may expect progress in the future since processing power of computers grows very quickly

Solutions to problems have to be currently looked

by inventing clever algorithms matched to specific

tasks