1

Multimedia Information and Mobile-Learning

Duc Phuong NguyenMartin GuggisbergHelmar Burkhart

University of BaselSwitzerland

2

Challenge 1: Content adaptation

Jobs done: 40Queues : 30Status: OKElapsed time: 1 hourRemain time: 2 hours

Short text can be sent via SMS

Client-side globus-hostname command is not returning a fully qualified domain name (FQN). The submission scripts use globus-hostname to send information to the server about how to contact the client. The fix is to make globus-hostname return the FQDN.There are firewalls and ports (GRAM: 2119, LDAP: 2135) which are not open. Check that GLOBUS_TCP_PORT_RANGE is set to 3000 to 3090 on the client (or another range acceptable for the server to contact the client on), and that all intermediate firewalls are open for this range in both directions. GRAM Job submission failed because the executable file permissions do not allow execution".Probable cause: You submitted a script, e.g., globus-job-run symphony:2119/jobmanager-loadleveler $PWD/simple.ll but simple.ll does not have the execute permission bit set.The error msg "Authentication Failed remote certificate signed by unknown CA" means the client cannot find the CA for the cert presented by the server.Make sure the CA cert is present in the trusted cert dir set with setup-gsi. Also, check the time is consistent across the machines.The error msg "GRAM Job submission failed because the connection to the server failed (check host and port) Make sure that the gatekeeper service is enabled.The error msg "GRAM Job submission failed because data transfer to the server failed" means the job manager has problems opening some file. Make sure $GLOBUS_LOCATION is readable by all the users (mode 755)

What about this text?

How can we sent this information to mobile users?

3

The second message started on 10.12.2006 at 17h …

…………………………………………………

Challenge 2: Content overview

Difficult to get the most important message first:„You have 10 voice messages in your mail box…The first message

started on 10.12.2006 at 14h ………………………………………………………………………………………………………

The ninth message started on 10.12.2006 at 17h …

…………………………………………………The last message is actually what important to me in this example“

4

Challenge 3: Integration

Suppose that we have a system that solves

Challenge 1 (content adaptation)Challenge 2 (content overview)

How can we integrate it into our E-Learning system?

5

Solution 1: Content adaptation

CoMobile

Speech1.wav

Speech2.wavLong texts

Audio

Voice call

• Quite difficult to convert mathematic formulae to speech

6

Solution 2: Content overview

CoMobile Keywords extraction(Post processing)

Audio

1. Lunch2. Food3. Project meeting4. Concert……………..

Speech1.wav

Speech2.wav

Voice call Recorded lectures

Text

List of keyword can be used to identify the important voice messages

SMS

7

CoMobile framework- Uni Basel

- Access learning materials - Search, notification- Collaboration (sharing, discussion) - Self-assessment, voting, quiz

8

CoMobile framework- Uni Basel

- Access learning materials - Search, notification- Collaboration (sharing, discussion) - Review, self-assessment, voting,

quiz

9

CoMobile-Example

CoMobile

Voice message

Alice

Voice message

Bob

Text message

Christian

Voice and Text

Pengi

10

CoMobile-Example

CoMobile

Voice messageVoice message

Voice and Text

Text message

12

4

3

11

CoMobile framework-Overview

1. Content adaptation: long texts are converted into speech and then can be accessed by mobile phones

2. Search on audio data

3. Platforms and languages independence

4. Flexible: easy to integrate via CoMobile-services

12

CoMobile framework-Components (I)

All components are open source software:- CoMobile main component

- Plug-in components: - Text to Speech: Festival- Speech recognition: Sphinx- Optical character recognition: GOCR

Plug-in components can be replaced without many modifications.

13

CoMobile framework-Components (II)

Service-oriented: works on document‘s level (e.g: input=audio files, output=text files)

CoMobile calls the component‘s public service

The component invokes CoMobile callback service when it finish the task

Communication between CoMobile and the plug-in components are done via HTTP

14

CoMobile framework- Easy to integrate

Service-oriented simplifies the intergration process:

PHP-based system with CoMobile Java-based system with CoMobile

15

CoMobile framework-Components (III)

Festival-Text to Speech: support many languages (English, German, French, Spanish, etc.)

Sphinx-Speech recognition: free acoustic model for English (64.000 words; WER = 18%)

Optical Character Recognition: GOCR, tesseract (supported by Google). Recognition rate is still quite low for handwritten notes (accuracy rate= appr. 20% with empirical tests).

16

Open issue: Recognition of notes taken by mobile phone camera

Handwritten notes and the low resolution of mobile phone‘s camera decrease the accuracy of OCR systems significantly

17

CoMobile example: Existing Web-based applications can use mobile phones to access and submit information by integrating with CoMobile services

http://informatik.unibas.ch/courses/multimedia/lecture1.mpg

Text retrieval, features extraction, vector space retrieval, inverted list, ranking

18

Uni Basel-CoMobile features

Content adaptation and content overview for mobile devices.

Data submission: send information (text, images, audio, video) via SMS, MMS, voice recording.

SMS notification service (replies, reminds, alerts).

Access information via various channels: Voice, SMS, MMS, WAP, Bluetooth.

19

Transformation issue

Information loss in transformation chain Recognition accuracy Quality of search: keywords, semantic

Text Audio

Graphic

20

Conclusion

Content transformation is an important issues in mobile applications.

Uni Basel-CoMobile: an easy to integrate and flexible framework that support collaborative learning with mobile devices.

The CoMobile framework is open source and look forward for contributions.

21

Uni Basel-CoMobile-Demo

CoMobile demonstration

22

Discussion

Cost: Users have to pay for SMS, MMS, as well as voice call. „Flat rate“ package will be common in the near future.

Storage: 1 minute video = 1 MB (approximately). There exists 2GB flash card for mobile phone

Bandwidth: 14.4 kbps (GSM), 140 kbps (GPRS), 1920 kbps (UMTS)

23

SMS gateway - Kannel

RS-232AT Commands

AT+CNMI=1,2,0,1,1SMS Text

HTTP request to forward SMS Text

24

MMS gateway - Mbuni

RS-232AT Commands

AT+CNMI=1,2,0,1,1MMS

Text, Picture, Audio

HTTP request to forward MMS

Mbuni

25

PBX Asterisk

PCI socketVoice

AGI: Perl, Java, PHP scripts are called

Telephone line