immr template studio courseware

©2011 Immersion Corporation–Confidential

NASDAQ: IMMR

Using Haptic Studio to Create Custom Haptic Effects

May 3, 2012

©2011 Immersion Corporation–Confidential 2

What you will learn in this course

Who should be taking this course? What you should have installed before starting the class What will be provided to you in class Why custom haptic effects? Introduction to creating custom haptic effects Exploring the sample application Understanding audio files Creating basis effects to compliment the features of audio files Combining multiple basis effects to create a complex timeline effect Exporting haptic effects for implementation into an application Adding custom haptic effects to an Android application Haptic Studio tips and tricks


Who should be taking this course?

This course targets experienced Android developers who have completed the course UHL: Adding haptic effects to your Android application. In order to implement custom effects created using Haptic Studio, developers should have practical experience with:Android architecture

The life cycle of components Event handling Tween and frame-by-frame animation Debugging an application Deploying an application to a device


What you should have installed before starting the class

As an experienced Android developer, you should attend class with these items installed on your computer:

Java SE Eclipse Android SDK Android Developer Tools

– From within Eclipse’s “Install new Software” option, point to Haptic Studio

IMPORTANT NOTE: Before starting the class, you should ensure that Haptic Studio is able to see and communicate with your Android device via Haptic Bridge.

http://dl-ssl.google.com/android/eclipse/


What will be provided to you in class

This course focuses on developing and implementing custom haptic effects. You will be provided the code for a simple sample application as well as four audio files that you will use as part of the exercises:– AirplaneCrash_Starter.zip (Eclipse project)– AirplaneCrash_Starter.apk– AirplaneCrash_Solution.zip (Eclipse project)– airplane_snd.wav– running_snd.wav– dying_snd.wav– impact_snd.wav

You are also provided a reference library of Haptic Studio effects that can be used as-is or modified for use in your future Haptic Studio effect design projects:Haptic_Studio_Reference_Library.ivs


Why custom haptic effects?

Haptic Studio allows you to dramatically enhance the quality of the mobile user's communications experience with custom haptic effects

Go beyond the simple haptic effects

Allows you to design effects that are perfectly tuned to compliment your project’s audio or visual elements


Introduction to creating custom haptic effects

In this course you will be creating not only custom basis effects, but also timeline effects

Timeline effects combine basis effects along a timeline

Haptic Studio allows you to create custom, timeline effects in one of two ways:– Manually - using the tools within Haptic Studio– Dynamically - using Haptic Studio’s MIDI conversion feature

This course if focused on the manual creation of custom haptic effects


Introduction to creating custom haptic effects

Haptic Studio simplifies the manual creation of haptic effects to compliment media files by allowing you to import and reference a media file on the same timeline that you are using to create the effect

Supported formats are:– AAC (.acc)– K3G (.k3g)– MPEG-4 Audio (.m4a)– MIDI (.mid)– MP3 (.mp3)– MPEG-4 Video (mp4)– SKM (.skm)– Wave (.wav)


Exploring the sample application

In the following exercises, you will create custom haptic effects and implement them into a sample application that is based on the animation of an aircraft in flight

You will be designing custom haptic effects that allow the user to feel the running engine and the impact of the crash.


Exercise 1: Exploring the sample application

In this exercise you will:

Explore the Airplane Crash sample application with and without haptic effects

Install and run the Airplane Crash starter file

Install and run the Airplane Crash solution file


Understanding audio files

Important that you be able to examine audio files and identify key features

The Y-axis represents amplitude (volume) and the X-axis represents frequency (the number of cycles/vibrations per a specified amount of time)

Higher frequency vibrations are sensed as higher pitched sounds and lower frequency vibrations are sensed as low pitch sounds

Easy to shape haptic effects to match audio waveforms Haptic Studio allows you to view audio waveforms and haptic

waveforms side-by-side


Discussion: Identifying key features of an audio file and developing a design strategy

Keep effects as short as possible

Reduces the impact of drift—a phenomena that can cause long effects to become out of sync’ with audio or other media files that they accompany

Consider all of the component parts (audio, animation, haptics, interactions, controls, etc.) in advance

Then develop a haptic design strategy that will maximize the benefits of haptic feedback



Let’s revisit the sound track from our sample application Airplane Crash

Looking at the soundtrack waveform, you should be able to easily identify these key features: engine running, engine dying, silence and impact



Better strategy would be to break it down into three separate haptic effects

Elected to divide the soundtrack into three separate sound files—each representing a key feature

Straight forward approach will allow our application to simply call the appropriate haptic effect as each individual sound file is played


Creating basis effects to compliment the features of audio files

Basis effects can be used alone or combined to form more complex, timeline effect

There are three types of basis effects:– Periodic effects are cyclic or pulsing and played at a

specific (though variable) frequency – MagSweep effects are constant as opposed to cyclic– WaveForm effects can capture a very high degree of

detail from an audio file and require TouchSense 5000 for replay


Creating basis effects to compliment the features of audio files

When you add a Periodic or MagSweep effect to your project you will be able to control certain attributes

In this course you will deal with:– Magnitude - the strength of the effect—like volume is to

audio – Duration - the length of the effect—how long it will play– Attack Time - the amount of time to fade in an effect– Fade Time - the amount of time to fade out an effect

When dealing with Periodic effects, you also have control over:– Period - the amount of time from cycle to cycle


Exercise 2: Creating simple basis effects to match the sounds of the running and dying engine


Select appropriate basis effects for each of the audio files

Modify these effects to compliment the sound file

Use a timeline to combine audio and haptic effects for testing



Launching Haptic Studio Plug your Android device into your computer and launch

Haptic Bridge– Do this BEFORE launching Haptic Studio.

Launch Haptic Studio Select your device from the drop-down menu

From the View menu, select Details



Select and modify a basis effect to compliment the sound of the running engine

You will adjust the parameters to create a strong, steady pulse with an abrupt start (attack) and stop (fade)

Click on the New Timeline button on the toolbar:

Place your cursor in the object name area and rename it to running_tl

Click on the New Multimedia button on the toolbar:



Locate and select the file running_snd.wav Click on the New Periodic button on the toolbar:

Place your cursor in the object name area and rename the effect running_he



Double click on the new running_he object to open the effect editor view:



Adjust the Magnitude parameter to the maximum level (10000)

Set the Attack Time and Fade Time parameters both to 0 so that the effect will start and stop abruptly

Use the Play button to sample the effect



Adjust the Period parameter to 50 ms and play the effect again



Since this is a single-magnitude effect (sudden start/stop) you can specify an infinite duration and control the start and stop at the application level

Click on the Infinite button:

Double click on the running_tl object to open the timeline Drag an instance of the running_snd object into the timeline Drag an instance of the running_he object into the timeline



Your timeline should now look like this:



Click on the New Timeline button on the toolbar and name the new timeline object dying_tl

Click on the New Multimedia button on the toolbar and select the file dying_snd.wav from your hard drive

Double click on the dying_snd object to produce the Wave Form inspector:



Click on the running_he object in list view to highlight it:

Select Copy (Ctrl+C) and then Paste > Effects and Referenced Effects (Ctrl+V) from the Edit menu

Rename the new object dying_he



Double click on the new dying object to open the effect editor and change the duration to 1620 ms

Now adjust the Fade Time parameter to 1620 ms so that it matches the duration-- producing a steady, fading, magnitude slope from beginning to end



Double click on the dying_tl object to open the timeline Drag the dying_snd object into the timeline Drag the dying_he object into the timeline Because you specified the duration of the effect to match the

duration of the sound file the two should match up perfectly on the timeline



Play the timeline and experience the sound and haptic effect together

Select Save > Save As and name your Haptic Studio project Airplane Crash



Combining multiple basis effects to create a complex timeline effect

Complex haptic effects are created by arranging one or more basis effects (like building blocks) on a timeline

In the case of complex effects, the application will call the timeline and not the individual effect

A timeline can be simple—a single basis effect repeated to create some sort of rhythm for example, or complex—multiple basis effects that may overlap



Example of a single effect (simple) timeline

Example of a multiple effect (complex) timeline



You can place an audio file on the timeline as a development aid

When a complex haptic effect timeline is exported, it is treated as a single effect.


Exercise 3: Creating a complex, timeline effect to match the sound of impact


Select appropriate basis effects for each part of the impact audio file

Modify these effects to compliment the sound they will accompany

Combine the basis effects on a timeline



Return to your Airplane Crash Haptic Studio project. Add a new timeline object and rename it impact_tl Use the New Multimedia button to import the file

impact_snd.wav Double click on the impact_snd object to display the audio

waveform (Figure 22). Listen carefully as you play the file and notice that there are really two components to the sound. There is the initial impact followed by a fading aftershock. Notice that the entire duration is 2.332 seconds (2332 milliseconds)



Estimated duration of initial impact: 279 ms (1/8th of the total) Estimate duration of aftershock: 2053 ms (7/8th of the total)



Click on the New MagSweep button on the toolbar to insert a MagSweep object

Rename the new object impact_a Double click on the impact_a object to open the effects

editor and apply theses settings:– Magnitude: 10000– Duration: 279 ms– Attack Time: 0– Fade Time: 0



Your settings should look something like this:



Click the New Periodic button on the toolbar to introduce a new Periodic object and rename it impact_b

Double click the new impact_b object to produce the object editor and apply these settings:– Magnitude: 10000– Period: 70 ms (a slower, deeper, pulsing than the running

engine)– Duration: 2053 ms (estimate determined earlier)– Attack Time: 0– Fade Time: 2053 ms



Double click on the impact_tl to open the timeline Drag the impact_snd object into the timeline Drag the impact_a object into the timeline Drag the impact_b object into the timeline Select the impact_a object on the timeline

– Slowly move your cursor over the right boundary until the 4-way arrow turns into a 2-way arrow

Select impact_b effect to produce the 4-way arrow Use your cursor and the 2-way arrow to adjust the length of

this effect to match the sound wave form (approximately 2113 ms)



Play the timeline and experience the complex effect and sound playing together

Select File > Save


Exporting haptic effects for implementation into an application

Media files are not embedded in Haptic Studio projects and data exported from Haptic Studio contains no information about media files—they are strictly for development purposes

Effects and media files must be re-synchronized at the application level

In the case of a complex effect that includes multiple basis effects, the timeline will be called and not the individual effects

When you export from Haptic Studio for Android applications, you will either export as an IVT file, or as Java code

If you are exporting Java code, omit the effect names if possible


Exporting haptic effects for implementation into an application

This is a segment of the Java file produced:


Exercise 4: Exporting haptic effects for implementation into the Airplane Crash application


Export your Haptic Studio created effects to a Java file



Return to your Airplane Crash Haptic Studio project. Select File > Export > Java Code and save the file as

AirplaneEffects.java to your hard drive. When asked about omitting effect names, say no



Adding custom haptic effects to an Android application

Adding a custom haptic effect is a multi-step process that includes:– Adding the UHL package,– Adding the native library file– Reading custom haptic effects information into a buffer– Playing the haptic effects on your target device


Playing custom effects

To play a custom haptic effect, you must first create a Device object

You also need to create an IVTBuffer object before playing the custom effects (IVT is the native file format for Haptic Studio)

Once these objects are created and initialized, you can play the actual effect by using the playIVTEffect() method

device = Device.newDevice(context);

IVTBuffer ivt = new IVTBuffer(myeffects.ivt);

device.playIVTEffect(ivt, myeffects.Periodic);


Stopping custom effects

To stop the play of custom haptic effects, you must use the following code:

The Device must be closed after it has been used to play and stop the haptic effects. This ensures that there are no resource leaks in the application.

device.stopAllPlayingEffects();

device.close();


Exercise 5: Adding your custom haptic effects to the Airplane Crash application


Set up an Eclipse project using the Airplane Crash starter file provided

Add the custom haptic effects you have created to the sample application



Set up an Eclipse project

In Eclipse, select File > Import > General > Existing Projects into Workspace

Select Next and then choose the radio button for Select archive file

Enter the path to the file AirplaneCrash_Starter.zip that is provided with this course

Click Finish



Add the Universal Haptic Layer (UHL) package

Unzip the UHL package that you downloaded earlier. You will see three files:– UHL.jar– libEmmulatorJ.so– UHLdoc.zip

Copy the UHL.jar and UHLdoc.zip files and paste them in the libs folder for the project.



Add the jar file

Right-click on the AirplaneCrash_Starter project in the Eclipse Package Explorer and choose Build Path > Configure Build Path

Select the Libraries tab and click on Add JARs

Browse to the project's libs folder and select the UHL.jar file Notice the UHL.jar file added to the project



Add the Javadoc for the UHL

Expand the UHL.jar directory and select Javadoc location. Click Edit. In the Javadoc For ‘UHL.jar’ window, select the Javadoc in

archive radio button. Select the Workspace file radio button. To set the Archive path, browse to your project and then the

libs folder and select UHLdoc.zip. Click OK.



Add the native library file

Create a new folder armeabi, inside the libs folder in your project directory

Copy the libImmEmulatorJ.so file from the UHL package and paste it into this new armeabi folder



Adjust the application’s manifest file Open AndroidManifest.xml in the root directory of your

Eclipse project Locate the tag <uses-sdk android:minSdkVersion=”3”/>

and change the value of the minSdkVersion to 7. Your code should look like this:

After the application tag block, add the following code for the VIBRATE permission:

Save the file

<uses-sdk android:minSdkVersion=”7”/>

<uses-permission android:name="android.permission.VIBRATE" />



Disabling automatic haptic feedback

Add the following code to the layout file in your project to disable the haptic effects that are automatically applied to certain components in the Android user interface:

<Buttonandroid:id="@+id/replayButton"android:layout_centerInParent="true"android:layout_height="wrap_content"android:layout_width="wrap_content"android:text="@string/replay"android:onClick="replayAnimation"android:hapticFeedbackEnabled="false"/>



Add the custom haptic effect you exported earlier

Copy the AirplaneEffects.java file from your hard drive to the src directory in your Eclipse project

Add the package declaration to the top of the Java file;

Open AirplaneCrashActivity.java class in the IDE for editing Declare the Device class variable in your Activity:

package com.immersion.haptic;

Device device;



Add the custom haptic effect you exported earlier

Create an IVTBuffer object with the exported effect definitions in your Activity;

IVTBuffer ivtBuffer = new IVTBuffer(Airplane.ivt);



In the onCreate() method of your Activity class, create a Device object;

// Instantiate a device object

try { device = Device.newDevice(getApplicationContext()); } catch(Exception ex) { Log.i("AirplaneCrashActivity.class", "Error in instantiating device object for haptics!"+ex.getMessage());}



In the Running() method within the Activity, add code to play the running_he haptic effect after the mediaPlayer.start() statement

// Play the running haptic effect try{ device.playIVTEffect(ivtBuffer, Airplane.running_he);}catch(Exception e){ Log.i("AirplaneCrashActivity.class","Catch all exceptions!"+e.getMessage());}



In the listener for the media player, when the running_snd.wav audio file is playing, add code to stop the running_he haptic effect when the audio stops:

mediaPlayer.setOnCompletionListener(new OnCompletionListener() { @Override public void onCompletion(MediaPlayer mp) { // TODO Auto-generated method stub device.stopAllPlayingEffects(); mediaPlayer.stop(); mediaPlayer.release(); Dying(); } });



In the Dying() method within the Activity, add code to play the dying_he haptic effect after the mediaPlayer.start() statement:

// Play the dying haptic effect try{ device.playIVTEffect(ivtBuffer, Airplane.dying_he);}catch(Exception e){ Log.i("AirplaneCrashActivity.class","Catch all exceptions!"+e.getMessage());}



In the listener for the media player, when the dying_snd.wav audio file is playing, add code to stop the dying_he haptic effect once the audio stops:

mediaPlayer.setOnCompletionListener(new OnCompletionListener() { @Override public void onCompletion(MediaPlayer mp) { // TODO Auto-generated method stub device.stopAllPlayingEffects(); mediaPlayer.stop(); mediaPlayer.release(); } });



In the override function onAnimationEnd() of the planeAnimation3.setAnimationListener() method, add code to play the impact_tl effect after the mediaPlayer.start() statement:

// Play the impact haptic effect try{ device.playIVTEffect(ivtBuffer, Airplane.impact_tl);}catch(Exception e){ Log.i("AirplaneCrashActivity.class","Catch all exceptions!"+e.getMessage());}



In the listener for the media player, when the impact_snd.wav audio file is playing, add code to stop the impact_tl haptic effect when the audio stops:

mediaPlayer.setOnCompletionListener(new OnCompletionListener() { @Override public void onCompletion(MediaPlayer mp) { device.stopAllPlayingEffects(); mediaPlayer.stop(); mediaPlayer.release(); } });



Add import statements for the Device and IVTBuffer objects (if they weren’t added automatically):

In the onDestroy() method of the Activity, add code to close the device:

import com.immersion.uhl.Device;import com.immersion.uhl.IVTBuffer;

/* When the device is not needed anymore, it must be closed to avoid resource leak */device.close();



Add code to close the device again in the onKeyDown() method in the Activity:

public boolean onKeyDown(int keyCode, KeyEvent event) { if ((keyCode == KeyEvent.KEYCODE_BACK) || (keyCode == KeyEvent.KEYCODE_MENU)) {

// Finish the activity finish();

// close the device device.close();

return true; } return super.onKeyDown(keyCode, event); }


Appendix A: Development environment setup

Installing Android SDK on your PC

If you haven’t already download and install the latest Android SDK, do so now:http://developer.android.com/sdk/1.5_r1/index.html

Within the Android SDK, navigate to the file adb.exe Add the path of the folder containing the executable adb.exe

to your PATH environment variable – This step is critical in enabling Haptic Studio to connect to

your device Install the ADB driver for your device

http://developer.android.com/sdk/1.5_r1/index.html

http://developer.android.com/sdk/1.5_r1/index.html



Installing Haptic Studio on your PC

Log on to your computer using an account with Administrator privileges (Haptic Studio requires Administrator privileges to install certain required components)

Download Haptic Studio from the SDK area

Run the Haptic Studio installer (.exe file) to install the application



Linking your Android device to Haptic Studio

On the handset, enable USB Debugging in the Settings/Applications/Development menu.

Connect the handset to the PC using the USB cable that came with your device.

Open a command prompt and type: adb devices



Installing and configuring Haptic Bridge on your Android device

Go to http://market.android.com/details?id=com.immersion.designerbridge on your device and download the Haptic Bridge application, or search for 'Haptic Bridge' in the Android Market

Install the Haptic Bridge application to your Android device Reboot your device (this ensures that the Haptic Bridge

service has started) Locate the Haptic Bridge application in the application

drawer by looking for the Haptic Bridge icon

http://market.android.com/details?id=com.immersion.designerbridge

http://market.android.com/details?id=com.immersion.designerbridge



Launch the application Launch Haptic Studio

– If Studio is able to communicate with the handset, the connection status displayed in the Haptic Bridge application will change to connected

Click on Reset Connection button in the application if the Haptic Bridge doesn’t connect to Haptic Studio

Haptic Studio will also display the device name in the toolbar– If you are using a TouchSense-enabled handset, the string that

appears in the toolbar will be the version of the TouchSense Player in the handset

– If the handset uses the TouchSense Emulator functionality of the UHL, TS Emulator 3000 will be displayed


Appendix B: Haptic Studio user interface

Haptic Studio menu items

File menu—open, save and close files, create new files, convert audio files

Edit menu—cut, copy and paste View menu

– Icon—shows only effect icons– List—this is essentially the Icon view reduced in size– Detail—lists effects and provides detailed information

Help menu—launches help documentation in a new window. New, Open and Save icons—shortcuts to File menu operations Cut, Copy and Paste icons—shortcuts to Edit menu

operations


Appendix B: Haptic Studio user interface

View icon—shortcut to View menu operations Basis Effects icons—inserts new Periodic, MagSweep and

WaveForm effects Timeline icon—creates a timeline into which basis effects can

be placed and manipulated Multimedia icon—imports an audio file for reference

purposes Play and Stop icons—shortcuts to Playback menu Device List—allows you to select your connected device Refresh icon—refreshes device list Help icon —shortcut to Help menu functionality


Appendix C: Haptic Studio device connection trouble shooting

1. Save your work

2. Close the Studio

3. Unplug your device

4. Open the Haptic Bridge application on your device and restart the service

5. Re-plug in your device

6. Open your Windows Programs menu and go to your “Administrative Tools” folder

7. Open your “Services” control application that is found in the “Administrative Tools” folder

8. Scroll down the list of services until you find the “Immersion TouchSense Service”

9. Select the “Immersion TouchSense Service” and stop that service, then restart the service from this same window


Appendix C: Haptic Studio device connection trouble shooting

10. Next, keep the “Services” control window open (in case you need to restart the services again) and re-launch Haptic Studio

11. Check your device drop-down list to find your device connected

12. If your device does not show connected, press the device refresh icon

13. Wait a few seconds and check your device list again

14. If you still do not see your device, keep your device connected but open your Haptic Bridge application on your device and restart Haptic Bridge software

15. Now try steps 11-12 again


Appendix D: Haptic Studio tips and tricks

Shifting All Timeline Elements – To shift all Timeline elements one way or the other, use “Ctrl+A” to select all elements within the Timeline and then you use your cursor to drag all the elements left or right in the Timeline as you please.

Sorting Effects within Timelines – While working within the Timeline view, you may find you want to sort the listed order of multiple objects to for instance, better align vibe events to a referenced waveform profile. Simply select the object name (not the Timeline events) and press your Delete key. Then use the Undo edit option. This will return your deleted Timeline object to the bottom of the Object list. In this way you can reorder the Object list in the Timeline anyway you like.

Sorting Effects within the List View – Since Haptic Studio lists effects in the order they were created, you may find you want to reorder the Object in the List View. You can do this by opening a new project and dragging Objects over to the new project in any order that you might want.



Timelines vs. Non-Timeline Basis effects - It is good practice place each of your new Basis effects, whether they are Periodic or MagSweeps, into Timelines and to rename the Timelines describing their purpose, like “Menu Click” or “Engine.” Once your effect is in a Timeline you have more design options, like:– The ability to browse to a sound file and use that sound file as a

reference during your design– The ability to use the repeat looping function– The ability to use the pre-roll function– The ability to use the Basis effect Override function and reuse

those Basis effects for multiple Timelines without changing the parameters of the original Basis effect

You can reference the “Haptic_Studio_Reference_Library.ivs” effect file to see a good example of this.



Single Effect Files vs. Multi-Effect Files - It is not necessary to create a new effect file for each effect. You can group any number of effects within one Studio file. The only exception is for ringtone specific effects. The TouchSense API mechanism for playing back an effect Timeline with a ringtone on a handset requires that the effect file have only one Timeline. If there is more than one Timeline in the referenced file, only the first Timeline will be played.

Periodic vs MagSweep effects – While MagSweep effects are easier to work with (since they lack the added parameter that allows pulsing), they can limit your creativity. You may want to stick with using only Periodic Basis effects unless you are certain the effect you are creating will remain very smooth (without the need for pulsing or changing frequency parameters).



Sharp vs Strong Effect Styles – Although Haptic Studio offers three Style modes within the Basis effect editing windows, (Strong, Smooth and Sharp), you will likely find yourself only using the Sharp and Strong modes because they are so different from each other. The Smooth mode is a subtle difference from Strong mode that most users will not be able to differentiate.

Periodic Wave Types – Within the Periodic editing window, you have the option of choosing the type of vibration waveform the TouchSense player will simulate. This function is for very high fidelity (like piezo) actuators. Do not choose a Wave Type if you are not designing for piezo actuators, just leave it as it is. But if you are designing for piezo actuators, it is best to use the Sine Wave Type. Sine waves produce strong, yet quiet effects.

Actuator Index – If you are not designing for multi-actuator devices, you will never have to change this. This option would allow you to trigger effects on two (or more) actuators at the same time from a single Timeline.


Additional Resources

Android Developer website: http://developer.android.com/index.html

http://www.immersion.com/developers/index.php?option=com_content&view=article&id=505&Itemid=831


http://developer.android.com/index.html







immr template studio courseware

Technology

simple haptic effects

andwithout haptic effects

timeline haptic studio

audio waveforms haptic

timeline effects inone

createcustom haptic

multiple basis effects

haptic studios midi