MM-in-CSEDMM-in-CSED 04/19/2304/19/23 11

Teaching Teaching Multimedia Multimedia

Software Software DevelopmentDevelopment(among other (among other

topics)topics)

UCSB Center for UCSB Center for Research in Electronic Research in Electronic

Art TechnologyArt Technology

Stephen Travis Popestp@create.ucsb.edu

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 22

OutlineOutline Questions Problems More Questions My Experience at UCSB/MAT Examples: recent student work @ UCSB

Some advice and a few conclusions …and yes, a Squeak pitch after all

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 33

What are the Issues?What are the Issues? Are we using MM to teach CS or

teaching MM software development?

RFP for MM-in-CSED: integrating MM assignments in CS education, “modern software like Squeak makes multimedia approachable and understandable”

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 44

What’s the Focus?What’s the Focus? Teaching programming and procedural

thinking -- low-level languages Teaching OO analysis/design --

knowledge- domain-modeling Teaching IDE and CASE tool usage Teaching real-world job (“vocational”) skills Preparing students for a comprehensive

CS curriculum Teaching non-CS-majors to build domain-

specific languages and SW tools

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 55

Teaching Software DesignTeaching Software Design In the traditional CS curriculum,

methodology, CASE/CM, IDE tools, and A/D are taught after programming(!)

In ST curricula, A/D is taught first.

What does the CS freshman need?

What does the MM artist/technician need?

What do other knowledge workers (science/humanities/arts) need?

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 66

Teaching Teaching Real-WorldReal-World

ToolsTools ST as a niche language…

UofW experience with ADA (1990s)

The ubiquity of C, C++, Java (C*)

The ubiquity of file/project-based

IDEs

The matter of principle (buck or go)

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 77

The Role of The Role of

Smalltalk/SqueakSmalltalk/Squeak Perfect OOA/D teaching platform

Simple concepts and syntax

Low-overhead IDE (?)

Platform-independent

Libraries/APIs for many application

domains, including MM

Not “mainstream,” however...

Music: Gates Still Open (2000-02)

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 88

Smalltalk as a First Smalltalk as a First

LanguageLanguage

My experience with teaching ST to many groups

Non-programmers have an easier time than programmers

Programmers get frustrated by the A/D focus and the IDE

This is good news!

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 99

Multiple Planes and Paradigm Multiple Planes and Paradigm ShiftsShifts

The two technologies that have the most profound impact on musical practice in the last 100 years were both invented before 1920: recording and broadcasting

“The impact of digital technology on music has been as shallow as it is broad” (F. R. Moore)

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 1010

Generations of Computer Generations of Computer TechnologyTechnology

Mainframes Mini-computers Engineering workstations PCs The Internet The Next Big Things: Distributed Systems Immersive user interfaces Multimedia integration

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 1111

Dimensions of Moore’s LawDimensions of Moore’s Law(With thanks to L. Rabiner)

log performance

time

RAM density(doubles every 18 months)

Disk access time

Disk throughput

LAN speed

WAN speed

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 1212

CS Education and CS Education and “vocalitionalism”“vocalitionalism”

Preparing CS majors to build the next-generation systems”• WA Distributed• Media rich, immersive UIs• Thin/wireless clients• Parallel or thread-based simulations• Non-sequential: ES, GA, NN, CSP, IFS, …• DSCP (distributed, sensing,

computation, and presentation)• Model-based “biology in the computer”

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 1313

Good NewsGood News

We know what to do (see below)

E.g., GaTech CS2340• Course on design for novices

• Build interactive maps

• Team-focus

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 1414

CS Language OptionsCS Language Options Scheme/CommonLISP ML/Haskell ES Shells Uniformly high-level Java

• OOA/D focus• Use of reflection• Still quite painful

Various Smalltalks

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 1515

CS @ UCSBCS @ UCSB Java, Scheme, C on UNIX/Linux Several integrated labs

• PSL, SCC, Clusters Wireless infrastructure Media classrooms VE centers Integration with application domains: MAT,

CREATE, dStudio, eStudio, BioInformatics, CompChem, MaterialsIT, CNSI, CITS

Dual appointments

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 1616

The UCSB MAT Program The UCSB MAT Program Two-year interdisciplinary Master’s degree

program in Media Arts and Technology (MAT) Partner depts: Music, Art, CS, EE Five core courses, electives, thesis project;

optional internship, independent study, RA-ship Emphasis on group projects Three areas of specialization:

• Multimedia engineering• Electronic music and sound design• Visual and spatial arts

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 1717

MAT StudentsMAT Students Generally have BA/BS in

• Music composition or performance• Visual arts• Computer science• Electrical engineering• Recent exceptions include architecture,

design, mechanical engr., film studies, etc.

• “Crossovers” possible (several already)

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 1818

MAT CoursesMAT Courses Five Core Courses

• History/literature of electronic music• Digital media art and culture• Multimedia networking• Digital media signal processing• Survey of media technology and

engineering Elective courses taken from the four

partner departments and related fields

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 1919

New MAT ElectivesNew MAT Electives Digital audio programming

sequence (six quarters) Maya sequence (three quarters) Camera tracking and user input

(project-oriented) Sound/music programming in

SuperCollider New seminar series in each of the

partner departments

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 2020

What to teach in MAT?What to teach in MAT? CS/EE students

• Already know C*• Let them eat Squeak! (and SC, and

others) Arts students

• Some (though not many) know no programming language at all

• Some know at least a scripting language (MEL, Lingo, etc.)

• Some know C*• Teach them Java (+2D, 3D, Sound, etc.)

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 2121

Example: MAT 240a-fExample: MAT 240a-f Digital audio programming: the series

Six quarters on various topics: using commercial APIs, frequency-domain processing, spatial audio and surround sound, synthesis techniques, multi-rate control and processing, etc.

Use Linux, Windows, and Mac platforms

Mainly in C* (with ObjC, SC, ST, etc.)

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 2222

Example: MAT 220Example: MAT 220

Sound/music system programming Focus on complex sound design,

interactive performance, and compositional algorithms

Students are mostly undergrad- or grad-level composers

Use SuperCollider (a dialect of ST) on Macs

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 2323

Example: MAT 41Example: MAT 41

Beginning (minimal)

programming skills for non-

programmers in MAT

(Not part of a comprehensive CS

curriculum)

Intro to Java

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 2424

Example: MAT 596Example: MAT 596

Independent R&D

Several ST/SC projects (Siren,

OSC, grammars, performance

rules, CSP, NN, etc.)

Lots of distributed projects

(CORBA, OSC, CO, etc.)

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 2525

So, why this way?So, why this way?

The “mainstream” issue

The learning curve issue

≥ Two languages needed

Squeak vs. other ST platforms

C* APIs: 2D, 3D graphics, spatial sound, CORBA

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 2626

An Answer!An Answer!

Teach ST as a “first” language (assuming most frosh know C*)

Integrate MM applications from day one (see below)

Stress methods, CASE, A/D, tools

Segue as needed to C*

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 2727

Examples: MAT Student Examples: MAT Student

ProjectsProjects Multimedia data/signal processing Robotics Multimedia databases Hardware input devices Virtual environments Advanced GUIs Work in Smalltalk, MATLab, Java,

C++, SuperCollider, Lingo, MEL

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 2828

RKRobot (Gary Thomas)RKRobot (Gary Thomas)

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 2929

Woon-gramsWoon-grams

QuickTime™ and aMotion JPEG A decompressor

are needed to see this picture.

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 3030

Matrix Controller (Dan Matrix Controller (Dan

Overholt)Overholt)

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 3131

Steganography Steganography (Gilroy (Gilroy Menezes)Menezes)

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 3232

Obsessive Obsessive RemixerRemixer (C. (C. Ramakrishnan)Ramakrishnan)

Sound example(Autumn Leaves)

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 3333

Smart TextShredderSmart TextShredder (Brian (Brian O’Reilly)O’Reilly)

public class TextShredder{ public static void main(String args[])

throws IOException, FileNotFoundException {System.out.println("Starting Shredder!");

// Instantiate shredder/cut up objectsWordBank wb =new WordBank();Jumble j = new Jumble();StringMover mv = new StringMover();

…

Output:I was also a hunter of birds In this, the share of reaction to family

members who want to live outside of a life of toil is great I believe this has been touched upon in a novel as a "first rebellion" by an author from Turkey As long as there was no provocation, animosity would never develop and an exemplary brotherly coexistence prevailed....

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 3434

LLCH Models (Brent LLCH Models (Brent Yokota)Yokota)

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 3535

CREATE Auralizer CREATE Auralizer (240C)(240C)

Sources, geometry, processing, output Current: Stereo, 4 KLOC, selectable cues

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 3636

Creatophone Installations in Creatophone Installations in LLCHLLCH

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 3737

The ATON VE The ATON VE (group)(group)

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 3838

DRIVE DRIVE UserUser

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 3939

DSCP Example: DSCP Example: Sensing/Speaking SpaceSensing/Speaking Space

Multimedia Installation with camera tracking of visitors controlling video and sound synthesis and spatialization

InteractionInteractionSpaceSpace

VideoVideoAnalysisAnalysis

SoundSoundSpatializationSpatialization

VideoVideoSynthesisSynthesis

SoundSoundSynthesisSynthesis

CameraCamera

ScreenScreen LoudspeakersLoudspeakers

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 4040

Sensing/Speaking SpaceSensing/Speaking Space at at SFMoMASFMoMA

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 4141

Opera Browser (Pierre Roy Opera Browser (Pierre Roy et al.)et al.)

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 4242

Squeak Tools and Squeak Tools and ApplicationsApplications

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 4343

Siren Siren Composer’s DB Composer’s DB & Tools& Tools

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 4444

Siren Speech SegmenterSiren Speech Segmenter

I CE M E L T S I CE M E L T S

Music: Gates Still Open (2000-02)

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 4545

Analysis for Music/Sound Analysis for Music/Sound DBsDBs

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 4646

DPE in DPE in

VWNCVWNC

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 4747

PulsarGenerator GUI in PulsarGenerator GUI in SCSC

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 4848

CS CurriculumCS Curriculum

Currently: ≥ two languages are still needed

Teach ST first• OOA/D, CASE, methods, CM, tools

focus

Use C* as appropriate• R-T, OS, networking courses (I.e., low-

level C)

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 4949

(Current) MM Curriculum(Current) MM Curriculum

One language: Java or ST Teach Java first

• APIs, portability, apps, level of documentation

• Real-world-proximity

Use ST for A/D, CASE, etc.• Beyond Q1

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 5050

CS for Knowledge MajorsCS for Knowledge Majors

Teach modeling, simulation,

media integration, application

composition

Provide low-overhead development

and distribution tools

Duh! -- Squeak

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 5151

The Squeak Pitch ThingyThe Squeak Pitch Thingy Language syntax (blocks as objects) 4th-generation class libraries “Rapid turn around” compilers Workspace-based system with

multiple desktops/projects IDE tools, SCCS, CM, etc. Many compatible implementations SqueakOS Object/costume framework “Stuff is easy” “Parallel thinking”

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 5252

ConclusionsConclusions It’s just not a perfect world ST has strengths and weaknesses C* have strengths and weaknesses Teaching CS involves a lot of different

topics at several levels We also need to teach some subset of

these skills to non-CS majors We can design innovative and effective

curricula for several target audiences

MM-in-CSEDMM-in-CSED 04/19/2304/19/23 5353