multimedia and the future of knowledge - hp labs · • opportunities and responsibilities of...

© Copyright 2015 HP Inc. The information contained herein is subject to change without notice.

Multimedia and the Future of KnowledgeBay Area Multi-Media Forum (BAMMF)5 November 2015

Steven SimskeHP Labs, Fort Collins CO USA

© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Abstract• Information from analog to digital and from linear to non-linear• Multimedia and electronic publishing are on the forefront of the

analog to digital conversion of the world’s information• Our communications, our finances, our social interactions, our

education, and effectively our entire culture is now online• Threats to privacy, security and data integrity• Threats to the human mind• Media for knowledge and thinking and society affect the way

people think, and the way society functions• Opportunities and responsibilities of multimedia experts to craft

the past, present and future of content in a way which both preserves its value, and allows its value to be appreciated by people of different age, gender, culture and purpose

The original multi-media (text+graphics) in a modern multimedia (app for text and graphics)

© Copyright 2015 HP Inc. The information contained herein is subject to change without notice.3

Ancient Culture was Multi-Media

http://www.moellerhaus.com/rosetta/rosetta.htm

http://ancientworldwonders.com/the-hanging-gardens-of-babylon.html

http://www.timetrips.co.uk/hieroglyphs_written_on.htm

The Rosetta Stone combined three languages—the spoken and written multiple media

The Hanging Gardens of Babylon combined the visual with the olfactory and perhaps the auditory (waterfalls)—sensory multimedia

Ancient sarcophaguses included hieroglyphics (written medium) and representative artifacts (including olfactory, gustatory and somatosensory media)

http://www.moellerhaus.com/rosetta/rosetta.htm

http://ancientworldwonders.com/the-hanging-gardens-of-babylon.html

http://www.timetrips.co.uk/hieroglyphs_written_on.htm


Medieval Culture was Multi-Media

http://ireland-calling.com/book-of-kells/, https://www.flickr.com/photos/aplumb/121285772

http://www.ancient-origins.net/ancient-places-africa-history-important-events/destruction-great-library-alexandria-001644

With the invention of long-lasting dyes, books advanced from grayscale to rich, multi-colored palimpsests and vellum

The Gutenberg Press made cultural preservation much easier. It also democratized the creation of mass multi-media. The distribution problem remained.

Cultural preservation:Backing up data was a manual, slow, inefficient and error-ridden process

http://ireland-calling.com/book-of-kells/

https://www.flickr.com/photos/aplumb/121285772

http://www.ancient-origins.net/ancient-places-africa-history-important-events/destruction-great-library-alexandria-001644


Renaissance & Enlightenment Culture was Multi-Media

http://www.orpheon.org/OldSite/Seiten/shop/sale.htm

http://movingclassics.tv/tv/classical-music-and-salon-culture/

Salons were used to illustrate the gamut of media of the modern mind:Paintings, musical instruments, musical scores, vivid dyes for clothing, wallpaper (invented 1785). Everything was still mechanical only

In the enlightenment, the variety of sounds exploded:Trombone c. 1500Violin, viola, cello, c. 1530Oboe c. 1655Pianoforte c. 1700Clarinet c. 1740French horn 1753Saxophone 1846




Early Modern Culture was Multi-Media

https://en.wikipedia.org/wiki/Georges_M%C3%A9li%C3%A8s

http://phonojack.com/Edison%20Collection.htm

Electricity changed everything. Media and culture became powered

With power came the ability to combine sensory modalities—sounds and image, for example

The Industrial Revolution allowed simultaneous hybridization, miniaturization and specialization of functionality… a multiple of multi-media!




A Short Dictionary of Multimedia ExamplesThe media primitives:1. Text2. Graphics3. Photo4. Music (Audio)

The multi-media derivatives:1. Documents (Text, Graphics, Photo)2. Movies (Audio, Photo)3. Web Pages (Text, Audio, Graphics, Photo, Video)4. Video (Audio, Photo)

Multi-media are any combinations of two or more primary media. The relative value of a given multi-media is tied to its function. If it is to entertain, it should be tied to the sensory preferences of the customer. If it is to educate, it should be tied to the learning preferences of the student.


The Transition from Analog to Digital

The past several decades have focused on the transition from analog to digital. This has happened in all of the following media:

1. Newspapers – from the NY Times to USA Today to cnn.com and commondreams.org2. Radio – from AM/FM to live streaming3. Printing – from gravure and flexo to inkjet, laserjet and digital presses4. Photography – from silver halide to digital cameras5. Copiers – from Xeroxing to faxing to digital sending6. Manufacturing – from assembly line to Legos to 3D printing/additive manufacturing

The advantages of digital are many—one digital design can be reproduced ad infinitum


Going Non-linear

For the majority of human history, culture and cultural events have been linear:1. Novels2. Symphonies3. Course curricula

The digital universe allows for nearly limitless connections between data, and so we have innately non-linear culture and cultural events:1. Browsing2. Interactive documents3. Mash-ups

Everything is clickable, swipe-able, and connected


What do the Media Represent?

Media appeal to one or more senses

1. Vision came first (text, drawings, photos)2. Auditory came next (music)3. Somatosensory (haptics)4. Hybrid (scratch and sniff, video, movies, etc.)

All media appeal to one or more senses. And all are in one way or another an abstraction of the underlying data. We de-reference at various levels depending on the immediacy of the media.

http://charliebroadway.blogspot.com/2013/03/contra-atheismum-part-5-tower-of-babel.html

Transitioning from the brain as a library (holder of facts) to the brain as a browser (holder of references)

http://charliebroadway.blogspot.com/2013/03/contra-atheismum-part-5-tower-of-babel.html


Culture may Change, but not the Senses that Enjoy Them

The earliest cave art was viewed by eyes and interpreted by brains not structurally different from the ones we have today

Ancient hominids, however, generally did not saturate their senses—they had bandwidth in reserve which certainly connected them more to their environment

Their television was hunting, dancing, observing nature

Their music was communal

Their vision was light-limited

Their Internet was their shared experiences

We share common means for culture, but not necessarily common culture…


What is On-Line?The real question is no longer “what is on-line”—

it is “what’s not on-line”?

Do digitization efforts really preserve linear and analog (pre-Internet) multi-media sufficiently?

Should they?

What are the deficiencies of multi-media? Have they changed during the Internet era?

Every technology begins with specifications being the most important comparison—when experience becomes the key, how do we measure?


Bringing Multiple Media Together

Culture is “the arts and other manifestations of human intellectual achievement regarded collectively”, or “the beliefs, customs, arts, etc., of a particular society, group, place, or time”. Key to these definitions is the plurality. Culture is a summation of the arts, or by definition multi-media.


What is the Brain for?

Brain as multiplexer

Brain as integrator

Brain as the original Big Data

Brain as temporal cloud of our past experiences

But now…Brain as indexing systemBrain as filterBrain as content switch

How significant is this change? How important?Compare/contrast the brain reaction…


Media and the Brain

How are the multi-media specifications and experiences chosen?

Specifications: generally based on the filtering model of human sensory (peripheral nervous system) and interpretative (central nervous system) capabilities.

Once bandwidth is saturated, the processing (feelings, associations, etc.) and the storage (how much can be conveyed and how quickly) are emphasized.

Does the brain change the media or do the media change the brain? What if we dropped a caveman in front of a television (a caveman other than your Uncle Phil that is)?

What would I gain if I could (a) smell this scene, (b) touch this scene, (c) hear this scene, (d) turn this scene into video, (e) augment this scene temporally?


The Cost of Distraction

Improper associations (links) stereotyping

Self-selecting from a sea of content self-justification

Where is the knowledge? Do we remember the source? How is the reference validated?

What do we really know versus “feel”?

At what point is everything just noise?

It’s not just driving (although technology continues to change significantly there)…e.g. 100 bikers hit by cars each year in Fort Collins


How do People First Learn of Cultural Items?From Classical to Kitsch

A quick survey

Beethoven’s 9th…the movie Get Smart

Hamlet…YouTube

Immanuel Kant…Wikipedia

Cuisine…Siri

Linear culture has been “sound bitten” and in the future it will be harder and harder to discern what the actual value of a cultural icon is. The on-ramp, or introduction, has changed for everything. Has the meaning also changed?

This R2-D2 appeared at the Loveland, CO, Mini Maker Faire in 2014 (assuming no royalties were paid!)


The Future

http://www.redbubble.com/people/icefaerie/works/6929122-celtic-ouroboros?p=sticker

For the majority of human history, culture and cultural events have been linear

That is not the case anymore

Multi-media is pervasive, and invasive:EducationReligionTransportationDiningGoverning

Everything is inter-connectedMedia are interrelated and referential

http://www.redbubble.com/people/icefaerie/works/6929122-celtic-ouroboros?p=sticker


The Realities

Technology is no longer processor-limited

Technology is no longer storage-limited

Technology is no longer truly bandwidth-limited (though more so than the above)

Technology is time-limited…the focus is on the now

Do we already have an orphaned history (1990-2000, for example, when the transition from pre-Internet processes to Internet processes occurred)?

Is the Way Back Machine sufficient?


How do We Enrich Multi-Media?Extraction is good…

But variances add

Every image processing task is noisy, lossy and imperfect

Metadata is more precise and provides only a single source of variance, but…

It is expensive

How do we get the creators of multi-media to build it to last?


Everything We Hold Dear is On-Line…and that is a very tenuous line

• Our entertainment• Our communications• Our finances• Our social interactions• Our dictionaries• Our encyclopedias• Our education• Book burning is rife again• Effectively, our entire culture is now on-line• Who ultimately decides what is truth?• The threats of self-selection, convenience, narcissism, and triviality


Threats to Privacy, Security and Data Integrity

• We can’t keep the internet locked up inside a watertight oratory with only one way in and one way out

• Threats can migrate and the attack surface is the entire planet• Open Source mitigates some problems• But how do you know what to trust?• Who is the ultimate authority on the world’s data?

• What if the Internet was destroyed tomorrow? Could you…(a) Pay your bills?(b) Purchase your food?(c) Get the news?(d) Phone your friends?

We are putting all of our media (newspaper, television, phone) together with the possibility of a single critical-point failure…should we?

Beware of Doug courtesy of the “Far Side” by Gary Larson


Media and Culture, and Threats to the Human Mind

• Media for knowledge affect the meme—the way in which a person, place, or event is culturally understood and referenced.

• Media for thinking affects the balance between an internal versus referential record.

• Media for society affects our development—think of the age at which you first saw adult situations on television and compare to your parents and your children?

• By allowing children to be in adult situations earlier and earlier, what do they lose? Should the creators of multi-media be thinking this through?

• Media IS our culture, are we taking it seriously enough?

Our culture is precarious….


The ResponsibilitiesMulti-Media as Universal Acid: Once We Realize the Importance of Multi-Media, We Realize Its Pervasiveness and Its Power

• Opportunities and responsibilities of multimedia experts to craft the past, present and future of content in a way which both preserves its value, and allows its value to be appreciated by people of different age, gender, culture and purpose—what are the standards?

• We need to rate the validity of media• We need to reinforce the knowledge• We need time stamps and validation• We have to come together and realize the crucial role of

tagging and flagging• It is “time” for a maturation in multi-media• Multi-media IS our culture—we need to act accordingly

We know this isn’t real, but only from our own context….

http://urbanlegends.about.com/library/bl_bush_gone_fishing.htm

http://urbanlegends.about.com/library/bl_bush_gone_fishing.htm