creative ai & multimodality: looking ahead
TRANSCRIPT
Creative AI & multimodality:looking aheadRoelof Pieters
@graphificImperial College London,
1 Dec 2015
[email protected]://artificialexperience.com/http://www.csc.kth.se/~roelof/
AICreative
AI
I kinda expect the audience to know AI & Machine Learning Let’s move on shall we ?
AI
All references to:- Arxiv or - GitXiv if the “code” or “dataset” is available
Collaborative Open Computer Sciencemore info (Medium)
AI > today’s focus
AI > today’s focus
“Deep learning is a set of algorithms in machine learning that attempt to learn in multiple levels, corresponding to different levels of abstraction.”
AI > today’s focus
use of several modes (media) to create a single artifact.
Multimodality
“Mode”Socially and culturally shaped resource for making meaning.— Gunther Kress
Creativity
Creativity
• Many definitions: philosophical, sociological, historical, practical
Creativity
1. Making unfamiliar combinations of familiar ideas.
2. Explore a structured conceptual space
3. (Radically) transforming ones structured conceptual space
“Exploration”
“Remix”
“The Creative Mind”— Margaret Boden
“Transformation”
• Skill
• Appreciation
• Imagination
• Learning
• Innovation
• Accountability,
• Subjectivity
• Intentionality.
Creativity > “Traits” software has to exhibit in order to avoid easy criticism of being “non-creative”.
(Simon Colton)
• Skill
• Appreciation
• Imagination
• Learning
• Innovation
• Accountability,
• Subjectivity
• Intentionality
Creativity > software traits
• Skill
• Appreciation
• Imagination
• Learning
• Innovation
• Accountability,
• Subjectivity
• Intentionality
Creativity > software traits
• Skill
• Appreciation
• Imagination
• Learning
• Innovation
• Accountability,
• Subjectivity
• Intentionality
Creativity > software traits
• Skill
• Appreciation
• Imagination
• Learning
• Innovation
• Accountability,
• Subjectivity
• Intentionality
Creativity > software traits
• Skill
• Appreciation
• Imagination
• Learning
• Innovation
• Accountability,
• Subjectivity
• Intentionality
Creativity > software traits
• Skill
• Appreciation
• Imagination
• Learning
• Innovation
• Accountability,
• Subjectivity
• Intentionality
Creativity > software traits
• Skill
• Appreciation
• Imagination
• Learning
• Innovation
• Accountability,
• Subjectivity
• Intentionality
Creativity > software traits
• Skill
• Appreciation
• Imagination
• Learning
• Innovation
• Accountability,
• Subjectivity
• Intentionality
Creativity > software traits
AICreative
Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-Encoders
• Attention-based Models
• Generative Adversarial Nets
Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-Encoders
• Attention-based Models
• Generative Adversarial Nets
Creative AI > Current possibilities > Appropriating “standard” nets for creative use Deep Dream
see also: www.csc.kth.se/~roelof/deepdream/
Creative AI > Current possibilities > Appropriating “standard” nets for creative use Deep Dream
see also: www.csc.kth.se/~roelof/deepdream/ codeyoutubeRoelof Pieters 2015
Creative AI > Current possibilities > Appropriating “standard” nets for creative use Deep Dream
see also: www.csc.kth.se/~roelof/deepdream/
C.M.Kosemen & Roelof Pieters (2015)Gizmodo
Creative AI > Current possibilities > Appropriating “standard” nets for creative use
Leon A. Gatys, Alexander S. Ecker, Matthias Bethge , 2015. A Neural Algorithm of Artistic Style (GitXiv)
Style Net
Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-Encoders
• Attention-based Models
• Generative Adversarial Nets
Creative AI > Current possibilities > Reinforcement Learning
• AMN: Emilio Parisotto, Jimmy Lei Ba, Ruslan Salakhutdinov 2015, Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning (arxiv)
• DQN: Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A., Veness, Joel, Bellemare, Marc G., Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K., Ostrovski, Georg, Petersen, Stig, Beattie, Charles, Sadik, Amir, Antonoglou, Ioannis, King, Helen, Kumaran, Dharshan, Wierstra, Daan, Legg, Shane, and Hassabis, Demis. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
Creative AI > Current possibilities > Reinforcement Learning
Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, Raul Vicente, 2015 Multiagent Cooperation and Competition with Deep Reinforcement Learning (GitXiv)
(YouTube)
Reinforcement Learning
Ning Xie, Hirotaka Hachiya, Masashi Sugiyama, 2013 , Artist Agent: A Reinforcement Learning Approach to Automatic Stroke Generation in Oriental Ink Painting (Paper, Lecture, YouTube)
Ning Xie, Hirotaka Hachiya, Masashi Sugiyama, 2013Artist Agent: A Reinforcement Learning Approach to Automatic Stroke Generation
in Oriental Ink Painting (Paper, Lecture, YouTube)
Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-Encoders
• Attention-based Models
• Generative Adversarial Nets
Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-Encoders
• Attention-based Models
• Generative Adversarial Nets
Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-encoders
• Attention-based Models
• Generative Adversarial Nets
Creative AI > Current possibilities
• Standard (“denoising”) Autoencoders
• Variational Autoencoder (VAE) / Stochastic Gradient VB
• Deep Convolutional Inverse Graphics Network
• Variational RNN (VRNN)
Vincent et al, 2010. Stacked Denoising Autoencoders: Learning Useful Representations ina Deep Network with a Local Denoising Criterion (paper) (code)
Creative AI > Current possibilities
• Standard “denoising” Autoencoders
• Variational Autoencoder (VAE) / Stochastic Gradient VB
• Deep Convolutional Inverse Graphics Network
• Variational RNN (VRNN)
• Diederik P Kingma, Max Welling, 2013. Auto-Encoding Variational Bayes (GitXiv)
Creative AI > Current possibilities
• Standard “denoising” Autoencoders
• Variational Autoencoder (VAE)
• Deep Convolutional Inverse Graphics Network (modified VAE)
• Variational RNN (VRNN)
Tejas D. Kulkarni, Will Whitney, Pushmeet Kohli, Joshua B. Tenenbaum, 2015 Deep Convolutional Inverse Graphics Network (GitXiv)
Creative AI > Current possibilities
• Standard “denoising” Autoencoders
• Variational Autoencoder (VAE)
• Deep Convolutional Inverse Graphics Network
• Variational RNN (VRNN) (VAE at every time step)
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, Yoshua Bengio, 2015 A Recurrent Latent Variable Model for Sequential Data (GitXiv)
VAEVAEVAE
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, Yoshua Bengio , 2015. A Recurrent Latent Variable Model for Sequential Data (GitXiv) (Audio Samples)
Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-Encoders
• Attention-based Models
• Generative Adversarial Nets
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, Daan Wierstra, 2015DRAW: A Recurrent Neural Network For Image Generation (GitXiv)
Variational Auto-Encoder Deep Recurrent Attentive Writer (DRAW) Network
Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-Encoders
• Attention-based Models
• Generative Adverserial Nets
Emily Denton, Soumith Chintala, Arthur Szlam, Rob Fergus, 2015. Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks (GitXiv)
Alec Radford, Luke Metz, Soumith Chintala , 2015. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (GitXiv)
Alec Radford, Luke Metz, Soumith Chintala , 2015. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (GitXiv)
”turn” vector created from four averaged samples of faces looking left vs looking right.
Alec Radford, Luke Metz, Soumith Chintala , 2015. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (GitXiv)
walking through the manifold
top: unmodified samplesbottom: same samples dropping out ”window” filters
Autonomy Supervision
Creativity?- unsupervised training- generator/discrimator- latent/z space- auto encoders- multimodality- query - target/class
Creativity?
Process Result
Creative AI > Needs as I see it
Creative AI as a “tool”
or “brush” to paint with
A system which marries the need for a creative process with the need for a creative output
• with as less human input as possible (data)
• with its own style
• with the possibility for human level supervision for rapid experimentation
Creative AI > a “brush”
A system which marries the need for a creative process with the need for a creative output
• with as less human input as possible ( )
• with its own style
• with the possibility for human level supervision for rapid experimentation
Creative AI > a “brush”
data
Creative AI > a “brush” > data
• reuse nets as much as possible
• combining unsupervised & supervised
• multiple modalities
• plug in external knowledge bases
Creative AI > a “brush” > data input
• unlabeled & labeled data
• external knowledge bases (dbpedia, wikipedia)
• one-shot learning
• zero-shot learning
Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D. Manning, Andrew Y. Ng, 2013 Zero-Shot Learning Through Cross-Modal Transfer
a zero-shot model that can predict both seen and unseen classes
Creative AI > a “brush” > data input
Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D. Manning, Andrew Y. Ng, 2013 Zero-Shot Learning Through Cross-Modal Transfer
(slides)
Creative AI > a “brush” > data input
Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D. Manning, Andrew Y. Ng, 2013 Zero-Shot Learning Through Cross-Modal Transfer
(slides)
Creative AI > a “brush” > data input
Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D. Manning, Andrew Y. Ng, 2013 Zero-Shot Learning Through Cross-Modal Transfer
(slides)
A system which marries the need for a creative process with the need for a creative output
• with as less human input as possible (data)
• with its own style
• with the possibility for human level for rapid experimentation
Creative AI > a “brush”
supervision
Creative AI > a “brush” > data
• “rich” latent (“z”) space
• easy user supervision over output:
• priors
• constrain network (units, layers, etc)
• guided input
• mixed input
• latent space
Creative AI > a “brush” > data
• “rich” latent (“z”) space
• easy user supervision over output:
• priors
• constrain network (units, layers, etc)
• guided input
• mixed input
• latent space
Creative AI > a “brush” > dataDeep Dream
Alexander Mordvintsev, Christopher Olah, Mike Tyka, 2015. Inceptionism: Going Deeper into Neural Networks
Google Research Blog
Creative AI > a “brush” > dataDeep Dream
Roelof Pieters, 2015 DeepDream - Class visualization Experiment (link)
Roelof Pieters, 2015 DeepDream - Class visualization Experiment (link)
Creative AI > a “brush” > data
• “rich” latent (“z”) space
• easy user supervision over output:
• priors
• constrain network (units, layers, etc)
• guided input
• mixed input
• latent space
Creative AI > a “brush” > dataDeep Dream
Roelof Pieters, 2015 DeepDream - Overview of standard bvlc googlenet (inception) layers (link)
Constrain Layers
Creative AI > a “brush” > dataDeep Dream
Roelof Pieters, 2015 Single Unit Activations (early layer) (Flickr Album)
Constrain Units
Creative AI > a “brush” > data
• “rich” latent (“z”) space
• easy user supervision over output:
• priors
• constrain network (units, layers, etc)
• guided input
• mixed input
• latent space
Creative AI > a “brush” > dataDeep Dream
Roelof Pieters, 2015 DeepDream Video (GitHub)
Creative AI > a “brush” > data
• “rich” latent (“z”) space
• easy user supervision over output:
• priors
• constrain network (units, layers, etc)
• guided input
• mixed input
• latent space
Creative AI > a “brush” > dataStyle Net
Roelof Pieters (graphific) (tweet) Roelof Pieters (graphific) (tweet)
Creative AI > a “brush” > data
• “rich” latent (“z”) space
• easy user supervision over output:
• priors
• constrain network (units, layers, etc)
• guided input
• mixed input
• latent space
Image -> Text
“A person riding a motorcycle on a dirt road.”???
Image -> Text
“Two hockey players are fighting over the puck.”???
Image -> Text
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (arxiv) (info) (code)
Andrej Karpathy Li Fei-Fei , 2015. Deep Visual-Semantic Alignments for Generating Image Descriptions (pdf) (info) (code)
Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan , 2015. Show and Tell: A Neural Image Caption Generator (arxiv)
Text -> Image “A stop sign is flying in blue skies.”
“A herd of elephants flying in the blue skies.”
Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, Ruslan Salakhutdinov, 2015. Generating Images from Captions with Attention (arxiv) (examples)
Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, Ruslan Salakhutdinov, 2015. Generating Images from Captions with Attention (arxiv) (examples)
Text -> Image
Subhashini Venugopalan, Marcus Rohrbach, Jeff Donahue, Raymond Mooney, Trevor Darrell, Kate Saenko , 2015. Sequence to Sequence -- Video to Text (GitXiv)
Video -> Text
A system which marries the need for a creative process with the need for a creative output
• with as less human input as possible (data)
• with its own style
• with the possibility for human level supervision for
Creative AI > a “brush”
rapid experimentation
Creative AI > a “brush” > rapid experimentation
Widening
Deepening
Tianqi Chen, Ian Goodfellow, Jonathon Shlens, 2015. Net2Net: Accelerating Learning via Knowledge Transfer (arxiv) / code (torch)
Reusing Nets:
Bigger Net
Teacher and Student net Hint training
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, Yoshua Bengio, 2014. FitNets: Hints for Thin Deep Nets (arxiv)
Knowledge distillation
SVHN Error MNIST Error
Reusing Nets:
Smaller Net
Hashed Net
Wenlin Chen, James T. Wilson, Stephen Tyree, Kilian Q. Weinberger, Yixin Chen, 2015. Compressing Neural Networks with the Hashing Trick (arxiv)
Shrinking Nets:
Hashing
Song Han, Huizi Mao, William J. Dally, 2015. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding (arxiv)
Shrinking Nets:
Pruning, Quantization & Huffman coding
Creative AI > a “brush” > rapid experimentation
• experiments need “tooling”, specialised design software to
• try things
• explore latent spaces (z-space)
• push the AI in the right direction
• be surprised by AI
Creative AI > a “brush” > rapid experimentation
human-machine collaboration
Creative AI > a “brush” > rapid experimentation
(YouTube, Paper)
Creative AI > a “brush” > rapid experimentation
(YouTube, Paper)
Creative AI > a “brush” > rapid experimentation
(Vimeo, Paper)
Creative AI > a “brush” > rapid experimentation
• Advertising and marketing• Architecture• Crafts• Design: product, graphic and fashion design• Film, TV, video, radio and photography• IT, software and computer services• Publishing• Museums, galleries and libraries• Music, performing and visual arts
Questions?
love letters? existential dilemma’s? academic questions? gifts? find me at: www.csc.kth.se/~roelof/