henry lieberman mit media lab ai in end-user software engineering henry lieberman, mit (presenting)...

Henry Lieberman • MIT Media Lab

AI in End-User Software Engineering

Henry Lieberman, MIT (presenting)

Raphael Hoffmann, Michael Toomim, U. Washington,

Seattle

Henry Lieberman • MIT Media Lab

Roles for AI in End-User SoftEng

Supply knowledge end-user doesn't have

Recognize user intent

Plan generation and plan recognition

Automate common patterns of use

Generalize & Specialize

Manage multiple representations

Error checking and sanity checking

Henry Lieberman • MIT Media Lab

U. Washington SoftBots & more

Planning-based software agents

Version-space Programming by Demonstration

Sequence prediction via Logic & Probability

Recommendation from software libraries

Henry Lieberman • MIT Media Lab

SmartEdit - Lau

Henry Lieberman • MIT Media Lab

Version Spaces

Henry Lieberman • MIT Media Lab

Linked Editing

Alternative to content abstraction facilities

Powerpoint Master Slides

Style Sheets

"Symbols" vs. instances in Flash

Web templates, components

User stays within WYSIWYG paradigm

Henry Lieberman • MIT Media Lab

Linked Editing

End-users "underuse" abstractions

Linked Editing

User edits concrete example

Generalizing over content rather than actions

Finds other instances of description

Performs "analogous" edits

Program code, Web pages, Vector graphics, Sheets

Henry Lieberman • MIT Media Lab

Linked Editing

Henry Lieberman • MIT Media Lab

Linked Editing

Henry Lieberman • MIT Media Lab

Assieme - Customization Recommendation

Goal: Recommending scripts for application extensions: Emacs, Firefox

Collect keystrokes, file names, commands

Provide Web service for recommendation

Collab filtering, learning classifier

Henry Lieberman • MIT Media Lab

Assieme - Customization Recommendation

Henry Lieberman • MIT Media Lab

The crazy dream: Programming in English

Programming languages are too difficult for human beings

People can express procedures in English (or other natural language)

Why not program in natural language?

That was the original idea behind Cobol

(whereas Fortran tried to make programming like math)

Henry Lieberman • MIT Media Lab

Childrens’ description of Pac-Man

Henry Lieberman • MIT Media Lab

But what can you do if you can only partially understand English?

Need to make the interaction fail-soft

Rely on dialogue to resolve ambiguities and errors

Maybe don’t eliminate code entirely

Emphasize correspondence between language and code

An "outliner" for code

Application: Programming MOOs

Henry Lieberman • MIT Media Lab

New resources for Common Sense Reasoning

Open Mind Common Sense knowledge base

ConceptNet Semantic Net

MontyLingua/LangUtils

Henry Lieberman • MIT Media Lab

Metafor Interface

Henry Lieberman • MIT Media Lab

Programmatic Semantics

English to “scaffolding” code

Roughly,

Turns nouns into classes and/or instances;

Verbs into functions

Adjectives into instance variables

Resolves anaphora

Untangles conditional and loops

Propagates context

Henry Lieberman • MIT Media Lab

Narrative stances

I want to make a bar with a customer. (1st person programmer)

There is a customer in the bar. (3rd person objective)

I am a customer sitting on a stool. (1st person program)

The bartender said, “Here is a customer” (3rd person subjective)

Henry Lieberman • MIT Media Lab

Refactoring

There is a bar. (atom)The bar contains two customers. (unimorphic list)It also contains a waiter. (unimorphic wrt. persons)It also contains some stools. (polymorphic list)The bar opens and closes. (class / agent)The bar is a kind of store. (inheritance class)Some bars close at 6pm. (subclass or instantiatable)

Henry Lieberman • MIT Media Lab

Shameless Plug