program synthesis with types peter-michael osera posera posera@cis.upenn.edu

Program Synthesis with TypesPeter-Michael Osera

http://www.cis.upenn.edu/~poseraposera@cis.upenn.edu

Programming is hard!

[3, 5, 2]

[3, 3, 5, 5, 2, 2]

?

[3, 5, 2]

[3, 3, 5, 5, 2, 2]

public static List<Integer> stutter(List<Integer> l) { List<Integer> ret = new LinkedList<Integer>(); for (int i = 0; i < l.size(); i++) { ret.add(l.get(i)); ret.add(l.get(i)); } return ret;}

Program SynthesisThe automatic generation of

programs from specification

(Input from sensors)

Brake (or not)

?

Correctness by Construction

PacemakersBanking SystemsMissile LaunchersAvionicsInsulin InjectorsLaser Eye Surgery MachinesCompilers and OptimizersDrug Delivery SystemsFinance and Trading SoftwareSilicon DesignsLunar Rover Navigation SystemsInfrastructure SoftwarePassword DatabasesCritical Runtime SystemsConference Management SystemsOS KernelsMedical Record DatabasesLife Support RegulatorsNuclear Reactor Management SystemsSupply Chain Management SystemsOnline Payment SystemsNetwork ProtocolsRocket ShipsMRI MachinesAcademic Record Databases

00101101010...

00010110...

?

Program Assistance

Bitsketching[Solar-Lezama

2005]

?

Broadening Participation

Flashfill [Gulwani

2011]

// ...List<Integer> ret = new LinkedList<Integer>();for (int i = 0; i < l.size(); i++) { ret.add(l.get(i)); ret.add(l.get(i));}return ret;// ...

// ... List<Integer> ret = new LinkedList<Integer>();for (int i : l) { ret.add(i); ret.add(i);}return ret;// ...

public static List<Integer>stutter(List<Integer> l) { return l; }

public static List<Integer>stutter(List<Integer> l) { return 5; }

// ... List<Integer> ret = new LinkedList<Integer>();for (int i = 0; i < l.size(); i++) { int x = l.get(i); ret.add(x); ret.add(x); i = 0;}return ret;// ...

Specification(i.e., “intent”)

Logical Specification

Input-output Examples[3, 5, 2] -> [3, 3, 5, 5, 2, 2]

Complete, hard to specify

Incomplete, easy to specify

Version space algebras, genetic prog. (Maching learning)[Lau 2000, Weimer 2009]

SMT solvers (Formal methods)[Solar-Lezama 2008, Kneuss 2013,

Torlak 2014]

Satisfiability Modulo Theories

Brute-force search (AI/logic)[Green 1969, Summers 1976,

Kitzelmann 2010, Albarghouthi 2013]

• Type theory,• Proof theory, and• Functional

programming[Osera 2015]

Functional Programming Logic

2. Exploit the inherent structure of typed, FPs

1. Synthesize programs using higher-order data(lambda x: x + 1)

Structure and Synthesis

Why does functional programming matter? [Huges 1990]

??

?? ?? ??

Pattern Matching

Higher-order Functions

“This is the key to functional programming’s power — it

allows improved modularization.”

Structure and Synthesis

Why does functional programming matter for synthesis?

??

?? ?? ??

Independent Reasoning

Restricted Search Space

Modularity simple, tractable synthesis techniques

1 2 3 4 5 6 7 8 9 10 111

10

100

1000

10000

100000

1000000

10000000

100000000

1000000000

10000000000

Untyped Programs Typed ProgramsProgram Size (AST nodes)

Nu

mb

er

of

Pro

gra

ms

Why Types?1,949,031,274

Untyped Programs

201,998Typed

Programs

21,704Typed, Normal

Programs

stutter

How do we write functional programs?

How do we synthesize functional programs?

Program Synthesis With Types

stutter = ?

type list =| Nil| Cons of int * list

Algebraic Datatype

Constructors

Constructor ArgumentsExample values of type

listNil

Cons (0, Nil)

Cons (1, Nil)

Cons (0, Cons (0, Nil))

Nil

Nil0

Nil1

Nil00

stutter : list -> list

type list =| Nil| Cons of int * list

Example values of type listNil

Cons (0, Nil)

Cons (1, Nil)

Cons (0, Cons (0, Nil))

[]

[0]

[1]

[0, 0]

stutter takes a list and produces a list

stutter : list -> list

: list -> list

Context:

Available Variables

stutter : list -> list

let rec stutter (l:list) : list = : list

Context:l : list

stutter : list -> list

stutter : list -> list

let rec stutter (l:list) : list = l

Context:l : list

stutter : list -> list

[3, 5, 2]

[3, 5, 2]

[3, 3, 5, 5, 2, 2]

stutter : list -> list

let rec stutter (l:list) : list = stutter l

Context:l : list

stutter : list -> list

[3, 5, 2]

[3, 3, 5, 5, 2, 2]

Loop!

stutter : list -> list

let rec stutter (l:list) : list = match l with | Nil -> : list | Cons (x, l’) -> : list

Context:l : list

stutter : list -> list

stutter : list -> list

let rec stutter (l:list) : list = match l with | Nil -> l | Cons (x, l’) -> : list

Context:l : list

stutter : list -> list

[]

[]

[]

stutter : list -> list

let rec stutter (l:list) : list = match l with | Nil -> [] | Cons (x, l’) -> : list

Context:l : list

stutter : list -> list

stutter : list -> list

let rec stutter (l:list) : list = match l with | Nil -> [] | Cons (x, l’) -> : list

Context:l : list

stutter : list -> list

x : int

l’ : list

[3, 5, 2]

Cons (3, Cons (5, Cons 2, Nil)))

stutter : list -> list

let rec stutter (l:list) : list = match l with | Nil -> [] | Cons (x, l’) -> : list

Context:l : list

stutter : list -> list

x : int

l’ : list

[3, 5, 2]

3 [5, 2]

Cons (x, Cons (x, )) stutter(l’)

[3, 3, 5, 5, 2, 2]

stutterCons (x2)

stutter : list -> list

let rec stutter (l:list) : list = match l with | Nil -> [] | Cons (x, l’) -> Cons (x, Cons (x, stutter l’))

kambing-mobile:synml posera$ ./synml.native tests/natlist/stutter.mllet stutter (l1:list) : list = match l1 with | Nil -> [] | Cons (n1, l2) -> Cons (n1, Cons (n1, stutter l2))

Type-directed Program Synthesiswith Examples

1. Refine program (and examples) by type

2. Guess a variable or function application

3. Learn more by pattern matching

stutter : list -> list

: list -> list

Goal Examples:[] => []

[0] => [0, 0]

[1, 0] => [1, 1, 0, 0]

Context

(Refine by type)

stutter : list -> list

let rec stutter (l:list) : list = : list

Goal Examples:[] => []

[0] => [0, 0]

[1, 0] => [1, 1, 0, 0]

Context

(Refine by type)

stutter : list -> list

let rec stutter (l:list) : list = : list

Goal Examples:[]

[0, 0]

[1, 1, 0, 0]

Contextl=[]

l=[0]

l=[1, 0]

(Guess a variable or application)

Example “World”

stutter : list -> list

let rec stutter (l:list) : list = l

Goal Examples:[]

[0, 0]

[1, 1, 0, 0]

Contextl=[]

l=[0]

l=[1, 0]

(Guess a variable or application)

1. Generate an expression*

2. Evaluate and check against each example world*We don’t generate stutter l – syntactic restriction on recursive

calls.

stutter : list -> list

let rec stutter (l:list) : list =

Goal Examples:[]

[0, 0]

[1, 1, 0, 0]

Contextl=[]

l=[0]

l=[1, 0]

(Learn more by pattern matching)let rec stutter (l:list) : list =

match l with | Nil -> : list | Cons (x, l’) -> : list

stutter : list -> list

let rec stutter (l:list) : list =

Goal Examples:[]

Contextl=[]

(Refine by type)

let rec stutter (l:list) : list = match l with | Nil -> : list | Cons (x, l’) -> : list

stutter : list -> list

let rec stutter (l:list) : list =

Goal Examples:[]

Contextl=[]

(Refine by type)

let rec stutter (l:list) : list = match l with | Nil -> Nil | Cons (x, l’) -> : list

stutter : list -> list

let rec stutter (l:list) : list =

Goal Examples:[0, 0]

[1, 1, 0, 0]

Contextl=[0], ...

l=[1, 0], ...

let rec stutter (l:list) : list = match l with | Nil -> Nil | Cons (x, l’) -> : list

stutter : list -> list

let rec stutter (l:list) : list =

Goal Examples:[0, 0]

[1, 1, 0, 0]

Contextl=[0], x=0, l’=[]

l=[1, 0], x=1, l’=[0]

let rec stutter (l:list) : list = match l with | Nil -> Nil | Cons (x, l’) -> : list

stutter : list -> list

let rec stutter (l:list) : list =

Goal Examples:Cons (0, Cons (0, ...))

Cons (1, Cons (1, ...))

Contextl=[0], x=0, l’=[]

l=[1, 0], x=1, l’=[0]

(Refine by type + guess a variable x2)let rec stutter (l:list) : list =

match l with | Nil -> Nil | Cons (x, l’) -> : list

Cons (x, Cons (x, ...))

stutter : list -> list

let rec stutter (l:list) : list =

Goal Examples:[]

[0, 0]

Contextl=[0], x=0, l’=[]

l=[1, 0], x=1, l’=[0]

(Guess a variable or application)let rec stutter (l:list) : list =

match l with | Nil -> Nil | Cons (x, l’) -> Cons (x, Cons (x, : list))

want: stutter l’

try: l, l’, ...

stutter : list -> list

let rec stutter (l:list) : list =

Goal Examples:[]

[0, 0]

Contextl=[0], x=0, l’=[]

l=[1, 0], x=1, l’=[0]

(Guess a variable or application)let rec stutter (l:list) : list =

match l with | Nil -> Nil | Cons (x, l’) -> Cons (x, Cons (x, stutter l’))

stutter=( [] => [] | [0] => [0, 0] | [1, 0] => [1, 1, 0, 0] )

stutter : list -> list

let rec stutter (l:list) : list =

Goal Examples:[]

[0, 0]

Contextl=[0], x=0, l’=[]

l=[1, 0], x=1, l’=[0]

(Guess a variable or application)let rec stutter (l:list) : list =

match l with | Nil -> Nil | Cons (x, l’) -> Cons (x, Cons (x, stutter l’))

stutter [] = []

stutter=( [] => [] | [0] => [0, 0] | [1, 0] => [1, 1, 0, 0] )

stutter : list -> list

let rec stutter (l:list) : list =

Goal Examples:[]

[0, 0]

Contextl=[0], x=0, l’=[]

l=[1, 0], x=1, l’=[0]

(Guess a variable or application)let rec stutter (l:list) : list =

match l with | Nil -> Nil | Cons (x, l’) -> Cons (x, Cons (x, stutter l’))

stutter [0] = [0, 0]

stutter=( [] => [] | [0] => [0, 0] | [1, 0] => [1, 1, 0, 0] )

stutter : list -> list

let rec stutter (l:list) : list =

Goal Examples:[]

[0, 0]

Contextl=[0], x=0, l’=[]

l=[1, 0], x=1, l’=[0]

let rec stutter (l:list) : list = match l with | Nil -> Nil | Cons (x, l’) -> Cons (x, Cons (x, stutter l’))

stutter=( [] => [] | [0] => [0, 0] | [1, 0] => [1, 1, 0, 0] )

1. Type Refinement

2. Guessing3. Pattern

Matching

...

Program Synthesis With Types

P -> P

x:P|-P

x:P|-P

x:P, n:Qx’:P|-P

x:P, n:Qx’:P|-P

x:P|-P

(Case Analysis

on x)

(Curry-Howard Isomorphism)

Synthesis=

Proof Search+

Example Refinement

λsyn, a logical foundation for program synthesis

Myth, a program synthesizer for typed, functional programs (OCaml)

let map : (nat -> nat) -> list -> list = fun (f2:nat -> nat) -> let rec f3 (l1:list) : list = match l1 with | Nil -> [] | Cons (n1, l2) -> Cons (f2 n1, f3 l2) in f3;;

real 0m0.040suser 0m0.023ssys 0m0.012s

map: higher-order functions

f2 is a function value!

let fvs_large : exp -> list = let rec f1 (e1:exp) : list = match e1 with | Unit -> [] | BVar n1 -> [n1] | FVar n1 -> [] | Lam (n1, e2) -> f1 e2 | App (e2, e3) -> append (f1 e2) (f1 e3) | Pair (e2, e3) -> append (f1 e2) (f1 e3) | Fst e2 -> f1 e2 | Snd e2 -> f1 e2 | Inl e2 -> f1 e2 | Inr e2 -> f1 e2 | Match (e2, n1, e3, n2, e4) -> (match f1 e2 with | Nil -> append (f1 e4) (f1 e3) | Cons (n3, l1) -> Cons (n3, append (f1 e3) (f1 e4))) | Const n1 -> [] | Binop (e2, b1, e3) -> append (f1 e3) (f1 e2) in f1;;

real 0m3.282suser 0m2.848ssys 0m0.413s

fvs_large: really big functions

75 AST nodes!(3.28s)

let pairwise_swap : list -> list = let rec f1 (l1:list) : list = match l1 with | Nil -> [] | Cons (n1, l2) -> (match f1 l2 with | Nil -> (match l2 with | Nil -> [] | Cons (n2, l3) -> Cons (n2, Cons (n1, f1 l3))) | Cons (n2, l3) -> []) in f1;;

real 0m1.315suser 0m1.212ssys 0m0.089s

pairwise_swap: surprising results

Ex. [0, 1, 0, 1] => [1, 0, 1, 0][0, 1, 0] => []

Inside-out recursion

• Evaluation: 43 benchmark tests + extended examples

• Compared to previous work: Benchmarks contains tests of similar complexity/size

Myth operates at comparable-or-better speeds

• Context size (number of variables) matters!

Test #/Examples

Size (AST)

T (min, s) T (ctx, s)

bool_xor 4 9 0.002 0.003

list_rev_snoc 5 11 0.006 0.018

list_fold 9 13 0.139 0.504

list_pairwise_swap

20 19 0.007 10.207

nat_iseven 4 10 0.001 0.014

tree_binsert 20 31 0.374 9.034

tree_nodes_at_level

24 22 1.093 4.917

arith 22 47 11.139 ---

...

My Mission

Help people harness the power of computation in its many forms, in particular, through computer

programming.

Programming Language Theory<->

Practical Tools and Implementation

Language interoperability

Low-level Memory Safety

Program Synthesis

PL + Computer Science

Education

Program Synthesis and Polymorphism

type ‘a list =| Nil| Cons of ‘a * list

Context:l1 : ‘a list

l2 : ‘a list

append : ‘a list -> ‘a list -> ‘a list

Terms of type list:l1l2

append l1 l2append l1 (append l1 l2)

...

append l1 [0]append l1 [1]append l2 [0]

...

Polymorphic list

Program Synthesis and Richer Types

How can richer type systems:

• Make synthesis more efficient?

• Allow us to synthesize richer classes of correct-by-construction programs?

Polymorphic Types

Linear Types

Dependent Types

Generic libraries, DSLs

Concurrency, security

Arbitrary properties

Semi-automated Programming

How do we integrate this into an editor?

Type-Directed Programming

module FreeMonad where data Free f a = Pure a | Free (f (Free f a)) instance Functor f => Monad (Free f) where return a = Pure a Pure a >>= f = _ Free f >>= g = _

(Example from https://wiki.haskell.org/GHC/Typed_holes)

FreeMonad.hs:9:18: Found hole ‘_’ with type: Free f b Where: ‘f’ is a rigid type variable bound by the instance declaration at FreeMonad.hs:7:10 ‘b’ is a rigid type variable bound by the type signature for (>>=) :: Free f a -> (a -> Free f b) -> Free f b at FreeMonad.hs:9:10 Relevant bindings include f :: a -> Free f b (bound at FreeMonad.hs:9:14) a :: a (bound at FreeMonad.hs:9:8) (>>=) :: Free f a -> (a -> Free f b) -> Free f b (bound at FreeMonad.hs:9:3) In the expression: _ In an equation for ‘>>=’: (Pure a) >>= f = _ In the instance declaration for ‘Monad (Free f)’

Type-Directed Programming

module FreeMonad where data Free f a = Pure a | Free (f (Free f a)) instance Functor f => Monad (Free f) where return a = Pure a Pure a >>= f = _ Free f >>= g = _

Found hole ‘_’ with type: Free f b...f :: a -> Free f ba :: a(>>=) :: Free f a -> (a -> Free f b) -> Free f b

f a :: Free f b

Type-Directed Programming

module FreeMonad where data Free f a = Pure a | Free (f (Free f a)) instance Functor f => Monad (Free f) where return a = Pure a Pure a >>= f = f a Free f >>= g = Free (fmap (>>= g) f)

How do we surface type-directed synthesis to the user?

How effective are these tools vs. “regular” programming?

InductFun: Synthesis and Education

How can synthesis be applied to programming education tools?

InductFun!

[Osera 2013]

An online tutorial system for learning proof by induction in the context of functional

programming.

http://inductfun.org

Typed program synthesis feedback for proofs

(e.g., introductory programs [Singh 2013])

Steve Zdancewic

David Walker(Princeton)

Jonathan Frankle

(Princeton)

Rohan Shah

Help people harness the power of computation in its many forms, in particular, through computer

programming.

http://www.cis.upenn.edu/~poseraposera@cis.upenn.edu

Thanks!

My Mission

Programming Language Theory<->

Practical Tools and Implementation

Synthesizing Normal Forms

e ::= x | e1 e2 | fun x -> e | C (e1, .., en) | match e with p1 -> e1 .. pn -> en

VariablesApplicationFunctionsConstructo

rs

Pattern Matching

e ::= x | e1 e2 | fun x -> e | C (e1, .., en) | match e with p1 -> e1 .. pn -> en

t1 -> t2Datatypes

Introduction Forms (type-directed)

Types direct intro form and example decomposition

e ::= x | e1 e2 | fun x -> e | C (e1, .., en) | match e with p1 -> e1 .. pn -> en

????

Elimination Forms (Not type-directed)

Must guess-and-check all possible (well-typed) elim forms

E ::= x | E II ::= E | fun x -> I | C (I1, .., In) | match E with p1 -> I1 .. pn -> In

λsyn: synthesis foundations

Elimination Forms

Introduction Forms

• Soundness: we always synthesis something that is well-typed and agrees with the examples

• Synthesized terms are always in normal form

(Inspired by proof search andbidirectional typechecking [Pfenning 2004])

An expression in normal form cannot evaluate any further

Synthesizing Normal Forms

Not in normal form Normal form

(fun x -> x) 5 5

match Nil with| Nil -> 0...

0

x

stutter l’

An expression in normal form cannot evaluate any further

Synthesizing Normal Forms

Not in normal form Normal form

(fun x -> x) 5 5

match Nil with| Nil -> 0...

0

We only synthesize these!

x

stutter l’

1 2 3 4 5 6 7 8 9 10 111

10

100

1000

10000

100000

1000000

10000000

100000000

1000000000

10000000000

Untyped Programs Typed ProgramsProgram Size (AST nodes)

Nu

mb

er

of

Pro

gra

ms

1,949,031,274

201,998

21,704

Refinement Trees

Dictated by types and examples

Generating and evaluating many

terms (exponential wall)

Pre-compute type refinements and pattern

matches

Localize efficient typed-

term generation