structure discovery in nonparametric regression through...
TRANSCRIPT
Structure Discovery in Nonparametric Regression throughCompositional Kernel SearchSE × ( Lin + Lin × ( Per + RQ ) )
1950 1952 1954 1956 1958 1960 1962
−200
0
200
400
600
=
SE × Lin
1950 1952 1954 1956 1958 1960 1962
−200
−100
0
100
200
300
400
SE × Lin × Per
1950 1952 1954 1956 1958 1960 1962
−200
−100
0
100
200
300
+SE × Lin × RQ
1950 1952 1954 1956 1958 1960 1962
−100
−50
0
50
100
Residuals
1950 1952 1954 1956 1958 1960 1962
−5
0
5
David Duvenaud, James Robert Lloyd, Roger Grosse,Joshua B. Tenenbaum, Zoubin Ghahramani
KERNEL CHOICE IS IMPORTANT
0 0
Squared-exp(SE)
localvariation
Periodic(PER)
repeatingstructure
0
0
Linear (LIN)linearfunctions
Rational-quadratic(RQ)
multi-scalevariation
IDENTIFYING STRUCTURE IS CRUCIAL FOR
EXTRAPOLATION
local smoothingLocal smooth structure
0 5 10 15−3
−2
−1
0
1
2
3
KERNELS CAN BE COMPOSED
0 0
LIN × LINquadraticfunctions
SE × PERlocallyperiodic
0
0
LIN + PERperiodicwith trend
SE + PERperiodicwith noise
IDENTIFYING STRUCTURE IS CRUCIAL FOR
EXTRAPOLATION
local smoothingLocal smooth structure
0 5 10 15−3
−2
−1
0
1
2
3
structured modelTrue structure
0 5 10 15−4
−3
−2
−1
0
1
2
3
COMPOSITIONAL STRUCTURE SEARCH
I We define simple grammar over kernels:I K → K + KI K → K × KI K → SE | RQ | LIN | PER
I Can automatically search open-ended space of kernels byapplying production rules, then checking model fit(approximate marginal likelihood).
COMPOSITIONAL STRUCTURE SEARCH
RQ
2000 2005 2010−20
0
20
40
60
Start
SE RQ
SE + RQ . . . PER + RQ
SE + PER + RQ . . . SE × (PER + RQ)
. . . . . . . . .
. . .
. . . PER × RQ
LIN PER
COMPOSITIONAL STRUCTURE SEARCH
( Per + RQ )
2000 2005 20100
10
20
30
40
Start
SE RQ
SE + RQ . . . PER + RQ
SE + PER + RQ . . . SE × (PER + RQ)
. . . . . . . . .
. . .
. . . PER × RQ
LIN PER
COMPOSITIONAL STRUCTURE SEARCH
SE × ( Per + RQ )
2000 2005 201010
20
30
40
50
Start
SE RQ
SE + RQ . . . PER + RQ
SE + PER + RQ . . . SE × (PER + RQ)
. . . . . . . . .
. . .
. . . PER × RQ
LIN PER
COMPOSITIONAL STRUCTURE SEARCH
( SE + SE × ( Per + RQ ) )
2000 2005 20100
20
40
60
Start
SE RQ
SE + RQ . . . PER + RQ
SE + PER + RQ . . . SE × (PER + RQ)
. . . . . . . . .
. . .
. . . PER × RQ
LIN PER
DECOMPOSING THE POSTERIORLIN × SE + SE × (PER + RQ)( Lin × SE + SE × ( Per + RQ ) )
1960 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010
−40
−20
0
20
40
60
=
LIN × SE Lin × SE
1960 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010
−40
−20
0
20
40
60SE × PERSE × Per
1984 1985 1986 1987 1988 1989
−5
0
5
+SE × RQSE × RQ
1960 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010
−4
−2
0
2
4ResidualsResiduals
1960 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010
−0.5
0
0.5
EXAMPLE: RADIO CRITICAL FREQUENCY(SE + RQ)× (PER + CS)
( CS × CS + ( SE + RQ × CS ) × ( Per + CS × CS ) )
1935 1940 1945 1950 1955−8
−6
−4
−2
0
2
4
6
=
SESE × CS × CS
1935 1940 1945 1950 1955−4
−2
0
2
4
6 PER × RQRQ × CS × Per
1935 1940 1945 1950 1955−5
0
5
+PER × SESE × Per
1935 1940 1945 1950 1955−3
−2
−1
0
1
2
3
4 RQRQ × CS × CS × CS
1935 1940 1945 1950 1955−6
−4
−2
0
2
4
6
AUTOMATED MODEL CONSTRUCTION
( RQ + RQ × ( CS + Lin × Per ) )
2005.44 2005.46 2005.48 2005.5 2005.52 2005.54 2005.56 2005.58 2005.6−4
−2
0
2
4
6x 10
10SE × CS × ( SE + CS + Lin × Per )
1972 1974 1976 1978 1980 1982 1984−4000
−2000
0
2000
4000
6000
8000
10000Per × RQ × RQ × ( Lin + Lin × ( Lin + Lin ) )
1955 1960 1965 1970 1975 1980 1985 1990 1995 2000−5
0
5
10
15x 10
4
CS × ( Per + Lin × SE × CS + Per × CS )
1973 1974 1975 1976 1977 1978 1979 1980−3000
−2000
−1000
0
1000
2000
3000
4000 RQ × CS × ( CS + Lin × CS + Lin × Per )
1955 1960 1965 1970 1975 1980 1985 1990 1995 2000−400
−200
0
200
400
600RQ × ( SE + Per + Lin × SE × RQ )
1950 1955 1960 1965 1970 1975 1980 1985−2
−1.5
−1
−0.5
0
0.5
1
1.5
2x 10
6
( SE + RQ ) × ( Per + Lin × CS )
1950 1955 1960 1965 1970 1975 1980 1985−2000
−1000
0
1000
2000
3000
4000 RQ × CS × CS × ( Lin × CS + Per × CS )
1960 1965 1970 1975−400
−200
0
200
400
600
800CS × ( SE + CS × CS × Per )
1966 1968 1970 1972 1974 1976 1978−30
−20
−10
0
10
20
30
CS × ( SE × CS × CS + CS × Per )
1820 1840 1860 1880 1900 1920 1940−4000
−2000
0
2000
4000
6000
8000 Lin × ( SE + CS × CS × CS × Per )
1700 1750 1800 1850 1900 1950 2000−150
−100
−50
0
50
100
150
200( Per + Lin × RQ × RQ × CS × CS × CS )
1962 1964 1966 1968 1970 1972 1974 1976 1978−1000
−500
0
500
1000