modelli basati su alberi e - loro interpretazione
Post on 27-Dec-2021
5 Views
Preview:
TRANSCRIPT
Outline
2
Decision trees
Regression and classification trees
Bagging and Random forest
Conditional inference trees and forest
Bart
Interpreting and understanding
Remember: as these algorithms are not set
according to theoretical assumptions, you
need to use
- training/testing sets
- cross-validation
General setting
3
- A vector of explanatory variables X = (X1, … Xp) and a response Y are observed
on a sample of iid statistical units
- We want to predict Y assuming that
f(X) = E[Y | X]<latexit sha1_base64="N9gOGcforqnnaj+7PFn5/jhED50=">AAACrHicfVFNTxsxEHW2LR+h0FCOXFwCElyiXVQJLkgIVKlHQASCsqvI9s6ChT9WtheIVnvgl/Ta/qT+G7zJChFSdUaWn968mfF4aC64dWH4txV8+PhpYXFpub3yeXXtS2f965XVhWHQZ1poM6DEguAK+o47AYPcAJFUwDW9P63j1w9gLNfq0o1zSCS5VTzjjDhPjTob2e5gDx/hH0N8g2PJUzxIRp1u2AsnhudB1IAuauxstN56jlPNCgnKMUGsHUZh7pKSGMeZgKodFxZywu7JLQw9VESCTcrJ6yu845kUZ9r4oxyesG8zSiKtHUvqlZK4O/s+VpP/jFHpaapFOtvfZYdJyVVeOFBs2j4rBHYa1x+EU26AOTH2gDDD/QSY3RFDmPPf2I4VPDItJVFpGXOVVmUsqX4qt+McTB5/m3gNt6tqVqz+p27g1C04yVVh50pcABGvJfztJ6YUX1S10C8ter+ieXC134vCXnT+vXt80qxvCW2iLbSLInSAjtFPdIb6iKEx+oV+oz9BL7gMhkEylQatJmcDzViQvQCcLNIK</latexit><latexit sha1_base64="N9gOGcforqnnaj+7PFn5/jhED50=">AAACrHicfVFNTxsxEHW2LR+h0FCOXFwCElyiXVQJLkgIVKlHQASCsqvI9s6ChT9WtheIVnvgl/Ta/qT+G7zJChFSdUaWn968mfF4aC64dWH4txV8+PhpYXFpub3yeXXtS2f965XVhWHQZ1poM6DEguAK+o47AYPcAJFUwDW9P63j1w9gLNfq0o1zSCS5VTzjjDhPjTob2e5gDx/hH0N8g2PJUzxIRp1u2AsnhudB1IAuauxstN56jlPNCgnKMUGsHUZh7pKSGMeZgKodFxZywu7JLQw9VESCTcrJ6yu845kUZ9r4oxyesG8zSiKtHUvqlZK4O/s+VpP/jFHpaapFOtvfZYdJyVVeOFBs2j4rBHYa1x+EU26AOTH2gDDD/QSY3RFDmPPf2I4VPDItJVFpGXOVVmUsqX4qt+McTB5/m3gNt6tqVqz+p27g1C04yVVh50pcABGvJfztJ6YUX1S10C8ter+ieXC134vCXnT+vXt80qxvCW2iLbSLInSAjtFPdIb6iKEx+oV+oz9BL7gMhkEylQatJmcDzViQvQCcLNIK</latexit><latexit sha1_base64="N9gOGcforqnnaj+7PFn5/jhED50=">AAACrHicfVFNTxsxEHW2LR+h0FCOXFwCElyiXVQJLkgIVKlHQASCsqvI9s6ChT9WtheIVnvgl/Ta/qT+G7zJChFSdUaWn968mfF4aC64dWH4txV8+PhpYXFpub3yeXXtS2f965XVhWHQZ1poM6DEguAK+o47AYPcAJFUwDW9P63j1w9gLNfq0o1zSCS5VTzjjDhPjTob2e5gDx/hH0N8g2PJUzxIRp1u2AsnhudB1IAuauxstN56jlPNCgnKMUGsHUZh7pKSGMeZgKodFxZywu7JLQw9VESCTcrJ6yu845kUZ9r4oxyesG8zSiKtHUvqlZK4O/s+VpP/jFHpaapFOtvfZYdJyVVeOFBs2j4rBHYa1x+EU26AOTH2gDDD/QSY3RFDmPPf2I4VPDItJVFpGXOVVmUsqX4qt+McTB5/m3gNt6tqVqz+p27g1C04yVVh50pcABGvJfztJ6YUX1S10C8ter+ieXC134vCXnT+vXt80qxvCW2iLbSLInSAjtFPdIb6iKEx+oV+oz9BL7gMhkEylQatJmcDzViQvQCcLNIK</latexit><latexit sha1_base64="N9gOGcforqnnaj+7PFn5/jhED50=">AAACrHicfVFNTxsxEHW2LR+h0FCOXFwCElyiXVQJLkgIVKlHQASCsqvI9s6ChT9WtheIVnvgl/Ta/qT+G7zJChFSdUaWn968mfF4aC64dWH4txV8+PhpYXFpub3yeXXtS2f965XVhWHQZ1poM6DEguAK+o47AYPcAJFUwDW9P63j1w9gLNfq0o1zSCS5VTzjjDhPjTob2e5gDx/hH0N8g2PJUzxIRp1u2AsnhudB1IAuauxstN56jlPNCgnKMUGsHUZh7pKSGMeZgKodFxZywu7JLQw9VESCTcrJ6yu845kUZ9r4oxyesG8zSiKtHUvqlZK4O/s+VpP/jFHpaapFOtvfZYdJyVVeOFBs2j4rBHYa1x+EU26AOTH2gDDD/QSY3RFDmPPf2I4VPDItJVFpGXOVVmUsqX4qt+McTB5/m3gNt6tqVqz+p27g1C04yVVh50pcABGvJfztJ6YUX1S10C8ter+ieXC134vCXnT+vXt80qxvCW2iLbSLInSAjtFPdIb6iKEx+oV+oz9BL7gMhkEylQatJmcDzViQvQCcLNIK</latexit>
- The best predictor is the function that minimises among all the possible functions
g(X) a loss function, such as for instance the mean squared error (if Y continuous)
- Let’s call such function [f(X)
<latexit sha1_base64="zNZ//GeNWKi9QLtZL09xwL6oDbA=">AAACqHicfVFLbxMxEJ4srxIeTemRi2mLVECKdrm0xwouPYaKNEF1FNne2caqHyvb2zZa7YHfwYUD/Cj+Dd4kQqRBzMjyp2++mfF4eKmkD2n6q5Pcu//g4aOtx90nT5893+7tvDj3tnICh8Iq68aceVTS4DDIoHBcOmSaKxzxq49tfHSNzktrPod5iRPNLo0spGAhUtPeNr2ROc5YqIvD8ZuGTHv7aT9dGNkE2Qrsn+zRd98BYDDd6XyluRWVRhOEYt5fZGkZJjVzQQqFTZdWHksmrtglXkRomEY/qRcvb8jryOSksC4eE8iC/TujZtr7ueZRqVmY+buxlvxnjOtIc6vy9f6hOJ7U0pRVQCOW7YtKkWBJ+zkklw5FUPMImHAyTkDEjDkmQvzCLjV4I6zWzOQ1lSZvaqq5va0PaImupK8W3sKDplkXm/+pV3DpHoOWpvIbJc6QqT8l4h0n5pycNa0wLi27u6JNcP6+n6X97FPc3gdY2ha8hD04hAyO4AROYQBDEFDBN/gBP5O3ySAZJV+W0qSzytmFNUv4b6BU06A=</latexit><latexit sha1_base64="ZT8RpmwreS7ZYZhnB5ThZbfD7mw=">AAACqHicfVFLbxMxEHaWVwmPpnDkYtoiFZCi3V7oMYILx1CRJqiOIts721j1Y2XPtkSrPfAfuHGFH9V/gzeJEGkQM7L86ZtvZjweUWoVME1vOsmdu/fuP9h52H30+MnT3d7es7PgKi9hJJ12fiJ4AK0sjFChhknpgRuhYSwuP7Tx8RX4oJz9jIsSpoZfWFUoyTFSs94uu1Y5zDnWxdHkdUNnvYO0ny6NboNsDQ4G++zt95vBYjjb63xjuZOVAYtS8xDOs7TEac09Kqmh6bIqQMnlJb+A8wgtNxCm9fLlDX0VmZwWzsdjkS7ZvzNqbkJYGBGVhuM83I615D9jwkRaOJ1v9sfiZForW1YIVq7aF5Wm6Gj7OTRXHiTqRQRcehUnoHLOPZcYv7DLLFxLZwy3ec2UzZuaGeG+1oesBF+yl0tv4WHTbIrt/9RruPIAaJStwlaJU+D6T4l4x4mFoKdNK4xLy26vaBucHfeztJ99itt7T1a2Q16QfXJEMvKODMhHMiQjIklFfpCf5FfyJhkm4+TLSpp01jnPyYYl4je17dUm</latexit><latexit sha1_base64="ZT8RpmwreS7ZYZhnB5ThZbfD7mw=">AAACqHicfVFLbxMxEHaWVwmPpnDkYtoiFZCi3V7oMYILx1CRJqiOIts721j1Y2XPtkSrPfAfuHGFH9V/gzeJEGkQM7L86ZtvZjweUWoVME1vOsmdu/fuP9h52H30+MnT3d7es7PgKi9hJJ12fiJ4AK0sjFChhknpgRuhYSwuP7Tx8RX4oJz9jIsSpoZfWFUoyTFSs94uu1Y5zDnWxdHkdUNnvYO0ny6NboNsDQ4G++zt95vBYjjb63xjuZOVAYtS8xDOs7TEac09Kqmh6bIqQMnlJb+A8wgtNxCm9fLlDX0VmZwWzsdjkS7ZvzNqbkJYGBGVhuM83I615D9jwkRaOJ1v9sfiZForW1YIVq7aF5Wm6Gj7OTRXHiTqRQRcehUnoHLOPZcYv7DLLFxLZwy3ec2UzZuaGeG+1oesBF+yl0tv4WHTbIrt/9RruPIAaJStwlaJU+D6T4l4x4mFoKdNK4xLy26vaBucHfeztJ99itt7T1a2Q16QfXJEMvKODMhHMiQjIklFfpCf5FfyJhkm4+TLSpp01jnPyYYl4je17dUm</latexit><latexit sha1_base64="vEeKIGcWKlkqjcY/p73KIenx9iY=">AAACqHicfVFNbxMxEHWWQkv4aApHLi4pUuEQ7fbSHiu49Bgq0qTqRtHYO9tY9cfK9rZEqz3wO7jSH8W/wZusEGlQZ2T56c2bGY+HFVI4H8e/O9GTrafPtneed1+8fPV6t7f35sKZ0nIccSONnTBwKIXGkRde4qSwCIpJHLObL018fIvWCaO/+UWBUwXXWuSCgw/UrLeb3okM5+Cr/HDysaazXj8exEujmyBpQZ+0NpztdX6kmeGlQu25BOeukrjw0wqsF1xi3U1LhwXwG7jGqwA1KHTTavnymn4ITEZzY8PRni7ZfzMqUM4tFAtKBX7uHsYa8r8xpgLNjMzW+/v8ZFoJXZQeNV+1z0tJvaHN59BMWOReLgIAbkWYgPI5WOA+fGE31XjHjVKgsyoVOqurVDHzvTpIC7RFur/0Bh7U9bpYP6Zu4codeiV06TZKnCPIvyXCHSZmjJ7XjTAsLXm4ok1wcTRI4kHyNe6ffm7Xt0PekffkkCTkmJySMzIkI8JJSX6SX+Q++hQNo3F0uZJGnTbnLVmziP0Bf5bSFw==</latexit>
Mean
⇣Y � g(X)
⌘2| X = x
�
<latexit sha1_base64="dZwMprb1ep0YLx4HRur6d8i0HnQ=">AAAC13icfVFNbxMxEHWWrxK+UjhyMaRI6YFot0Kil6KqXLgglYq0qeIlsr2zG6u2d2V7IdFqhbggrvwHfg1XuPBv8CYrRBrEjCw/v3kzo/GwQgrrwvBXJ7hy9dr1G1s3u7du37l7r7d9/9TmpeEw4rnMzZhRC1JoGDnhJIwLA1QxCWfs4mUTP3sPxopcv3WLAmJFMy1Swanz1LT3gjiYu+o1UF0TCanDE0yORIYH+Bw/xdlgvLt6777bw0SJBI8P5hgTI7KZw/G01w+H4dLwJoha0EetHU+3O59IkvNSgXZcUmsnUVi4uKLGCS6h7pLSQkH5Bc1g4qGmCmxcLQet8RPPJDjNjT/a4SX7d0ZFlbULxbxSUTezl2MN+c8YU55muUzW+7t0P66ELkoHmq/ap6XELsfNX+JEGOBOLjyg3Ag/AeYzaih3/se7RMMHnitFdVIRoZO6Iorl82qHFGAK8mjpDdyp63Wx/p+6hSu34JTQpd0ocQJU/inhbz8xY/ikboR+adHlFW2C071hFA6jN8/6h0ft+rbQQ/QYDVCEnqND9AodoxHi6Bv6jn6gn8F58DH4HHxZSYNOm/MArVnw9TeXmOHx</latexit><latexit sha1_base64="dZwMprb1ep0YLx4HRur6d8i0HnQ=">AAAC13icfVFNbxMxEHWWrxK+UjhyMaRI6YFot0Kil6KqXLgglYq0qeIlsr2zG6u2d2V7IdFqhbggrvwHfg1XuPBv8CYrRBrEjCw/v3kzo/GwQgrrwvBXJ7hy9dr1G1s3u7du37l7r7d9/9TmpeEw4rnMzZhRC1JoGDnhJIwLA1QxCWfs4mUTP3sPxopcv3WLAmJFMy1Swanz1LT3gjiYu+o1UF0TCanDE0yORIYH+Bw/xdlgvLt6777bw0SJBI8P5hgTI7KZw/G01w+H4dLwJoha0EetHU+3O59IkvNSgXZcUmsnUVi4uKLGCS6h7pLSQkH5Bc1g4qGmCmxcLQet8RPPJDjNjT/a4SX7d0ZFlbULxbxSUTezl2MN+c8YU55muUzW+7t0P66ELkoHmq/ap6XELsfNX+JEGOBOLjyg3Ag/AeYzaih3/se7RMMHnitFdVIRoZO6Iorl82qHFGAK8mjpDdyp63Wx/p+6hSu34JTQpd0ocQJU/inhbz8xY/ikboR+adHlFW2C071hFA6jN8/6h0ft+rbQQ/QYDVCEnqND9AodoxHi6Bv6jn6gn8F58DH4HHxZSYNOm/MArVnw9TeXmOHx</latexit><latexit sha1_base64="dZwMprb1ep0YLx4HRur6d8i0HnQ=">AAAC13icfVFNbxMxEHWWrxK+UjhyMaRI6YFot0Kil6KqXLgglYq0qeIlsr2zG6u2d2V7IdFqhbggrvwHfg1XuPBv8CYrRBrEjCw/v3kzo/GwQgrrwvBXJ7hy9dr1G1s3u7du37l7r7d9/9TmpeEw4rnMzZhRC1JoGDnhJIwLA1QxCWfs4mUTP3sPxopcv3WLAmJFMy1Swanz1LT3gjiYu+o1UF0TCanDE0yORIYH+Bw/xdlgvLt6777bw0SJBI8P5hgTI7KZw/G01w+H4dLwJoha0EetHU+3O59IkvNSgXZcUmsnUVi4uKLGCS6h7pLSQkH5Bc1g4qGmCmxcLQet8RPPJDjNjT/a4SX7d0ZFlbULxbxSUTezl2MN+c8YU55muUzW+7t0P66ELkoHmq/ap6XELsfNX+JEGOBOLjyg3Ag/AeYzaih3/se7RMMHnitFdVIRoZO6Iorl82qHFGAK8mjpDdyp63Wx/p+6hSu34JTQpd0ocQJU/inhbz8xY/ikboR+adHlFW2C071hFA6jN8/6h0ft+rbQQ/QYDVCEnqND9AodoxHi6Bv6jn6gn8F58DH4HHxZSYNOm/MArVnw9TeXmOHx</latexit><latexit sha1_base64="dZwMprb1ep0YLx4HRur6d8i0HnQ=">AAAC13icfVFNbxMxEHWWrxK+UjhyMaRI6YFot0Kil6KqXLgglYq0qeIlsr2zG6u2d2V7IdFqhbggrvwHfg1XuPBv8CYrRBrEjCw/v3kzo/GwQgrrwvBXJ7hy9dr1G1s3u7du37l7r7d9/9TmpeEw4rnMzZhRC1JoGDnhJIwLA1QxCWfs4mUTP3sPxopcv3WLAmJFMy1Swanz1LT3gjiYu+o1UF0TCanDE0yORIYH+Bw/xdlgvLt6777bw0SJBI8P5hgTI7KZw/G01w+H4dLwJoha0EetHU+3O59IkvNSgXZcUmsnUVi4uKLGCS6h7pLSQkH5Bc1g4qGmCmxcLQet8RPPJDjNjT/a4SX7d0ZFlbULxbxSUTezl2MN+c8YU55muUzW+7t0P66ELkoHmq/ap6XELsfNX+JEGOBOLjyg3Ag/AeYzaih3/se7RMMHnitFdVIRoZO6Iorl82qHFGAK8mjpDdyp63Wx/p+6hSu34JTQpd0ocQJU/inhbz8xY/ikboR+adHlFW2C071hFA6jN8/6h0ft+rbQQ/QYDVCEnqND9AodoxHi6Bv6jn6gn8F58DH4HHxZSYNOm/MArVnw9TeXmOHx</latexit>
Example : Multiple linear regression
4
MSE =RSS
n<latexit sha1_base64="KhtGCUkucezpZCk8hgu1Rm15wEs=">AAACdXicfVFdaxQxFM2OH63rR7f2UYToruLTNrNabZFCUQQRhOq6bWGzLJnMnTY0H0OSUZcwv6Wv+pP8Jb6a2Y5iRb2XwLnnnptcTrJSCucJ+dZJLl2+cnVl9Vr3+o2bt9Z667cPnKkshwk30tijjDmQQsPECy/hqLTAVCbhMDt92fQPP4J1wugPflHCTLFjLQrBmY/UvLfxdvwK72JaWMbD+/G4Drqe9/pkuEXSnacpJkOyjAhGW2Rnm+C0Zfqojf35eucNzQ2vFGjPJXNumpLSzwKzXnAJdZdWDkrGT9kxTCPUTIGbheX2NX4QmRwXxsajPV6yv08EppxbqCwqFfMn7s9eQ/6tN618sT0LQpeVB83PHyoqib3BjRU4Fxa4l4sIGLci7or5CYtG+GhYl2r4xI1STOeBCp3XgarMfA4DWoIt6b02m2JQ1xfl+v/6X0VMTB14JXTlMH3eXBTd/2kx/jc4GA3Tx8PRuyf9vRftP6yiO+g+eoRS9AztoddoH00QRwt0hr6gr53vyd1kkDw8lyaddmYDXYhk8wcVZL0x</latexit>
Decision trees
5
X1<c
X2<r
Yes
Yes
No
No
—> IDEA: To find a piecewise-constant
approximation of f(X) to predict Y, chosen
in a way that mimics how decisions are actually taken
—> EXAMPLE:
Y = treatment
X1 = Fever
X2 = Pain
Example with only 1 covariate
6
First split
Terminal nodes / leaves
Internal/decision nodes
Induced partitioning
Example : simulated data, 2 covariates
A reg
7
A decision tree is a structure organised hierarchically
The tree structure is equivalent to partition the joint space of the explanatory variables into M
(= n. leaves) subspaces
The number in each leaf is the mean of the response for the observations that fall there.
Y = sin(X1) · sin(X2) + ✏, ✏ ⇠ N(0, 1)<latexit sha1_base64="vmF9ZPX75PLTPRkAIqTloVLBW9Q=">AAAC33icfVFLbxMxEHaWVwmvFI5cDClSIqpot0KiF6QKJMQJlYq0QXUU2d5Ja9Wv2t5CtNoDJ26IK/+BX8MNwY/BmywVaRAzsvz5m29mNB5mpfAhTX+0kkuXr1y9tna9fePmrdt3Out3970pHIchN9K4EaMepNAwDCJIGFkHVDEJB+zkRR0/OAPnhdFvw8zCWNEjLaaC0xCpSeflO/wMEy90bzTJ+pjw3IQ/760+fowJWC+k0ZuYnJ4WNMfnTC1T+HUv3cz6k043HaRzw6sga0AXNbY7WW99JLnhhQIduKTeH2apDeOSuiC4hKpNCg+W8hN6BIcRaqrAj8v5wBV+FJkcT42LRwc8Z//OKKnyfqZYVCoajv3FWE3+M8ZUpJmR+XL/MN0el0LbIoDmi/bTQuJgcP2nOBcOeJCzCCh3Ik6A+TF1lIf4822i4T03SlGdl0TovCqJYuZDuUEsOEsezL2GG1W1LNb/Uzdw4R6CErrwKyX2gMrzEvGOEzOG96paGJeWXVzRKtjfGmTpIHvzpLvzvFnfGrqPHqIeytBTtINeoV00RBx9Q9/RT/Qrocmn5HPyZSFNWk3OPbRkydffFtfk3w==</latexit><latexit sha1_base64="vmF9ZPX75PLTPRkAIqTloVLBW9Q=">AAAC33icfVFLbxMxEHaWVwmvFI5cDClSIqpot0KiF6QKJMQJlYq0QXUU2d5Ja9Wv2t5CtNoDJ26IK/+BX8MNwY/BmywVaRAzsvz5m29mNB5mpfAhTX+0kkuXr1y9tna9fePmrdt3Out3970pHIchN9K4EaMepNAwDCJIGFkHVDEJB+zkRR0/OAPnhdFvw8zCWNEjLaaC0xCpSeflO/wMEy90bzTJ+pjw3IQ/760+fowJWC+k0ZuYnJ4WNMfnTC1T+HUv3cz6k043HaRzw6sga0AXNbY7WW99JLnhhQIduKTeH2apDeOSuiC4hKpNCg+W8hN6BIcRaqrAj8v5wBV+FJkcT42LRwc8Z//OKKnyfqZYVCoajv3FWE3+M8ZUpJmR+XL/MN0el0LbIoDmi/bTQuJgcP2nOBcOeJCzCCh3Ik6A+TF1lIf4822i4T03SlGdl0TovCqJYuZDuUEsOEsezL2GG1W1LNb/Uzdw4R6CErrwKyX2gMrzEvGOEzOG96paGJeWXVzRKtjfGmTpIHvzpLvzvFnfGrqPHqIeytBTtINeoV00RBx9Q9/RT/Qrocmn5HPyZSFNWk3OPbRkydffFtfk3w==</latexit><latexit sha1_base64="vmF9ZPX75PLTPRkAIqTloVLBW9Q=">AAAC33icfVFLbxMxEHaWVwmvFI5cDClSIqpot0KiF6QKJMQJlYq0QXUU2d5Ja9Wv2t5CtNoDJ26IK/+BX8MNwY/BmywVaRAzsvz5m29mNB5mpfAhTX+0kkuXr1y9tna9fePmrdt3Out3970pHIchN9K4EaMepNAwDCJIGFkHVDEJB+zkRR0/OAPnhdFvw8zCWNEjLaaC0xCpSeflO/wMEy90bzTJ+pjw3IQ/760+fowJWC+k0ZuYnJ4WNMfnTC1T+HUv3cz6k043HaRzw6sga0AXNbY7WW99JLnhhQIduKTeH2apDeOSuiC4hKpNCg+W8hN6BIcRaqrAj8v5wBV+FJkcT42LRwc8Z//OKKnyfqZYVCoajv3FWE3+M8ZUpJmR+XL/MN0el0LbIoDmi/bTQuJgcP2nOBcOeJCzCCh3Ik6A+TF1lIf4822i4T03SlGdl0TovCqJYuZDuUEsOEsezL2GG1W1LNb/Uzdw4R6CErrwKyX2gMrzEvGOEzOG96paGJeWXVzRKtjfGmTpIHvzpLvzvFnfGrqPHqIeytBTtINeoV00RBx9Q9/RT/Qrocmn5HPyZSFNWk3OPbRkydffFtfk3w==</latexit><latexit sha1_base64="vmF9ZPX75PLTPRkAIqTloVLBW9Q=">AAAC33icfVFLbxMxEHaWVwmvFI5cDClSIqpot0KiF6QKJMQJlYq0QXUU2d5Ja9Wv2t5CtNoDJ26IK/+BX8MNwY/BmywVaRAzsvz5m29mNB5mpfAhTX+0kkuXr1y9tna9fePmrdt3Out3970pHIchN9K4EaMepNAwDCJIGFkHVDEJB+zkRR0/OAPnhdFvw8zCWNEjLaaC0xCpSeflO/wMEy90bzTJ+pjw3IQ/760+fowJWC+k0ZuYnJ4WNMfnTC1T+HUv3cz6k043HaRzw6sga0AXNbY7WW99JLnhhQIduKTeH2apDeOSuiC4hKpNCg+W8hN6BIcRaqrAj8v5wBV+FJkcT42LRwc8Z//OKKnyfqZYVCoajv3FWE3+M8ZUpJmR+XL/MN0el0LbIoDmi/bTQuJgcP2nOBcOeJCzCCh3Ik6A+TF1lIf4822i4T03SlGdl0TovCqJYuZDuUEsOEsezL2GG1W1LNb/Uzdw4R6CErrwKyX2gMrzEvGOEzOG96paGJeWXVzRKtjfGmTpIHvzpLvzvFnfGrqPHqIeytBTtINeoV00RBx9Q9/RT/Qrocmn5HPyZSFNWk3OPbRkydffFtfk3w==</latexit>
X1, X2 ⇠ N(0, 1)<latexit sha1_base64="vuZjzxA94BrYvpAa8+VvAliLuLg=">AAACrXicfVFNTxsxEHWW0tL0K9DeuBhCJZBQuosqlSMql54QRQQi4bCyvZNg4Y+V7S2E1R76T3ql/6j/pt5kVTUEMSPLT2/ezHg8LJfC+Tj+04qWni0/f7Hysv3q9Zu37zqra2fOFJZDnxtp7IBRB1Jo6HvhJQxyC1QxCefs+rCOn/8A64TRp36Sw1DRsRYjwakPVNr5MEiTXTxI9zBxQuGj7Xg32Uk73bgXTw0vgqQBXdTYcbra+kkywwsF2nNJnbtI4twPS2q94BKqNikc5JRf0zFcBKipAjcsp8+v8MfAZHhkbDja4yn7f0ZJlXMTxYJSUX/lHsZq8tEYU4FmRmbz/f1of1gKnRceNJ+1HxUSe4PrH8KZsMC9nARAuRVhAsyvqKXch39sEw033ChFdVYSobOqJIqZ23KL5GBzsjH1Gm5V1bxYP6Vu4MwdeCV04RZKnACV/0qEO0zMGD6pamFYWvJwRYvgbK+XxL3k++fuwddmfStoHW2ibZSgL+gAfUPHqI84ukO/0D36HX2K+hGJLmfSqNXkvEdzFo3/Asbg0hI=</latexit><latexit sha1_base64="vuZjzxA94BrYvpAa8+VvAliLuLg=">AAACrXicfVFNTxsxEHWW0tL0K9DeuBhCJZBQuosqlSMql54QRQQi4bCyvZNg4Y+V7S2E1R76T3ql/6j/pt5kVTUEMSPLT2/ezHg8LJfC+Tj+04qWni0/f7Hysv3q9Zu37zqra2fOFJZDnxtp7IBRB1Jo6HvhJQxyC1QxCefs+rCOn/8A64TRp36Sw1DRsRYjwakPVNr5MEiTXTxI9zBxQuGj7Xg32Uk73bgXTw0vgqQBXdTYcbra+kkywwsF2nNJnbtI4twPS2q94BKqNikc5JRf0zFcBKipAjcsp8+v8MfAZHhkbDja4yn7f0ZJlXMTxYJSUX/lHsZq8tEYU4FmRmbz/f1of1gKnRceNJ+1HxUSe4PrH8KZsMC9nARAuRVhAsyvqKXch39sEw033ChFdVYSobOqJIqZ23KL5GBzsjH1Gm5V1bxYP6Vu4MwdeCV04RZKnACV/0qEO0zMGD6pamFYWvJwRYvgbK+XxL3k++fuwddmfStoHW2ibZSgL+gAfUPHqI84ukO/0D36HX2K+hGJLmfSqNXkvEdzFo3/Asbg0hI=</latexit><latexit sha1_base64="vuZjzxA94BrYvpAa8+VvAliLuLg=">AAACrXicfVFNTxsxEHWW0tL0K9DeuBhCJZBQuosqlSMql54QRQQi4bCyvZNg4Y+V7S2E1R76T3ql/6j/pt5kVTUEMSPLT2/ezHg8LJfC+Tj+04qWni0/f7Hysv3q9Zu37zqra2fOFJZDnxtp7IBRB1Jo6HvhJQxyC1QxCefs+rCOn/8A64TRp36Sw1DRsRYjwakPVNr5MEiTXTxI9zBxQuGj7Xg32Uk73bgXTw0vgqQBXdTYcbra+kkywwsF2nNJnbtI4twPS2q94BKqNikc5JRf0zFcBKipAjcsp8+v8MfAZHhkbDja4yn7f0ZJlXMTxYJSUX/lHsZq8tEYU4FmRmbz/f1of1gKnRceNJ+1HxUSe4PrH8KZsMC9nARAuRVhAsyvqKXch39sEw033ChFdVYSobOqJIqZ23KL5GBzsjH1Gm5V1bxYP6Vu4MwdeCV04RZKnACV/0qEO0zMGD6pamFYWvJwRYvgbK+XxL3k++fuwddmfStoHW2ibZSgL+gAfUPHqI84ukO/0D36HX2K+hGJLmfSqNXkvEdzFo3/Asbg0hI=</latexit><latexit sha1_base64="vuZjzxA94BrYvpAa8+VvAliLuLg=">AAACrXicfVFNTxsxEHWW0tL0K9DeuBhCJZBQuosqlSMql54QRQQi4bCyvZNg4Y+V7S2E1R76T3ql/6j/pt5kVTUEMSPLT2/ezHg8LJfC+Tj+04qWni0/f7Hysv3q9Zu37zqra2fOFJZDnxtp7IBRB1Jo6HvhJQxyC1QxCefs+rCOn/8A64TRp36Sw1DRsRYjwakPVNr5MEiTXTxI9zBxQuGj7Xg32Uk73bgXTw0vgqQBXdTYcbra+kkywwsF2nNJnbtI4twPS2q94BKqNikc5JRf0zFcBKipAjcsp8+v8MfAZHhkbDja4yn7f0ZJlXMTxYJSUX/lHsZq8tEYU4FmRmbz/f1of1gKnRceNJ+1HxUSe4PrH8KZsMC9nARAuRVhAsyvqKXch39sEw033ChFdVYSobOqJIqZ23KL5GBzsjH1Gm5V1bxYP6Vu4MwdeCV04RZKnACV/0qEO0zMGD6pamFYWvJwRYvgbK+XxL3k++fuwddmfStoHW2ibZSgL+gAfUPHqI84ukO/0D36HX2K+hGJLmfSqNXkvEdzFo3/Asbg0hI=</latexit>
X1<c
X2<r
Yes
Yes
No
No
Regression trees
8
—> We need to learn:
- structure of the tree
- which variable splits and where
- predictions in each leaf/region
Recursive binary splitting: a top-down, greedy approach
The approach is top-down because it begins at the top of the tree (at which
point all observations belong to a single region) and then successively splits the
covariate space
Binary: Each split is indicated via two new branches further down on the tree
It is greedy because at each step of the tree-building process, the best split is
made at that particular step, rather than looking ahead and picking a split that
will lead to a better tree in some future step.
9
Regression trees from a regression perspective
11
(1) Start with M = 1, R1 = Rp<latexit sha1_base64="2MPgHA2f9xx4CwF6K56FayMie/Q=">AAACgHicfVFdTxQxFO2OqLh+LfBoYqqLCQ9mnSIgakhIeCEkJEhcINmum07nLjT0i7aDbibz5q/xFf6M/8bOMhIw6r1pcu655/bjNLNS+JCmP1vJnZm79+7PPmg/fPT4ydPO3PyBN4Xj0OdGGneUMQ9SaOgHESQcWQdMZRIOs9Otun94Ds4Loz+HiYWhYsdajAVnIVKjzvPdDfIa07OzguV4f0TwBlUsnGRZuV99sXjU6aa9dBr4BlhNyfs1gknDdFETe6O51g7NDS8U6MAl835AUhuGJXNBcAlVmxYeLOOn7BgGEWqmwA/L6UMq/CoyOR4bF5cOeMrenCiZ8n6isqisb+n/7NXk33qDIozXh6XQtgig+dVB40LiYHDtCs6FAx7kJALGnYh3xfyEOcZD9K5NNXzlRimm85IKnVclVZn5Vi5SC87SF03WxWJV3Zbr/+uvi5iYeghK6MJj+rHeKLr/22L8b3Cw3CNve8ufVrqb680/zKJn6CVaQgS9Q5toG+2hPuLoO/qBLtBlkiRLyZuEXEmTVjOzgG5F8uEXAzS/rg==</latexit>
(2) Search for the first split:
(j, s1) : R1 = {(X1, . . . , Xp) 2 X : Xj s1}, R2 = {(X1, . . . , Xp) 2 X : Xj > s1}<latexit sha1_base64="C+13GHF5DEj49Ec1VS3s/3CjM6o=">AAAC9XicjVFNb9QwEHXCV1k+uoUjBwa2SFtptUqWFkoRqBIXxKkgto20XkWO423d2k5qO5SVlV/BmRviyu/hp3DD2aaoRYCYka2ZN2+erZmsFNzYKPoehJcuX7l6bel658bNW7eXuyt3dk1RacrGtBCFTjJimOCKjS23giWlZkRmgu1lR6+a+t4Hpg0v1Hs7L9lUkn3FZ5wS66G0+6l/OACTxmuAB7DVXO/SGF4Adv0kjQeARV5YAwNI0tJzuAIsiT2gRLik3vKoO6w9iR03Irj2RHxckdyrjP5XBc5kXrYaabcXDaOFwblgI4qfPYkhbpEeam0nXQne4LyglWTKUkGMmcRRaaeOaMupYHUHV4aVhB6RfTbxoSKSmalbTK+GRx7JYVZof5SFBXq+wxFpzFxmntl82vxea8A/1SaVnW1OHVdlZZmipw/NKgG2gGYVkHPNqBVzHxCquf8r0AOiCbV+YR2s2AktpCQqd35iee2wzIqPbhWXTJf4QetNslrXF+nq3/xfiXfAhlnJVWUAP2+E/PTPRgx/D3ZHw/jxcPR2vbe92e5hCd1DD1Efxegp2kav0Q4aI4p+BPeDfrAWnoSfwy/h11NqGLQ9d9EFC7/9BBXm5Jc=</latexit>
minj,s1
2
4minµ1
X
i:xij2R1
(yi � µ1)2 + min
µ2
X
i:xij2R2
(yi � µ2)2
3
5
<latexit sha1_base64="lUOUG9ySK6cckuindVfiSRiqeS8=">AAADEHicfVHLbtQwFHXCqwyvKbBjc2GKVASMkvAqZVOpLBCrgpi20niIHMeZcWs7ke1AR1F+gg9hzQ6x5Q/4G5xMoJ2Kci1b5557fG3dkxSCGxsEvzz/3PkLFy+tXO5duXrt+o3+6s1dk5eashHNRa73E2KY4IqNLLeC7ReaEZkItpccbjf1vU9MG56rD3ZesIkkU8UzTol1VNz/iiVXcXXwCEwc1oAFyyyMYcFiWbakKWVc8c0jdxy4lCt4fyxeh3nM4XGrBaz5dGbhwccI8Ct42BzHraIzWkWnWkErXuq1gBOI+4NgGLQBJ8CzIHz5PISwYwaoi5141XuL05yWkilLBTFmHAaFnVREW04Fq3u4NKwg9JBM2dhBRSQzk6odbA33HZNClmu3lYWWPXmjItKYuUycUhI7M6drDfmv2ri02cak4qooLVN08VBWCrA5NC5ByjWjVswdIFRz91egM6IJtc7LHlbsM82lJCqt3AjT2o03yY+qNVwwXeC73WqStbpelqv/6/8mbjm3mHXulcb52DRy0/8zYjgb7EbD8Mkwevd0sLXR+bCC7qB7aB2F6AXaQm/QDhoh6t32Nr1t77X/xf/mf/d/LKS+1925hZbC//kbHALx4Q==</latexit>
This corresponds to finding j and s1 that minimise the MSE of the one-factor regression model
Yi = µ1 I{Xijsj} + µ2 I{Xij>sj}"i<latexit sha1_base64="3jqSnVhYBaqamChltnq39FcWWtA=">AAAC1XicfVFNbxMxEPUuX2n4aApHLgMpEhIo2g0tFCFQJS7AqUikCYqjldfrtG5t72J7C5HlG+LK7+uvAW+yVC0qzMjS85v3/DGTV4IbmySnUXzl6rXrNzpr3Zu3bt9Z723c3TdlrSkb0VKUepITwwRXbGS5FWxSaUZkLtg4P37b1McnTBteqk92UbGZJAeKzzklNlBZz3zOOLwGACzrzKUe8NMGE3uY5+69zxx2MMkcPwoVwb6AyY4Aew/wZOUYLh2XG96cqfEJ0awyXIQrOWS9fjJIlgHnwHaSvnyeQtoyfdTGXrYRfcBFSWvJlKWCGDNNk8rOHNGWU8F8F9eGVYQekwM2DVARyczMLbvj4VFgCpiXOixlYcmedzgijVnIPCibb5i/aw15WW1a2/nOzHFV1ZYpurpoXguwJTSthoJrRq1YBECo5uGtQA+JJtSGgXSxYl9pKSVRhcNcFd5hmZff3CaumK7wgzabzab3F+Xq//qzTUjAhlnJVW0Av2oOCt3/02L4N9gfDtJng+HHrf7uTjuHDrqPHqLHKEUv0C56h/bQCFF0in5FnWgtHsc+/h7/WEnjqPXcQxci/vkb27re6w==</latexit>
) bµm = yRm m = 1, 2<latexit sha1_base64="hn++SOZYSWZuxqw8MnslktjmHxc=">AAACnnicfVFdaxNBFJ1s/ajxo2l97MtoKvggYTdam2ILBUFEEGsxbSATlruzN8nQnZntzKwxLPtn/DW+6pv/xtl0FSvqvQyce+6583EmyTNhXRh+bwVr167fuLl+q337zt17G53NrVOrC8NxyHWmzSgBi5lQOHTCZTjKDYJMMjxLzl/W/bOPaKzQ6oNb5jiRMFNiKjg4T8WdA3YiZnMHxugFZRcFpJQtRIpzcCWTRRVLekhZAoYu4/IkllUjkofRk37c6Ya93TDafx7RsBeuwoP+brg/CGnUMF3SxHG82XrDUs0LicrxDKwdR2HuJiUYJ3iGVZsVFnPg5zDDsYcKJNpJuXpmRR95JqVTbfxSjq7Y3ydKkNYuZeKVEtzc/tmryb/1xoWbDialUHnhUPHLg6ZFRp2mtWc0FQa5y5YeADfC35XyORjgzjvbZgoXXEsJKi2ZUGnljUv0p3KH5Why9qDJutipqqty9X/9r8InZRadFKqwlL2oN/Lu/7SY/huc9nvR017//bPu0aD5h3WyTR6SxyQie+SIvCbHZEg4+Uy+kK/kW0CDV8Hb4N2lNGg1M/fJlQhGPwAwC8y+</latexit>
Estimate/Prediction
j=1, …, p
+
Regression trees from a regression perspective
12
(3) Search for the second split: repeat the procedure (2) within R1 or R2
This corresponds to finding k and s2 that minimise the MSE in one of the one-factor
regression models
Estimate/Prediction
R1 , R2 —> R1 , R2 , R3 minimising the loss function (MSE)
Yi = µ1 I{Xijs1} + µ2 I{Xij>s1}I{Xiks2} + µ3 I{Xij>s1}I{Xik>s2} + "i<latexit sha1_base64="aWhe8QCQb//B3Q++djGFB60LeAY=">AAADQ3icnVFdb9MwFHXC1yhfHTzycqFDQgJVScfGEGKaxAMfT0OiW1FdRY7jbqa2E2yno7Ly8/gRPPLMG+IVCafNpg4VHrhWpONzzj2OfdNCcGOj6GsQXrh46fKVtauta9dv3LzVXr99YPJSU9anucj1ICWGCa5Y33Ir2KDQjMhUsMN08rLWD6dMG56r93ZWsJEkR4qPOSXWU0n724eEwwsALMvExRXgxzUm9jhN3ZsqcdjBIHH8o1cE+wQmiQFXFcCjRUdv3rG6YffUvUKenOX15nlN3Ob/x+0uZ02JZoXhwt+QJ+1O1I3mBUtgK4qfbccQN0wHNbWfrAdvcZbTUjJlqSDGDOOosCNHtOVUsKqFS8MKQifkiA09VEQyM3LzWVTwwDMZjHPtP2Vhzi53OCKNmcnUO+t7mD+1mlylDUs73hk5rorSMkUXB41LATaHerCQcc2oFTMPCNXc/yvQY6IJtX78LazYCc2lJCpzmKusclim+We3gQumC3yvWfVmo6rO29W//WcbvwAbZiVXpQH8vA7yr3/6xPB3cNDrxpvd3rsnnb2dZg5r6C66jx6iGD1Fe+g12kd9RINXgQymwUn4Jfwe/gh/Lqxh0PTcQecq/PUb/yEIPw==</latexit>
Yi = µ1 I{Xijs1}I{Xiks2} + µ2 I{Xijs1}I{Xik>s2} + µ3 I{Xij>s1} + "i,<latexit sha1_base64="ZwSsHDzZp1hhnluWoRq1433gYPs=">AAADSHicjVLLbhMxFPVMeZTwaApLNhdSJCSqaCZtoQhRVWIDiEWRSBsURyOPx2lNbM9gewqRmR/kD/gB1uwQOzzJNGpRQFzL0vG55x4/rtNCcGOj6FsQrly6fOXq6rXW9Rs3b621128fmrzUlPVpLnI9SIlhgivWt9wKNig0IzIV7CidvKjzR6dMG56rd3ZasJEkx4qPOSXWU0n7+/uEw3MALMvExRXgzRoTe5Km7lWVOOxgkDj+wWcE+wgmiQFX1VLNZKHp1ZpHc8/e/3oud9w7s4PGb2vmt9xub3E+Lz4lmhWGC39Lvpm0O1E3mgWcAztR/PRxDHHDdFATB8l68BpnOS0lU5YKYswwjgo7ckRbTgWrWrg0rCB0Qo7Z0ENFJDMjN2tIBQ88k8E4134qCzP2fIUj0pipTL2yvoj5M1eTy3LD0o53R46rorRM0flG41KAzaHuLmRcM2rF1ANCNfdnBXpCNKHW/4EWVuwTzaUkKnOYq6xyWKb5Z7eBC6YLfK8Z9WKjqi7K1b/1i4UfgA2zkqvSAH5WG/nXP3ti+Ds47HXjrW7v7XZnf7fpwyq6i+6jhyhGT9A+eokOUB/R4E2gAxd8Cb+GP8Kf4a+5NAyamjvoQqyEvwH1nQke</latexit>
) bµm = yRm m = 1, . . . , 3<latexit sha1_base64="Icuy3eKqWhfY/RiR2q0UYDs/ZO4=">AAACpXicfVHbbhMxEHWWWwmXpvDIiyFF8FBFuymFVKhSpb4gJKRSNWmlOFrNep3Eqi+LPUuIVvtBfA2v9G9w0gVRBMzI0pkzZ3w5zgolPcbxZSu6cfPW7Tsbd9v37j94uNnZejTytnRcDLlV1p1n4IWSRgxRohLnhROgMyXOsoujVf/ss3BeWnOKy0JMNMyMnEoOGKi0c8RO5GyO4JxdUPaphJyyhczFHLBiuqxTTQ8oy8DRZVqdpLpuRPog2WEqt+h3dtNON+7txcn+64TGvXgdAfT34v1BTJOG6ZImjtOt1nuWW15qYZAr8H6cxAVOKnAouRJ1m5VeFMAvYCbGARrQwk+q9Wtr+jwwOZ1aF5ZBumZ/n6hAe7/UWVBqwLn/s7ci/9YblzgdTCppihKF4VcHTUtF0dKVdTSXTnBUywCAOxnuSvkcHHAMBreZEQtutQaTV0yavA7+ZfZLtc0K4Qr2tMlVsV3X1+Xm//pfRUjKvEAtTekpe7vaKLj/02L6bzDq95LdXv/jq+7hoPmHDfKEPCMvSULekEPyjhyTIeHkK/lGvpPL6EX0ITqNRlfSqNXMPCbXIkp/ADGvz7M=</latexit>
The tree model : assigning a value to each terminal node
13
conditional means
E[Y | X = x] =MX
m=1
µmI{x2Rm}<latexit sha1_base64="Y2jmsOr295e4AWR92gXAVO8I2Uo=">AAACqXicfVFtixMxEE7Xt7O+9fSjX0Z7giAs2dLqFSkcivgCwp3Yu2rTW7LZ9C7cJrskWW0J+Un+Gj8J+l/M9nriiTpD4JmZZ5LJM1lVCGMx/taKLly8dPnKxtX2tes3bt7qbN7eN2WtGR+zsij1JKOGF0LxsRW24JNKcyqzgh9kJ8+b+sEnro0o1Xu7rPhM0iMl5oJRG1Jp5yWR1B5nmXvhpx+ASJHDZLSAGYyAmFqmTo4Sf/g2VOpUwhn5tU8dcQsgQsG7VBLv004Xx7i3Pej3AMe9AR4mwwAGOBk+7kMS45V10dp2083WG5KXrJZcWVZQY6YJruzMUW0FK7hvk9rwirITesSnASoquZm51Y89PAiZHOalDkdZWGV/73BUGrOUWWA2I5s/a03yb7VpbefbMydUVVuu2OlD87oAW0IjH+RCc2aLZQCUaRFmBXZMNWU2iNwmin9mpZRU5S5ok3tHZFYu3BapuK7IvbU3wZb35+nq//xfQfCwGm6lULUB8rS5KKh/JjH8G+z34gTHyV6/u/NsvYcNdBfdRw9Rgp6gHfQK7aIxYugL+oq+ox/Ro2gvmkQfT6lRa91zB52ziP0Ehs7RKw==</latexit><latexit sha1_base64="Y2jmsOr295e4AWR92gXAVO8I2Uo=">AAACqXicfVFtixMxEE7Xt7O+9fSjX0Z7giAs2dLqFSkcivgCwp3Yu2rTW7LZ9C7cJrskWW0J+Un+Gj8J+l/M9nriiTpD4JmZZ5LJM1lVCGMx/taKLly8dPnKxtX2tes3bt7qbN7eN2WtGR+zsij1JKOGF0LxsRW24JNKcyqzgh9kJ8+b+sEnro0o1Xu7rPhM0iMl5oJRG1Jp5yWR1B5nmXvhpx+ASJHDZLSAGYyAmFqmTo4Sf/g2VOpUwhn5tU8dcQsgQsG7VBLv004Xx7i3Pej3AMe9AR4mwwAGOBk+7kMS45V10dp2083WG5KXrJZcWVZQY6YJruzMUW0FK7hvk9rwirITesSnASoquZm51Y89PAiZHOalDkdZWGV/73BUGrOUWWA2I5s/a03yb7VpbefbMydUVVuu2OlD87oAW0IjH+RCc2aLZQCUaRFmBXZMNWU2iNwmin9mpZRU5S5ok3tHZFYu3BapuK7IvbU3wZb35+nq//xfQfCwGm6lULUB8rS5KKh/JjH8G+z34gTHyV6/u/NsvYcNdBfdRw9Rgp6gHfQK7aIxYugL+oq+ox/Ro2gvmkQfT6lRa91zB52ziP0Ehs7RKw==</latexit><latexit sha1_base64="Y2jmsOr295e4AWR92gXAVO8I2Uo=">AAACqXicfVFtixMxEE7Xt7O+9fSjX0Z7giAs2dLqFSkcivgCwp3Yu2rTW7LZ9C7cJrskWW0J+Un+Gj8J+l/M9nriiTpD4JmZZ5LJM1lVCGMx/taKLly8dPnKxtX2tes3bt7qbN7eN2WtGR+zsij1JKOGF0LxsRW24JNKcyqzgh9kJ8+b+sEnro0o1Xu7rPhM0iMl5oJRG1Jp5yWR1B5nmXvhpx+ASJHDZLSAGYyAmFqmTo4Sf/g2VOpUwhn5tU8dcQsgQsG7VBLv004Xx7i3Pej3AMe9AR4mwwAGOBk+7kMS45V10dp2083WG5KXrJZcWVZQY6YJruzMUW0FK7hvk9rwirITesSnASoquZm51Y89PAiZHOalDkdZWGV/73BUGrOUWWA2I5s/a03yb7VpbefbMydUVVuu2OlD87oAW0IjH+RCc2aLZQCUaRFmBXZMNWU2iNwmin9mpZRU5S5ok3tHZFYu3BapuK7IvbU3wZb35+nq//xfQfCwGm6lULUB8rS5KKh/JjH8G+z34gTHyV6/u/NsvYcNdBfdRw9Rgp6gHfQK7aIxYugL+oq+ox/Ro2gvmkQfT6lRa91zB52ziP0Ehs7RKw==</latexit><latexit sha1_base64="Y2jmsOr295e4AWR92gXAVO8I2Uo=">AAACqXicfVFtixMxEE7Xt7O+9fSjX0Z7giAs2dLqFSkcivgCwp3Yu2rTW7LZ9C7cJrskWW0J+Un+Gj8J+l/M9nriiTpD4JmZZ5LJM1lVCGMx/taKLly8dPnKxtX2tes3bt7qbN7eN2WtGR+zsij1JKOGF0LxsRW24JNKcyqzgh9kJ8+b+sEnro0o1Xu7rPhM0iMl5oJRG1Jp5yWR1B5nmXvhpx+ASJHDZLSAGYyAmFqmTo4Sf/g2VOpUwhn5tU8dcQsgQsG7VBLv004Xx7i3Pej3AMe9AR4mwwAGOBk+7kMS45V10dp2083WG5KXrJZcWVZQY6YJruzMUW0FK7hvk9rwirITesSnASoquZm51Y89PAiZHOalDkdZWGV/73BUGrOUWWA2I5s/a03yb7VpbefbMydUVVuu2OlD87oAW0IjH+RCc2aLZQCUaRFmBXZMNWU2iNwmin9mpZRU5S5ok3tHZFYu3BapuK7IvbU3wZb35+nq//xfQfCwGm6lULUB8rS5KKh/JjH8G+z34gTHyV6/u/NsvYcNdBfdRw9Rgp6gHfQK7aIxYugL+oq+ox/Ro2gvmkQfT6lRa91zB52ziP0Ehs7RKw==</latexit>
R1 : x1 < 3 ^ x2 < 1.5 ) µ1 = 60<latexit sha1_base64="KnSd0jQICzsmRk5PD+V+o7aoKY0=">AAACnnicfVHbbhMxEHWWS9twS+GxLwMpEg9otdvSGwSpAgkhJESpSBspjlZe7ySxurZXtpc0Wu3P8DW8wht/gzcNiHKbkaVzZs74cpwWubAuir61gitXr11fWV1r37h56/adzvrdE6tLw7HPda7NIGUWc6Gw74TLcVAYZDLN8TQ9e9n0Tz+isUKrD25e4EiyiRJjwZnzpaTTO05ieAr0GZwncQ+2gc4wm6BnWz2Iwx2gj4Eei8nUMWP0bEFl6Weew26UdLpRGB3sR9sH8CeIw2gRXbKMo2S99YZmmpcSleM5s3YYR4UbVcw4wXOs27S0WDB+xiY49FAxiXZULZ5Zw0NfyWCsjV/KwaL660TFpLVzmXqlZG5qf+81xb/1hqUb748qoYrSoeIXB43LHJyGxjPIhEHu8rkHjBvh7wp8ygzjzjvbpgpnXEvJVFZRobK6ojLV59UmLdAU9P4yG7JZ15fl6v/6n8QnUItOClVa/1vNRt79HxbDv8HJVhhHYfz+SffwxfIfVskGeUAekZjskUPymhyRPuHkE/lMvpCvAQSvgrfBuwtp0FrO3COXIhh8B6ajyFI=</latexit><latexit sha1_base64="KnSd0jQICzsmRk5PD+V+o7aoKY0=">AAACnnicfVHbbhMxEHWWS9twS+GxLwMpEg9otdvSGwSpAgkhJESpSBspjlZe7ySxurZXtpc0Wu3P8DW8wht/gzcNiHKbkaVzZs74cpwWubAuir61gitXr11fWV1r37h56/adzvrdE6tLw7HPda7NIGUWc6Gw74TLcVAYZDLN8TQ9e9n0Tz+isUKrD25e4EiyiRJjwZnzpaTTO05ieAr0GZwncQ+2gc4wm6BnWz2Iwx2gj4Eei8nUMWP0bEFl6Weew26UdLpRGB3sR9sH8CeIw2gRXbKMo2S99YZmmpcSleM5s3YYR4UbVcw4wXOs27S0WDB+xiY49FAxiXZULZ5Zw0NfyWCsjV/KwaL660TFpLVzmXqlZG5qf+81xb/1hqUb748qoYrSoeIXB43LHJyGxjPIhEHu8rkHjBvh7wp8ygzjzjvbpgpnXEvJVFZRobK6ojLV59UmLdAU9P4yG7JZ15fl6v/6n8QnUItOClVa/1vNRt79HxbDv8HJVhhHYfz+SffwxfIfVskGeUAekZjskUPymhyRPuHkE/lMvpCvAQSvgrfBuwtp0FrO3COXIhh8B6ajyFI=</latexit><latexit sha1_base64="KnSd0jQICzsmRk5PD+V+o7aoKY0=">AAACnnicfVHbbhMxEHWWS9twS+GxLwMpEg9otdvSGwSpAgkhJESpSBspjlZe7ySxurZXtpc0Wu3P8DW8wht/gzcNiHKbkaVzZs74cpwWubAuir61gitXr11fWV1r37h56/adzvrdE6tLw7HPda7NIGUWc6Gw74TLcVAYZDLN8TQ9e9n0Tz+isUKrD25e4EiyiRJjwZnzpaTTO05ieAr0GZwncQ+2gc4wm6BnWz2Iwx2gj4Eei8nUMWP0bEFl6Weew26UdLpRGB3sR9sH8CeIw2gRXbKMo2S99YZmmpcSleM5s3YYR4UbVcw4wXOs27S0WDB+xiY49FAxiXZULZ5Zw0NfyWCsjV/KwaL660TFpLVzmXqlZG5qf+81xb/1hqUb748qoYrSoeIXB43LHJyGxjPIhEHu8rkHjBvh7wp8ygzjzjvbpgpnXEvJVFZRobK6ojLV59UmLdAU9P4yG7JZ15fl6v/6n8QnUItOClVa/1vNRt79HxbDv8HJVhhHYfz+SffwxfIfVskGeUAekZjskUPymhyRPuHkE/lMvpCvAQSvgrfBuwtp0FrO3COXIhh8B6ajyFI=</latexit><latexit sha1_base64="KnSd0jQICzsmRk5PD+V+o7aoKY0=">AAACnnicfVHbbhMxEHWWS9twS+GxLwMpEg9otdvSGwSpAgkhJESpSBspjlZe7ySxurZXtpc0Wu3P8DW8wht/gzcNiHKbkaVzZs74cpwWubAuir61gitXr11fWV1r37h56/adzvrdE6tLw7HPda7NIGUWc6Gw74TLcVAYZDLN8TQ9e9n0Tz+isUKrD25e4EiyiRJjwZnzpaTTO05ieAr0GZwncQ+2gc4wm6BnWz2Iwx2gj4Eei8nUMWP0bEFl6Weew26UdLpRGB3sR9sH8CeIw2gRXbKMo2S99YZmmpcSleM5s3YYR4UbVcw4wXOs27S0WDB+xiY49FAxiXZULZ5Zw0NfyWCsjV/KwaL660TFpLVzmXqlZG5qf+81xb/1hqUb748qoYrSoeIXB43LHJyGxjPIhEHu8rkHjBvh7wp8ygzjzjvbpgpnXEvJVFZRobK6ojLV59UmLdAU9P4yG7JZ15fl6v/6n8QnUItOClVa/1vNRt79HxbDv8HJVhhHYfz+SffwxfIfVskGeUAekZjskUPymhyRPuHkE/lMvpCvAQSvgrfBuwtp0FrO3COXIhh8B6ajyFI=</latexit>
R2 : x1 < 3 ^ x2 � 1.5 ) µ2 = 100<latexit sha1_base64="zJUHIZd6JAKg/Lb2eZxwJ5zcyyk=">AAACo3icfVHbbhMxEHWWWwm3FB55GUgr8YBWuymoLRepwAuClxI1baV4FXm9k43Vtb21vaTRar+Hr+EVxN/gTQOi3GZkac6ZM74cp2UhrIuib53g0uUrV6+tXe/euHnr9p3e+t1DqyvDccR1oc1xyiwWQuHICVfgcWmQybTAo/TkTds/+ojGCq0O3KLERLJciangzHlq0ns1nAzgGdDncDaJX8AW0DlmOXo0AJrjKcThU6CPgQ5FPnPMGD1fQll5wUuIo2jS60dhtLsTbe3Cn0UcRsvok1XsT9Y772imeSVROV4wa8dxVLqkZsYJXmDTpZXFkvETluPYl4pJtEm9fGsDm57JYKqNX8rBkv11ombS2oVMvVIyN7O/91ryb71x5aY7SS1UWTlU/PygaVWA09AaB5kwyF2x8AXjRvi7Ap8xw7jz9napwjnXUjKV1VSorKmpTPVZvUFLNCV9sMoWbDTNRbn6v/4n8AnUopNCVdZ/WbuRd/+HxfDv4nAQxlEYf3jS33u9+oc1cp88JI9ITLbJHnlL9smIcPKJfCZfyNdgM3gfDIODc2nQWc3cIxciSL4DF9bKLg==</latexit><latexit sha1_base64="zJUHIZd6JAKg/Lb2eZxwJ5zcyyk=">AAACo3icfVHbbhMxEHWWWwm3FB55GUgr8YBWuymoLRepwAuClxI1baV4FXm9k43Vtb21vaTRar+Hr+EVxN/gTQOi3GZkac6ZM74cp2UhrIuib53g0uUrV6+tXe/euHnr9p3e+t1DqyvDccR1oc1xyiwWQuHICVfgcWmQybTAo/TkTds/+ojGCq0O3KLERLJciangzHlq0ns1nAzgGdDncDaJX8AW0DlmOXo0AJrjKcThU6CPgQ5FPnPMGD1fQll5wUuIo2jS60dhtLsTbe3Cn0UcRsvok1XsT9Y772imeSVROV4wa8dxVLqkZsYJXmDTpZXFkvETluPYl4pJtEm9fGsDm57JYKqNX8rBkv11ombS2oVMvVIyN7O/91ryb71x5aY7SS1UWTlU/PygaVWA09AaB5kwyF2x8AXjRvi7Ap8xw7jz9napwjnXUjKV1VSorKmpTPVZvUFLNCV9sMoWbDTNRbn6v/4n8AnUopNCVdZ/WbuRd/+HxfDv4nAQxlEYf3jS33u9+oc1cp88JI9ITLbJHnlL9smIcPKJfCZfyNdgM3gfDIODc2nQWc3cIxciSL4DF9bKLg==</latexit><latexit sha1_base64="zJUHIZd6JAKg/Lb2eZxwJ5zcyyk=">AAACo3icfVHbbhMxEHWWWwm3FB55GUgr8YBWuymoLRepwAuClxI1baV4FXm9k43Vtb21vaTRar+Hr+EVxN/gTQOi3GZkac6ZM74cp2UhrIuib53g0uUrV6+tXe/euHnr9p3e+t1DqyvDccR1oc1xyiwWQuHICVfgcWmQybTAo/TkTds/+ojGCq0O3KLERLJciangzHlq0ns1nAzgGdDncDaJX8AW0DlmOXo0AJrjKcThU6CPgQ5FPnPMGD1fQll5wUuIo2jS60dhtLsTbe3Cn0UcRsvok1XsT9Y772imeSVROV4wa8dxVLqkZsYJXmDTpZXFkvETluPYl4pJtEm9fGsDm57JYKqNX8rBkv11ombS2oVMvVIyN7O/91ryb71x5aY7SS1UWTlU/PygaVWA09AaB5kwyF2x8AXjRvi7Ap8xw7jz9napwjnXUjKV1VSorKmpTPVZvUFLNCV9sMoWbDTNRbn6v/4n8AnUopNCVdZ/WbuRd/+HxfDv4nAQxlEYf3jS33u9+oc1cp88JI9ITLbJHnlL9smIcPKJfCZfyNdgM3gfDIODc2nQWc3cIxciSL4DF9bKLg==</latexit><latexit sha1_base64="zJUHIZd6JAKg/Lb2eZxwJ5zcyyk=">AAACo3icfVHbbhMxEHWWWwm3FB55GUgr8YBWuymoLRepwAuClxI1baV4FXm9k43Vtb21vaTRar+Hr+EVxN/gTQOi3GZkac6ZM74cp2UhrIuib53g0uUrV6+tXe/euHnr9p3e+t1DqyvDccR1oc1xyiwWQuHICVfgcWmQybTAo/TkTds/+ojGCq0O3KLERLJciangzHlq0ns1nAzgGdDncDaJX8AW0DlmOXo0AJrjKcThU6CPgQ5FPnPMGD1fQll5wUuIo2jS60dhtLsTbe3Cn0UcRsvok1XsT9Y772imeSVROV4wa8dxVLqkZsYJXmDTpZXFkvETluPYl4pJtEm9fGsDm57JYKqNX8rBkv11ombS2oVMvVIyN7O/91ryb71x5aY7SS1UWTlU/PygaVWA09AaB5kwyF2x8AXjRvi7Ap8xw7jz9napwjnXUjKV1VSorKmpTPVZvUFLNCV9sMoWbDTNRbn6v/4n8AnUopNCVdZ/WbuRd/+HxfDv4nAQxlEYf3jS33u9+oc1cp88JI9ITLbJHnlL9smIcPKJfCZfyNdgM3gfDIODc2nQWc3cIxciSL4DF9bKLg==</latexit>
R4 : x1 � 3 ^ x1 � 4 ) µ4 = 30<latexit sha1_base64="eEqsyyfxJKZHKOvUf2Pin8/BR04=">AAACo3icfVHbbhMxEHWWWwm3FB55GUgr8YCiXRKJtgipwAuClxI1baV4FXm9k43Vtb21vaTRar+Hr+EVxN/g3QZEuc3I0pkzZ3w5TopcWBeG3zrBlavXrt/YuNm9dfvO3Xu9zftHVpeG44TrXJuThFnMhcKJEy7Hk8Igk0mOx8npm6Z//BGNFVodulWBsWSZEnPBmfPUrPdqPBvBHtAXcD6LaIZnMAS6xDTDhoCWGQF9CnQssoVjxuhlW8rSD76EYTjr9cNBuLsTDnfhTxANwjb6ZB0Hs83OO5pqXkpUjufM2mkUFi6umHGC51h3aWmxYPyUZTj1UDGJNq7at9aw7ZkU5tr4pRy07K8TFZPWrmTilZK5hf2915B/601LN9+JK6GK0qHiFwfNyxychsY4SIVB7vKVB4wb4e8KfMEM487b26UKl1xLyVRaUaHSuqIy0efVFi3QFPTROptiq64vy9X/9T8Ln0AtOilUaf2XNRt5939YDP8GR88GUTiIPoz6+6/X/7BBHpLH5AmJyHOyT96SAzIhnHwin8kX8jXYDt4H4+DwQhp01jMPyKUI4u/X1ssA</latexit><latexit sha1_base64="eEqsyyfxJKZHKOvUf2Pin8/BR04=">AAACo3icfVHbbhMxEHWWWwm3FB55GUgr8YCiXRKJtgipwAuClxI1baV4FXm9k43Vtb21vaTRar+Hr+EVxN/g3QZEuc3I0pkzZ3w5TopcWBeG3zrBlavXrt/YuNm9dfvO3Xu9zftHVpeG44TrXJuThFnMhcKJEy7Hk8Igk0mOx8npm6Z//BGNFVodulWBsWSZEnPBmfPUrPdqPBvBHtAXcD6LaIZnMAS6xDTDhoCWGQF9CnQssoVjxuhlW8rSD76EYTjr9cNBuLsTDnfhTxANwjb6ZB0Hs83OO5pqXkpUjufM2mkUFi6umHGC51h3aWmxYPyUZTj1UDGJNq7at9aw7ZkU5tr4pRy07K8TFZPWrmTilZK5hf2915B/601LN9+JK6GK0qHiFwfNyxychsY4SIVB7vKVB4wb4e8KfMEM487b26UKl1xLyVRaUaHSuqIy0efVFi3QFPTROptiq64vy9X/9T8Ln0AtOilUaf2XNRt5939YDP8GR88GUTiIPoz6+6/X/7BBHpLH5AmJyHOyT96SAzIhnHwin8kX8jXYDt4H4+DwQhp01jMPyKUI4u/X1ssA</latexit><latexit sha1_base64="eEqsyyfxJKZHKOvUf2Pin8/BR04=">AAACo3icfVHbbhMxEHWWWwm3FB55GUgr8YCiXRKJtgipwAuClxI1baV4FXm9k43Vtb21vaTRar+Hr+EVxN/g3QZEuc3I0pkzZ3w5TopcWBeG3zrBlavXrt/YuNm9dfvO3Xu9zftHVpeG44TrXJuThFnMhcKJEy7Hk8Igk0mOx8npm6Z//BGNFVodulWBsWSZEnPBmfPUrPdqPBvBHtAXcD6LaIZnMAS6xDTDhoCWGQF9CnQssoVjxuhlW8rSD76EYTjr9cNBuLsTDnfhTxANwjb6ZB0Hs83OO5pqXkpUjufM2mkUFi6umHGC51h3aWmxYPyUZTj1UDGJNq7at9aw7ZkU5tr4pRy07K8TFZPWrmTilZK5hf2915B/601LN9+JK6GK0qHiFwfNyxychsY4SIVB7vKVB4wb4e8KfMEM487b26UKl1xLyVRaUaHSuqIy0efVFi3QFPTROptiq64vy9X/9T8Ln0AtOilUaf2XNRt5939YDP8GR88GUTiIPoz6+6/X/7BBHpLH5AmJyHOyT96SAzIhnHwin8kX8jXYDt4H4+DwQhp01jMPyKUI4u/X1ssA</latexit><latexit sha1_base64="eEqsyyfxJKZHKOvUf2Pin8/BR04=">AAACo3icfVHbbhMxEHWWWwm3FB55GUgr8YCiXRKJtgipwAuClxI1baV4FXm9k43Vtb21vaTRar+Hr+EVxN/g3QZEuc3I0pkzZ3w5TopcWBeG3zrBlavXrt/YuNm9dfvO3Xu9zftHVpeG44TrXJuThFnMhcKJEy7Hk8Igk0mOx8npm6Z//BGNFVodulWBsWSZEnPBmfPUrPdqPBvBHtAXcD6LaIZnMAS6xDTDhoCWGQF9CnQssoVjxuhlW8rSD76EYTjr9cNBuLsTDnfhTxANwjb6ZB0Hs83OO5pqXkpUjufM2mkUFi6umHGC51h3aWmxYPyUZTj1UDGJNq7at9aw7ZkU5tr4pRy07K8TFZPWrmTilZK5hf2915B/601LN9+JK6GK0qHiFwfNyxychsY4SIVB7vKVB4wb4e8KfMEM487b26UKl1xLyVRaUaHSuqIy0efVFi3QFPTROptiq64vy9X/9T8Ln0AtOilUaf2XNRt5939YDP8GR88GUTiIPoz6+6/X/7BBHpLH5AmJyHOyT96SAzIhnHwin8kX8jXYDt4H4+DwQhp01jMPyKUI4u/X1ssA</latexit>
45 30R3 : x1 � 3 ^ x1 < 4 ) µ3 = 45
<latexit sha1_base64="W4+yVGs8uGqm30UKpoJrTZcJsxs=">AAACoHicfVHLbhMxFHWmQEt4pWXJ5tIUiQWKZkgqWl6qYFOxoa1IWxFHI4/nZmJ1bA+2hzQazd/wNWxhw9/gSQOivO6VpXPPPdeP46TIhXVh+K0VrFy5em117Xr7xs1bt+901jeOrS4NxyHXuTanCbOYC4VDJ1yOp4VBJpMcT5Kz103/5CMaK7R65+YFjiXLlJgIzpyn4s7Lo7gPT4E+g/M4ohl+gD7QGaYZNgQ8hwHQR0CPRDZ1zBg9W5Sy9FMvYLAdd7phL9zdCfu78CeIeuEiumQZB/F66w1NNS8lKsdzZu0oCgs3rphxgudYt2lpsWD8jGU48lAxiXZcLR5awwPPpDDRxi/lYMH+OlExae1cJl4pmZva33sN+bfeqHSTnXElVFE6VPzioEmZg9PQuAapMMhdPveAcSP8XYFPmWHceW/bVOGMaymZSisqVFpXVCb6vNqiBZqC3l9mU2zV9WW5+r/+Z+ETqEUnhSqt/69mI+/+D4vh3+D4cS8Ke9HhoLv3avkPa+Qe2SQPSUSekD2yTw7IkHDyiXwmX8jXYDPYD94GhxfSoLWcuUsuRfD+O3B1yYk=</latexit><latexit sha1_base64="W4+yVGs8uGqm30UKpoJrTZcJsxs=">AAACoHicfVHLbhMxFHWmQEt4pWXJ5tIUiQWKZkgqWl6qYFOxoa1IWxFHI4/nZmJ1bA+2hzQazd/wNWxhw9/gSQOivO6VpXPPPdeP46TIhXVh+K0VrFy5em117Xr7xs1bt+901jeOrS4NxyHXuTanCbOYC4VDJ1yOp4VBJpMcT5Kz103/5CMaK7R65+YFjiXLlJgIzpyn4s7Lo7gPT4E+g/M4ohl+gD7QGaYZNgQ8hwHQR0CPRDZ1zBg9W5Sy9FMvYLAdd7phL9zdCfu78CeIeuEiumQZB/F66w1NNS8lKsdzZu0oCgs3rphxgudYt2lpsWD8jGU48lAxiXZcLR5awwPPpDDRxi/lYMH+OlExae1cJl4pmZva33sN+bfeqHSTnXElVFE6VPzioEmZg9PQuAapMMhdPveAcSP8XYFPmWHceW/bVOGMaymZSisqVFpXVCb6vNqiBZqC3l9mU2zV9WW5+r/+Z+ETqEUnhSqt/69mI+/+D4vh3+D4cS8Ke9HhoLv3avkPa+Qe2SQPSUSekD2yTw7IkHDyiXwmX8jXYDPYD94GhxfSoLWcuUsuRfD+O3B1yYk=</latexit><latexit sha1_base64="W4+yVGs8uGqm30UKpoJrTZcJsxs=">AAACoHicfVHLbhMxFHWmQEt4pWXJ5tIUiQWKZkgqWl6qYFOxoa1IWxFHI4/nZmJ1bA+2hzQazd/wNWxhw9/gSQOivO6VpXPPPdeP46TIhXVh+K0VrFy5em117Xr7xs1bt+901jeOrS4NxyHXuTanCbOYC4VDJ1yOp4VBJpMcT5Kz103/5CMaK7R65+YFjiXLlJgIzpyn4s7Lo7gPT4E+g/M4ohl+gD7QGaYZNgQ8hwHQR0CPRDZ1zBg9W5Sy9FMvYLAdd7phL9zdCfu78CeIeuEiumQZB/F66w1NNS8lKsdzZu0oCgs3rphxgudYt2lpsWD8jGU48lAxiXZcLR5awwPPpDDRxi/lYMH+OlExae1cJl4pmZva33sN+bfeqHSTnXElVFE6VPzioEmZg9PQuAapMMhdPveAcSP8XYFPmWHceW/bVOGMaymZSisqVFpXVCb6vNqiBZqC3l9mU2zV9WW5+r/+Z+ETqEUnhSqt/69mI+/+D4vh3+D4cS8Ke9HhoLv3avkPa+Qe2SQPSUSekD2yTw7IkHDyiXwmX8jXYDPYD94GhxfSoLWcuUsuRfD+O3B1yYk=</latexit><latexit sha1_base64="W4+yVGs8uGqm30UKpoJrTZcJsxs=">AAACoHicfVHLbhMxFHWmQEt4pWXJ5tIUiQWKZkgqWl6qYFOxoa1IWxFHI4/nZmJ1bA+2hzQazd/wNWxhw9/gSQOivO6VpXPPPdeP46TIhXVh+K0VrFy5em117Xr7xs1bt+901jeOrS4NxyHXuTanCbOYC4VDJ1yOp4VBJpMcT5Kz103/5CMaK7R65+YFjiXLlJgIzpyn4s7Lo7gPT4E+g/M4ohl+gD7QGaYZNgQ8hwHQR0CPRDZ1zBg9W5Sy9FMvYLAdd7phL9zdCfu78CeIeuEiumQZB/F66w1NNS8lKsdzZu0oCgs3rphxgudYt2lpsWD8jGU48lAxiXZcLR5awwPPpDDRxi/lYMH+OlExae1cJl4pmZva33sN+bfeqHSTnXElVFE6VPzioEmZg9PQuAapMMhdPveAcSP8XYFPmWHceW/bVOGMaymZSisqVFpXVCb6vNqiBZqC3l9mU2zV9WW5+r/+Z+ETqEUnhSqt/69mI+/+D4vh3+D4cS8Ke9HhoLv3avkPa+Qe2SQPSUSekD2yTw7IkHDyiXwmX8jXYDPYD94GhxfSoLWcuUsuRfD+O3B1yYk=</latexit>
The tree model
14
E[Y | X = x] =MX
m=1
µmI{x2Rm}<latexit sha1_base64="Y2jmsOr295e4AWR92gXAVO8I2Uo=">AAACqXicfVFtixMxEE7Xt7O+9fSjX0Z7giAs2dLqFSkcivgCwp3Yu2rTW7LZ9C7cJrskWW0J+Un+Gj8J+l/M9nriiTpD4JmZZ5LJM1lVCGMx/taKLly8dPnKxtX2tes3bt7qbN7eN2WtGR+zsij1JKOGF0LxsRW24JNKcyqzgh9kJ8+b+sEnro0o1Xu7rPhM0iMl5oJRG1Jp5yWR1B5nmXvhpx+ASJHDZLSAGYyAmFqmTo4Sf/g2VOpUwhn5tU8dcQsgQsG7VBLv004Xx7i3Pej3AMe9AR4mwwAGOBk+7kMS45V10dp2083WG5KXrJZcWVZQY6YJruzMUW0FK7hvk9rwirITesSnASoquZm51Y89PAiZHOalDkdZWGV/73BUGrOUWWA2I5s/a03yb7VpbefbMydUVVuu2OlD87oAW0IjH+RCc2aLZQCUaRFmBXZMNWU2iNwmin9mpZRU5S5ok3tHZFYu3BapuK7IvbU3wZb35+nq//xfQfCwGm6lULUB8rS5KKh/JjH8G+z34gTHyV6/u/NsvYcNdBfdRw9Rgp6gHfQK7aIxYugL+oq+ox/Ro2gvmkQfT6lRa91zB52ziP0Ehs7RKw==</latexit><latexit sha1_base64="Y2jmsOr295e4AWR92gXAVO8I2Uo=">AAACqXicfVFtixMxEE7Xt7O+9fSjX0Z7giAs2dLqFSkcivgCwp3Yu2rTW7LZ9C7cJrskWW0J+Un+Gj8J+l/M9nriiTpD4JmZZ5LJM1lVCGMx/taKLly8dPnKxtX2tes3bt7qbN7eN2WtGR+zsij1JKOGF0LxsRW24JNKcyqzgh9kJ8+b+sEnro0o1Xu7rPhM0iMl5oJRG1Jp5yWR1B5nmXvhpx+ASJHDZLSAGYyAmFqmTo4Sf/g2VOpUwhn5tU8dcQsgQsG7VBLv004Xx7i3Pej3AMe9AR4mwwAGOBk+7kMS45V10dp2083WG5KXrJZcWVZQY6YJruzMUW0FK7hvk9rwirITesSnASoquZm51Y89PAiZHOalDkdZWGV/73BUGrOUWWA2I5s/a03yb7VpbefbMydUVVuu2OlD87oAW0IjH+RCc2aLZQCUaRFmBXZMNWU2iNwmin9mpZRU5S5ok3tHZFYu3BapuK7IvbU3wZb35+nq//xfQfCwGm6lULUB8rS5KKh/JjH8G+z34gTHyV6/u/NsvYcNdBfdRw9Rgp6gHfQK7aIxYugL+oq+ox/Ro2gvmkQfT6lRa91zB52ziP0Ehs7RKw==</latexit><latexit sha1_base64="Y2jmsOr295e4AWR92gXAVO8I2Uo=">AAACqXicfVFtixMxEE7Xt7O+9fSjX0Z7giAs2dLqFSkcivgCwp3Yu2rTW7LZ9C7cJrskWW0J+Un+Gj8J+l/M9nriiTpD4JmZZ5LJM1lVCGMx/taKLly8dPnKxtX2tes3bt7qbN7eN2WtGR+zsij1JKOGF0LxsRW24JNKcyqzgh9kJ8+b+sEnro0o1Xu7rPhM0iMl5oJRG1Jp5yWR1B5nmXvhpx+ASJHDZLSAGYyAmFqmTo4Sf/g2VOpUwhn5tU8dcQsgQsG7VBLv004Xx7i3Pej3AMe9AR4mwwAGOBk+7kMS45V10dp2083WG5KXrJZcWVZQY6YJruzMUW0FK7hvk9rwirITesSnASoquZm51Y89PAiZHOalDkdZWGV/73BUGrOUWWA2I5s/a03yb7VpbefbMydUVVuu2OlD87oAW0IjH+RCc2aLZQCUaRFmBXZMNWU2iNwmin9mpZRU5S5ok3tHZFYu3BapuK7IvbU3wZb35+nq//xfQfCwGm6lULUB8rS5KKh/JjH8G+z34gTHyV6/u/NsvYcNdBfdRw9Rgp6gHfQK7aIxYugL+oq+ox/Ro2gvmkQfT6lRa91zB52ziP0Ehs7RKw==</latexit><latexit sha1_base64="Y2jmsOr295e4AWR92gXAVO8I2Uo=">AAACqXicfVFtixMxEE7Xt7O+9fSjX0Z7giAs2dLqFSkcivgCwp3Yu2rTW7LZ9C7cJrskWW0J+Un+Gj8J+l/M9nriiTpD4JmZZ5LJM1lVCGMx/taKLly8dPnKxtX2tes3bt7qbN7eN2WtGR+zsij1JKOGF0LxsRW24JNKcyqzgh9kJ8+b+sEnro0o1Xu7rPhM0iMl5oJRG1Jp5yWR1B5nmXvhpx+ASJHDZLSAGYyAmFqmTo4Sf/g2VOpUwhn5tU8dcQsgQsG7VBLv004Xx7i3Pej3AMe9AR4mwwAGOBk+7kMS45V10dp2083WG5KXrJZcWVZQY6YJruzMUW0FK7hvk9rwirITesSnASoquZm51Y89PAiZHOalDkdZWGV/73BUGrOUWWA2I5s/a03yb7VpbefbMydUVVuu2OlD87oAW0IjH+RCc2aLZQCUaRFmBXZMNWU2iNwmin9mpZRU5S5ok3tHZFYu3BapuK7IvbU3wZb35+nq//xfQfCwGm6lULUB8rS5KKh/JjH8G+z34gTHyV6/u/NsvYcNdBfdRw9Rgp6gHfQK7aIxYugL+oq+ox/Ro2gvmkQfT6lRa91zB52ziP0Ehs7RKw==</latexit>
The tree-building algorithm: stopping rule
STOPPING RULE
On the training data:
To avoid to have leaves with only one unit - perfect (over)fitting
- stop if the node contains less than a pre-specified minimum node size (usually 5, but depends on n)
- stop if the pre-specified maximum tree depth limit is reached
15
Pruning a tree
This process may produce good predictions on the training set, but is
likely to overfit the data, leading to poor test set performance.
A smaller tree with fewer splits (fewer regions R1 … RJ) might lead to
lower variance and better interpretation at the cost of a little bias.
Good strategy: grow a very large tree then prune it back
18
Pruning a tree : cost-complexity measure
20
Cost-complexity pruning :
Construct a sequence of sub-trees, pruned at different depth d, having
numbers of nodes varying from 1 to |Td|
Compute the cost complexity measure for the tree, which is based on
CP (d) =
|Td|X
m=1
X
i:xi2Rm
(yi � µm)2 + ↵|Td|<latexit sha1_base64="X34oKPUyfzHHZcEkmlVzLHMwL5M=">AAACtHicfVHbbhMxEHWWWwmXpu0jLwMpUipEtFuqXkCVKvqCeApV01aK05XXO2ms2t6V7aWNtvtf/AovvMJn4E0XRLmNZenMmTO+nElyKawLw8+t4NbtO3fvLdxvP3j46PFiZ2n5yGaF4TjkmczMScIsSqFx6ISTeJIbZCqReJyc79f1449orMj0oZvlOFbsTIuJ4Mx5Ku4c7A966RrsArWFiku1G1Wn5dVhnF5VQN80rHgNl7EAKjQcxKqC3sxnL4GqIlZrp+vwAiiT+ZTBvDHudMN+uLMdvtqBP0HUD+fRJU0M4qXWe5pmvFCoHZfM2lEU5m5cMuMEl1i1aWExZ/ycneHIQ80U2nE5/3wFzz2TwiQzfmsHc/bXjpIpa2cq8UrF3NT+XqvJv9VGhZtsj0uh88Kh5tcXTQoJLoPaSUiFQe7kzAPGjfBvBT5lhnHn/W5TjRc8U4rptPTGpVVJVZJdlqs0R5PTp82qk9WquinX/9f/TPzyE0KnhC6sH1d9kHf/h8Xwb3C03o/CfvRho7v3tpnDAnlCnpEeicgW2SPvyIAMCSefyBfylXwLNgMa8ACvpUGr6VkhNyLQ3wF0t9NS</latexit><latexit sha1_base64="X34oKPUyfzHHZcEkmlVzLHMwL5M=">AAACtHicfVHbbhMxEHWWWwmXpu0jLwMpUipEtFuqXkCVKvqCeApV01aK05XXO2ms2t6V7aWNtvtf/AovvMJn4E0XRLmNZenMmTO+nElyKawLw8+t4NbtO3fvLdxvP3j46PFiZ2n5yGaF4TjkmczMScIsSqFx6ISTeJIbZCqReJyc79f1449orMj0oZvlOFbsTIuJ4Mx5Ku4c7A966RrsArWFiku1G1Wn5dVhnF5VQN80rHgNl7EAKjQcxKqC3sxnL4GqIlZrp+vwAiiT+ZTBvDHudMN+uLMdvtqBP0HUD+fRJU0M4qXWe5pmvFCoHZfM2lEU5m5cMuMEl1i1aWExZ/ycneHIQ80U2nE5/3wFzz2TwiQzfmsHc/bXjpIpa2cq8UrF3NT+XqvJv9VGhZtsj0uh88Kh5tcXTQoJLoPaSUiFQe7kzAPGjfBvBT5lhnHn/W5TjRc8U4rptPTGpVVJVZJdlqs0R5PTp82qk9WquinX/9f/TPzyE0KnhC6sH1d9kHf/h8Xwb3C03o/CfvRho7v3tpnDAnlCnpEeicgW2SPvyIAMCSefyBfylXwLNgMa8ACvpUGr6VkhNyLQ3wF0t9NS</latexit><latexit sha1_base64="X34oKPUyfzHHZcEkmlVzLHMwL5M=">AAACtHicfVHbbhMxEHWWWwmXpu0jLwMpUipEtFuqXkCVKvqCeApV01aK05XXO2ms2t6V7aWNtvtf/AovvMJn4E0XRLmNZenMmTO+nElyKawLw8+t4NbtO3fvLdxvP3j46PFiZ2n5yGaF4TjkmczMScIsSqFx6ISTeJIbZCqReJyc79f1449orMj0oZvlOFbsTIuJ4Mx5Ku4c7A966RrsArWFiku1G1Wn5dVhnF5VQN80rHgNl7EAKjQcxKqC3sxnL4GqIlZrp+vwAiiT+ZTBvDHudMN+uLMdvtqBP0HUD+fRJU0M4qXWe5pmvFCoHZfM2lEU5m5cMuMEl1i1aWExZ/ycneHIQ80U2nE5/3wFzz2TwiQzfmsHc/bXjpIpa2cq8UrF3NT+XqvJv9VGhZtsj0uh88Kh5tcXTQoJLoPaSUiFQe7kzAPGjfBvBT5lhnHn/W5TjRc8U4rptPTGpVVJVZJdlqs0R5PTp82qk9WquinX/9f/TPzyE0KnhC6sH1d9kHf/h8Xwb3C03o/CfvRho7v3tpnDAnlCnpEeicgW2SPvyIAMCSefyBfylXwLNgMa8ACvpUGr6VkhNyLQ3wF0t9NS</latexit><latexit sha1_base64="X34oKPUyfzHHZcEkmlVzLHMwL5M=">AAACtHicfVHbbhMxEHWWWwmXpu0jLwMpUipEtFuqXkCVKvqCeApV01aK05XXO2ms2t6V7aWNtvtf/AovvMJn4E0XRLmNZenMmTO+nElyKawLw8+t4NbtO3fvLdxvP3j46PFiZ2n5yGaF4TjkmczMScIsSqFx6ISTeJIbZCqReJyc79f1449orMj0oZvlOFbsTIuJ4Mx5Ku4c7A966RrsArWFiku1G1Wn5dVhnF5VQN80rHgNl7EAKjQcxKqC3sxnL4GqIlZrp+vwAiiT+ZTBvDHudMN+uLMdvtqBP0HUD+fRJU0M4qXWe5pmvFCoHZfM2lEU5m5cMuMEl1i1aWExZ/ycneHIQ80U2nE5/3wFzz2TwiQzfmsHc/bXjpIpa2cq8UrF3NT+XqvJv9VGhZtsj0uh88Kh5tcXTQoJLoPaSUiFQe7kzAPGjfBvBT5lhnHn/W5TjRc8U4rptPTGpVVJVZJdlqs0R5PTp82qk9WquinX/9f/TPzyE0KnhC6sH1d9kHf/h8Xwb3C03o/CfvRho7v3tpnDAnlCnpEeicgW2SPvyIAMCSefyBfylXwLNgMa8ACvpUGr6VkhNyLQ3wF0t9NS</latexit>
where α is a non-negative regularization parameter controlling the trade-off
between the tree complexity and its fitting
Pruning a tree : Choosing the best subtree
21
Choose 𝜶:
- 𝜶 controls the trade-off between complexity and fitting
- optimal value chosen using cross-validation
- Then fit the tree on full data using the chosen optimal value
X1<c
X2<r
Yes
Yes
No
No
Classification
Very similar to a regression tree,
For qualitative responses rather than a
quantitative one
Response classes: 1, …, K
23
Prediction at each node: the most
common class in the corresponding
Need to change the loss function
IDEA: the more homogeneous the units in
the leaves the better
24
Splitting Rule : purity/ impurity measures
pmk m = 1, . . .M k = 1, . . .K<latexit sha1_base64="Mxci4lmwtiFpdOLDwyyw88dpO/k=">AAACznicfVHbbhMxEHWWWwm3FB55MaRIPKBoNxBoH5AqeEFCSKEibaU6imzvpLHiW21vIVqtEG/8A1/DK3wBf4M32aKG24wsH505M6OZYVYKH9L0Ryu5cPHS5SsbV9vXrt+4eauzeXvfm8JxGHEjjTtk1IMUGkZBBAmH1gFVTMIBm7+s4wen4Lww+l1YWBgreqzFVHAaIjXpDMiMBmwnpZpXmJycFDTH6nn2CBOZm+DxmzNyfo58Pel0094gzXaeZjjtpUuLoD9Id7ZTnDVMFzU2nGy2PpHc8EKBDlxS74+y1IZxSV0QXELVJoUHS/mcHsNRhJoq8ONyOV+FH0Qmx1Pj4tMBL9nzGSVV3i8Ui0pFw8z/HqvJv8aYijQzMl/vH6bb41JoWwTQfNV+WkgcDK5XiHPhgAe5iIByJ+IEmM+oozzERbeJhvfcKEV1XhKh86okipkP5Rax4Cy5t/QablXVulj/T93AlXsISujC/1FiD6j8VSL+cWLG8F5VC+PRzi6D/w32+73sca//9kl390Vzvg10F91HD1GGnqFd9AoN0Qhx9AV9Rd/Q92SYnCZV8nElTVpNzh20Zsnnn9qt4FA=</latexit>
-> Proportion of units in node m having Y = k
-> Gini index for node mVariance of a Bernoulli distribution
Total Variance
-> Cross entropy for node m
Gm =KX
k=1
pmk(1� pmk)<latexit sha1_base64="VXdJ14C2DvmF+wdsqMqFufYH4Q8=">AAACzHicfVHbbhMxEHWWWwm3FB55MaRI5YFotxcoEpUq8QASEioVaSt1w8r2ThorvqxsbyGy/MAL/8DX8AqfwN/gTRbUcJuR5aMzZ2Y0M7QS3Lo0/d5JLly8dPnKytXutes3bt7qrd4+tLo2DIZMC22OKbEguIKh407AcWWASCrgiE6fN/GjMzCWa/XWzSoYSXKq+Jgz4iJV9DZfFBLv4tzWsvDT3Sy8868CzifE4arwchrwOs4eLREPi14/HaRzw+fAdpo9fZzhrGX6qLX9YrXzMS81qyUoxwSx9iRLKzfyxDjOBIRuXluoCJuSUziJUBEJduTn0wX8IDIlHmsTn3J4zp7P8ERaO5M0KiVxE/t7rCH/GqMy0lSLcrm/G++MPFdV7UCxRftxLbDTuFkgLrkB5sQsAsIMjxNgNiGGMBfX3M0VvGdaSqJKn3NVBp9Lqj/4tbwCU+X35t7AtRCWxep/6hYu3IKTXNX2jxIHQMSvEvGPE1OKD0IjjEf7eRn8b3C4Mcg2Bxtvtvp7z9rzraC76D5aRxl6gvbQS7SPhoihz+gL+oq+Ja8Tl/gkLKRJp825g5Ys+fQDOyTfPw==</latexit>
Dm = �KX
k=1
pmk log pmk
<latexit sha1_base64="iDIg4964Ahn9Kk+LukO1kZ9qMSs=">AAACzHicfVHbbhMxEHWWWwm3FB55MaRIvBDtttCCRKVK8ICEhEpF2krdsLK9k8SKLyvbC0SWH3jhH/gaXuET+Bu8yYISbjOyfHTmzIxmhlaCW5em3zvJufMXLl7auNy9cvXa9Ru9zZvHVteGwZBpoc0pJRYEVzB03Ak4rQwQSQWc0NmzJn7yDozlWr1x8wpGkkwUH3NGXKSK3s7zQuJ9/ADntpaFn+1n4a1/GXA+JQ5XhZeziIWerBJFr58O0oXhFfAozZ7sZjhrmT5q7bDY7HzMS81qCcoxQaw9y9LKjTwxjjMBoZvXFirCZmQCZxEqIsGO/GK6gO9FpsRjbeJTDi/Y1QxPpLVzSaNSEje1v8ca8q8xKiNNtSjX+7vx45HnqqodKLZsP64Fdho3C8QlN8CcmEdAmOFxAsymxBDm4pq7uYL3TEtJVOlzrsrgc0n1B7+VV2Cq/M7CG7gVwrpY/U/dwqVbcJKr2v5R4giI+FUi/nFiSvFRaITxaD8vg/8NjrcH2c5g+/XD/sHT9nwb6Da6i+6jDO2hA/QCHaIhYugz+oK+om/Jq8QlPglLadJpc26hNUs+/QBa3uA4</latexit>
-> Misclassification error for node m Em = 1�maxk
pmk<latexit sha1_base64="fapc2jHpwRwqoRiroIUJeBqFGPk=">AAACuHicfVFLbxMxEHaWVwmvtBw5YEiRuBDtFvoAgVQJIXEsFWkr1dFieyeJFT9Wtpc2svbAgV/DFX4M/wZvsqCG14wsf/rmmxnNDCulcD5Nv3eSS5evXL22dr174+at23d66xtHzlSWw5AbaewJow6k0DD0wks4KS1QxSQcs9nrJn78EawTRr/38xJGik60GAtOfaTy3v03ucKvcIafYKLoeR5mNSZT6nGZBxVx3uung3Rh+ALYTrPnOxnOWqaPWjvI1zufSGF4pUB7Lqlzp1la+lGg1gsuoe6SykFJ+YxO4DRCTRW4UVhMUuNHkSnw2Nj4tMcL9mJGoMq5uWJRqaifut9jDfnXGFORZkYWq/39eG8UhC4rD5ov248rib3BzbJwISxwL+cRUG5FnADzKbWU+7jSLtFwxo1SVBeBCF3UgShmzsMmKcGW5MHCG7hZ16ti/T91C5fuwCuhK/dHiUOg8leJ+MeJGcOHdSOMR/t5GfxvcLQ1yJ4Ott496++/bM+3hu6hh+gxytAu2kdv0QEaIo4+oy/oK/qWvEg+JJNELKVJp825i1YssT8A48fXjg==</latexit>
Splitting and stopping rules
Gini index and cross-entropy are quite similar numerically
Cross-entropy and the Gini index are more sensitive to changes in the node probabilities than the misclassification rate
Gini index or the cross-entropy are typically used to evaluate the quality of a particular split
Any of these three approaches might be used when pruning the tree but Classification error rate is preferable for comparing the prediction accuracy of the final pruned tree
26
Many beats one
Trees are someway easier to explain to people than other predictive procedures : the tree plot makes them easy to understand
Trees can naturally deal with interactions and non-linearities, continuous and continuous predictors and responses
But they cannot boast a great predictive performance
Can we use more trees?
28
Bagging
BAGGING = Bootstrap AGGregation
To have more trees we need to introduce some variability among the trees: grow each tree on a different bootstrap sample
Averaging many trees reduces the variability of the prediction
Bagging grows B trees, taking advantage of resampling techniques
30
Many beat one
Trees are someway easier to explain to people than other predictive procedures : The tree plot makes them easy to interprete
They can naturally deal with interactions and non-linearities, continuous and continuous predictors and responses
But they cannot boast a great predictive performance
Can we have more trees?
31
Out-of-Bag prediction
On average, each bagged tree uses of around 2/3 of the observations
The remaining 1/3 of the units, not used to fit a bagged tree, are called the out-of-
bag (OOB) sample
We can predict the Yi using each of the trees in which unit i was OOB
This yields around B/3 predictions for the i-th unit
To obtain a single prediction for the ith observation we average those predicted
responses (quantitative), or take a majority vote (qualitative)
33
OOB error estimate
Since an OOB prediction can be computed for all n units, we can
compute an overall OOB MSE or classification error rate
It can be shown that with B sufficiently large, OOB error is virtually
equivalent to leave-one-out cross-validation error
Price to pay for bagging : interpretation
34
From bagging to Random Forest
The bagged trees based on the bootstrapped samples often look quite similar to
each other. They are therefore often highly correlated
Averaging uncorrelated trees can lead to a larger reduction in variance
To de-correlate the trees, random forests build a number of trees on
bootstrapped data using a random sample of mtry predictors
36
Random Forest
Two parameters to be tuned:
38
(2) mtry = n. variables sampled at each split
BAGGING
NB: It depends on the unknown number of good predictors
Conditional inference trees
In conditional inference trees (CTREE), we perform a Fisher permutation test for
independence between the response and each predictor
A split is possible only if the p-value (adjusted for multiple comparisons) is
smaller than a pre-specified nominal level
No need to prune the tree!
40
(1) Perform all the independence tests
(2) Choose the variable with lowest p-value and split maximising the contrast
(3) Stop when no adjusted p-values are below the threshold
BART = Bayesian Additive Regression Trees
Bayesian “almost" nonparametric
Sum of tree model, no bagging, no variable sampling, no pruning
The trees are grown via MCMC and regularised by ad hoc priors
Each tree is evaluated in its entirety via the leaves parameters
Very good performance
42
E[Y | X = x] =⌧X
t=1
T (X;Rt,�t) =⌧X
t=1
MtX
m=1
µmtI{x2Rmt}<latexit sha1_base64="EtHyiQQzNe3VUJpcE4DNNilbO8s=">AAADPXicfVHdbtMwGHXC3yh/HVxy80GHNCRUJeN3QKVJCAkukEa1bkV1FzmO21mznSh2oJXlC96F9+A5eADuELfc4qYBrTD4rEjH5zvHdr6TFoJrE0VfgvDM2XPnL6xdbF26fOXqtfb69X2dVyVlA5qLvBymRDPBFRsYbgQbFiUjMhXsID1+segfvGel5rnaM/OCjSWZKj7hlBhPJe1PWBJzlKb2pRu9Ayx5BjiVMOzNxtADrCuZWNOL3aHFhlQOYG+z7j+D2kiJsH2XmHu1C0+JlCQxd0+1LhlZM28S4wlZ+X0Nlm947RKLLcwAcwX9puccJO1O1I3qghPgYRRvP4ohbpgOamo3WQ8+4iynlWTKUEG0HsVRYcaWlIZTwVwLV5oVhB6TKRt5qIhkemzrcTq445kMJnnpP2WgZk86LJFaz2XqlYuH6z97C/LUXio9neYiW73fTJ6MLVdFZZiiy+snlQCTwyIxyHjJqBFzDwgtuf8DoEekJNT4XFtYsQ8092NXmfVjy5zFMs1ndgMXrCzwrXot4IZzq2L1P3UDl0szI7mq9F9H9BkRv4+wTYzQdwuhD+1XMvBvsL/Vje93t94+6Ow8b+JbQzfRbbSJYvQY7aBXaBcNEA1aQRRsB0/Dz+HX8Fv4fSkNg8ZzA61U+OMnFHoJzA==</latexit>
BART = Bayesian Additive Regression Trees
No greedy search, but Backfitting MCMC algorithm
At each step, we sample from the full-conditional using the residuals given the other trees
A move in the tree structure consists of Growing, Pruning, Changing
43
E[Y | X = x] =⌧X
t=1
T (X;Rt,�t) =⌧X
t=1
MtX
m=1
µmtI{x2Rmt}<latexit sha1_base64="EtHyiQQzNe3VUJpcE4DNNilbO8s=">AAADPXicfVHdbtMwGHXC3yh/HVxy80GHNCRUJeN3QKVJCAkukEa1bkV1FzmO21mznSh2oJXlC96F9+A5eADuELfc4qYBrTD4rEjH5zvHdr6TFoJrE0VfgvDM2XPnL6xdbF26fOXqtfb69X2dVyVlA5qLvBymRDPBFRsYbgQbFiUjMhXsID1+segfvGel5rnaM/OCjSWZKj7hlBhPJe1PWBJzlKb2pRu9Ayx5BjiVMOzNxtADrCuZWNOL3aHFhlQOYG+z7j+D2kiJsH2XmHu1C0+JlCQxd0+1LhlZM28S4wlZ+X0Nlm947RKLLcwAcwX9puccJO1O1I3qghPgYRRvP4ohbpgOamo3WQ8+4iynlWTKUEG0HsVRYcaWlIZTwVwLV5oVhB6TKRt5qIhkemzrcTq445kMJnnpP2WgZk86LJFaz2XqlYuH6z97C/LUXio9neYiW73fTJ6MLVdFZZiiy+snlQCTwyIxyHjJqBFzDwgtuf8DoEekJNT4XFtYsQ8092NXmfVjy5zFMs1ndgMXrCzwrXot4IZzq2L1P3UDl0szI7mq9F9H9BkRv4+wTYzQdwuhD+1XMvBvsL/Vje93t94+6Ow8b+JbQzfRbbSJYvQY7aBXaBcNEA1aQRRsB0/Dz+HX8Fv4fSkNg8ZzA61U+OMnFHoJzA==</latexit>
Interpreting tree-based models : longing transparency
46
Black box
X y = f(x) + ✏<latexit sha1_base64="lmdalk2ezbeczQt3wGyxgXAH/pk=">AAACrHicfVFLT9wwEPamL9i+oBy5uF0qUVVaOagUOFRC9NIjRSwgbSJkOxOw8Eu2Q4miHPpLem1/Ev+mzm5adfuakeVP33wz4/EwK4UPhNwOkjt3791/sLQ8fPjo8ZOnK6vPTrypHIcJN9K4M0Y9SKFhEkSQcGYdUMUknLKr91389BqcF0Yfh9pCruiFFqXgNETqfGWtxu9wuXnzCr/GGVgvZMeOyHibpHtvU0zGZGYRbG2TvV2C054Zod4Oz1cHn7PC8EqBDlxS76cpsSFvqAuCS2iHWeXBUn5FL2AaoaYKfN7MXt/il5EpcGlcPDrgGftrRkOV97ViUalouPS/xzryrzGmIs2MLBb7h3I3b4S2VQDN5+3LSuJgcPdBuBAOeJB1BJQ7ESfA/JI6ykP8xmGm4RM3SlFdNJnQRdtkipmbZiOz4Gz2fOYd3GjbRbH+n7qHc/cQlNCV/6PEEVD5s0S848SM4aO2E8al/dgM/jc42RqnEX98M9o/6Ne3hNbRC7SJUrSD9tEHdIgmiKMafUFf0bdknBwn0ySfS5NBn7OGFiwpvwOSwNLu</latexit><latexit sha1_base64="lmdalk2ezbeczQt3wGyxgXAH/pk=">AAACrHicfVFLT9wwEPamL9i+oBy5uF0qUVVaOagUOFRC9NIjRSwgbSJkOxOw8Eu2Q4miHPpLem1/Ev+mzm5adfuakeVP33wz4/EwK4UPhNwOkjt3791/sLQ8fPjo8ZOnK6vPTrypHIcJN9K4M0Y9SKFhEkSQcGYdUMUknLKr91389BqcF0Yfh9pCruiFFqXgNETqfGWtxu9wuXnzCr/GGVgvZMeOyHibpHtvU0zGZGYRbG2TvV2C054Zod4Oz1cHn7PC8EqBDlxS76cpsSFvqAuCS2iHWeXBUn5FL2AaoaYKfN7MXt/il5EpcGlcPDrgGftrRkOV97ViUalouPS/xzryrzGmIs2MLBb7h3I3b4S2VQDN5+3LSuJgcPdBuBAOeJB1BJQ7ESfA/JI6ykP8xmGm4RM3SlFdNJnQRdtkipmbZiOz4Gz2fOYd3GjbRbH+n7qHc/cQlNCV/6PEEVD5s0S848SM4aO2E8al/dgM/jc42RqnEX98M9o/6Ne3hNbRC7SJUrSD9tEHdIgmiKMafUFf0bdknBwn0ySfS5NBn7OGFiwpvwOSwNLu</latexit><latexit sha1_base64="lmdalk2ezbeczQt3wGyxgXAH/pk=">AAACrHicfVFLT9wwEPamL9i+oBy5uF0qUVVaOagUOFRC9NIjRSwgbSJkOxOw8Eu2Q4miHPpLem1/Ev+mzm5adfuakeVP33wz4/EwK4UPhNwOkjt3791/sLQ8fPjo8ZOnK6vPTrypHIcJN9K4M0Y9SKFhEkSQcGYdUMUknLKr91389BqcF0Yfh9pCruiFFqXgNETqfGWtxu9wuXnzCr/GGVgvZMeOyHibpHtvU0zGZGYRbG2TvV2C054Zod4Oz1cHn7PC8EqBDlxS76cpsSFvqAuCS2iHWeXBUn5FL2AaoaYKfN7MXt/il5EpcGlcPDrgGftrRkOV97ViUalouPS/xzryrzGmIs2MLBb7h3I3b4S2VQDN5+3LSuJgcPdBuBAOeJB1BJQ7ESfA/JI6ykP8xmGm4RM3SlFdNJnQRdtkipmbZiOz4Gz2fOYd3GjbRbH+n7qHc/cQlNCV/6PEEVD5s0S848SM4aO2E8al/dgM/jc42RqnEX98M9o/6Ne3hNbRC7SJUrSD9tEHdIgmiKMafUFf0bdknBwn0ySfS5NBn7OGFiwpvwOSwNLu</latexit><latexit sha1_base64="lmdalk2ezbeczQt3wGyxgXAH/pk=">AAACrHicfVFLT9wwEPamL9i+oBy5uF0qUVVaOagUOFRC9NIjRSwgbSJkOxOw8Eu2Q4miHPpLem1/Ev+mzm5adfuakeVP33wz4/EwK4UPhNwOkjt3791/sLQ8fPjo8ZOnK6vPTrypHIcJN9K4M0Y9SKFhEkSQcGYdUMUknLKr91389BqcF0Yfh9pCruiFFqXgNETqfGWtxu9wuXnzCr/GGVgvZMeOyHibpHtvU0zGZGYRbG2TvV2C054Zod4Oz1cHn7PC8EqBDlxS76cpsSFvqAuCS2iHWeXBUn5FL2AaoaYKfN7MXt/il5EpcGlcPDrgGftrRkOV97ViUalouPS/xzryrzGmIs2MLBb7h3I3b4S2VQDN5+3LSuJgcPdBuBAOeJB1BJQ7ESfA/JI6ykP8xmGm4RM3SlFdNJnQRdtkipmbZiOz4Gz2fOYd3GjbRbH+n7qHc/cQlNCV/6PEEVD5s0S848SM4aO2E8al/dgM/jc42RqnEX98M9o/6Ne3hNbRC7SJUrSD9tEHdIgmiKMafUFf0bdknBwn0ySfS5NBn7OGFiwpvwOSwNLu</latexit>
Accurate predictions No (few) assumptions
by = df(x)<latexit sha1_base64="jRiB5LFxU/DlwBcnk0Mzicvc6Mc=">AAACtHicfVFLTxsxEHa2L0gfhPbIxSVUopfIi4DCoRIqF44UNYCEo8j2zhILP1a2txCt9sCv6bX9Of03eJOlalraGVn+/M03MxoPL5T0gZCfneTR4ydPny0td5+/ePlqpbf6+tTb0gkYCqusO+fMg5IGhkEGBeeFA6a5gjN+ddjEz76C89KaL2FawEizSyNzKViI1Li3Rq9lBhMW8BR/xPePKt+8eV+Pe30y2CHp/m6KyYDMLIKtHbK/R3DaMn3U2vF4tXNLMytKDSYIxby/SEkRRhVzQQoFdZeWHgomrtglXERomAY/qmZT1PhdZDKcWxePCXjG/p5RMe39VPOo1CxM/J+xhnwwxnWkuVXZYv+Q740qaYoygBHz9nmpcLC4+SicSQciqGkETDgZJ8BiwhwTIX5nlxq4FlZrZrKKSpPVFdXc3lQbtABX0Lczb+BGXS+Kzf/ULZy7h6ClKf1fJU6AqV8l4h0n5hyf1I0wLu1+M/jf4HRrkEb8ebt/8Kld3xJaQ+toE6XoAzpAR+gYDZFAt+gb+o5+JLsJTUQCc2nSaXPeoAVLzB2RgtcZ</latexit><latexit sha1_base64="jRiB5LFxU/DlwBcnk0Mzicvc6Mc=">AAACtHicfVFLTxsxEHa2L0gfhPbIxSVUopfIi4DCoRIqF44UNYCEo8j2zhILP1a2txCt9sCv6bX9Of03eJOlalraGVn+/M03MxoPL5T0gZCfneTR4ydPny0td5+/ePlqpbf6+tTb0gkYCqusO+fMg5IGhkEGBeeFA6a5gjN+ddjEz76C89KaL2FawEizSyNzKViI1Li3Rq9lBhMW8BR/xPePKt+8eV+Pe30y2CHp/m6KyYDMLIKtHbK/R3DaMn3U2vF4tXNLMytKDSYIxby/SEkRRhVzQQoFdZeWHgomrtglXERomAY/qmZT1PhdZDKcWxePCXjG/p5RMe39VPOo1CxM/J+xhnwwxnWkuVXZYv+Q740qaYoygBHz9nmpcLC4+SicSQciqGkETDgZJ8BiwhwTIX5nlxq4FlZrZrKKSpPVFdXc3lQbtABX0Lczb+BGXS+Kzf/ULZy7h6ClKf1fJU6AqV8l4h0n5hyf1I0wLu1+M/jf4HRrkEb8ebt/8Kld3xJaQ+toE6XoAzpAR+gYDZFAt+gb+o5+JLsJTUQCc2nSaXPeoAVLzB2RgtcZ</latexit><latexit sha1_base64="jRiB5LFxU/DlwBcnk0Mzicvc6Mc=">AAACtHicfVFLTxsxEHa2L0gfhPbIxSVUopfIi4DCoRIqF44UNYCEo8j2zhILP1a2txCt9sCv6bX9Of03eJOlalraGVn+/M03MxoPL5T0gZCfneTR4ydPny0td5+/ePlqpbf6+tTb0gkYCqusO+fMg5IGhkEGBeeFA6a5gjN+ddjEz76C89KaL2FawEizSyNzKViI1Li3Rq9lBhMW8BR/xPePKt+8eV+Pe30y2CHp/m6KyYDMLIKtHbK/R3DaMn3U2vF4tXNLMytKDSYIxby/SEkRRhVzQQoFdZeWHgomrtglXERomAY/qmZT1PhdZDKcWxePCXjG/p5RMe39VPOo1CxM/J+xhnwwxnWkuVXZYv+Q740qaYoygBHz9nmpcLC4+SicSQciqGkETDgZJ8BiwhwTIX5nlxq4FlZrZrKKSpPVFdXc3lQbtABX0Lczb+BGXS+Kzf/ULZy7h6ClKf1fJU6AqV8l4h0n5hyf1I0wLu1+M/jf4HRrkEb8ebt/8Kld3xJaQ+toE6XoAzpAR+gYDZFAt+gb+o5+JLsJTUQCc2nSaXPeoAVLzB2RgtcZ</latexit><latexit sha1_base64="jRiB5LFxU/DlwBcnk0Mzicvc6Mc=">AAACtHicfVFLTxsxEHa2L0gfhPbIxSVUopfIi4DCoRIqF44UNYCEo8j2zhILP1a2txCt9sCv6bX9Of03eJOlalraGVn+/M03MxoPL5T0gZCfneTR4ydPny0td5+/ePlqpbf6+tTb0gkYCqusO+fMg5IGhkEGBeeFA6a5gjN+ddjEz76C89KaL2FawEizSyNzKViI1Li3Rq9lBhMW8BR/xPePKt+8eV+Pe30y2CHp/m6KyYDMLIKtHbK/R3DaMn3U2vF4tXNLMytKDSYIxby/SEkRRhVzQQoFdZeWHgomrtglXERomAY/qmZT1PhdZDKcWxePCXjG/p5RMe39VPOo1CxM/J+xhnwwxnWkuVXZYv+Q740qaYoygBHz9nmpcLC4+SicSQciqGkETDgZJ8BiwhwTIX5nlxq4FlZrZrKKSpPVFdXc3lQbtABX0Lczb+BGXS+Kzf/ULZy7h6ClKf1fJU6AqV8l4h0n5hyf1I0wLu1+M/jf4HRrkEb8ebt/8Kld3xJaQ+toE6XoAzpAR+gYDZFAt+gb+o5+JLsJTUQCc2nSaXPeoAVLzB2RgtcZ</latexit>
by<latexit sha1_base64="FnE4oyBTKg2X6CtyIjxvJ9uvArY=">AAAConicfVFLbxMxEHaWVwmvFo5cDAkSp2oXIcGxggvi1BbSVoqjauydNFb9WNmzlGi1B34EV/hd/Bu8yQqRBjEjy5+++WbG45GV0ZHy/Ncgu3Hz1u07O3eH9+4/ePhod+/xSfR1UDhR3vhwJiGi0Q4npMngWRUQrDR4Ki/fd/HTLxii9u4zLSucWbhweq4VUKKm4kqXuADiS36+O8r385XxbVD0YMR6OzzfG3wTpVe1RUfKQIzTIq9o1kAgrQy2Q1FHrEBdwgVOE3RgMc6a1Ztb/iIxJZ/7kI4jvmL/zmjAxri0Mikt0CJej3XkP2PSJlp6U272p/nbWaNdVRM6tW4/rw0nz7tv4aUOqMgsEwAVdJqAqwUEUJQ+bygcXilvLbiyEdqVbSOs9F+bsagwVOLZyjs4bttNsfufuodrj0hWuzpulThGMH9KpDtNLCU/bjthWlpxfUXb4OTVfpHw0evRwbt+fTvsKXvOXrKCvWEH7AM7ZBOmmGff2Q/2MxtnH7Oj7NNamg36nCdswzLxGyLt0FQ=</latexit><latexit sha1_base64="FnE4oyBTKg2X6CtyIjxvJ9uvArY=">AAAConicfVFLbxMxEHaWVwmvFo5cDAkSp2oXIcGxggvi1BbSVoqjauydNFb9WNmzlGi1B34EV/hd/Bu8yQqRBjEjy5+++WbG45GV0ZHy/Ncgu3Hz1u07O3eH9+4/ePhod+/xSfR1UDhR3vhwJiGi0Q4npMngWRUQrDR4Ki/fd/HTLxii9u4zLSucWbhweq4VUKKm4kqXuADiS36+O8r385XxbVD0YMR6OzzfG3wTpVe1RUfKQIzTIq9o1kAgrQy2Q1FHrEBdwgVOE3RgMc6a1Ztb/iIxJZ/7kI4jvmL/zmjAxri0Mikt0CJej3XkP2PSJlp6U272p/nbWaNdVRM6tW4/rw0nz7tv4aUOqMgsEwAVdJqAqwUEUJQ+bygcXilvLbiyEdqVbSOs9F+bsagwVOLZyjs4bttNsfufuodrj0hWuzpulThGMH9KpDtNLCU/bjthWlpxfUXb4OTVfpHw0evRwbt+fTvsKXvOXrKCvWEH7AM7ZBOmmGff2Q/2MxtnH7Oj7NNamg36nCdswzLxGyLt0FQ=</latexit><latexit sha1_base64="FnE4oyBTKg2X6CtyIjxvJ9uvArY=">AAAConicfVFLbxMxEHaWVwmvFo5cDAkSp2oXIcGxggvi1BbSVoqjauydNFb9WNmzlGi1B34EV/hd/Bu8yQqRBjEjy5+++WbG45GV0ZHy/Ncgu3Hz1u07O3eH9+4/ePhod+/xSfR1UDhR3vhwJiGi0Q4npMngWRUQrDR4Ki/fd/HTLxii9u4zLSucWbhweq4VUKKm4kqXuADiS36+O8r385XxbVD0YMR6OzzfG3wTpVe1RUfKQIzTIq9o1kAgrQy2Q1FHrEBdwgVOE3RgMc6a1Ztb/iIxJZ/7kI4jvmL/zmjAxri0Mikt0CJej3XkP2PSJlp6U272p/nbWaNdVRM6tW4/rw0nz7tv4aUOqMgsEwAVdJqAqwUEUJQ+bygcXilvLbiyEdqVbSOs9F+bsagwVOLZyjs4bttNsfufuodrj0hWuzpulThGMH9KpDtNLCU/bjthWlpxfUXb4OTVfpHw0evRwbt+fTvsKXvOXrKCvWEH7AM7ZBOmmGff2Q/2MxtnH7Oj7NNamg36nCdswzLxGyLt0FQ=</latexit><latexit sha1_base64="FnE4oyBTKg2X6CtyIjxvJ9uvArY=">AAAConicfVFLbxMxEHaWVwmvFo5cDAkSp2oXIcGxggvi1BbSVoqjauydNFb9WNmzlGi1B34EV/hd/Bu8yQqRBjEjy5+++WbG45GV0ZHy/Ncgu3Hz1u07O3eH9+4/ePhod+/xSfR1UDhR3vhwJiGi0Q4npMngWRUQrDR4Ki/fd/HTLxii9u4zLSucWbhweq4VUKKm4kqXuADiS36+O8r385XxbVD0YMR6OzzfG3wTpVe1RUfKQIzTIq9o1kAgrQy2Q1FHrEBdwgVOE3RgMc6a1Ztb/iIxJZ/7kI4jvmL/zmjAxri0Mikt0CJej3XkP2PSJlp6U272p/nbWaNdVRM6tW4/rw0nz7tv4aUOqMgsEwAVdJqAqwUEUJQ+bygcXilvLbiyEdqVbSOs9F+bsagwVOLZyjs4bttNsfufuodrj0hWuzpulThGMH9KpDtNLCU/bjthWlpxfUXb4OTVfpHw0evRwbt+fTvsKXvOXrKCvWEH7AM7ZBOmmGff2Q/2MxtnH7Oj7NNamg36nCdswzLxGyLt0FQ=</latexit>X
X y = f(x) + ✏<latexit sha1_base64="lmdalk2ezbeczQt3wGyxgXAH/pk=">AAACrHicfVFLT9wwEPamL9i+oBy5uF0qUVVaOagUOFRC9NIjRSwgbSJkOxOw8Eu2Q4miHPpLem1/Ev+mzm5adfuakeVP33wz4/EwK4UPhNwOkjt3791/sLQ8fPjo8ZOnK6vPTrypHIcJN9K4M0Y9SKFhEkSQcGYdUMUknLKr91389BqcF0Yfh9pCruiFFqXgNETqfGWtxu9wuXnzCr/GGVgvZMeOyHibpHtvU0zGZGYRbG2TvV2C054Zod4Oz1cHn7PC8EqBDlxS76cpsSFvqAuCS2iHWeXBUn5FL2AaoaYKfN7MXt/il5EpcGlcPDrgGftrRkOV97ViUalouPS/xzryrzGmIs2MLBb7h3I3b4S2VQDN5+3LSuJgcPdBuBAOeJB1BJQ7ESfA/JI6ykP8xmGm4RM3SlFdNJnQRdtkipmbZiOz4Gz2fOYd3GjbRbH+n7qHc/cQlNCV/6PEEVD5s0S848SM4aO2E8al/dgM/jc42RqnEX98M9o/6Ne3hNbRC7SJUrSD9tEHdIgmiKMafUFf0bdknBwn0ySfS5NBn7OGFiwpvwOSwNLu</latexit><latexit sha1_base64="lmdalk2ezbeczQt3wGyxgXAH/pk=">AAACrHicfVFLT9wwEPamL9i+oBy5uF0qUVVaOagUOFRC9NIjRSwgbSJkOxOw8Eu2Q4miHPpLem1/Ev+mzm5adfuakeVP33wz4/EwK4UPhNwOkjt3791/sLQ8fPjo8ZOnK6vPTrypHIcJN9K4M0Y9SKFhEkSQcGYdUMUknLKr91389BqcF0Yfh9pCruiFFqXgNETqfGWtxu9wuXnzCr/GGVgvZMeOyHibpHtvU0zGZGYRbG2TvV2C054Zod4Oz1cHn7PC8EqBDlxS76cpsSFvqAuCS2iHWeXBUn5FL2AaoaYKfN7MXt/il5EpcGlcPDrgGftrRkOV97ViUalouPS/xzryrzGmIs2MLBb7h3I3b4S2VQDN5+3LSuJgcPdBuBAOeJB1BJQ7ESfA/JI6ykP8xmGm4RM3SlFdNJnQRdtkipmbZiOz4Gz2fOYd3GjbRbH+n7qHc/cQlNCV/6PEEVD5s0S848SM4aO2E8al/dgM/jc42RqnEX98M9o/6Ne3hNbRC7SJUrSD9tEHdIgmiKMafUFf0bdknBwn0ySfS5NBn7OGFiwpvwOSwNLu</latexit><latexit sha1_base64="lmdalk2ezbeczQt3wGyxgXAH/pk=">AAACrHicfVFLT9wwEPamL9i+oBy5uF0qUVVaOagUOFRC9NIjRSwgbSJkOxOw8Eu2Q4miHPpLem1/Ev+mzm5adfuakeVP33wz4/EwK4UPhNwOkjt3791/sLQ8fPjo8ZOnK6vPTrypHIcJN9K4M0Y9SKFhEkSQcGYdUMUknLKr91389BqcF0Yfh9pCruiFFqXgNETqfGWtxu9wuXnzCr/GGVgvZMeOyHibpHtvU0zGZGYRbG2TvV2C054Zod4Oz1cHn7PC8EqBDlxS76cpsSFvqAuCS2iHWeXBUn5FL2AaoaYKfN7MXt/il5EpcGlcPDrgGftrRkOV97ViUalouPS/xzryrzGmIs2MLBb7h3I3b4S2VQDN5+3LSuJgcPdBuBAOeJB1BJQ7ESfA/JI6ykP8xmGm4RM3SlFdNJnQRdtkipmbZiOz4Gz2fOYd3GjbRbH+n7qHc/cQlNCV/6PEEVD5s0S848SM4aO2E8al/dgM/jc42RqnEX98M9o/6Ne3hNbRC7SJUrSD9tEHdIgmiKMafUFf0bdknBwn0ySfS5NBn7OGFiwpvwOSwNLu</latexit><latexit sha1_base64="lmdalk2ezbeczQt3wGyxgXAH/pk=">AAACrHicfVFLT9wwEPamL9i+oBy5uF0qUVVaOagUOFRC9NIjRSwgbSJkOxOw8Eu2Q4miHPpLem1/Ev+mzm5adfuakeVP33wz4/EwK4UPhNwOkjt3791/sLQ8fPjo8ZOnK6vPTrypHIcJN9K4M0Y9SKFhEkSQcGYdUMUknLKr91389BqcF0Yfh9pCruiFFqXgNETqfGWtxu9wuXnzCr/GGVgvZMeOyHibpHtvU0zGZGYRbG2TvV2C054Zod4Oz1cHn7PC8EqBDlxS76cpsSFvqAuCS2iHWeXBUn5FL2AaoaYKfN7MXt/il5EpcGlcPDrgGftrRkOV97ViUalouPS/xzryrzGmIs2MLBb7h3I3b4S2VQDN5+3LSuJgcPdBuBAOeJB1BJQ7ESfA/JI6ykP8xmGm4RM3SlFdNJnQRdtkipmbZiOz4Gz2fOYd3GjbRbH+n7qHc/cQlNCV/6PEEVD5s0S848SM4aO2E8al/dgM/jc42RqnEX98M9o/6Ne3hNbRC7SJUrSD9tEHdIgmiKMafUFf0bdknBwn0ySfS5NBn7OGFiwpvwOSwNLu</latexit>
by = df(x)<latexit sha1_base64="jRiB5LFxU/DlwBcnk0Mzicvc6Mc=">AAACtHicfVFLTxsxEHa2L0gfhPbIxSVUopfIi4DCoRIqF44UNYCEo8j2zhILP1a2txCt9sCv6bX9Of03eJOlalraGVn+/M03MxoPL5T0gZCfneTR4ydPny0td5+/ePlqpbf6+tTb0gkYCqusO+fMg5IGhkEGBeeFA6a5gjN+ddjEz76C89KaL2FawEizSyNzKViI1Li3Rq9lBhMW8BR/xPePKt+8eV+Pe30y2CHp/m6KyYDMLIKtHbK/R3DaMn3U2vF4tXNLMytKDSYIxby/SEkRRhVzQQoFdZeWHgomrtglXERomAY/qmZT1PhdZDKcWxePCXjG/p5RMe39VPOo1CxM/J+xhnwwxnWkuVXZYv+Q740qaYoygBHz9nmpcLC4+SicSQciqGkETDgZJ8BiwhwTIX5nlxq4FlZrZrKKSpPVFdXc3lQbtABX0Lczb+BGXS+Kzf/ULZy7h6ClKf1fJU6AqV8l4h0n5hyf1I0wLu1+M/jf4HRrkEb8ebt/8Kld3xJaQ+toE6XoAzpAR+gYDZFAt+gb+o5+JLsJTUQCc2nSaXPeoAVLzB2RgtcZ</latexit><latexit sha1_base64="jRiB5LFxU/DlwBcnk0Mzicvc6Mc=">AAACtHicfVFLTxsxEHa2L0gfhPbIxSVUopfIi4DCoRIqF44UNYCEo8j2zhILP1a2txCt9sCv6bX9Of03eJOlalraGVn+/M03MxoPL5T0gZCfneTR4ydPny0td5+/ePlqpbf6+tTb0gkYCqusO+fMg5IGhkEGBeeFA6a5gjN+ddjEz76C89KaL2FawEizSyNzKViI1Li3Rq9lBhMW8BR/xPePKt+8eV+Pe30y2CHp/m6KyYDMLIKtHbK/R3DaMn3U2vF4tXNLMytKDSYIxby/SEkRRhVzQQoFdZeWHgomrtglXERomAY/qmZT1PhdZDKcWxePCXjG/p5RMe39VPOo1CxM/J+xhnwwxnWkuVXZYv+Q740qaYoygBHz9nmpcLC4+SicSQciqGkETDgZJ8BiwhwTIX5nlxq4FlZrZrKKSpPVFdXc3lQbtABX0Lczb+BGXS+Kzf/ULZy7h6ClKf1fJU6AqV8l4h0n5hyf1I0wLu1+M/jf4HRrkEb8ebt/8Kld3xJaQ+toE6XoAzpAR+gYDZFAt+gb+o5+JLsJTUQCc2nSaXPeoAVLzB2RgtcZ</latexit><latexit sha1_base64="jRiB5LFxU/DlwBcnk0Mzicvc6Mc=">AAACtHicfVFLTxsxEHa2L0gfhPbIxSVUopfIi4DCoRIqF44UNYCEo8j2zhILP1a2txCt9sCv6bX9Of03eJOlalraGVn+/M03MxoPL5T0gZCfneTR4ydPny0td5+/ePlqpbf6+tTb0gkYCqusO+fMg5IGhkEGBeeFA6a5gjN+ddjEz76C89KaL2FawEizSyNzKViI1Li3Rq9lBhMW8BR/xPePKt+8eV+Pe30y2CHp/m6KyYDMLIKtHbK/R3DaMn3U2vF4tXNLMytKDSYIxby/SEkRRhVzQQoFdZeWHgomrtglXERomAY/qmZT1PhdZDKcWxePCXjG/p5RMe39VPOo1CxM/J+xhnwwxnWkuVXZYv+Q740qaYoygBHz9nmpcLC4+SicSQciqGkETDgZJ8BiwhwTIX5nlxq4FlZrZrKKSpPVFdXc3lQbtABX0Lczb+BGXS+Kzf/ULZy7h6ClKf1fJU6AqV8l4h0n5hyf1I0wLu1+M/jf4HRrkEb8ebt/8Kld3xJaQ+toE6XoAzpAR+gYDZFAt+gb+o5+JLsJTUQCc2nSaXPeoAVLzB2RgtcZ</latexit><latexit sha1_base64="jRiB5LFxU/DlwBcnk0Mzicvc6Mc=">AAACtHicfVFLTxsxEHa2L0gfhPbIxSVUopfIi4DCoRIqF44UNYCEo8j2zhILP1a2txCt9sCv6bX9Of03eJOlalraGVn+/M03MxoPL5T0gZCfneTR4ydPny0td5+/ePlqpbf6+tTb0gkYCqusO+fMg5IGhkEGBeeFA6a5gjN+ddjEz76C89KaL2FawEizSyNzKViI1Li3Rq9lBhMW8BR/xPePKt+8eV+Pe30y2CHp/m6KyYDMLIKtHbK/R3DaMn3U2vF4tXNLMytKDSYIxby/SEkRRhVzQQoFdZeWHgomrtglXERomAY/qmZT1PhdZDKcWxePCXjG/p5RMe39VPOo1CxM/J+xhnwwxnWkuVXZYv+Q740qaYoygBHz9nmpcLC4+SicSQciqGkETDgZJ8BiwhwTIX5nlxq4FlZrZrKKSpPVFdXc3lQbtABX0Lczb+BGXS+Kzf/ULZy7h6ClKf1fJU6AqV8l4h0n5hyf1I0wLu1+M/jf4HRrkEb8ebt/8Kld3xJaQ+toE6XoAzpAR+gYDZFAt+gb+o5+JLsJTUQCc2nSaXPeoAVLzB2RgtcZ</latexit>
Accurate estimates Assumptions needed
by = f(x, b✓)<latexit sha1_base64="f8PoFNp6AX+VUnUNbcIJligTQxY=">AAACu3icfVFLb9NAEN6YR0t4pXDksjRFKhKK7KrPQ6UKLhxLRdpKdRTtrsfN0n1Yu2PayPKht/4arvBX+DesExcRXjNa7advvpnRzPBCSY9x/L0T3bl77/7S8oPuw0ePnzztrTw79rZ0AobCKutOOfOgpIEhSlRwWjhgmis44RfvmvjJZ3BeWvMRpwWMNDs3MpeCYaDGvdX0UmYwYUindJ/m61dvbokqxQkgq1+Pe/14sBUne9sJjQfxzALY2Ir3dmOatEyftHY4Xulcp5kVpQaDQjHvz5K4wFHFHEqhoO6mpYeCiQt2DmcBGqbBj6rZMDV9FZiM5taFZ5DO2F8zKqa9n2oelJrhxP8ea8i/xrgONLcqW+yP+e6okqYoEYyYt89LRdHSZl80kw4EqmkATDgZJqBiwhwTGLbaTQ1cCqs1M1mVSpPVVaq5varW0gJckb6ceQPX6npRbP6nbuHcPaCWpvR/lDgCpn6WCH+YmHN6VDfCcLTby9B/g+ONQRLwh83+wdv2fMvkBVkl6yQhO+SAvCeHZEgEuSFfyFfyLdqPRPQpUnNp1GlznpMFi8ofp7PZ/Q==</latexit><latexit sha1_base64="f8PoFNp6AX+VUnUNbcIJligTQxY=">AAACu3icfVFLb9NAEN6YR0t4pXDksjRFKhKK7KrPQ6UKLhxLRdpKdRTtrsfN0n1Yu2PayPKht/4arvBX+DesExcRXjNa7advvpnRzPBCSY9x/L0T3bl77/7S8oPuw0ePnzztrTw79rZ0AobCKutOOfOgpIEhSlRwWjhgmis44RfvmvjJZ3BeWvMRpwWMNDs3MpeCYaDGvdX0UmYwYUindJ/m61dvbokqxQkgq1+Pe/14sBUne9sJjQfxzALY2Ir3dmOatEyftHY4Xulcp5kVpQaDQjHvz5K4wFHFHEqhoO6mpYeCiQt2DmcBGqbBj6rZMDV9FZiM5taFZ5DO2F8zKqa9n2oelJrhxP8ea8i/xrgONLcqW+yP+e6okqYoEYyYt89LRdHSZl80kw4EqmkATDgZJqBiwhwTGLbaTQ1cCqs1M1mVSpPVVaq5varW0gJckb6ceQPX6npRbP6nbuHcPaCWpvR/lDgCpn6WCH+YmHN6VDfCcLTby9B/g+ONQRLwh83+wdv2fMvkBVkl6yQhO+SAvCeHZEgEuSFfyFfyLdqPRPQpUnNp1GlznpMFi8ofp7PZ/Q==</latexit><latexit sha1_base64="f8PoFNp6AX+VUnUNbcIJligTQxY=">AAACu3icfVFLb9NAEN6YR0t4pXDksjRFKhKK7KrPQ6UKLhxLRdpKdRTtrsfN0n1Yu2PayPKht/4arvBX+DesExcRXjNa7advvpnRzPBCSY9x/L0T3bl77/7S8oPuw0ePnzztrTw79rZ0AobCKutOOfOgpIEhSlRwWjhgmis44RfvmvjJZ3BeWvMRpwWMNDs3MpeCYaDGvdX0UmYwYUindJ/m61dvbokqxQkgq1+Pe/14sBUne9sJjQfxzALY2Ir3dmOatEyftHY4Xulcp5kVpQaDQjHvz5K4wFHFHEqhoO6mpYeCiQt2DmcBGqbBj6rZMDV9FZiM5taFZ5DO2F8zKqa9n2oelJrhxP8ea8i/xrgONLcqW+yP+e6okqYoEYyYt89LRdHSZl80kw4EqmkATDgZJqBiwhwTGLbaTQ1cCqs1M1mVSpPVVaq5varW0gJckb6ceQPX6npRbP6nbuHcPaCWpvR/lDgCpn6WCH+YmHN6VDfCcLTby9B/g+ONQRLwh83+wdv2fMvkBVkl6yQhO+SAvCeHZEgEuSFfyFfyLdqPRPQpUnNp1GlznpMFi8ofp7PZ/Q==</latexit><latexit sha1_base64="f8PoFNp6AX+VUnUNbcIJligTQxY=">AAACu3icfVFLb9NAEN6YR0t4pXDksjRFKhKK7KrPQ6UKLhxLRdpKdRTtrsfN0n1Yu2PayPKht/4arvBX+DesExcRXjNa7advvpnRzPBCSY9x/L0T3bl77/7S8oPuw0ePnzztrTw79rZ0AobCKutOOfOgpIEhSlRwWjhgmis44RfvmvjJZ3BeWvMRpwWMNDs3MpeCYaDGvdX0UmYwYUindJ/m61dvbokqxQkgq1+Pe/14sBUne9sJjQfxzALY2Ir3dmOatEyftHY4Xulcp5kVpQaDQjHvz5K4wFHFHEqhoO6mpYeCiQt2DmcBGqbBj6rZMDV9FZiM5taFZ5DO2F8zKqa9n2oelJrhxP8ea8i/xrgONLcqW+yP+e6okqYoEYyYt89LRdHSZl80kw4EqmkATDgZJqBiwhwTGLbaTQ1cCqs1M1mVSpPVVaq5varW0gJckb6ceQPX6npRbP6nbuHcPaCWpvR/lDgCpn6WCH+YmHN6VDfCcLTby9B/g+ONQRLwh83+wdv2fMvkBVkl6yQhO+SAvCeHZEgEuSFfyFfyLdqPRPQpUnNp1GlznpMFi8ofp7PZ/Q==</latexit>
...<latexit sha1_base64="DS2CiyTtBeyi7XrtuJnrYrmOpLs=">AAACnnicfVFLbxMxEHaWAiW8WjhycUmROEXeqoX2VoGEekG0VdNE6kaV7Z20VvxY2bOFaLUHfgJX+GX9N3iTpWp4zcjyp2++mfF4RKFVQMauO8mdlbv37q8+6D589PjJ07X1Z6fBlV7CQDrt/EjwAFpZGKBCDaPCAzdCw1BM3zfx4RX4oJw9wVkBY8MvrJooyTFSw+wqdxi652s91t9h6d6blLI+m1sEWztsb5fRtGV6pLXD8/XO1yx3sjRgUWoewlnKChxX3KOSGupuVgYouJzyCziL0HIDYVzN31vTV5HJ6cT5eCzSOXs7o+ImhJkRUWk4XobfYw3515gwkRZO58v9cbI7rpQtSgQrF+0npaboaPMlNFceJOpZBFx6FSeg8pJ7LjF+XDez8Fk6Y7jNq0zZvK4yI9yXajMrwBfZxtwbuFnXy2L7P3ULFx4AjbJl+KPEMXB9UyLecWIh6HHdCOPSfm2G/hucbvXTiI+2e/vv2vWtkhfkJXlNUvKW7JMDckgGRJIp+Ua+kx8JTT4kH5NPC2nSaXOekyVLRj8Bzp/O9Q==</latexit><latexit sha1_base64="DS2CiyTtBeyi7XrtuJnrYrmOpLs=">AAACnnicfVFLbxMxEHaWAiW8WjhycUmROEXeqoX2VoGEekG0VdNE6kaV7Z20VvxY2bOFaLUHfgJX+GX9N3iTpWp4zcjyp2++mfF4RKFVQMauO8mdlbv37q8+6D589PjJ07X1Z6fBlV7CQDrt/EjwAFpZGKBCDaPCAzdCw1BM3zfx4RX4oJw9wVkBY8MvrJooyTFSw+wqdxi652s91t9h6d6blLI+m1sEWztsb5fRtGV6pLXD8/XO1yx3sjRgUWoewlnKChxX3KOSGupuVgYouJzyCziL0HIDYVzN31vTV5HJ6cT5eCzSOXs7o+ImhJkRUWk4XobfYw3515gwkRZO58v9cbI7rpQtSgQrF+0npaboaPMlNFceJOpZBFx6FSeg8pJ7LjF+XDez8Fk6Y7jNq0zZvK4yI9yXajMrwBfZxtwbuFnXy2L7P3ULFx4AjbJl+KPEMXB9UyLecWIh6HHdCOPSfm2G/hucbvXTiI+2e/vv2vWtkhfkJXlNUvKW7JMDckgGRJIp+Ua+kx8JTT4kH5NPC2nSaXOekyVLRj8Bzp/O9Q==</latexit><latexit sha1_base64="DS2CiyTtBeyi7XrtuJnrYrmOpLs=">AAACnnicfVFLbxMxEHaWAiW8WjhycUmROEXeqoX2VoGEekG0VdNE6kaV7Z20VvxY2bOFaLUHfgJX+GX9N3iTpWp4zcjyp2++mfF4RKFVQMauO8mdlbv37q8+6D589PjJ07X1Z6fBlV7CQDrt/EjwAFpZGKBCDaPCAzdCw1BM3zfx4RX4oJw9wVkBY8MvrJooyTFSw+wqdxi652s91t9h6d6blLI+m1sEWztsb5fRtGV6pLXD8/XO1yx3sjRgUWoewlnKChxX3KOSGupuVgYouJzyCziL0HIDYVzN31vTV5HJ6cT5eCzSOXs7o+ImhJkRUWk4XobfYw3515gwkRZO58v9cbI7rpQtSgQrF+0npaboaPMlNFceJOpZBFx6FSeg8pJ7LjF+XDez8Fk6Y7jNq0zZvK4yI9yXajMrwBfZxtwbuFnXy2L7P3ULFx4AjbJl+KPEMXB9UyLecWIh6HHdCOPSfm2G/hucbvXTiI+2e/vv2vWtkhfkJXlNUvKW7JMDckgGRJIp+Ua+kx8JTT4kH5NPC2nSaXOekyVLRj8Bzp/O9Q==</latexit><latexit sha1_base64="DS2CiyTtBeyi7XrtuJnrYrmOpLs=">AAACnnicfVFLbxMxEHaWAiW8WjhycUmROEXeqoX2VoGEekG0VdNE6kaV7Z20VvxY2bOFaLUHfgJX+GX9N3iTpWp4zcjyp2++mfF4RKFVQMauO8mdlbv37q8+6D589PjJ07X1Z6fBlV7CQDrt/EjwAFpZGKBCDaPCAzdCw1BM3zfx4RX4oJw9wVkBY8MvrJooyTFSw+wqdxi652s91t9h6d6blLI+m1sEWztsb5fRtGV6pLXD8/XO1yx3sjRgUWoewlnKChxX3KOSGupuVgYouJzyCziL0HIDYVzN31vTV5HJ6cT5eCzSOXs7o+ImhJkRUWk4XobfYw3515gwkRZO58v9cbI7rpQtSgQrF+0npaboaPMlNFceJOpZBFx6FSeg8pJ7LjF+XDez8Fk6Y7jNq0zZvK4yI9yXajMrwBfZxtwbuFnXy2L7P3ULFx4AjbJl+KPEMXB9UyLecWIh6HHdCOPSfm2G/hucbvXTiI+2e/vv2vWtkhfkJXlNUvKW7JMDckgGRJIp+Ua+kx8JTT4kH5NPC2nSaXOekyVLRj8Bzp/O9Q==</latexit>
...<latexit sha1_base64="DS2CiyTtBeyi7XrtuJnrYrmOpLs=">AAACnnicfVFLbxMxEHaWAiW8WjhycUmROEXeqoX2VoGEekG0VdNE6kaV7Z20VvxY2bOFaLUHfgJX+GX9N3iTpWp4zcjyp2++mfF4RKFVQMauO8mdlbv37q8+6D589PjJ07X1Z6fBlV7CQDrt/EjwAFpZGKBCDaPCAzdCw1BM3zfx4RX4oJw9wVkBY8MvrJooyTFSw+wqdxi652s91t9h6d6blLI+m1sEWztsb5fRtGV6pLXD8/XO1yx3sjRgUWoewlnKChxX3KOSGupuVgYouJzyCziL0HIDYVzN31vTV5HJ6cT5eCzSOXs7o+ImhJkRUWk4XobfYw3515gwkRZO58v9cbI7rpQtSgQrF+0npaboaPMlNFceJOpZBFx6FSeg8pJ7LjF+XDez8Fk6Y7jNq0zZvK4yI9yXajMrwBfZxtwbuFnXy2L7P3ULFx4AjbJl+KPEMXB9UyLecWIh6HHdCOPSfm2G/hucbvXTiI+2e/vv2vWtkhfkJXlNUvKW7JMDckgGRJIp+Ua+kx8JTT4kH5NPC2nSaXOekyVLRj8Bzp/O9Q==</latexit><latexit sha1_base64="DS2CiyTtBeyi7XrtuJnrYrmOpLs=">AAACnnicfVFLbxMxEHaWAiW8WjhycUmROEXeqoX2VoGEekG0VdNE6kaV7Z20VvxY2bOFaLUHfgJX+GX9N3iTpWp4zcjyp2++mfF4RKFVQMauO8mdlbv37q8+6D589PjJ07X1Z6fBlV7CQDrt/EjwAFpZGKBCDaPCAzdCw1BM3zfx4RX4oJw9wVkBY8MvrJooyTFSw+wqdxi652s91t9h6d6blLI+m1sEWztsb5fRtGV6pLXD8/XO1yx3sjRgUWoewlnKChxX3KOSGupuVgYouJzyCziL0HIDYVzN31vTV5HJ6cT5eCzSOXs7o+ImhJkRUWk4XobfYw3515gwkRZO58v9cbI7rpQtSgQrF+0npaboaPMlNFceJOpZBFx6FSeg8pJ7LjF+XDez8Fk6Y7jNq0zZvK4yI9yXajMrwBfZxtwbuFnXy2L7P3ULFx4AjbJl+KPEMXB9UyLecWIh6HHdCOPSfm2G/hucbvXTiI+2e/vv2vWtkhfkJXlNUvKW7JMDckgGRJIp+Ua+kx8JTT4kH5NPC2nSaXOekyVLRj8Bzp/O9Q==</latexit><latexit sha1_base64="DS2CiyTtBeyi7XrtuJnrYrmOpLs=">AAACnnicfVFLbxMxEHaWAiW8WjhycUmROEXeqoX2VoGEekG0VdNE6kaV7Z20VvxY2bOFaLUHfgJX+GX9N3iTpWp4zcjyp2++mfF4RKFVQMauO8mdlbv37q8+6D589PjJ07X1Z6fBlV7CQDrt/EjwAFpZGKBCDaPCAzdCw1BM3zfx4RX4oJw9wVkBY8MvrJooyTFSw+wqdxi652s91t9h6d6blLI+m1sEWztsb5fRtGV6pLXD8/XO1yx3sjRgUWoewlnKChxX3KOSGupuVgYouJzyCziL0HIDYVzN31vTV5HJ6cT5eCzSOXs7o+ImhJkRUWk4XobfYw3515gwkRZO58v9cbI7rpQtSgQrF+0npaboaPMlNFceJOpZBFx6FSeg8pJ7LjF+XDez8Fk6Y7jNq0zZvK4yI9yXajMrwBfZxtwbuFnXy2L7P3ULFx4AjbJl+KPEMXB9UyLecWIh6HHdCOPSfm2G/hucbvXTiI+2e/vv2vWtkhfkJXlNUvKW7JMDckgGRJIp+Ua+kx8JTT4kH5NPC2nSaXOekyVLRj8Bzp/O9Q==</latexit><latexit sha1_base64="DS2CiyTtBeyi7XrtuJnrYrmOpLs=">AAACnnicfVFLbxMxEHaWAiW8WjhycUmROEXeqoX2VoGEekG0VdNE6kaV7Z20VvxY2bOFaLUHfgJX+GX9N3iTpWp4zcjyp2++mfF4RKFVQMauO8mdlbv37q8+6D589PjJ07X1Z6fBlV7CQDrt/EjwAFpZGKBCDaPCAzdCw1BM3zfx4RX4oJw9wVkBY8MvrJooyTFSw+wqdxi652s91t9h6d6blLI+m1sEWztsb5fRtGV6pLXD8/XO1yx3sjRgUWoewlnKChxX3KOSGupuVgYouJzyCziL0HIDYVzN31vTV5HJ6cT5eCzSOXs7o+ImhJkRUWk4XobfYw3515gwkRZO58v9cbI7rpQtSgQrF+0npaboaPMlNFceJOpZBFx6FSeg8pJ7LjF+XDez8Fk6Y7jNq0zZvK4yI9yXajMrwBfZxtwbuFnXy2L7P3ULFx4AjbJl+KPEMXB9UyLecWIh6HHdCOPSfm2G/hucbvXTiI+2e/vv2vWtkhfkJXlNUvKW7JMDckgGRJIp+Ua+kx8JTT4kH5NPC2nSaXOekyVLRj8Bzp/O9Q==</latexit>
...<latexit sha1_base64="DS2CiyTtBeyi7XrtuJnrYrmOpLs=">AAACnnicfVFLbxMxEHaWAiW8WjhycUmROEXeqoX2VoGEekG0VdNE6kaV7Z20VvxY2bOFaLUHfgJX+GX9N3iTpWp4zcjyp2++mfF4RKFVQMauO8mdlbv37q8+6D589PjJ07X1Z6fBlV7CQDrt/EjwAFpZGKBCDaPCAzdCw1BM3zfx4RX4oJw9wVkBY8MvrJooyTFSw+wqdxi652s91t9h6d6blLI+m1sEWztsb5fRtGV6pLXD8/XO1yx3sjRgUWoewlnKChxX3KOSGupuVgYouJzyCziL0HIDYVzN31vTV5HJ6cT5eCzSOXs7o+ImhJkRUWk4XobfYw3515gwkRZO58v9cbI7rpQtSgQrF+0npaboaPMlNFceJOpZBFx6FSeg8pJ7LjF+XDez8Fk6Y7jNq0zZvK4yI9yXajMrwBfZxtwbuFnXy2L7P3ULFx4AjbJl+KPEMXB9UyLecWIh6HHdCOPSfm2G/hucbvXTiI+2e/vv2vWtkhfkJXlNUvKW7JMDckgGRJIp+Ua+kx8JTT4kH5NPC2nSaXOekyVLRj8Bzp/O9Q==</latexit><latexit sha1_base64="DS2CiyTtBeyi7XrtuJnrYrmOpLs=">AAACnnicfVFLbxMxEHaWAiW8WjhycUmROEXeqoX2VoGEekG0VdNE6kaV7Z20VvxY2bOFaLUHfgJX+GX9N3iTpWp4zcjyp2++mfF4RKFVQMauO8mdlbv37q8+6D589PjJ07X1Z6fBlV7CQDrt/EjwAFpZGKBCDaPCAzdCw1BM3zfx4RX4oJw9wVkBY8MvrJooyTFSw+wqdxi652s91t9h6d6blLI+m1sEWztsb5fRtGV6pLXD8/XO1yx3sjRgUWoewlnKChxX3KOSGupuVgYouJzyCziL0HIDYVzN31vTV5HJ6cT5eCzSOXs7o+ImhJkRUWk4XobfYw3515gwkRZO58v9cbI7rpQtSgQrF+0npaboaPMlNFceJOpZBFx6FSeg8pJ7LjF+XDez8Fk6Y7jNq0zZvK4yI9yXajMrwBfZxtwbuFnXy2L7P3ULFx4AjbJl+KPEMXB9UyLecWIh6HHdCOPSfm2G/hucbvXTiI+2e/vv2vWtkhfkJXlNUvKW7JMDckgGRJIp+Ua+kx8JTT4kH5NPC2nSaXOekyVLRj8Bzp/O9Q==</latexit><latexit sha1_base64="DS2CiyTtBeyi7XrtuJnrYrmOpLs=">AAACnnicfVFLbxMxEHaWAiW8WjhycUmROEXeqoX2VoGEekG0VdNE6kaV7Z20VvxY2bOFaLUHfgJX+GX9N3iTpWp4zcjyp2++mfF4RKFVQMauO8mdlbv37q8+6D589PjJ07X1Z6fBlV7CQDrt/EjwAFpZGKBCDaPCAzdCw1BM3zfx4RX4oJw9wVkBY8MvrJooyTFSw+wqdxi652s91t9h6d6blLI+m1sEWztsb5fRtGV6pLXD8/XO1yx3sjRgUWoewlnKChxX3KOSGupuVgYouJzyCziL0HIDYVzN31vTV5HJ6cT5eCzSOXs7o+ImhJkRUWk4XobfYw3515gwkRZO58v9cbI7rpQtSgQrF+0npaboaPMlNFceJOpZBFx6FSeg8pJ7LjF+XDez8Fk6Y7jNq0zZvK4yI9yXajMrwBfZxtwbuFnXy2L7P3ULFx4AjbJl+KPEMXB9UyLecWIh6HHdCOPSfm2G/hucbvXTiI+2e/vv2vWtkhfkJXlNUvKW7JMDckgGRJIp+Ua+kx8JTT4kH5NPC2nSaXOekyVLRj8Bzp/O9Q==</latexit><latexit sha1_base64="DS2CiyTtBeyi7XrtuJnrYrmOpLs=">AAACnnicfVFLbxMxEHaWAiW8WjhycUmROEXeqoX2VoGEekG0VdNE6kaV7Z20VvxY2bOFaLUHfgJX+GX9N3iTpWp4zcjyp2++mfF4RKFVQMauO8mdlbv37q8+6D589PjJ07X1Z6fBlV7CQDrt/EjwAFpZGKBCDaPCAzdCw1BM3zfx4RX4oJw9wVkBY8MvrJooyTFSw+wqdxi652s91t9h6d6blLI+m1sEWztsb5fRtGV6pLXD8/XO1yx3sjRgUWoewlnKChxX3KOSGupuVgYouJzyCziL0HIDYVzN31vTV5HJ6cT5eCzSOXs7o+ImhJkRUWk4XobfYw3515gwkRZO58v9cbI7rpQtSgQrF+0npaboaPMlNFceJOpZBFx6FSeg8pJ7LjF+XDez8Fk6Y7jNq0zZvK4yI9yXajMrwBfZxtwbuFnXy2L7P3ULFx4AjbJl+KPEMXB9UyLecWIh6HHdCOPSfm2G/hucbvXTiI+2e/vv2vWtkhfkJXlNUvKW7JMDckgGRJIp+Ua+kx8JTT4kH5NPC2nSaXOekyVLRj8Bzp/O9Q==</latexit>
...<latexit sha1_base64="DS2CiyTtBeyi7XrtuJnrYrmOpLs=">AAACnnicfVFLbxMxEHaWAiW8WjhycUmROEXeqoX2VoGEekG0VdNE6kaV7Z20VvxY2bOFaLUHfgJX+GX9N3iTpWp4zcjyp2++mfF4RKFVQMauO8mdlbv37q8+6D589PjJ07X1Z6fBlV7CQDrt/EjwAFpZGKBCDaPCAzdCw1BM3zfx4RX4oJw9wVkBY8MvrJooyTFSw+wqdxi652s91t9h6d6blLI+m1sEWztsb5fRtGV6pLXD8/XO1yx3sjRgUWoewlnKChxX3KOSGupuVgYouJzyCziL0HIDYVzN31vTV5HJ6cT5eCzSOXs7o+ImhJkRUWk4XobfYw3515gwkRZO58v9cbI7rpQtSgQrF+0npaboaPMlNFceJOpZBFx6FSeg8pJ7LjF+XDez8Fk6Y7jNq0zZvK4yI9yXajMrwBfZxtwbuFnXy2L7P3ULFx4AjbJl+KPEMXB9UyLecWIh6HHdCOPSfm2G/hucbvXTiI+2e/vv2vWtkhfkJXlNUvKW7JMDckgGRJIp+Ua+kx8JTT4kH5NPC2nSaXOekyVLRj8Bzp/O9Q==</latexit><latexit sha1_base64="DS2CiyTtBeyi7XrtuJnrYrmOpLs=">AAACnnicfVFLbxMxEHaWAiW8WjhycUmROEXeqoX2VoGEekG0VdNE6kaV7Z20VvxY2bOFaLUHfgJX+GX9N3iTpWp4zcjyp2++mfF4RKFVQMauO8mdlbv37q8+6D589PjJ07X1Z6fBlV7CQDrt/EjwAFpZGKBCDaPCAzdCw1BM3zfx4RX4oJw9wVkBY8MvrJooyTFSw+wqdxi652s91t9h6d6blLI+m1sEWztsb5fRtGV6pLXD8/XO1yx3sjRgUWoewlnKChxX3KOSGupuVgYouJzyCziL0HIDYVzN31vTV5HJ6cT5eCzSOXs7o+ImhJkRUWk4XobfYw3515gwkRZO58v9cbI7rpQtSgQrF+0npaboaPMlNFceJOpZBFx6FSeg8pJ7LjF+XDez8Fk6Y7jNq0zZvK4yI9yXajMrwBfZxtwbuFnXy2L7P3ULFx4AjbJl+KPEMXB9UyLecWIh6HHdCOPSfm2G/hucbvXTiI+2e/vv2vWtkhfkJXlNUvKW7JMDckgGRJIp+Ua+kx8JTT4kH5NPC2nSaXOekyVLRj8Bzp/O9Q==</latexit><latexit sha1_base64="DS2CiyTtBeyi7XrtuJnrYrmOpLs=">AAACnnicfVFLbxMxEHaWAiW8WjhycUmROEXeqoX2VoGEekG0VdNE6kaV7Z20VvxY2bOFaLUHfgJX+GX9N3iTpWp4zcjyp2++mfF4RKFVQMauO8mdlbv37q8+6D589PjJ07X1Z6fBlV7CQDrt/EjwAFpZGKBCDaPCAzdCw1BM3zfx4RX4oJw9wVkBY8MvrJooyTFSw+wqdxi652s91t9h6d6blLI+m1sEWztsb5fRtGV6pLXD8/XO1yx3sjRgUWoewlnKChxX3KOSGupuVgYouJzyCziL0HIDYVzN31vTV5HJ6cT5eCzSOXs7o+ImhJkRUWk4XobfYw3515gwkRZO58v9cbI7rpQtSgQrF+0npaboaPMlNFceJOpZBFx6FSeg8pJ7LjF+XDez8Fk6Y7jNq0zZvK4yI9yXajMrwBfZxtwbuFnXy2L7P3ULFx4AjbJl+KPEMXB9UyLecWIh6HHdCOPSfm2G/hucbvXTiI+2e/vv2vWtkhfkJXlNUvKW7JMDckgGRJIp+Ua+kx8JTT4kH5NPC2nSaXOekyVLRj8Bzp/O9Q==</latexit><latexit sha1_base64="DS2CiyTtBeyi7XrtuJnrYrmOpLs=">AAACnnicfVFLbxMxEHaWAiW8WjhycUmROEXeqoX2VoGEekG0VdNE6kaV7Z20VvxY2bOFaLUHfgJX+GX9N3iTpWp4zcjyp2++mfF4RKFVQMauO8mdlbv37q8+6D589PjJ07X1Z6fBlV7CQDrt/EjwAFpZGKBCDaPCAzdCw1BM3zfx4RX4oJw9wVkBY8MvrJooyTFSw+wqdxi652s91t9h6d6blLI+m1sEWztsb5fRtGV6pLXD8/XO1yx3sjRgUWoewlnKChxX3KOSGupuVgYouJzyCziL0HIDYVzN31vTV5HJ6cT5eCzSOXs7o+ImhJkRUWk4XobfYw3515gwkRZO58v9cbI7rpQtSgQrF+0npaboaPMlNFceJOpZBFx6FSeg8pJ7LjF+XDez8Fk6Y7jNq0zZvK4yI9yXajMrwBfZxtwbuFnXy2L7P3ULFx4AjbJl+KPEMXB9UyLecWIh6HHdCOPSfm2G/hucbvXTiI+2e/vv2vWtkhfkJXlNUvKW7JMDckgGRJIp+Ua+kx8JTT4kH5NPC2nSaXOekyVLRj8Bzp/O9Q==</latexit>
… Using machine learning for making decisions…
47
- Out of the Artificial Intelligence/technological framework …
- Sometimes accurate predictions are not enough for making good decisions
- To understand whether the algorithm is working in a sensible way, the black box has to be whitened
- It is not a matter of exactly understanding every bit and bytes of the model for all data points
- It is a matter of exactly understanding what drives the prediction, which are the discriminative predictors … are they reasonable?
How to interpret Trees and Forests?
One tree -> the tree plot makes clear how predictions have been made
Forests are less transparent -> Variable importance measures
Variable importance is a measure of the importance of each variable in predicting
the response
Several way to compute variable importance: the best is based on permutation of
the variable -> Gain in prediction
Importance in predicting is not importance in explaining or causing
48
50
- Think about Personalised medicine
- Or algorithms for banks to give a loan
- It is not just a matter of “to know how” but also of “Is it fair?”
Y
- Variable importance in predicting sometimes deviates from true causal mechanism/direct association
Tricky example
Generative/explanatory vs Predictive models
51
- Statistical/Machine learning is focused on predicting
- Computer science, text or image processing rely on predictive modelling: the
focus is on new/future observation
- Human sciences usually require generative/explanatory modelling: observed
data are used to assess causal/explanatory hypotheses
- Predicting is different from explaining
- Lack of understanding in many disciplines of this distinction
Thanks for your attention!Some references
- Breiman, L. (2001). Random forests. Machine Learning, 45:5–32.
- Chipman, H. A., George, E. I., and McCulloch, R. E. (2010). BART: Bayesian additive regression trees. Annals of Applied Statistics, 4:266–298.
- Breiman, L., Friedman, J. H., Olshen, R. A., and Stone, C. J. (1984). Classification and Regression Trees. Chapman & Hall/CRC.
- Hothorn, T., Hornik, K., and Zeileis, A. (2006). Unbiased recursive partitioning: a conditional inference framework. Journal of Computational and Graphical Statistics, 15:651–674
- Berk, R. A. (2008). Statistical learning from a regression perspective (2nd edition). New York: Springer.
- Friedman, J., Hastie, T., & Tibshirani, R. (2001). The elements of statistical learning (Vol. 1, No. 10). New York: Springer series in statistics.
Flexibility - interpretability trade-off
55
Flexibility/Accuracy
Inte
rpre
tabi
lity/
Tran
spar
ency
Linear/logistic regression
Deep learning
Random Forest
BART
Bagging
CART
SVM
Boosting
GAM
Training Error versus Test error
57
— The training and the test errors can be quietly different, and can vary a lot among
different partitions of the data
— Training error : it is the value of the loss measure computed on the training data
— Test error : it is the value of the loss measure computed predicting the statistical
learning method on the test data
top related