support vector machines and kernel methods machine learning march 25, 2010
TRANSCRIPT
Kernel Methods
• Points that are not linearly separable in 2 dimension, might be linearly separable in 3.
Kernel Methods
• Points that are not linearly separable in 2 dimension, might be linearly separable in 3.
Kernel Methods
• We will look at a way to add dimensionality to the data in order to make it linearly separable.
• In the extreme. we can construct a dimension for each data point
• May lead to overfitting.
7
Basis of Kernel Methods
• The decision process doesn’t depend on the dimensionality of the data.• We can map to a higher dimensionality of the data space.
• Note: data points only appear within a dot product.• The objective function is based on the dot product of data points – not
the data points themselves.
8
Basis of Kernel Methods
• Since data points only appear within a dot product.• Thus we can map to another space through a replacement
• The objective function is based on the dot product of data points – not the data points themselves.
Kernels
• The objective function is based on a dot product of data points, rather than the data points themselves.
• We can represent this dot product as a Kernel– Kernel Function, Kernel Matrix
• Finite (if large) dimensionality of K(xi,xj) unrelated to dimensionality of x
Kernels
• In general we don’t need to know the form of ϕ.
• Just specifying the kernel function is sufficient.• A good kernel: Computing K(xi,xj) is cheaper
than ϕ(xi)
Kernels
• Valid Kernels:– Symmetric– Must be decomposable into ϕ functions• Harder to show.• Gram matrix is positive semi-definite (psd).• Determining psd:
– all eigenvalues are positive– diagonal entries are larger than the sum of the abs.values of
the off diagonal entries in each row.
Kernels
• Given a valid kernels, K(x,z) and K’(x,z), more kernels can be made from them.– cK(x,z)– K(x,z)+K’(x,z)– K(x,z)K’(x,z)– exp(K(x,z))– …and more
18
Polynomial Kernels
• The dot product is related to a polynomial power of the original dot product.
• if c is large then focus on linear terms• if c is small focus on higher order terms• Very fast to calculate
19
Radial Basis Functions
• The inner product of two points is related to the distance in space between the two points.
• Placing a bump on each point.
20
String kernels
• Not a gaussian, but still a legitimate Kernel– K(s,s’) = difference in length– K(s,s’) = count of different letters– K(s,s’) = minimum edit distance
• Kernels allow for infinite dimensional inputs.– The Kernel is a FUNCTION defined over the input
space. Don’t need to specify the input space exactly
• We don’t need to manually encode the input.
21
Graph Kernels
• Define the kernel function based on graph properties– These properties must be computable in poly-time
• Walks of length < k• Paths• Spanning trees• Cycles
• Kernels allow us to incorporate knowledge about the input without direct “feature extraction”.– Just similarity in some space.
Where else can we apply Kernels?
• Anywhere that the dot product of x is used in an optimization.
• Perceptron:
Kernels in Clustering
• In clustering, it’s very common to define cluster similarity by the distance between points– k-nn (k-means)
• This distance can be replaced by a kernel.
• We’ll return to this more in the section on unsupervised techniques