-
Notifications
You must be signed in to change notification settings - Fork 0
0.3 Convolutional networks
Convolutional networks (LeCun, 1989), also known as convolutional neural networks, or CNNs, are a specialized kind of artificial neural network for processing data that has a known grid-like topology. Examples include time-series data, which can be thought of as a 1-D grid taking samples at regular time intervals, and image data, which can be thought of as a 2-D grid of pixels.
The name "convolutional neural networks" indicates that the network employs a mathematical operation called convolution. Convolution is a specialized kind of linear operation. Convolutional networks are simply networks that use convolution in place of general matrix multiplication in at least one of their layers.
Usually, the operation used in a convolutional networks does not correspond precisely to the definition of convolution as used in other fields, such as engineering or pure mathematics.
Convolutional networks stand out as an example of neuroscientific principles influencing deep learning.
In convolutional network terminology, the first argument (in this example let's say the function x) to the convolution is often referred to as the input, and the second argument (the function w) as the kernel. The output is sometimes referred to as the feature map.
In machine learning applications, the input is usually a multidimensional array of data, and the kernel is usually a multidimensional array of parameters that are adapted by the learning algorithm. We will refer to these multidimensional arrays as tensors.
It is rare for convolution to be used alone in machine learning; instead convolution is used simultaneously with other functions, and the combination of these functions does not commute regardless of whether the convolution operation flips its kernel or not.
Discrete convolution can be viewed as multiplication by a matrix, but the matrix has several entries constrained to be equal to other entries.
In two dimensions, a doubly block circulant matrix corresponds to convolution. In addition to these constraints, convolution usually corresponds to a very sparse matrix (a matrix whose entries are mostly equal to zero).
Convolution leverages three important ideas that can help improve a machine learning system: sparse interactions, parameter sharing and equivariant representations.
Sparse interactions (also referred to as sparse connectivity or sparse weights). This is accomplished by making the kernel smaller than the input when processing an image, the input image might have thousands or millions of pixels, but we can detect small, meaningful features such as edges with kernels that occupy only tens or hundreds of pixels. We need to store fewer parameters, which both reduces the memory that computing the output requires fewer operations.
As a synonym for parameter sharing, one can say that a network has tied weights, because the value of the weight applied to one input is tied to the value of a weight applied elsewhere.
The parameter sharing used by the convolution operation means that rather thn learning a spearte set of parmeters for every location, we learn only one set.
Convolutions is thus dramatically more efficient than dense matrix multiplication in terms of the memory requirements and statistical efficiency.
Convolution is an extremely efficient way of describing transformations that apply the same linear transformation of a small local region across the entire input.
In the case of convolution, the particular form of parameter sharing caused the layer to have a property called equivariance to translation. To say a function is equivariant means that if the input changes, the output changes in the same way.
Other operations besides convolutions are usually necessary to implement a convolution network. To perform learning, one must able to compute the gradient with respect to the kernel, given the gradient with respect to the outputs.
Recall that convolution is a linear operation and can this be described as a matrix multiplication. The matrix is sparse, and each element of the kernel is copied to several elements of the matrix. This view help us to derive some of the other operations needed to implement a convolution network.
Multiplication by the transpose of the matrix defined by convolution is one such operation. This is the operation needed to back-propagate error derivatives through a conolution layer, so it is needed to train convolutional networks that have more than one hidden layer.
These three operations, convolution, backprop from output to weights, and backprop from output to inputs are sufficient to compute all the gradients needed to train any depth of feedforward convolutional network.
It is also worth mentioning that neuroscience has told us relatively little about how to train convolutional networks. Model structures with parameter sharing across multiple spatial locations date back to early connectionist models of vision (Marr and Poggio, 1976), but these models did not use the modern back-propagation algorithm and gradient descent.
Convolutional nets were some of the first working deep networks train with back-propagaion. It is not entirely clear why convolutional networks succeeded when general back-propagation networks were considered to have failed. It may simple be that convolutional networks were more computational efficient than fully connected networks, so it was easier to run multiple experiments with them, and tune their implementation and hyperparameters.
It may be that primary barriers to the success of neural networks were psychological (practitioners did not expect neural networks to work, so they did not make serious effort to use neural networks).
Convolutional networks provide a way to specialize neural networks to work with data that has a clear grid-structured topology and to scale such models to very large size. This approach has been the most successful on a two-dimensional image topology.