Backpropagation

Backpropagation is a method of training neural networks to perform tasks more accurately. The algorithm was first used for this purpose in 1974 in papers published by Werbos, Rumelhart, Hinton, and Williams. The term backpropagation is short for "backward propagation of errors".

It works especially well for feed forward neural networks (networks without any loops) and problems that require supervised learning.

How it works

The idea is to test how wrong the neural network is and then correct it. This is repeated many times.

With a little more detail:

1. You create a loss function, which shows how far the answers from the neural net are from the real answers. (This is often done many times. After that you take the average)
2. You calculate how to adjust the parameters (weights and biases) inside the neural net through the derivative of the loss function. Specifically, the chain rule is used to find the derivative with respect to each parameter.
3. You adjust the parameters. How you adjust the parameters is based on your training method, with one of the simplest being gradient descent .

This is repeated until the neural network is good enough at its job -i.e., its error as measured by the loss function is low.

Backpropagation Media

• 

  A simple neural network with two input units (each with a single input) and one output unit (with two inputs)

• 

  Error surface of a linear neuron for a single training case

• 

  Diagram of an artificial neural network to illustrate the notation used here

• 

  Gradient descent may find a local minimum instead of the global minimum.