The early model of Neurons, ADALINE
In 1943, Warren McCulloch and Walter Pitss
first model brain cells.
Later, in 1958, Frank Rosenblatt added ‘weight’
to McCulloch and Pitss’s brain model
and Donald Olding Hebb’s theory about learning,
and came up with perceptrons.
Today’s story is about a model, rather than a person.
ADAINE, known as the early neural model,
is invented in 1960,
by Bernard Widrow and his student Ted Hoff at Stanford.
Perceptron classifies the value in the net input function
according to threshold, and returns -1 or 1.
ADALINE adapts weight
in order to minimize the error of misclassifying data.
We call this cost function J(w).
Then, sum deviation square in order to calculate the error.
This equation represents curvature(quadratic function)
rather than linear step function, thus, it is differentiable.
Looking into the graph for easier understanding-
This cost function J(w) is a quadratic function.
The J(w) is minimized as the w is updated
until the gradient of this curve converges to 0.
ADALINE uses this gradient descent,
and the equation of gradually updating the w is shown below.
We solve the derivative of J(w) like this.
The derivative of composite function is solved as follows.
In short, ADALINE boils down to a problem of finding the cost function
that minimizes the w value, by repeating the equation above.