AlexNet, Showing Overwhelming Talent of Deep Learning
Long time ago, Alfred Lothar Wegener argued that continents were once connected
and that they are still moving. He was ridiculed by the world at that time.
In 2019, during Geoffrey Hinton’s 2018 Turing Award speech,
he looked back on his research
In late 2000s, Alex Krizhevsky from Geoffrey Hinton’s team initiated various attempts
to prove how deep learning can solve various problems in reality.
First, they came up with the CIFAR data set in 2009,
to study image classification with deep learning.
In 2011, they were doing a great job in recognizing images of CIFAR data set with deep learning.
Ilya Sutskever from the same lab suggested Alex to participate on the ImageNet contest.
The ILSVRC(ImageNet Large Scale Visual Recognition Challenge) was compared
to the Olympics of the field of vision.
It gave the championship trophy to whatever team
with the lowest error rate in image recognition.
Starting from 2010, in 2011,
the SVM machine learning from Xerox lab won the championship.
Alex and Ilya started their research for the competition.
They added the GPU feature to enhance their performance
in a limited amount of time.
Despite their effort, there was a problem of overfitting,
which hindered further development.
Especially as deep learning used neural network, the issue was more challenging.
Nitish Srivastava, the co-member in the lab, solved this chronic problem.
It was the dropout technique.
It does not calculate all nodes.
Instead, it randomly drops neurons.
Later, this idea was registered as a patent for those four people,
and was published in NItish’s master dissertation paper.
ReLu(Rectified Linear Unit) from Yoshua Bengio’s further improved the performance.
Finally, in 2012, Alex won the ILSVRC contest with an astonishing record even to himself.
This showed the world that deep learning is now ready to come out
from the lab and change the human history.
After this contest, Hinton, Alex, and Ilya started their business called DNNResearch in 2012.learning
A year later, Google took over the company to scout them.