The History of AI Models - Part 2: The Deep Learning Revolution (2000s - 2016)

In the first part of our article series, we covered the birth of AI, the perceptrons that formed the basis of the first neural networks, and the AI winters. Although researchers made significant algorithmic progress in the late 1990s, they faced a major obstacle: insufficient processing power and a lack of data.

In our second part, we examine the Deep Learning Revolution that began in the 2000s and brought artificial intelligence into every aspect of our lives.

The Rise of GPUs and the Data Explosion

With the widespread use of the internet in the early 2000s, there was a massive explosion in digital data production (photos, texts, videos). This meant that the “fuel” needed to train machine learning models was now available in abundance.

However, processing this much data with standard processors (CPUs) took months or even years. The solution came from Graphics Processing Units (GPUs) developed for the gaming industry. Researchers discovered that the parallel processing capability of GPUs could perform the massive matrix multiplications in neural networks at extraordinary speeds. With Nvidia introducing the CUDA platform, GPUs became the engine of deep learning.

2012: ImageNet and the AlexNet Revolution

The moment that is considered a turning point for deep learning worldwide occurred in 2012. ImageNet was a massive dataset competition consisting of millions of labeled images. Until that year, traditional computer vision techniques had an error rate of around 25%.

Geoffrey Hinton and his team (Ilya Sutskever and Alex Krizhevsky) participated in the competition with a Convolutional Neural Network (CNN) model they named AlexNet. Trained on two GPUs, this deep model crushed its competitors by incredibly reducing the error rate to 15.3%. This success sent a shockwave through academia and the industry. Everyone started abandoning traditional methods and turning to deep neural networks.

First Steps in Natural Language Processing (NLP)

Deep learning was creating a revolution not only in visual processing but also in text processing (NLP). In 2013, Google researcher Tomas Mikolov and his team introduced the Word2Vec algorithm.

Word2Vec allowed the computer to understand the semantic relationships between words by converting words into mathematical vectors. (For example: King - Man + Woman = Queen). This innovation paved the way for texts to be processed as semantic structures rather than just strings of words.

During the same period, RNN (Recurrent Neural Networks) and LSTM (Long Short-Term Memory) models, which are effective on time series and texts, became popular. However, RNNs also had a weakness: they forgot past information in long texts and were not suitable for parallel training.

Towards the Golden Age of Artificial Intelligence

By 2016, deep learning was the focus of massive companies. AlphaGo, developed by Google’s DeepMind team, defeating the world’s best Go player Lee Sedol proved the power of Deep Reinforcement Learning.

However, on the language models side, the problem of consistently understanding and generating long texts still persisted.

The architecture that would solve this problem and reshape the world with artificial intelligence was on the way. In the third part of our series, “The Transformer Era and Large Language Models”, we will examine Google’s legendary “Attention is All You Need” paper and the rise of OpenAI’s GPT models.