The History of AI Models - Part 3: The Transformer Era & LLMs (2017 - 2022)

In the second part of our article series, we examined the deep learning revolution and how neural networks surged with GPUs. However, by 2017, RNN and LSTM models used especially in Natural Language Processing (NLP) were still experiencing serious bottlenecks in processing long texts and running in parallel.

In our third part, we discuss that historic paper that completely changed the course of the artificial intelligence world and the rise of Large Language Models (LLMs).

2017: Attention Is All You Need

One of the most important turning points in AI history is the paper titled “Attention Is All You Need”, published by Google researchers in 2017. This paper introduced a brand new neural network architecture they named the Transformer.

Older RNN models had to read a text word by word, in sequence. The Transformer architecture, thanks to its Self-Attention mechanism, could look at all the words in a sentence simultaneously and understand which ones were related to each other. This innovation provided two massive advantages:

1. Much Better Understanding of Context: The model could better grasp the distant relationships of words with each other.
2. Parallelization: Since the obligation to read sequentially was removed, models could be trained on GPUs on a massive scale, in parallel, and much faster.

The Birth of BERT and GPT

Following the announcement of the Transformer architecture, a massive race began among tech giants.

• Google’s BERT (2018): Google introduced the BERT model, which can read texts from both directions (bidirectional). BERT created a revolution in search engines understanding what people were asking and broke records in NLP tests.
• OpenAI and the GPT Series (2018 - 2020): OpenAI adopted a different approach: “Generative” models focused only on predicting the next word.
  • GPT-1 (2018): The first attempt with 117 million parameters.
  • GPT-2 (2019): A model reaching 1.5 billion parameters that generated such realistic texts that OpenAI initially hesitated to make the model fully open source, citing that it could be “too dangerous”.
  • GPT-3 (2020): Exactly 175 billion parameters! Trained with a very large part of the internet, GPT-3 showed that artificial intelligence could not only complete text but also write code, compose poetry, and make logical inferences.

2022: ChatGPT Shakes the World

Although large language models (LLMs) had incredible potential, it was quite difficult for an ordinary person to use them. The models worked only as autocomplete tools; they did not chat.

OpenAI trained these models with the Reinforcement Learning from Human Feedback (RLHF) technique. The goal was to transform the model into an honest, harmless, and helpful dialogue partner. In November 2022, ChatGPT, built on the GPT-3.5 architecture, was opened to the public.

The result was an unprecedented explosion in AI history. ChatGPT reached 100 million active users in just two months, becoming the fastest-growing consumer application in history. Artificial intelligence was no longer just in laboratories; it was in everyone’s pocket.

ChatGPT’s success had opened the door to a new era. In the final part of our series, “Multimodality and Autonomous Agents”, we will examine how GPT-4, Llama, today’s open-weight revolution, and AI agents are shaping our future.