Artificial intelligence (AI) is a vast scientific field that has existed for decades, encompassing many different techniques. Generative AI is a specific, powerful type of AI that can create new content, such as text or images, rather than just classifying existing data. This ability to generate novel outputs marks a significant evolution from earlier AI systems.
From AI’s Beginnings to Machine Learning
Artificial intelligence emerged as a scientific discipline shortly after the second world war. Its beginnings coincided with the advent of the first digital computers. For much of its history, progress in AI was very slow. This began to change in the current century.
AI is a broad field, but one class of techniques, machine learning, started to become practically useful around 2005. The name “machine learning” can be misleading. It does not mean a computer learns like a person reading a textbook. Instead, it refers to systems that learn from data to perform specific tasks.
How Machine Learning Recognizes Patterns
The simplest way machine learning works is through supervised learning. This method requires training data. Training data consists of input-output pairs. For example, a picture of a human face is the input, and the name of the person in the picture is the desired output. When people upload and label photos on social media, they contribute to this training data.
This process is often a classification task. The machine learning system classifies an input. For instance, it might classify a picture as “Alan Turing” or identify a tumor on an X-ray scan. This technology also enables self-driving cars to recognize stop signs, bicycles, or pedestrians. These classification tasks became powerful around 2005 and were supercharged around 2012.
At its core, machine learning often uses neural networks. These are software implementations inspired by the structure of animal brains. A human brain, for example, contains an estimated 86 billion neurons. Each neuron can connect to up to 8,000 other neurons. In a simplified model, each software neuron performs a tiny, simple pattern recognition task. It might look for a specific color, like red, in a pixel. When it detects its pattern, it sends a signal to its connections.
The idea of building such systems dates back to the 1940s. Researchers McCulloch and Pitts observed that brain structures resembled electrical circuits. The concept of implementing this in software was explored in the 1960s and again in the 1980s. However, it only became truly feasible this century.
The Three Pillars Driving Modern AI
The acceleration of AI this century, particularly machine learning, stems from three key developments. First, there were scientific advances in what is called deep learning. Deep learning involves neural networks with many layers, allowing them to learn more complex patterns.
Second, there was the availability of big data. To configure these neural networks to recognize patterns effectively, they need vast amounts of training data. The more data, the better the system can learn.
Third, affordable and scalable computational power became available. Training large neural networks requires immense processing capability. Graphics Processing Units (GPUs) proved to be particularly well-suited for the mathematical operations involved. GPUs, the same technology used in gaming computers, supercharged AI development around 2012. This made companies like Nvidia extremely valuable. Silicon Valley invested billions of dollars into AI, realizing that bigger neural networks, more data, and more compute power led to greater capabilities.
The Emergence of Generative AI
While classification tasks were driving AI forward, a new direction emerged around 2017 and 2018. A specific machine learning technology proved highly effective for this new era of large-scale AI. A pivotal research paper, “Attention Is All You Need,” introduced a neural network architecture called the Transformer. This architecture was designed for large language models (LLMs).
In June 2020, OpenAI released GPT-3, a large language model. GPT-3 represented a dramatic step change in capability compared to previous systems. Its scale was unprecedented. In neural network terms, GPT-3 had 175 billion parameters. Parameters refer to individual neurons or the connections between them. While not the same as brain neurons, this number is of a similar order of magnitude.
To train such a massive network, an enormous amount of data was needed. GPT-3 was trained on approximately 500 billion words of ordinary English text. This data was collected by downloading the entire World Wide Web, including text from PDF documents, government regulations, and university minutes. To put this scale into perspective, a person would take an extremely long time to read that much text. This shows that machine learning is much less efficient at learning than human beings are, requiring vastly more data.
The Scale and Impact of Large Language Models
The core distinction between earlier machine learning and generative AI lies in their output. Older machine learning systems primarily performed classification tasks. They identified what something is, like recognizing a face or a tumor. Generative AI, on the other hand, creates new content.
Large language models, built with Transformer architectures and trained on massive datasets, can generate human-like text. They can write essays, translate languages, summarize documents, or create different kinds of creative content. This ability to produce novel and coherent outputs, rather than just categorize existing ones, is what defines generative AI. The scale of these models, combined with vast training data and powerful computing, has enabled them to perform tasks that were once considered exclusive to human intelligence.
