Deep learning is machine learning done with neural networks that have many layers. Each layer turns its input into a slightly more abstract representation, and training adjusts the network's weights until its errors on examples shrink. Natural language processing (NLP) is the field that makes software work with human language, and its current systems are built on deep learning.
This guide explains how a neural network learns, with small worked examples instead of equations; why the approach took off after 2012; what convolutional networks, recurrent networks and transformers are each for; what embeddings are; how NLP moved from hand-written rules to large language models (LLMs); and what LLMs can and cannot do reliably. Two related articles cover the neighbouring questions: how classic machine learning algorithms scale to big data, and why AI matters in computer science today. Generative AI as a product category, with its uses and risks, is covered in the rise of generative AI; here it appears only where the technology underneath explains it.
How a neural network works
A neural network is a function with adjustable numbers inside it. It takes numbers in (pixel values, word IDs, sensor readings), passes them through layers of simple units, and produces numbers out, such as a score for each possible answer. Despite the name, the units are arithmetic, not biology. Each one multiplies every input by a weight, adds the results plus a constant called a bias, and passes the total through a simple non-linear function. A common choice is ReLU, which turns negative totals into zero and leaves positive ones alone.
Units are arranged in layers. An input layer receives the data, one or more hidden layers transform it, and an output layer gives the answer. The outputs of one layer are the inputs of the next. The weights and biases, together called parameters, are everything the network has learned; a real network has millions or billions of them.
A worked example: one unit that scores a ticket
Take a single unit that decides whether a support ticket is urgent. It gets two inputs: 1 if the message mentions an outage (otherwise 0), and 1 if the sender has a support contract (otherwise 0). Its current weights are 0.5 and 0.2, and its bias is −0.3.
- A ticket that mentions an outage, from a contract customer, scores 0.5 × 1 + 0.2 × 1 − 0.3 = 0.4.
- The correct label is urgent, written as 1, so the error is 1 − 0.4 = 0.6.
- With a step size (the learning rate) of 0.1, each weight moves by 0.1 × error × its input. Both weights rise by 0.06, to 0.56 and 0.26, and the bias rises by 0.06, to −0.24.
- The same ticket now scores 0.56 + 0.26 − 0.24 = 0.58. The error has fallen from 0.6 to 0.42.
Repeat that over thousands of labelled tickets and the weights settle where the errors across all of them are small. That is the whole idea of training; the rest is scale.
How training adjusts the weights
Training is a loop: show the network an example, measure how wrong its output is, and change every weight slightly in the direction that makes the error smaller. The measure of error is the loss, and stepping downhill on it, a little at a time, is gradient descent. In the single unit above, the update rule is exactly gradient descent on a squared-error loss.
A network with many layers needs one more step: working out how much each weight, deep inside, contributed to the error at the output. A 1986 paper in Nature by Rumelhart, Hinton and Williams described backpropagation as a procedure that adjusts the connection weights again and again to shrink the gap between the output the network gives and the output it should give. It passes the error backwards, layer by layer, and it is still the most common training algorithm for neural networks.
What training leaves behind is a set of weights, not a list of examples. That is why a trained network can handle inputs it has never seen, and also why it cannot reliably tell you where a particular answer came from.
What makes deep learning "deep"
Machine learning existed long before deep learning. The difference is who designs the features, the measurements a model learns from. In their 2015 review in Nature, LeCun, Bengio and Hinton explain that conventional machine learning depended on hand-built feature extractors, designed with careful engineering and deep domain knowledge, to turn raw data, such as the pixels of an image, into something a classifier could use. Deep learning is representation learning with many levels: the layers learn their own features from the raw data.
The review's worked example is an image. In a network trained on photos, the first layer typically responds to edges at particular orientations and positions. The second responds to motifs, particular arrangements of edges. The third assembles motifs into combinations that correspond to parts of familiar objects, and later layers detect objects as combinations of parts. Nobody programs those detectors. They are what the weights become when training reduces the error.
Depth pays because each layer builds on the one below, but it costs data and computation. That is why deep learning stayed on the margins until both became cheap enough.
Machine learning vs deep learning vs generative AI
The three terms nest rather than compete. Google's introduction describes machine learning as training software, called a model, to make predictions or generate content from data. Deep learning is the part of machine learning that uses neural networks of many layers. Generative AI is the class of models that create content; NIST's Generative AI Profile (July 2024) uses a definition of models that imitate the structure and characteristics of their input data to produce new, synthetic content such as text, images, audio and video. The generative models in wide use today, including large language models, are deep learning models. Artificial intelligence is the umbrella over all of it, including older approaches such as hand-written rules.
| Machine learning (classic) | Deep learning | Generative AI | |
|---|---|---|---|
| What it is | Models that learn patterns from examples | Machine learning with neural networks of many layers | Models that produce new text, images, audio or code |
| Where the features come from | People choose and engineer them | The layers learn them from raw data | Learned, as in deep learning |
| Typical input | Tables: rows and columns of business data | Images, sound, text and video | Very large collections of text, images or other media |
| Typical models | Linear and logistic regression, decision trees, XGBoost | Convolutional networks, recurrent networks, transformers | Large language models, image generators |
| Typical output | A number, a class or a ranking | A class, a number or a vector | New content in response to a prompt |
| Compute to train | Often an ordinary computer | GPUs or similar accelerators | Large clusters of accelerators |
| Example | Predicting which invoices will be paid late | Spotting a defect in a photo of a part | Drafting a reply to a customer email |
| Main weakness | Only as good as the features people chose | Needs a lot of data, and its decisions are hard to inspect | Fluent, confident errors (confabulation) |
Deep learning is not automatically better. On ordinary business tables, a 2022 benchmark across 45 datasets found that tree-based models such as XGBoost and random forests still beat deep learning on medium-sized tables of around 10,000 rows. The purchase, churn and lead scores behind machine learning in marketing are usually built on exactly that kind of customer table. Deep learning earns its cost where the input is raw and unstructured: pixels, sound and language.
Why deep learning took off after 2012
By 2012, neural networks and backpropagation were decades old. Three things arrived together.
- Data. Large collections of labelled examples appeared. ImageNet held over 15 million labelled high-resolution images in about 22,000 categories, and its annual challenge (ILSVRC) used a subset of roughly 1.2 million training images in 1,000 classes.
- GPUs. Graphics processors run the same arithmetic on many numbers in parallel, which is what a layer of a neural network does. The Nature review notes that GPUs that were convenient to program let researchers train networks 10 or 20 times faster.
- Training techniques. ReLU units, which train faster than the older S-shaped functions, and dropout, which switches off random units during training so the network cannot rely on any one of them, reduced overfitting.
The result was AlexNet, a deep convolutional network by Krizhevsky, Sutskever and Hinton. It won ILSVRC-2012 with a top-5 test error of 15.3%, against 26.2% for the second-best entry; top-5 error is the share of images whose correct label was not among the model's five best guesses. AlexNet had 60 million parameters, 650,000 neurons and eight learned layers, five convolutional and three fully connected. It took five to six days to train on two NVIDIA GTX 580 GPUs with 3 GB of memory each, and the authors found that removing any single convolutional layer made results worse. They also wrote that the network's size was limited mainly by GPU memory and training time.
Depth kept paying. In 2015, Microsoft Research's residual networks (ResNet) made it practical to train networks up to 152 layers deep, and an ensemble of them won ILSVRC 2015 with a 3.57% error.
The same pattern later appeared in language. In January 2020, OpenAI researchers reported that a language model's loss falls as a power law with model size, dataset size and training compute, with some trends spanning more than seven orders of magnitude. Predictable returns from scale are one reason language models grew to the 175 billion parameters of GPT-3 that same year.
The main deep learning architectures and what each is for
An architecture is the wiring of a network: which units connect to which, and which weights are shared. Three families cover most of deep learning.
| Architecture | Built for | How it reads the input | Main limit |
|---|---|---|---|
| Convolutional network (CNN) | Images, video, audio spectrograms | Small shared filters slide across a grid | Sees local patterns first; needs depth for the big picture |
| Recurrent network (RNN, LSTM) | Sequences: text, speech, sensor readings | One step at a time, carrying a state | Forgets over long sequences, and cannot run steps in parallel |
| Transformer | Text first; the basis of today's LLMs | All tokens at once, through attention | Cost grows with the square of the input length |
Convolutional networks for images
The Nature review names four ideas behind convolutional networks: local connections, shared weights, pooling and many layers. Each unit looks only at a small patch of the layer below, and every unit in a feature map uses the same set of weights, called a filter. Pooling then merges nearby responses, so a pattern is still found if it shifts slightly.
A worked example: picture a 3 × 3 filter whose weights are negative in the left column and positive in the right. Slid across a photo of a circuit board, it gives a large output wherever dark pixels meet bright ones from left to right, and close to zero on flat areas. Because the same nine weights are reused at every position, the network finds that edge anywhere in the image, and it learns nine numbers instead of a separate weight for every pixel. A real layer runs dozens of such filters, and training, not a person, sets their weights. By 2015, the review described convolutional networks as the dominant approach for almost all recognition and detection tasks in images.
Recurrent networks and their limits
A recurrent network reads a sequence one element at a time and carries a state vector that summarizes what it has seen so far. That suits language, where order matters. To classify "The backup job that finance scheduled for Friday night after the storage migration has failed", it has to carry "backup job" across ten words until "has failed" arrives.
Recurrent networks have three limits:
- They forget. The review explains that backpropagated gradients grow or shrink at each time step, so over many steps they typically explode or vanish, and long-range links are learned poorly. Long short-term memory (LSTM) networks, published in 1997, add a gated memory cell that holds information for longer, and they powered speech recognition and translation systems into the mid-2010s.
- They are sequential. Step 20 cannot start before step 19 finishes, a constraint that rules out parallel work within a single training example and makes long inputs slow to train.
- They squeeze. A recurrent translator encoded a whole sentence into one fixed-length vector before writing the translation. In 2014, Bahdanau, Cho and Bengio identified that vector as a bottleneck and let the decoder search back over the parts of the source sentence relevant to each word it produced. That mechanism is attention.
Transformers and attention
In June 2017, researchers at Google Brain, Google Research and the University of Toronto published "Attention Is All You Need". It proposed the transformer, an architecture built on attention alone, with no recurrence and no convolutions at all. In self-attention, every token scores how relevant every other token in the input is to it, then builds its new representation as a weighted mix of theirs. All tokens are processed at once, so training runs in parallel on GPUs: the paper's larger model set a new best translation score (28.4 BLEU on WMT 2014 English-to-German) after 3.5 days of training on eight GPUs.
A worked example: in "The server rejected the request because it was malformed", an attention head representing "it" can put most of its weight on "request". Change the last word to "overloaded" and the weight should move to "server". No rule says so; the association is learned from text in which such sentences resolve one way or the other. A transformer stacks many such layers, and each layer has several attention heads that pick up different relationships.
The price is in the word "every". Self-attention relates each token to all the others, so its cost grows with the square of the number of tokens in the context. That is why long inputs are expensive and why every model has a maximum context length.
Embeddings: how words become numbers
A network only does arithmetic, so text has to become numbers first. The naive way gives each word in the vocabulary its own position in a very long list of zeros, called one-hot encoding. It says nothing about meaning: "Tuesday" is as far from "Wednesday" as from "firewall". An embedding replaces that with a short vector of learned numbers per token, trained so that words used in similar ways end up close together. The Nature review gives the example of a network trained to predict the next word in news stories: its learned vectors for Tuesday and Wednesday came out very similar, as did those for Sweden and Norway.
In 2013, Google researchers published word2vec, which learned good word vectors from 1.6 billion words of text in under a day. The paper cites a striking property of such vectors: vector("King") − vector("Man") + vector("Woman") lands closest to the vector for "Queen". Directions in the space carry relationships.
Word2vec gives each word one vector, whatever the sentence. That fails on a word like "cell", which means different things in a spreadsheet, a phone plan and a battery. Contextual embeddings fix this by giving a word a different vector in each sequence, computed from the words around it, and a transformer's self-attention layers produce exactly that.
Embeddings also power semantic search, a worked example most people have already used. Turn every help article into a vector, turn the question "my laptop won't charge" into a vector the same way, and return the nearest articles. The top result can be an article titled "Battery not charging when plugged in", although the two share no keyword. The same retrieval step is how an assistant finds the passages it answers from.
What NLP is, and how it moved from rules to large language models
Natural language processing is the part of computer science that lets software work with human language. Typical tasks are classifying a message, extracting names, dates and amounts, translating, summarizing and answering questions; speech recognition and speech synthesis sit at its edges. The field has changed its method three times, and one small task shows each change: routing the email "I was charged twice for my March invoice" to the billing team.
| Era | How the system handles language | The billing email | Weakness |
|---|---|---|---|
| Hand-written rules | People write dictionaries, grammars and patterns | A rule fires on "invoice" or "charged" | Breaks on phrasing nobody anticipated; rules pile up |
| Statistical NLP (from about 1990) | Probabilities counted from large collections of text | Word and word-pair counts from past billing emails outweigh the others | Treats words as unrelated symbols |
| Neural NLP (from about 2013) | Embeddings and neural networks, one model trained per task | "charged twice" lands near "double payment" in vector space | Needs labelled data and a trained model for every task |
| Pretrained transformers and LLMs (2018 on) | One model pretrained on vast text, adapted by fine-tuning or a prompt | The model reads an instruction and the email, and answers "billing" | Can be fluently wrong; costs more per request |
From hand-written rules to statistics
Early language systems were written by hand: bilingual dictionaries, grammar rules, keyword patterns. Rules are exact and explainable, and they still suit narrow, predictable inputs, but language varies more than any team can write down.
In 1990, Brown and colleagues at IBM published A Statistical Approach to Machine Translation, which replaced grammar rules with probabilities learned from translated text. They note that Warren Weaver had suggested statistical methods for translation in 1949 and that researchers soon abandoned the idea, which they blamed on slow computers and too little machine-readable text. Their own data came from the Canadian Hansard, the record of Parliament that Canada keeps in both English and French: about 100 million words of English with the corresponding French. In an early test on 73 French sentences, with the English vocabulary limited to 1,000 words, 48% of the translations were judged reasonable.
Statistical language models of that period counted how often short word sequences, called n-grams, occur. As the Nature review points out, they treat each word as an atomic unit, so what is learned about "invoice" tells the model nothing about "bill".
Neural networks
Neural NLP replaced counts with embeddings and networks: word2vec in 2013, recurrent encoder-decoder translators, then attention in 2014. In September 2016, Google described GNMT, a deep LSTM network with 8 encoder and 8 decoder layers using attention, which splits rare words into common sub-word units it calls wordpieces. In a human side-by-side evaluation on isolated simple sentences, it cut translation errors by 60% on average compared with Google's phrase-based production system. Splitting text into sub-word pieces is still how language models read; the pieces are called tokens.
Pretrained transformers and large language models
From 2018 the pattern changed from one model per task to one pretrained model adapted many times. Google's BERT (October 2018) pretrained a transformer on unlabelled text and could then be fine-tuned for tasks such as question answering by adding just one output layer. OpenAI's GPT-3 (May 2020) went further: an autoregressive language model with 175 billion parameters, trained on 300 billion tokens, that performed many tasks from instructions and a few examples written in the prompt, with no further training.
Bigger was not automatically more useful. OpenAI's InstructGPT work (March 2022) fine-tuned GPT-3 with human feedback, and in human evaluations people preferred the outputs of a 1.3-billion-parameter InstructGPT to those of the 175-billion-parameter GPT-3. ChatGPT, introduced on November 30, 2022, is a sibling of InstructGPT trained with the same reinforcement learning from human feedback, adapted to dialogue.
So a large language model is a transformer trained to predict the next token over a very large body of text, usually tuned afterwards to follow instructions.
What large language models can and cannot do reliably
An LLM writes one token at a time, each a likely continuation of the text so far. NIST's Generative AI Profile notes that this kind of statistical prediction can produce accurate, consistent output, and also false content stated with confidence, which it calls confabulation and which most people call hallucination. The same document warns that models can confabulate the logic or citations that seem to justify an answer.
OpenAI's September 2025 research explains where the line falls. Patterns that are consistent across the training data, such as spelling and matching parentheses, stop producing errors as models grow, but rare, arbitrary facts cannot be inferred from patterns, and those are where hallucinations come from. It adds that evaluations scored on accuracy alone reward guessing over admitting uncertainty.
| Task | Reliable? | Why | What to do |
|---|---|---|---|
| Summarize, rewrite or translate text you provide | Usually | The answer is in the input | Spot-check against the source |
| Classify a message or extract fields into a fixed format | Usually, once tested | It matches patterns in the input | Validate against a schema; test on real cases |
| Draft emails, documents or code | As a first draft | Common patterns are learned well | A person reviews before anything ships |
| Recall specific facts: dates, prices, figures, citations | No | Arbitrary facts cannot be predicted from patterns | Supply the source documents and require citations |
| Know about events after its training data ends | No | Nothing newer is in its weights | Retrieve current data when the question is asked |
| Find one detail in a very long input | Varies | Accuracy drops when the detail sits in the middle | Put key material first or last; retrieve short passages |
| Exact arithmetic and long chains of logic | Not guaranteed | Errors compound over many generated steps | Let code do the calculation |
| Give the same answer twice | Not by default | Sampling chooses among likely tokens | Fix the settings and test repeatedly |
The long-input row comes from a 2023 study which found that models often do best when the relevant information is at the beginning or end of the input, and significantly worse when it sits in the middle of a long context.
A worked example makes the difference concrete. Paste a 20-page supplier contract into a model and ask for its termination terms: the answer is in the input, and you can check it in minutes. Ask the same model, with no documents, which clause of a regulation applies to your business: it may return a fluent answer with a clause number that does not exist.
Warning
Do not use a language model as the only source of a fact that matters. Give it the documents, make it cite them, and have a person check anything that will be acted on.
For more on where model understanding breaks down, see AI limitations in understanding; for multi-step logic and reasoning models, see how AI reasons, and where it fails. To try the ideas yourself, a first AI project in Python trains a small model step by step, and our AI frameworks comparison covers PyTorch, TensorFlow and JAX.
If you are deciding where a language model belongs in your own work, our AI & automation service uses a model only where a step needs judgment, such as reading a document or drafting a reply, and ordinary code everywhere else; nothing changes in your systems until a rule or a person approves it. The assistants it builds answer from your own pages, documents and live data, cite their sources and say when they do not know.
