# Deep learning and NLP explained: neural networks, transformers and LLMs

> Deep learning trains neural networks of many layers on data. How they learn, why they took off in 2012, and how NLP went from rules to transformers and LLMs.

- URL: https://computese.com/the-ai-and-machine-learning-revolution/
- Author: Duong Quan Nguyen, CEO, Computese
- Published: 2024-10-05
- Updated: 2026-09-25
- Topics: AI & automation

## In short
- Deep learning is machine learning with neural networks of many layers. Each layer turns its input into a slightly more abstract representation, and training adjusts the network's weights, step by step, until its errors on examples shrink.
- Deep learning took off in 2012: AlexNet, trained on 1.2 million labelled images with two GPUs, won the ImageNet challenge with a 15.3% top-5 error, against 26.2% for the next entry.
- Convolutional networks suit images; recurrent networks read sequences step by step but forget and train slowly; the transformer (2017) replaced recurrence with attention, so every token weighs every other token in parallel.
- Natural language processing (NLP) moved from hand-written rules to statistics learned from text, then to neural networks and embeddings, then to large language models (LLMs) pretrained on vast amounts of text.
- LLMs are dependable at reworking text you give them: summarizing, classifying, extracting, drafting. They are not a dependable source of facts, because they generate likely text and can state false things with confidence.

Deep learning is machine learning done with neural networks that have many layers. Each layer turns its input into a slightly more abstract representation, and training adjusts the network's weights until its errors on examples shrink. Natural language processing (NLP) is the field that makes software work with human language, and its current systems are built on deep learning.

This guide explains how a neural network learns, with small worked examples instead of equations; why the approach took off after 2012; what convolutional networks, recurrent networks and transformers are each for; what embeddings are; how NLP moved from hand-written rules to large language models (LLMs); and what LLMs can and cannot do reliably. Two related articles cover the neighbouring questions: how classic [machine learning algorithms scale to big data](https://computese.com/a-revolutionary-machine-learning-algorithm/), and [why AI matters in computer science today](https://computese.com/why-ai-in-computer-science-is-important-in-today/). Generative AI as a product category, with its uses and risks, is covered in [the rise of generative AI](https://computese.com/the-rise-of-generative-ai/); here it appears only where the technology underneath explains it.

## How a neural network works

A neural network is a function with adjustable numbers inside it. It takes numbers in (pixel values, word IDs, sensor readings), passes them through layers of simple units, and produces numbers out, such as a score for each possible answer. Despite the name, the units are arithmetic, not biology. Each one multiplies every input by a **weight**, adds the results plus a constant called a **bias**, and passes the total through a simple non-linear function. A common choice is ReLU, which turns negative totals into zero and leaves positive ones alone.

Units are arranged in layers. An input layer receives the data, one or more hidden layers transform it, and an output layer gives the answer. The outputs of one layer are the inputs of the next. The weights and biases, together called parameters, are everything the network has learned; a real network has millions or billions of them.

### A worked example: one unit that scores a ticket

Take a single unit that decides whether a support ticket is urgent. It gets two inputs: 1 if the message mentions an outage (otherwise 0), and 1 if the sender has a support contract (otherwise 0). Its current weights are 0.5 and 0.2, and its bias is −0.3.

1. A ticket that mentions an outage, from a contract customer, scores 0.5 × 1 + 0.2 × 1 − 0.3 = 0.4.
2. The correct label is urgent, written as 1, so the error is 1 − 0.4 = 0.6.
3. With a step size (the learning rate) of 0.1, each weight moves by 0.1 × error × its input. Both weights rise by 0.06, to 0.56 and 0.26, and the bias rises by 0.06, to −0.24.
4. The same ticket now scores 0.56 + 0.26 − 0.24 = 0.58. The error has fallen from 0.6 to 0.42.

Repeat that over thousands of labelled tickets and the weights settle where the errors across all of them are small. That is the whole idea of training; the rest is scale.

### How training adjusts the weights

Training is a loop: show the network an example, measure how wrong its output is, and change every weight slightly in the direction that makes the error smaller. The measure of error is the **loss**, and stepping downhill on it, a little at a time, is **gradient descent**. In the single unit above, the update rule is exactly gradient descent on a squared-error loss.

A network with many layers needs one more step: working out how much each weight, deep inside, contributed to the error at the output. A 1986 paper in _Nature_ by Rumelhart, Hinton and Williams described **backpropagation** as a procedure that [adjusts the connection weights again and again to shrink the gap](https://www.nature.com/articles/323533a0) between the output the network gives and the output it should give. It passes the error backwards, layer by layer, and it is still [the most common training algorithm for neural networks](https://developers.google.com/machine-learning/crash-course/neural-networks/backpropagation).

![A small neural network of circles in four columns joined by thin lines. A document enters on the left, an arrow lands just off a target on the right, and a dashed arrow returns to one orange connection.](https://computese.com/images/blog/the-ai-and-machine-learning-revolution/training.d3f3a3d67c-1536.webp)

*No weight is written by a programmer: each one ends up wherever many small corrections leave it.*

What training leaves behind is a set of weights, not a list of examples. That is why a trained network can handle inputs it has never seen, and also why it cannot reliably tell you where a particular answer came from.

## What makes deep learning "deep"

Machine learning existed long before deep learning. The difference is who designs the features, the measurements a model learns from. In their 2015 review in _Nature_, LeCun, Bengio and Hinton explain that conventional machine learning [depended on hand-built feature extractors, designed with careful engineering and deep domain knowledge](https://www.cs.toronto.edu/~hinton/absps/NatureDeepReview.pdf), to turn raw data, such as the pixels of an image, into something a classifier could use. Deep learning is representation learning with many levels: the layers learn their own features from the raw data.

The review's worked example is an image. In a network trained on photos, the first layer typically responds to edges at particular orientations and positions. The second responds to motifs, particular arrangements of edges. The third assembles motifs into combinations that correspond to parts of familiar objects, and later layers detect objects as combinations of parts. Nobody programs those detectors. They are what the weights become when training reduces the error.

Depth pays because each layer builds on the one below, but it costs data and computation. That is why deep learning stayed on the margins until both became cheap enough.

## Machine learning vs deep learning vs generative AI

The three terms nest rather than compete. Google's introduction describes machine learning as [training software, called a model, to make predictions or generate content from data](https://developers.google.com/machine-learning/intro-to-ml/what-is-ml). Deep learning is the part of machine learning that uses neural networks of many layers. Generative AI is the class of models that create content; NIST's Generative AI Profile (July 2024) uses a definition of [models that imitate the structure and characteristics of their input data to produce new, synthetic content](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf) such as text, images, audio and video. The generative models in wide use today, including large language models, are deep learning models. Artificial intelligence is the umbrella over all of it, including older approaches such as hand-written rules.

|                              | Machine learning (classic)                              | Deep learning                                              | Generative AI                                         |
| ---------------------------- | ------------------------------------------------------- | ---------------------------------------------------------- | ----------------------------------------------------- |
| What it is                   | Models that learn patterns from examples                | Machine learning with neural networks of many layers       | Models that produce new text, images, audio or code   |
| Where the features come from | People choose and engineer them                         | The layers learn them from raw data                        | Learned, as in deep learning                          |
| Typical input                | Tables: rows and columns of business data               | Images, sound, text and video                              | Very large collections of text, images or other media |
| Typical models               | Linear and logistic regression, decision trees, XGBoost | Convolutional networks, recurrent networks, transformers   | Large language models, image generators               |
| Typical output               | A number, a class or a ranking                          | A class, a number or a vector                              | New content in response to a prompt                   |
| Compute to train             | Often an ordinary computer                              | GPUs or similar accelerators                               | Large clusters of accelerators                        |
| Example                      | Predicting which invoices will be paid late             | Spotting a defect in a photo of a part                     | Drafting a reply to a customer email                  |
| Main weakness                | Only as good as the features people chose               | Needs a lot of data, and its decisions are hard to inspect | Fluent, confident errors (confabulation)              |

Deep learning is not automatically better. On ordinary business tables, a 2022 benchmark across 45 datasets found that [tree-based models such as XGBoost and random forests still beat deep learning on medium-sized tables](https://arxiv.org/abs/2207.08815) of around 10,000 rows. The purchase, churn and lead scores behind [machine learning in marketing](https://computese.com/machine-learning-in-marketing/) are usually built on exactly that kind of customer table. Deep learning earns its cost where the input is raw and unstructured: pixels, sound and language.

## Why deep learning took off after 2012

By 2012, neural networks and backpropagation were decades old. Three things arrived together.

- **Data.** Large collections of labelled examples appeared. [ImageNet held](https://papers.nips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf) over 15 million labelled high-resolution images in about 22,000 categories, and its annual challenge (ILSVRC) used a subset of roughly 1.2 million training images in 1,000 classes.
- **GPUs.** Graphics processors run the same arithmetic on many numbers in parallel, which is what a layer of a neural network does. The Nature review notes that GPUs that were convenient to program let researchers train networks 10 or 20 times faster.
- **Training techniques.** ReLU units, which train faster than the older S-shaped functions, and dropout, which switches off random units during training so the network cannot rely on any one of them, reduced overfitting.

The result was AlexNet, a deep convolutional network by Krizhevsky, Sutskever and Hinton. It [won ILSVRC-2012 with a top-5 test error of 15.3%, against 26.2% for the second-best entry](https://papers.nips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf); top-5 error is the share of images whose correct label was not among the model's five best guesses. AlexNet had 60 million parameters, 650,000 neurons and eight learned layers, five convolutional and three fully connected. It took five to six days to train on two NVIDIA GTX 580 GPUs with 3 GB of memory each, and the authors found that removing any single convolutional layer made results worse. They also wrote that the network's size was limited mainly by GPU memory and training time.

Depth kept paying. In 2015, Microsoft Research's residual networks (ResNet) made it practical to train [networks up to 152 layers deep](https://arxiv.org/abs/1512.03385), and an ensemble of them won ILSVRC 2015 with a 3.57% error.

The same pattern later appeared in language. In January 2020, OpenAI researchers reported that a language model's loss [falls as a power law with model size, dataset size and training compute](https://arxiv.org/abs/2001.08361), with some trends spanning more than seven orders of magnitude. Predictable returns from scale are one reason language models grew to the 175 billion parameters of GPT-3 that same year.

## The main deep learning architectures and what each is for

An architecture is the wiring of a network: which units connect to which, and which weights are shared. Three families cover most of deep learning.

| Architecture                  | Built for                                | How it reads the input                   | Main limit                                                    |
| ----------------------------- | ---------------------------------------- | ---------------------------------------- | ------------------------------------------------------------- |
| Convolutional network (CNN)   | Images, video, audio spectrograms        | Small shared filters slide across a grid | Sees local patterns first; needs depth for the big picture    |
| Recurrent network (RNN, LSTM) | Sequences: text, speech, sensor readings | One step at a time, carrying a state     | Forgets over long sequences, and cannot run steps in parallel |
| Transformer                   | Text first; the basis of today's LLMs    | All tokens at once, through attention    | Cost grows with the square of the input length                |

### Convolutional networks for images

The Nature review names four ideas behind convolutional networks: local connections, shared weights, pooling and many layers. Each unit looks only at a small patch of the layer below, and every unit in a feature map uses the same set of weights, called a filter. Pooling then merges nearby responses, so a pattern is still found if it shifts slightly.

A worked example: picture a 3 × 3 filter whose weights are negative in the left column and positive in the right. Slid across a photo of a circuit board, it gives a large output wherever dark pixels meet bright ones from left to right, and close to zero on flat areas. Because the same nine weights are reused at every position, the network finds that edge anywhere in the image, and it learns nine numbers instead of a separate weight for every pixel. A real layer runs dozens of such filters, and training, not a person, sets their weights. By 2015, the review described convolutional networks as the dominant approach for almost all recognition and detection tasks in images.

### Recurrent networks and their limits

A recurrent network reads a sequence one element at a time and carries a state vector that summarizes what it has seen so far. That suits language, where order matters. To classify "The backup job that finance scheduled for Friday night after the storage migration has failed", it has to carry "backup job" across ten words until "has failed" arrives.

Recurrent networks have three limits:

1. **They forget.** The review explains that backpropagated gradients grow or shrink at each time step, so over many steps they typically explode or vanish, and long-range links are learned poorly. Long short-term memory (LSTM) networks, published in 1997, add a gated memory cell that holds information for longer, and they powered speech recognition and translation systems into the mid-2010s.
2. **They are sequential.** Step 20 cannot start before step 19 finishes, a constraint that [rules out parallel work within a single training example](https://arxiv.org/abs/1706.03762) and makes long inputs slow to train.
3. **They squeeze.** A recurrent translator encoded a whole sentence into one fixed-length vector before writing the translation. In 2014, Bahdanau, Cho and Bengio identified that vector as a bottleneck and let the decoder [search back over the parts of the source sentence relevant to each word](https://arxiv.org/abs/1409.0473) it produced. That mechanism is attention.

### Transformers and attention

In June 2017, researchers at Google Brain, Google Research and the University of Toronto published "Attention Is All You Need". It proposed the transformer, an architecture built on attention alone, [with no recurrence and no convolutions at all](https://arxiv.org/abs/1706.03762). In **self-attention**, every token scores how relevant every other token in the input is to it, then builds its new representation as a weighted mix of theirs. All tokens are processed at once, so training runs in parallel on GPUs: the paper's larger model set a new best translation score (28.4 BLEU on WMT 2014 English-to-German) after 3.5 days of training on eight GPUs.

A worked example: in "The server rejected the request because it was malformed", an attention head representing "it" can put most of its weight on "request". Change the last word to "overloaded" and the weight should move to "server". No rule says so; the association is learned from text in which such sentences resolve one way or the other. A transformer stacks many such layers, and each layer has several attention heads that pick up different relationships.

![A row of seven cards stands for the tokens of a sentence. Arcs of different thickness run from one orange card to all the others, the thickest to a card near the start, above a second row of cards.](https://computese.com/images/blog/the-ai-and-machine-learning-revolution/attention.3c86d7e164-1536.webp)

*Every token weighs every other token in one step, so a transformer trains on whole passages in parallel.*

The price is in the word "every". Self-attention relates each token to all the others, so [its cost grows with the square of the number of tokens](https://developers.google.com/machine-learning/crash-course/llm/transformers) in the context. That is why long inputs are expensive and why every model has a maximum context length.

## Embeddings: how words become numbers

A network only does arithmetic, so text has to become numbers first. The naive way gives each word in the vocabulary its own position in a very long list of zeros, called one-hot encoding. It says nothing about meaning: "Tuesday" is as far from "Wednesday" as from "firewall". An **embedding** replaces that with a short vector of learned numbers per token, trained so that words used in similar ways end up close together. The Nature review gives the example of a network trained to predict the next word in news stories: its learned vectors for Tuesday and Wednesday came out very similar, as did those for Sweden and Norway.

In 2013, Google researchers published word2vec, which [learned good word vectors from 1.6 billion words of text in under a day](https://arxiv.org/abs/1301.3781). The paper cites a striking property of such vectors: vector("King") − vector("Man") + vector("Woman") lands closest to the vector for "Queen". Directions in the space carry relationships.

Word2vec gives each word one vector, whatever the sentence. That fails on a word like "cell", which means different things in a spreadsheet, a phone plan and a battery. [Contextual embeddings](https://developers.google.com/machine-learning/crash-course/embeddings/obtaining-embeddings) fix this by giving a word a different vector in each sequence, computed from the words around it, and a transformer's self-attention layers produce exactly that.

Embeddings also power semantic search, a worked example most people have already used. Turn every help article into a vector, turn the question "my laptop won't charge" into a vector the same way, and return the nearest articles. The top result can be an article titled "Battery not charging when plugged in", although the two share no keyword. The same retrieval step is how an assistant finds the passages it answers from.

![Grey dots on a flat grid form three loose clusters. An orange dot sits in one cluster inside a dashed circle with its four nearest dots, each linked to a small document, while two parallel arrows join dot pairs.](https://computese.com/images/blog/the-ai-and-machine-learning-revolution/embeddings.17bec4e298-1536.webp)

*When distance means difference in meaning, the vectors nearest a question are the passages most likely to answer it.*

## What NLP is, and how it moved from rules to large language models

Natural language processing is the part of computer science that lets software work with human language. Typical tasks are classifying a message, extracting names, dates and amounts, translating, summarizing and answering questions; speech recognition and speech synthesis sit at its edges. The field has changed its method three times, and one small task shows each change: routing the email "I was charged twice for my March invoice" to the billing team.

| Era                                        | How the system handles language                                       | The billing email                                                      | Weakness                                               |
| ------------------------------------------ | --------------------------------------------------------------------- | ---------------------------------------------------------------------- | ------------------------------------------------------ |
| Hand-written rules                         | People write dictionaries, grammars and patterns                      | A rule fires on "invoice" or "charged"                                 | Breaks on phrasing nobody anticipated; rules pile up   |
| Statistical NLP (from about 1990)          | Probabilities counted from large collections of text                  | Word and word-pair counts from past billing emails outweigh the others | Treats words as unrelated symbols                      |
| Neural NLP (from about 2013)               | Embeddings and neural networks, one model trained per task            | "charged twice" lands near "double payment" in vector space            | Needs labelled data and a trained model for every task |
| Pretrained transformers and LLMs (2018 on) | One model pretrained on vast text, adapted by fine-tuning or a prompt | The model reads an instruction and the email, and answers "billing"    | Can be fluently wrong; costs more per request          |

### From hand-written rules to statistics

Early language systems were written by hand: bilingual dictionaries, grammar rules, keyword patterns. Rules are exact and explainable, and they still suit narrow, predictable inputs, but language varies more than any team can write down.

In 1990, Brown and colleagues at IBM published [A Statistical Approach to Machine Translation](https://aclanthology.org/J90-2002/), which replaced grammar rules with probabilities learned from translated text. They note that Warren Weaver had suggested statistical methods for translation in 1949 and that researchers soon abandoned the idea, which they blamed on slow computers and too little machine-readable text. Their own data came from the Canadian Hansard, the record of Parliament that Canada keeps in both English and French: about 100 million words of English with the corresponding French. In an early test on 73 French sentences, with the English vocabulary limited to 1,000 words, 48% of the translations were judged reasonable.

Statistical language models of that period counted how often short word sequences, called n-grams, occur. As the Nature review points out, they treat each word as an atomic unit, so what is learned about "invoice" tells the model nothing about "bill".

### Neural networks

Neural NLP replaced counts with embeddings and networks: word2vec in 2013, recurrent encoder-decoder translators, then attention in 2014. In September 2016, Google described GNMT, a deep LSTM network with 8 encoder and 8 decoder layers using attention, which splits rare words into common sub-word units it calls wordpieces. In a human side-by-side evaluation on isolated simple sentences, it [cut translation errors by 60% on average](https://arxiv.org/abs/1609.08144) compared with Google's phrase-based production system. Splitting text into sub-word pieces is still how language models read; the pieces are called **tokens**.

### Pretrained transformers and large language models

From 2018 the pattern changed from one model per task to one pretrained model adapted many times. Google's BERT (October 2018) [pretrained a transformer on unlabelled text](https://arxiv.org/abs/1810.04805) and could then be fine-tuned for tasks such as question answering by adding just one output layer. OpenAI's GPT-3 (May 2020) went further: an [autoregressive language model with 175 billion parameters](https://arxiv.org/abs/2005.14165), trained on 300 billion tokens, that performed many tasks from instructions and a few examples written in the prompt, with no further training.

Bigger was not automatically more useful. OpenAI's InstructGPT work (March 2022) fine-tuned GPT-3 with human feedback, and in human evaluations people preferred [the outputs of a 1.3-billion-parameter InstructGPT to those of the 175-billion-parameter GPT-3](https://arxiv.org/abs/2203.02155). ChatGPT, [introduced on November 30, 2022](https://openai.com/index/chatgpt/), is a sibling of InstructGPT trained with the same reinforcement learning from human feedback, adapted to dialogue.

So a **large language model** is a transformer trained to predict the next token over a very large body of text, usually tuned afterwards to follow instructions.

## What large language models can and cannot do reliably

An LLM writes one token at a time, each a likely continuation of the text so far. NIST's Generative AI Profile notes that this kind of statistical prediction can produce accurate, consistent output, and also [false content stated with confidence](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf), which it calls confabulation and which most people call hallucination. The same document warns that models can confabulate the logic or citations that seem to justify an answer.

OpenAI's September 2025 research explains where the line falls. Patterns that are consistent across the training data, such as spelling and matching parentheses, stop producing errors as models grow, but [rare, arbitrary facts cannot be inferred from patterns](https://openai.com/index/why-language-models-hallucinate/), and those are where hallucinations come from. It adds that evaluations scored on accuracy alone reward guessing over admitting uncertainty.

| Task                                                     | Reliable?            | Why                                               | What to do                                              |
| -------------------------------------------------------- | -------------------- | ------------------------------------------------- | ------------------------------------------------------- |
| Summarize, rewrite or translate text you provide         | Usually              | The answer is in the input                        | Spot-check against the source                           |
| Classify a message or extract fields into a fixed format | Usually, once tested | It matches patterns in the input                  | Validate against a schema; test on real cases           |
| Draft emails, documents or code                          | As a first draft     | Common patterns are learned well                  | A person reviews before anything ships                  |
| Recall specific facts: dates, prices, figures, citations | No                   | Arbitrary facts cannot be predicted from patterns | Supply the source documents and require citations       |
| Know about events after its training data ends           | No                   | Nothing newer is in its weights                   | Retrieve current data when the question is asked        |
| Find one detail in a very long input                     | Varies               | Accuracy drops when the detail sits in the middle | Put key material first or last; retrieve short passages |
| Exact arithmetic and long chains of logic                | Not guaranteed       | Errors compound over many generated steps         | Let code do the calculation                             |
| Give the same answer twice                               | Not by default       | Sampling chooses among likely tokens              | Fix the settings and test repeatedly                    |

The long-input row comes from a 2023 study which found that models often do best when the relevant information is [at the beginning or end of the input, and significantly worse when it sits in the middle](https://arxiv.org/abs/2307.03172) of a long context.

A worked example makes the difference concrete. Paste a 20-page supplier contract into a model and ask for its termination terms: the answer is in the input, and you can check it in minutes. Ask the same model, with no documents, which clause of a regulation applies to your business: it may return a fluent answer with a clause number that does not exist.

> [!WARNING]
> Do not use a language model as the only source of a fact that matters. Give it the documents, make it cite them, and have a person check anything that will be acted on.

For more on where model understanding breaks down, see [AI limitations in understanding](https://computese.com/ai-limitations-in-understanding/); for multi-step logic and reasoning models, see [how AI reasons, and where it fails](https://computese.com/exploring-the-new-frontier-in-ai-logical/). To try the ideas yourself, [a first AI project in Python](https://computese.com/artificial-intelligence-with-python/) trains a small model step by step, and our [AI frameworks comparison](https://computese.com/latest-ai-tools-and-frameworks-a-comparative-analysis/) covers PyTorch, TensorFlow and JAX.

If you are deciding where a language model belongs in your own work, our [AI & automation service](https://computese.com/services/ai-automation/) uses a model only where a step needs judgment, such as reading a document or drafting a reply, and ordinary code everywhere else; nothing changes in your systems until a rule or a person approves it. The assistants it builds answer from your own pages, documents and live data, cite their sources and say when they do not know.

## Key terms
- **Neural network**: A function built from layers of simple units. Each unit multiplies its inputs by weights, adds a bias and applies a simple non-linear function; the outputs of one layer are the inputs of the next.
- **Weights (parameters)**: The adjustable numbers inside a network, set by training rather than written by a programmer. AlexNet had 60 million parameters; GPT-3 had 175 billion.
- **Backpropagation**: The procedure that works out how much each weight contributed to the error, from the output layer back towards the input, so gradient descent can adjust every weight to reduce it.
- **Deep learning**: Machine learning with neural networks of many layers, which learn their own features from raw data such as pixels, sound or text instead of relying on features people design.
- **Convolutional neural network (CNN)**: A network that slides small filters with shared weights across a grid of values, which suits images, video and audio spectrograms.
- **Recurrent neural network (RNN)**: A network that reads a sequence one step at a time and carries a state vector forward. LSTM networks add a gated memory cell so information survives over more steps.
- **Transformer and self-attention**: The architecture introduced in 2017 that processes all tokens at once. In self-attention, each token scores how relevant every other token is and mixes in their information accordingly.
- **Embedding**: A vector of learned numbers that represents a token, a document or an image, placed so that items with similar meanings sit close together.
- **Token**: The unit a language model reads and writes: a word, part of a word or a punctuation mark, each mapped to an ID number.
- **Large language model (LLM)**: A transformer trained to predict the next token over a very large body of text, usually tuned afterwards to follow instructions. GPT-3 and the models behind ChatGPT are examples.

## Common questions

### What is the difference between machine learning and deep learning?

Deep learning is a subset of machine learning. Classic machine learning usually works on features people choose, such as the columns of a table, and often wins on ordinary business data. Deep learning uses neural networks of many layers that learn their own features from raw images, sound or text, at the cost of more data and computing power.

### What is NLP in simple terms?

Natural language processing is the part of computer science that lets software work with human language: route an email, pull a name and an amount out of an invoice, translate, summarize or answer a question. Current systems, from BERT to chat assistants, are transformer-based neural networks.

### Is ChatGPT deep learning or NLP?

Both. ChatGPT is an NLP application built on a large language model, which is a deep learning model: a transformer trained to predict the next token, then fine-tuned with human feedback so that it follows instructions in a dialogue.

### How does a neural network learn?

By adjusting its weights. It makes a prediction on a training example, a loss function measures the error, backpropagation works out how each weight contributed, and gradient descent nudges every weight to reduce the error. Repeated over many examples, the weights settle on values that also work for inputs the network has not seen.

### Why did deep learning take off in 2012?

Large labelled datasets, programmable GPUs and better training techniques arrived together. AlexNet, trained on 1.2 million ImageNet images with two GPUs, won ILSVRC-2012 with a 15.3% top-5 error against 26.2% for the next entry, and by 2015 deep convolutional networks were the dominant approach to recognition tasks in images.

### Why do large language models make things up?

Because they generate likely text, not checked facts. Consistent patterns such as spelling are learned well, but arbitrary facts such as dates, figures and citations cannot be predicted from patterns, and evaluations that reward guessing make it worse. Give the model the source documents and check anything that will be acted on.

## Sources
1. [Learning representations by back-propagating errors](https://www.nature.com/articles/323533a0), Nature (Rumelhart, Hinton and Williams, 1986)
2. [Neural networks: Training using backpropagation](https://developers.google.com/machine-learning/crash-course/neural-networks/backpropagation), Google Machine Learning Crash Course
3. [Deep learning (Nature 521, 436 to 444, 2015), authors' copy](https://www.cs.toronto.edu/~hinton/absps/NatureDeepReview.pdf), Yann LeCun, Yoshua Bengio and Geoffrey Hinton (University of Toronto)
4. [What is machine learning?](https://developers.google.com/machine-learning/intro-to-ml/what-is-ml), Google for Developers
5. [Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1)](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf), NIST
6. [Why do tree-based models still outperform deep learning on tabular data?](https://arxiv.org/abs/2207.08815), arXiv (Grinsztajn, Oyallon and Varoquaux, 2022)
7. [ImageNet Classification with Deep Convolutional Neural Networks](https://papers.nips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf), NeurIPS 2012 (Krizhevsky, Sutskever and Hinton)
8. [Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385), arXiv (He et al., Microsoft Research, 2015)
9. [Scaling Laws for Neural Language Models](https://arxiv.org/abs/2001.08361), arXiv (Kaplan et al., OpenAI, 2020)
10. [Attention Is All You Need](https://arxiv.org/abs/1706.03762), arXiv (Vaswani et al., NIPS 2017)
11. [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473), arXiv (Bahdanau, Cho and Bengio, 2014)
12. [LLMs: What's a large language model?](https://developers.google.com/machine-learning/crash-course/llm/transformers), Google Machine Learning Crash Course
13. [Efficient Estimation of Word Representations in Vector Space (word2vec)](https://arxiv.org/abs/1301.3781), arXiv (Mikolov et al., Google, 2013)
14. [Embeddings: Obtaining embeddings](https://developers.google.com/machine-learning/crash-course/embeddings/obtaining-embeddings), Google Machine Learning Crash Course
15. [A Statistical Approach to Machine Translation](https://aclanthology.org/J90-2002/), Computational Linguistics, ACL Anthology (Brown et al., IBM, 1990)
16. [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://arxiv.org/abs/1609.08144), arXiv (Wu et al., Google, 2016)
17. [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805), arXiv (Devlin et al., Google, 2018)
18. [Language Models are Few-Shot Learners (GPT-3)](https://arxiv.org/abs/2005.14165), arXiv (Brown et al., OpenAI, 2020)
19. [Training language models to follow instructions with human feedback (InstructGPT)](https://arxiv.org/abs/2203.02155), arXiv (Ouyang et al., OpenAI, 2022)
20. [Introducing ChatGPT](https://openai.com/index/chatgpt/), OpenAI
21. [Why language models hallucinate](https://openai.com/index/why-language-models-hallucinate/), OpenAI
22. [Lost in the Middle: How Language Models Use Long Contexts](https://arxiv.org/abs/2307.03172), arXiv (Liu et al., 2023)