# AI continuous learning: why neural networks forget and stop learning

> AI continuous learning means a model keeps learning after deployment. Deep networks forget and lose plasticity, so teams retrain, fine-tune or use retrieval.

- URL: https://computese.com/breaking-new-ground-in-ai-continuous-learning-system/
- Author: Duong Quan Nguyen, CEO, Computese
- Published: 2024-08-30
- Updated: 2026-09-25
- Topics: AI & automation

## In short
- Continual (continuous, lifelong) learning means a model keeps learning from new data after deployment without losing what it knows. Standard deep learning is trained once and frozen, and substantial new data usually means retraining from scratch.
- Two failures stand in the way: catastrophic forgetting, where new training overwrites old skills, and loss of plasticity, where a network gradually loses the ability to learn at all.
- University of Alberta researchers showed loss of plasticity in Nature on August 21, 2024, and kept networks learning with continual backpropagation, which reinitializes a tiny fraction of the least-used units.
- Research methods against forgetting fall into three families: replay of old data, regularization such as EWC that protects important weights, and architectures that give each task its own capacity.
- Production systems still keep models current with monitoring, scheduled or triggered retraining, fine-tuning and retrieval (RAG). Treat every deployed model as a dated snapshot.

AI continuous learning, usually called continual or lifelong learning, is a model's ability to keep learning from new data after deployment without losing what it already knows. Standard deep learning does not do this well: networks are trained once and frozen, because further training tends to overwrite old skills and, over long runs, erodes the ability to learn.

That second failure is the subject of a study University of Alberta researchers published in Nature on August 21, 2024: standard deep learning methods gradually lose plasticity, their ability to learn, and a modified algorithm called continual backpropagation keeps it. This explainer covers what the study found, how it differs from the older problem of catastrophic forgetting, the main method families, how production systems keep models current today, and what it means if your business runs models.

## What continual learning is, and why models are frozen

Almost every deep learning system is built in two phases: training adjusts the network's weights on a fixed dataset, then the weights are held constant while the network is used. The authors of the [Nature paper](https://www.nature.com/articles/s41586-024-07711-7) point out that large language models follow the same pattern: pre-trained on a large generic dataset, fine-tuned on smaller ones for an application or for safety, then frozen before release.

Continual learning removes the line between the two phases. The model keeps updating from a stream of data whose distribution changes: new products, new fraud patterns, new vocabulary, a robot whose joints wear. Research papers call it continual or lifelong learning, or incremental learning when classes or tasks arrive in batches; product pages often say continuous learning.

Simply continuing to train on new data usually does not work with current methods, according to the same paper: the effect of the new data is either too large or too small, and not properly balanced with the old. So the common way to add substantial new data is to discard the network and train a new one from scratch on old and new data together. For a large language model, the authors note, each such retraining can cost millions of dollars in computation.

Researchers split the problem into three settings, following a [2022 paper in Nature Machine Intelligence](https://www.nature.com/articles/s42256-022-00568-3) by Gido van de Ven, Tinne Tuytelaars and Andreas Tolias:

| Setting            | What the model must do                                                         | A business example                                        |
| ------------------ | ------------------------------------------------------------------------------ | --------------------------------------------------------- |
| Task-incremental   | Learn a set of clearly distinct tasks, and it is told which task it faces      | One model that sorts tickets and, separately, tags photos |
| Domain-incremental | Solve the same kind of problem as the context changes                          | A defect detector moved to a line with different lighting |
| Class-incremental  | Tell apart a growing number of classes without being told which batch is which | A product classifier that gains new product lines         |

Every method balances two needs that pull against each other: stability keeps what was learned, plasticity lets the network learn what is new. A [2023 survey by Liyuan Wang and colleagues](https://arxiv.org/abs/2302.00487) sums up the goal of continual learning as a proper stability-plasticity trade-off. Catastrophic forgetting is a failure of stability; loss of plasticity is a failure of the other half.

## Catastrophic forgetting: the classic problem

Catastrophic forgetting is the older and better known failure. Michael McCloskey and Neal Cohen described it in 1989 as catastrophic interference, in a chapter subtitled "the sequential learning problem". A network trained on task A and then on task B loses task A, often abruptly, because the weights task A depended on are changed to serve task B.

The cause is that a neural network does not store each skill in its own place: the same weights serve many tasks. When the data for every task is available together, training finds weights that work for all of them, which is why ordinary training on a shuffled dataset does not forget. The [2017 PNAS paper on elastic weight consolidation](https://pmc.ncbi.nlm.nih.gov/articles/PMC5380101/) makes exactly this point about interleaving tasks. The trouble starts when tasks arrive in sequence and old data is not shown again, which is the situation of a deployed model learning from whatever arrives next.

Large language models are not immune. A [2023 study of continual instruction tuning](https://arxiv.org/abs/2308.08747) found catastrophic forgetting in models from 1 billion to 7 billion parameters, and within that range the forgetting grew more severe as the models got larger.

## Loss of plasticity: what the University of Alberta study found

"Loss of plasticity in deep continual learning" appeared in Nature (volume 632, pages 768 to 774) on August 21, 2024. Its authors are Shibhansh Dohare, J. Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A. Rupam Mahmood and Richard S. Sutton of the University of Alberta's Department of Computing Science; Mahmood and Sutton also hold Canada CIFAR AI Chairs at Amii, the Alberta Machine Intelligence Institute. Sutton shared the [2024 ACM A.M. Turing Award](https://awards.acm.org/about/2024-turing) with Andrew Barto for the foundations of reinforcement learning.

The paper's claim is blunt: in continual learning, standard deep learning methods gradually lose plasticity until they learn no better than a shallow network. This is a different failure from forgetting. Forgetting is doing badly on old examples that are no longer shown; loss of plasticity is doing badly on new ones, even when nothing old is being tested.

Showing it took three to four orders of magnitude more computation than training one network, by the paper's estimate, across three test beds:

| Test                        | Set-up                                                                                              | What happened with standard backpropagation                                                                      |
| --------------------------- | --------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| Continual ImageNet          | Thousands of two-class tasks built from pairs of ImageNet classes, such as cats versus houses       | Up to 88% correct on early tasks; by task 2,000, a large loss at every step size, some below a linear network    |
| Class-incremental CIFAR-100 | An 18-layer residual network adding 5 classes at a time up to 100, while still training on old ones | Better than retraining from scratch at first, worse after 40 classes, 5% less accurate at 100                    |
| Ant locomotion              | PPO, a standard reinforcement-learning algorithm, driving a simulated ant-like robot                | Worse after each friction change (every 2 million steps); with friction constant, collapse after about 3 million |

Looking inside the networks showed why. Many units went dormant (the paper counts units active less than 1% of the time), the rest became similar to each other, which the authors measure as a falling stable rank, and in the reinforcement-learning runs the weights grew steadily larger. A network in that state has lost the variety of features that let it learn in the first place; after 20 million steps with constant friction, the standard agent failed every episode.

Two findings matter for anyone training models. First, some everyday tools made it worse: in the paper's tests, the Adam optimizer, dropout and normalization all increased loss of plasticity. Second, duration matters. Plasticity loss is often severe when learning continues for many tasks and may not occur at all over a few, so a model fine-tuned a handful of times is far from the regime the paper studied, while a system meant to learn for years is squarely in it.

## How continual backpropagation keeps a network learning

Ordinary backpropagation has two parts: random initialization, which gives a new network its variety, and gradient descent on every step afterwards. Continual backpropagation keeps doing a little of the first part forever. As training runs, it scores every hidden unit by its contribution to the units it feeds (a running average of the unit's activation times the size of its outgoing weights), and from time to time it reinitializes a small fraction of the lowest-scoring units.

Three details keep this from damaging what the network knows:

- **Only mature units are candidates.** A freshly reset unit is protected for a set number of updates, so it has time to become useful.
- **A reset unit's outgoing weights start at zero.** It changes nothing about the network's current output until it has learned something.
- **The replacement rate is tiny.** In the CIFAR-100 test it was 10⁻⁵ per step, which in the 512-unit last layer means roughly one unit every 200 updates.

![A layered neural network in which a few pale hidden units have gone quiet. One orange unit has just been reset: fresh incoming links, and dashed outgoing links that carry nothing yet.](https://computese.com/images/blog/breaking-new-ground-in-ai-continuous-learning-system/reinit.93c1333a6a-1536.webp)

*Fresh units restore the variety a network had at initialization, without touching the units it relies on.*

With it, the networks kept learning. It outperformed the existing methods tested on Continual ImageNet, fully overcame loss of plasticity on class-incremental CIFAR-100 with almost no dead units, and, with a little L2 regularization added, kept PPO improving on the ant task. Simpler methods help too: L2 regularization, which pulls weights toward zero, and Shrink and Perturb, which adds small random changes on top, both reduced the loss. The authors conclude that gradient descent alone is not enough, and that sustained deep learning needs a random, non-gradient source of variability.

The university's [own write-up](https://www.ualberta.ca/en/folio/2024/10/deep-learning-methods-lose-their-ability-to-learn-over-time.html) (Folio, October 29, 2024) describes the method as ranking neurons by usefulness and resetting the least useful to restore the original level of plasticity; Mahmood compares it to neurogenesis in human and animal brains. It is not the only variant: a [2023 ICML paper on the dormant neuron phenomenon](https://arxiv.org/abs/2302.12902) proposed ReDo, which recycles dormant neurons during reinforcement-learning training, and the Nature paper expects the idea to need further development. The authors' code is public.

> [!NOTE]
> The experiments used image classifiers and a reinforcement-learning agent, not large language models. The authors write that a systematic study with a large language model was not possible, because a single training run costs millions of dollars. Whether continual backpropagation works at that scale is still open.

## Other approaches: replay, regularization and modular networks

The Wang survey groups continual learning methods into several families. Three cover most of what a practitioner will meet, and all three were designed against forgetting.

### Replay: keep training on some of the past

Replay mixes stored examples of old data into new training, so the network is again optimized for old and new data together, which removes the cause of forgetting. [Rolnick and colleagues (NeurIPS 2019)](https://arxiv.org/abs/1811.11682) gave a reinforcement-learning agent a replay buffer of past experience and substantially reduced forgetting on Atari and DMLab tasks; with limited storage, randomly discarding data let a small buffer perform almost as well as an unbounded one.

![New data cards travel along a conveyor into a neural network. Cards drop into an orange box below, and orange arrows lift two older cards back onto the conveyor between the new ones.](https://computese.com/images/blog/breaking-new-ground-in-ai-continuous-learning-system/replay.999d476652-1536.webp)

*Replay is the most direct defence against forgetting, provided you are allowed to keep the old data.*

When old data cannot be kept, [deep generative replay](https://arxiv.org/abs/1705.08690) (Shin and colleagues, NIPS 2017) trains a generator to produce stand-in samples of earlier tasks. In the van de Ven comparison, replay-based methods were among the top performers in all three settings. For another use of generated training data, see [how synthetic data trains robots](https://computese.com/ai-generated-data-for/).

### Regularization: protect the weights that matter

Elastic weight consolidation (EWC), published in PNAS in March 2017 by researchers at DeepMind and Imperial College London, estimates how important each weight was to the old task from the diagonal of the Fisher information matrix, and adds a quadratic penalty for moving important weights away from their old values. The paper pictures the penalty as a spring anchoring each weight, stiffer for the weights that matter most. It was demonstrated on a sequence of shuffled handwritten-digit tasks and on Atari 2600 games learned one after another. [Synaptic intelligence](https://arxiv.org/abs/1703.04200) (ICML 2017) works similarly, with each weight accumulating a measure of its relevance as training goes.

The weakness shows in the hardest setting. In van de Ven's comparison, EWC and synaptic intelligence did well when told which task they faced, but in class-incremental learning they scored about as low as plain sequential training, and the authors conclude that there they failed completely. They also trade plasticity for stability by design: EWC works by slowing learning on the weights it protects.

### Architecture: give new tasks their own capacity

[Progressive neural networks](https://arxiv.org/abs/1606.04671) (DeepMind, 2016) add a new column of layers for each task, freeze the old columns and connect them sideways so old features can be reused. Nothing old is overwritten, so they do not forget, but the network grows with every task, and picking the right column at inference needs the task label. [PackNet](https://arxiv.org/abs/1711.05769) (2017) prunes a trained network to free up weights and trains the next task on the freed ones only; its authors added three fine-grained classification tasks to one ImageNet-trained VGG-16 network with accuracy close to separately trained networks.

| Family         | Examples                             | How it limits forgetting                      | Main cost                                                   |
| -------------- | ------------------------------------ | --------------------------------------------- | ----------------------------------------------------------- |
| Replay         | Experience replay, generative replay | Trains on old and new data together           | Storing old data (and the right to keep it), or a generator |
| Regularization | EWC, synaptic intelligence           | Penalizes changes to weights old tasks needed | Weak when the model must tell old and new classes apart     |
| Architecture   | Progressive networks, PackNet        | Gives each task its own weights               | Growth or a fixed capacity; often needs the task label      |

## How production systems keep models current today

The Nature paper notes that almost every application trains a deep network in a separate phase and switches learning off once the network is in use. Production teams keep models current with four tools instead, often more than one at a time:

| Approach                            | What changes                                    | Suits                                                        | Watch for                                    |
| ----------------------------------- | ----------------------------------------------- | ------------------------------------------------------------ | -------------------------------------------- |
| Retraining on a schedule or trigger | A new model trained on recent and old data      | Predictive models whose patterns shift: demand, fraud, churn | Cost per run; a validation gate              |
| Continued pre-training              | A large model's weights, with old data replayed | Teams that own a large model and a lot of new text           | Forgetting if learning rate or replay is off |
| Fine-tuning, full or LoRA           | Some or all weights, on a small task dataset    | New behaviour: a format, a tone, a narrow task               | Lost performance outside the target domain   |
| Retrieval (RAG)                     | Only the documents the model can look up        | Facts that change: prices, policies, product data            | Answers are only as current as the index     |

### Retrain on a schedule, or when monitoring says so

Google Cloud's [MLOps guidance](https://docs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning) calls this continuous training: an automated pipeline retrains the model and deploys it. Its triggers are on demand; on a schedule, when new labelled data arrives daily, weekly or monthly; when new training data becomes available; when model performance degrades; and when the input data distribution changes significantly, which the guide calls concept drift. The pipeline validates the data before training and promotes a new model only if it performs better than the current one.

Cloud platforms ship the monitoring half. [Azure Machine Learning's model monitoring](https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring), for example, compares each feature's distribution in production with the training data and alerts when a test statistic or distance score crosses a threshold. Its built-in signals cover data drift, prediction drift, data quality, feature attribution drift and model performance, and Microsoft documents the follow-through: when a classifier's accuracy drops below a threshold, an Azure Event Grid event can start a retraining job on the ground-truth data collected.

Retraining from scratch sidesteps both failures, because each run starts from a fresh network trained on old and new data together. That is why it remains the default. Its price is compute and a pipeline to maintain.

### Continued pre-training with replay

For large language models, retraining from scratch is the expensive option. A [2024 study of continual pre-training](https://arxiv.org/abs/2403.08763) found a cheaper recipe: warm the learning rate up again, decay it again, and replay some of the previous data. That combination matched full retraining on all the data, by final loss and benchmark averages, at 405 million parameters and, for the milder data shift, at 10 billion, using a fraction of the compute.

### Fine-tuning, and why LoRA forgets less

Fine-tuning adapts a trained model to a narrower task with a small dataset. Full fine-tuning updates every weight. [LoRA](https://arxiv.org/abs/2106.09685) (low-rank adaptation, 2021) freezes the pre-trained weights and trains small low-rank matrices added to each Transformer layer; for GPT-3 175B, its authors report 10,000 times fewer trainable parameters than full fine-tuning. A [2024 comparison on code and mathematics](https://arxiv.org/abs/2405.09673) put the trade-off in its title, LoRA learns less and forgets less: it underperformed full fine-tuning on the target domain but kept more of the base model's performance outside it.

### Retrieval: update what the model can look up

Retrieval-augmented generation (RAG) leaves the model's weights alone. A retriever finds passages in an index, and the model answers from them. The [original RAG paper](https://arxiv.org/abs/2005.11401) (Lewis and colleagues, NeurIPS 2020) showed why this suits changing facts. The authors swapped a December 2016 Wikipedia index for a December 2018 one and asked about 82 world leaders who had changed between the two dates. With the index matching the year, RAG answered 70% (2016) and 68% (2018) correctly; with the indexes mismatched, 12% and 4%. The model itself was not retrained.

![A padlocked model box takes in a question and an orange stack of pages from a newly connected server, while an older server stands unplugged beside it. An answer leaves on the right.](https://computese.com/images/blog/breaking-new-ground-in-ai-continuous-learning-system/index.3fd51ce7e7-1536.webp)

*Swapping the index changes what a frozen model can answer, with no training run at all.*

Retrieval does not teach a model new skills, and it is only as good as its index and retriever. For facts that change weekly (prices, stock, policies, rules), though, it is usually the cheapest way to keep answers current, and the easiest to audit, because each answer can cite the passage it came from.

## Where continual learning research is heading

The problem is not closed. In November 2025, Google Research introduced [Nested Learning](https://research.google/blog/introducing-nested-learning-a-new-ml-paradigm-for-continual-learning/), from a NeurIPS 2025 paper, which treats a model as a set of nested optimization problems that update at different rates, aiming to reduce catastrophic forgetting; its proof-of-concept architecture is called Hope. The announcement names continual learning as one of the fundamental challenges that persist for large language models, whose knowledge is limited to their input window and what they learned in pre-training.

Mahmood describes the Alberta group's work in similar terms, as a testing ground for deep learning systems deployed in the real world that keep learning. Until such systems are routine, the practical question for most organizations is not how to make a model learn continually, but how to keep a frozen one from going stale. For the wider research picture, see [the future of AI in computer science](https://computese.com/the-future-of-ai-what-lies-ahead-in-computer-science/); for a related line of work, where models improve by reasoning over what they already know, see [AI that learns by thinking](https://computese.com/ai-that-learns-by-thinking/).

## What continual learning means for a business running models

If you run models, or buy products built on them, the research translates into a few working rules:

1. **Treat every model as a dated snapshot.** A deployed network does not learn from use. Hosted language models are versioned: Anthropic's [model overview](https://platform.claude.com/docs/en/models/overview), for example, lists a training data cutoff and a reliable knowledge cutoff for each model, and a date before which it will not be retired. Record which version you use and when it will change; our [Google Gemini explainer](https://computese.com/google-unveils-gemini-the-most-advanced-and-versatile-ai-model-yet/) shows how quickly one model family moves between versions.
2. **Match the fix to what goes stale.** Changing facts belong in retrieval. A new format, tone or narrow task is a fine-tuning job. Shifting patterns in your own data, such as demand, fraud or churn, call for retraining a predictive model.
3. **Monitor before you retrain.** Track input drift and, where you can collect ground truth, live accuracy. Set the retraining schedule by how fast your data changes and what a run costs, as Google's guide advises.
4. **Keep the old data, and the right to use it.** Retraining from scratch and replay both need past data. Retention rules and consent decide whether you may keep it, so check them before you design the pipeline.
5. **Gate every new model on a fixed evaluation set.** Promote a retrained or fine-tuned model only if it beats the current one on a test set that includes the old cases. That is where forgetting shows up before customers see it.
6. **Ask what "self-learning" means.** When a vendor says its model learns continuously, ask what exactly updates, how often, from which data, and how each update is tested before it goes live.

Our [AI and automation service](https://computese.com/services/ai-automation/) builds assistants that answer from your own documents through retrieval, and runs an evaluation set built from your real cases before any model, prompt or source change goes live. When the problem is the data feeding a model, our [data platform](https://computese.com/services/data-platform/) work builds pipelines with tests, freshness checks and data contracts, and serves features and datasets for machine learning. More on applied AI is on the [AI topic page](https://computese.com/category/ai/).

## Key terms
- **Continual learning**: Learning from a stream of data whose distribution changes over time, without losing earlier knowledge. Also called continuous, lifelong or incremental learning.
- **Catastrophic forgetting**: The abrupt loss of performance on earlier tasks when a network is trained on new ones, because weights the old tasks relied on are changed. Described as catastrophic interference in 1989.
- **Loss of plasticity**: The gradual decline of a network's ability to learn anything new during long continual training. Plasticity is that ability; the 2024 Nature paper showed standard deep learning methods lose it.
- **Stability-plasticity trade-off**: The tension between keeping what a model has learned (stability) and learning what is new (plasticity). Every continual learning method has to balance the two.
- **Continual backpropagation**: A variant of backpropagation from the 2024 Nature paper that keeps reinitializing a small fraction of the least-used hidden units, so the network keeps its variety and its ability to learn.
- **Elastic weight consolidation (EWC)**: A 2017 method that estimates how important each weight was to earlier tasks and penalizes moving the important ones, slowing learning exactly where it would cause forgetting.
- **Experience replay**: Mixing stored examples of past data into new training, so the network is optimized for old and new data together. Generative replay uses a trained generator instead of stored data.
- **Concept drift**: A change in the data a model meets in production, or in how that data relates to the outcome, that makes the model stale. Monitoring tools detect it by comparing production data with training data.
- **Retrieval-augmented generation (RAG)**: A design in which a retriever finds relevant passages in an index and the model answers from them, so knowledge can be updated by changing the index instead of the weights.
- **LoRA (low-rank adaptation)**: A fine-tuning method that freezes a model's pre-trained weights and trains small low-rank matrices added to each layer, updating far fewer parameters than full fine-tuning.

## Common questions

### What is continuous learning in AI?

Continuous learning, more often called continual or lifelong learning, is a model's ability to keep learning from new data after deployment without forgetting what it learned before. Most deep learning systems do not do it: they are trained, frozen and then retrained once enough new data has built up.

### What is the difference between catastrophic forgetting and loss of plasticity?

Catastrophic forgetting means a network does worse on old tasks after learning new ones, because shared weights are overwritten. Loss of plasticity means it gradually gets worse at learning anything new, even when old tasks are not being tested. A continual learner has to avoid both.

### What did the University of Alberta researchers discover?

In a Nature paper published on August 21, 2024, Shibhansh Dohare, Richard Sutton and colleagues showed that standard deep learning methods gradually lose plasticity when training continues over many tasks, until they learn no better than a shallow network. Their continual backpropagation algorithm, which reinitializes a tiny fraction of the least-used units, kept networks learning in their tests.

### Can a deployed AI model learn from new data on its own?

A standard deployed model does not: its weights stay fixed while it is used, and it changes only when someone retrains or fine-tunes it and deploys the new version. A product can also look up fresh information through retrieval, which changes what the model can read, not what it has learned.

### How often should a machine learning model be retrained?

It depends on how fast your data changes and what a training run costs. Google's MLOps guidance lists the usual triggers: on demand, on a schedule (daily, weekly or monthly), when new training data arrives, when performance degrades and when the input data distribution shifts. Monitoring tells you which one applies to you.

### Is RAG a form of continual learning?

Not in the research sense, because the model's weights do not change. It is the practical way to keep a model's facts current: in the original RAG paper, swapping a 2016 Wikipedia index for a 2018 one updated the model's answers about world leaders without any retraining.

## Sources
1. [Loss of plasticity in deep continual learning](https://www.nature.com/articles/s41586-024-07711-7), Nature
2. [Deep learning methods lose their ability to learn over time](https://www.ualberta.ca/en/folio/2024/10/deep-learning-methods-lose-their-ability-to-learn-over-time.html), University of Alberta (Folio)
3. [Three types of incremental learning](https://www.nature.com/articles/s42256-022-00568-3), Nature Machine Intelligence
4. [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://arxiv.org/abs/2302.00487), arXiv (IEEE TPAMI)
5. [Overcoming catastrophic forgetting in neural networks](https://pmc.ncbi.nlm.nih.gov/articles/PMC5380101/), PNAS (PubMed Central)
6. [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://arxiv.org/abs/2308.08747), arXiv
7. [ACM A.M. Turing Award Honors Two Researchers Who Led the Development of Cornerstone AI Technology](https://awards.acm.org/about/2024-turing), ACM
8. [The Dormant Neuron Phenomenon in Deep Reinforcement Learning](https://arxiv.org/abs/2302.12902), arXiv (ICML 2023)
9. [Experience Replay for Continual Learning](https://arxiv.org/abs/1811.11682), arXiv (NeurIPS 2019)
10. [Continual Learning with Deep Generative Replay](https://arxiv.org/abs/1705.08690), arXiv (NIPS 2017)
11. [Continual Learning Through Synaptic Intelligence](https://arxiv.org/abs/1703.04200), arXiv (ICML 2017)
12. [Progressive Neural Networks](https://arxiv.org/abs/1606.04671), arXiv (DeepMind)
13. [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://arxiv.org/abs/1711.05769), arXiv
14. [MLOps: Continuous delivery and automation pipelines in machine learning](https://docs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning), Google Cloud
15. [Model monitoring in production](https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring), Microsoft Learn
16. [Simple and Scalable Strategies to Continually Pre-train Large Language Models](https://arxiv.org/abs/2403.08763), arXiv
17. [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685), arXiv
18. [LoRA Learns Less and Forgets Less](https://arxiv.org/abs/2405.09673), arXiv (TMLR 2024)
19. [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401), arXiv (NeurIPS 2020)
20. [Introducing Nested Learning: A new ML paradigm for continual learning](https://research.google/blog/introducing-nested-learning-a-new-ml-paradigm-for-continual-learning/), Google Research
21. [Models overview](https://platform.claude.com/docs/en/models/overview), Anthropic