Welcome to the "Introduction" tutorial of the "From Zero to Hero" series. We will start our journey by taking a quick look at the Avalanche main modules to understand its general architecture.

As hinted in the getting started introduction Avalanche is organized in five main modules:

• Benchmarks: This module maintains a uniform API for data handling: mostly generating a stream of data from one or more datasets. It contains all the major CL benchmarks (similar to what has been done for torchvision).

• Training: This module provides all the necessary utilities concerning model training. This includes simple and efficient ways of implement new continual learning strategies as well as a set pre-implemented CL baselines and state-of-the-art algorithms you will be able to use for comparison!

• Evaluation: This module provides all the utilities and metrics that can help evaluate a CL algorithm with respect to all the factors we believe to be important for a continually learning system. It also includes advanced logging and plotting features, including native Tensorboard support.

• Models: In this module you'll find several model architectures and pre-trained models that can be used for your continual learning experiment (similar to what has been done in torchvision.models). Furthermore, we provide everything you need to implement architectural strategies, task-aware models, and dynamic model expansion.

• Logging: It includes advanced logging and plotting features, including native stdout, file and Tensorboard support (How cool it is to have a complete, interactive dashboard, tracking your experiment metrics in real-time with a single line of code?)

Avalanche
├── Benchmarks
│   ├── Classic
│   ├── Datasets
│   ├── Generators
│   ├── Scenarios
│   └── Utils
├── Evaluation
│   ├── Metrics
│   ├── Tensorboard
|   └── Utils
├── Training
│   ├── Strategies
│   ├── Plugins
|   └── Utils
├── Models
└── Loggers

In this series of tutorials, you'll get the chance to learn in-depth all the features offered by each module and sub-module of Avalanche, how to put them together and how to master Avalanche, for a stress-free continual learning prototyping experience!

🤝 Run it on Google Colab

Avalanche is an End-to-End Continual Learning Library based on , born within with the goal of providing a shared and collaborative open-source (MIT licensed) codebase for fast prototyping, training and of continual learning algorithms.

Avalanche can help Continual Learning researchers and practitioners in several ways:

• Write less code, prototype faster & reduce errors

• Improve reproducibility, modularity and reusability

• Increase code efficiency, scalability & portability

• Augment impact and usability of your research products

The library is organized in five main modules:

• Training: This module provides all the necessary utilities concerning model training. This includes simple and efficient ways of implement new continual learning strategies as well as a set pre-implemented CL baselines and state-of-the-art algorithms you will be able to use for comparison!

• Evaluation: This modules provides all the utilities and metrics that can help evaluate a CL algorithm with respect to all the factors we believe to be important for a continually learning system.

Let's make it together 👫 a wonderful ride! 🎈

Check out how your code changes when you start using Avalanche! 👇

🚦 Getting Started

We know that learning a new tool may be tough at first. This is why we made Avalanche as easy as possible to learn with a set of resources that will help you along the way.

For example, you may start with our 5-minutes guide that will let you acquire the basics about Avalanche and how you can use it in your research project:

We have also prepared for you a large set of examples & snippets you can plug-in directly into your code and play with:

Having completed these two sections, you will already feel with superpowers ⚡, this is why we have also created an in-depth tutorial that will cover all the aspect of Avalanche in details and make you a true Continual Learner! 👨‍🎓️

📑 Cite Avalanche

🗂️ Maintained by ContinualAI Lab

Avalanche is a framework in constant development. Thanks to the support of the community and its active members we plan to extend its features and improve its usability based on the demands of our research community! A the moment, Avalanche is in Beta (v0.3.1). We support a large number of Benchmarks, Strategies and Metrics, that makes it, we believe, the best tool out there for your continual learning research! 💪

Benchmarks and Datasets

You find a complete list of the features on the .

🖼️ Datasets

Avalanche supports all the most popular computer vision datasets used in Continual Learning. Some of them are available in , while other have been integrated by us. Most datasets can be automatically downloaded by Avalanche.

• Toy datasets: MNIST, Fashion MNIST, KMNIST, EMNIST, QMNIST.

• CIFAR: CIFAR10, CIFAR100.

• ImageNet: TinyImagenet, MiniImagenet, Imagenet.

• Others: EndlessCLDataset, CUB200, OpenLORIS, Stream-51, INATURALIST2018, Omniglot, CLEARImage, ...

📚 Benchmarks

All the major continual learning benchmarks are available and ready to use. Benchmarks split the datasets and create the train and test streams:

• MNIST: SplitMNIST, RotatedMNIST, PermutedMNIST, SplitFashionMNIST.

• CIFAR10: SplitCIFAR10, SplitCIFAR100, SplitCIFAR110.

• CORe50: all the CORe50 scenarios are supported.

• Others: SplitCUB200, CLStream51, CLEAR.

📈 Continual Learning Strategies

• Baselines: Naive, JointTraining, Cumulative.

• Rehearsal: Replay with reservoir sampling and balanced buffers, GSS greedy, CoPE, Generative Replay.

• Regularization: EWC, LwF, GEM, AGEM, CWR*, Synaptic Intelligence, MAS.

• Architectural: Progressive Neural Networks, multi-head, incremental classifier.

• Others: GDumb, iCaRL, AR1, Streaming LDA, LFL.

Models

Avalanche uses and extends pytorch nn.Module to define continual learning models:

• support for nn.Modules and torchvision models.

• Dynamic output heads for class-incremental scenarios and multi heads for task-incremental scenarios.

• support for architectural strategies and dynamically expanding models such as progressive neural networks.

📊 Metrics and Evaluation

Avalanche provides continuous evaluation of CL strategies with a large set of Metrics. They are collected and logged automatically by the strategy during the training and evaluation loops.

• Standard Performance Metrics: accuracy, loss, confusion (averaged over streams or experiences).

• CL-Metrics: backward/forward transfer, forgetting.

• Computational Resources: CPU and RAM usage, MAC, execution times.

A Short Guide for Researchers on the Run

Avalanche is mostly about making the life of a continual learning researcher easier.

Below, you can see the main Avalanche modules and how they interact with each other.

What are the three pillars of any respectful continual learning research project?

1. Benchmarks: Machine learning researchers need multiple benchmarks with efficient data handling utils to design and prototype new algorithms. Quantitative results on ever-changing benchmarks has been one of the driving forces of Deep Learning.

2. Training: Efficient implementation and training of continual learning algorithms; comparisons with other baselines and state-of-the-art methods become fundamental to asses the quality of an original algorithmic proposal.

3. Evaluation: Training utils and Benchmarks are not enough alone to push continual learning research forward. Comprehensive and sound evaluation protocols and metrics need to be employed as well.

With Avalanche, you can find all these three fundamental pieces together and much more, in a single and coherent, well-maintained codebase.

Let's take a quick tour on how you can use Avalanche for your research projects with a 5-minutes guide, for researchers on the run!

🏛️ General Architecture

Avalanche is organized in five main modules:

2. Training: This module provides all the necessary utilities concerning model training. This includes simple and efficient ways of implement new continual learning strategies as well as a set pre-implemented CL baselines and state-of-the-art algorithms you will be able to use for comparison!

3. Evaluation: This modules provides all the utilities and metrics that can help in evaluating a CL algorithm with respect to all the factors we believe to be important for a continually learning system.

In the graphic below, you can see how Avalanche sub-modules are available and organized as well:

We will learn more about each of them during this tutorial series, but keep in mind that the [Avalanche API documentation](https://avalanche-api.continualai.org/en/latest/) is your friend as well!

All right, let's start with the benchmarks module right away 👇

📚 Benchmarks

The benchmark module offers three main features:

1. Datasets: a comprehensive list of PyTorch Datasets ready to use (It includes all the Torchvision Datasets and more!).

2. Classic Benchmarks: a set of classic Continual Learning Benchmarks ready to be used (there can be multiple benchmarks based on a single dataset).

3. Generators: a set of functions you can use to generate your own benchmark starting from any PyTorch Dataset!

Datasets

Datasets can be imported in Avalanche as simply as:

Of course, you can use them as you would use any PyTorch Dataset.

Benchmarks Basics

The Avalanche benchmarks (instances of the Scenario class), contains several attributes that describe the benchmark. However, the most important ones are the train and test streams.

In Avalanche we often suppose to have access to these two parallel stream of data (even though some benchmarks may not provide such feature, but contain just a unique test set).

Each of these streams are iterable, indexable and sliceable objects that are composed of experiences. Experiences are batch of data (or "tasks") that can be provided with or without a specific task label.

Classic Benchmarks

Avalanche maintains a set of commonly used benchmarks built on top of one or multiple datasets.

Benchmarks Generators

What if we want to create a new benchmark that is not present in the "Classic" ones? Well, in that case Avalanche offers a number of utilities that you can use to create your own benchmark with maximum flexibility: the benchmark generators!

The specific scenario generators are useful when starting from one or multiple PyTorch datasets and you want to create a "New Instances" or "New Classes" benchmark: i.e. it supports the easy and flexible creation of a Domain-Incremental, Class-Incremental or Task-Incremental scenarios among others.

Finally, if your ideal benchmark does not fit well in the aforementioned Domain-Incremental, Class-Incremental or Task-Incremental scenarios, you can always use our generic generators:

• filelist_benchmark

• paths_benchmark

• dataset_benchmark

• tensors_benchmark

You can read more about how to use them the full Benchmarks module tutorial!

💪 Training

The training module in Avalanche is build on modularity and it has two main goals:

1. provide a set of standard continual learning baselines that can be easily run for comparison;

2. provide the necessary utilities to implement and run your own strategy in the most efficient and simple way possible thanks to the building blocks we already prepared for you.

Strategies

If you want to compare your strategy with other classic continual learning algorithms or baselines, in Avalanche this is as simple as creating an object:

Create your own Strategy

The simplest way to build your own strategy is to create a python class that implements the main train and eval methods.

Let's define our Continual Learning algorithm "MyStrategy" as a simple python class:

Then, we can use our strategy as we would do for the pre-implemented ones:

While this is the easiest possible way to add your own strategy, Avalanche supports more sophisticated modalities (based on callbacks) that lets you write more neat, modular and reusable code, inheriting functionality from a parent classes and using pre-implemented plugins.

Check out more details about what Avalanche can offer in this module following the "Training" chapter of the "From Zero to Hero" tutorial!

📈 Evaluation

The evaluation module is quite straightforward: it offers all the basic functionalities to evaluate and keep track of a continual learning experiment.

This is mostly done through the Metrics and the Loggers. The Metrics provide a set of classes which implements the main continual learning metrics like Accuracy, Forgetting, Memory Usage, Running Times, etc. Metrics should be created via the utility functions (e.g. accuracy_metrics, timing_metrics and others) specifying in the arguments when those metrics should be computed (after each minibatch, epoch, experience etc...). The Loggers specify a way to report the metrics (e.g. with Tensorboard, on console or others). Loggers are created by instantiating the respective class.

Metrics and loggers interact via the Evaluation Plugin: this is the main object responsible of tracking the experiment progress. Metrics and loggers are directly passed to the EvaluationPlugin instance. You will see the output of the loggers automatically during training and evaluation! Let's see how to put this together in few lines of code:

For more details about the evaluation module (how to write new metrics/loggers, a deeper tutorial on metrics) check out the extended guide in the "Evaluation" chapter of the "From Zero to Hero" Avalanche tutorial!

🔗 Putting all Together

You've learned how to install Avalanche, how to create benchmarks that can suit your needs, how you can create your own continual learning algorithm and how you can evaluate its performance.

Here we show how you can use all these modules together to design your experiments as quantitative supporting evidence for your research project or paper.

🤝 Run it on Google Colab

Welcome to the "Training" tutorial of the "From Zero to Hero" series. In this part we will present the functionalities offered by the training module.

First, let's install Avalanche. You can skip this step if you have installed it already.

💪 The Training Module

The training module in Avalanche is designed with modularity in mind. Its main goals are to:

1. provide a set of popular continual learning baselines that can be easily used to run experimental comparisons;

2. provide simple abstractions to create and run your own strategy as efficiently and easily as possible starting from a couple of basic building blocks we already prepared for you.

At the moment, the training module includes three main components:

• Templates: these are high level abstractions used as a starting point to define the actual strategies. The templates contain already implemented basic utilities and functionalities shared by a group of strategies (e.g. the BaseSGDTemplate contains all the implemented methods to deal with strategies based on SGD).

• Strategies: these are popular baselines already implemented for you which you can use for comparisons or as base classes to define a custom strategy.

• Plugins: these are classes that allow adding some specific behavior to your own strategy. The plugin system allows defining reusable components which can be easily combined (e.g. a replay strategy, a regularization strategy). They are also used to automatically manage logging and evaluation.

Keep in mind that many Avalanche components are independent of Avalanche strategies. If you already have your own strategy which does not use Avalanche, you can use Avalanche's benchmarks, models, data loaders, and metrics without ever looking at Avalanche's strategies!

📈 How to Use Strategies & Plugins

If you want to compare your strategy with other classic continual learning algorithms or baselines, in Avalanche you can instantiate a strategy with a couple of lines of code.

Strategy Instantiation

Most strategies require only 3 mandatory arguments:

• model: this must be a torch.nn.Module.

• optimizer: torch.optim.Optimizer already initialized on your model.

• loss: a loss function such as those in torch.nn.functional.

Additional arguments are optional and allow you to customize training (batch size, number of epochs, ...) or strategy-specific parameters (memory size, regularization strength, ...).

Training & Evaluation

Each strategy object offers two main methods: train and eval. Both of them, accept either a single experience(Experience) or a list of them, for maximum flexibility.

We can train the model continually by iterating over the train_stream provided by the scenario.

Adding Plugins

Most continual learning strategies follow roughly the same training/evaluation loops, i.e. a simple naive strategy (a.k.a. finetuning) augmented with additional behavior to counteract catastrophic forgetting. The plugin systems in Avalanche is designed to easily augment continual learning strategies with custom behavior, without having to rewrite the training loop from scratch. Avalanche strategies accept an optional list of plugins that will be executed during the training/evaluation loops.

For example, early stopping is implemented as a plugin:

In Avalanche, most continual learning strategies are implemented using plugins, which makes it easy to combine them together. For example, it is extremely easy to create a hybrid strategy that combines replay and EWC together by passing the appropriate plugins list to the SupervisedTemplate:

Beware that most strategy plugins modify the internal state. As a result, not all the strategy plugins can be combined together. For example, it does not make sense to use multiple replay plugins since they will try to modify the same strategy variables (mini-batches, dataloaders), and therefore they will be in conflict.

📝 A Look Inside Avalanche Strategies

If you arrived at this point you already know how to use Avalanche strategies and are ready to use it. However, before making your own strategies you need to understand a little bit the internal implementation of the training and evaluation loops.

In Avalanche you can customize a strategy in 2 ways:

1. Plugins: Most strategies can be implemented as additional code that runs on top of the basic training and evaluation loops (e.g. the Naive strategy). Therefore, the easiest way to define a custom strategy such as a regularization or replay strategy, is to define it as a custom plugin. The advantage of plugins is that they can be combined, as long as they are compatible, i.e. they do not modify the same part of the state. The disadvantage is that in order to do so you need to understand the strategy loop, which can be a bit complex at first.

2. Subclassing: In Avalanche, continual learning strategies inherit from the appropriate template, which provides generic training and evaluation loops. The most high level template is the BaseTemplate, from which all the Avalanche's strategies inherit. Most template's methods can be safely overridden (with some caveats that we will see later).

Keep in mind that if you already have a working continual learning strategy that does not use Avalanche, you can use most Avalanche components such as benchmarks, evaluation, and models without using Avalanche's strategies!

Training and Evaluation Loops

As we already mentioned, Avalanche strategies inherit from the appropriate template (e.g. continual supervised learning strategies inherit from the SupervisedTemplate). These templates provide:

1. Basic Training and Evaluation loops which define a naive (finetuning) strategy.

2. Callback points, which are used to call the plugins at a specific moments during the loop's execution.

3. A set of variables representing the state of the loops (current model, data, mini-batch, predictions, ...) which allows plugins and child classes to easily manipulate the state of the training loop.

The training loop has the following structure:

The evaluation loop is similar:

Methods starting with before/after are the methods responsible for calling the plugins. Notice that before the start of each experience during training we have several phases:

• dataset adaptation: This is the phase where the training data can be modified by the strategy, for example by adding other samples from a separate buffer.

• dataloader initialization: Initialize the data loader. Many strategies (e.g. replay) use custom dataloaders to balance the data.

• model adaptation: Here, the dynamic models (see the models tutorial) are updated by calling their adaptation method.

• optimizer initialization: After the model has been updated, the optimizer should also be updated to ensure that the new parameters are optimized.

Strategy State

The strategy state is accessible via several attributes. Most of these can be modified by plugins and subclasses:

• self.clock: keeps track of several event counters.

• self.experience: the current experience.

• self.adapted_dataset: the data modified by the dataset adaptation phase.

• self.dataloader: the current dataloader.

• self.mbatch: the current mini-batch. For supervised classification problems, mini-batches have the form <x, y, t>, where x is the input, y is the target class, and t is the task label.

• self.mb_output: the current model's output.

• self.loss: the current loss.

• self.is_training: True if the strategy is in training mode.

How to Write a Plugin

Plugins provide a simple solution to define a new strategy by augmenting the behavior of another strategy (typically the Naive strategy). This approach reduces the overhead and code duplication, improving code readability and prototyping speed.

Creating a plugin is straightforward. As with strategies, you must create a class that inherits from the corresponding plugin template (BasePlugin, BaseSGDPlugin, SupervisedPlugin) and implements the callbacks that you need. The exact callback to use depend on the aim of your plugin. You can use the loop shown above to understand what callbacks you need to use. For example, we show below a simple replay plugin that uses after_training_exp to update the buffer after each training experience, and the before_training_exp to customize the dataloader. Notice that before_training_exp is executed after make_train_dataloader, which means that the Naive strategy already updated the dataloader. If we used another callback, such as before_train_dataset_adaptation, our dataloader would have been overwritten by the Naive strategy. Plugin methods always receive the strategy as an argument, so they can access and modify the strategy's state.

Check base plugin's documentation for a complete list of the available callbacks.

How to Write a Custom Strategy

You can always define a custom strategy by overriding the corresponding template methods. However, There is an important caveat to keep in mind. If you override a method, you must remember to call all the callback's handlers (the methods starting with before/after) at the appropriate points. For example, train calls before_training and after_training before and after the training loops, respectively. The easiest way to avoid mistakes is to start from the template's method that you want to override and modify it based on your own needs without removing the callbacks handling.

Notice that the EvaluationPlugin (see evaluation tutorial) uses the strategy callbacks.

As an example, the SupervisedTemplate, for continual supervised strategies, provides the global state of the loop in the strategy's attributes, which you can safely use when you override a method. For instance, the Cumulative strategy trains a model continually on the union of all the experiences encountered so far. To achieve this, the cumulative strategy overrides adapt_train_dataset and updates `self.adapted_dataset' by concatenating all the previous experiences with the current one.

Easy, isn't it? :-)

In general, we recommend to implement a Strategy via plugins, if possible. This approach is the easiest to use and requires minimal knowledge of the strategy templates. It also allows other people to re-use your plugin and facilitates interoperability among different strategies.

For example, replay strategies can be implemented as a custom strategy or as plugins. However, creating a plugin allows using the replay in conjunction with other strategies or plugins, making it possible to combine different approaches to build the ultimate continual learning algorithm!

This completes the "Training" chapter for the "From Zero to Hero" series. We hope you enjoyed it!

🤝 Run it on Google Colab

Avalanche was born within ContinualAI with a clear goal in mind:

Pushing Continual Learning to the next level, providing a shared and collaborative library for fast prototyping, training and reproducible evaluation of continual learning algorithms.

As a powerful avalanche, a Continual Learning agent incrementally improves its knowledge and skills over time, building upon the previously acquired ones and learning how to interact with the external world.

We hope Avalanche may trigger the same positive reinforcement loop within our community, moving towards a more collaborative and inclusive way of doing research and helping us tackle bigger problems, faster and better, but together! 👪

💪The Avalanche Advantage

Avalanche has several advantages:

• Shared & Coherent Codebase: Aren't you tired of re-inventing the wheel in continual learning? We are. Re-producing paper results has always been daunting in machine learning and it is even more so in continual learning. Avalanche makes you stop re-write your (and other people) code all over again with a coherent and shared codebase that provides already all the utilities, benchmark, metrics and baselines you may need for your next great continual learning research project!

• Errors Reduction: The more code we write, the more bugs we introduce in our code. This is the rule, not the exception. Avalanche, let you focus on what really matters: defining your CL solution. Benchmarks preparation to training, evaluation and comparison with other methods will be already there for you. This in turn, massively reduce the amount of errors introduced and the time needed to debug your code.

• Faster Prototyping: As researchers or data scientists, we have dozens ideas every day and time is always too little to execute them. However, if we think about it, most of the time spent in bringing our ideas to life is consumed in installing software, preparing and cleaning our data, preparing the experiments code infrastructure and so on. Avalanche lets you focus just on the original algorithmic proposal, taking care of most of the rest!

• Improved Reproducibility & Portability: One of the great features of Avalanche, is the possibility of reproducing experimental results easily and on any OS. Researchers can simply plug-in their algorithm into the codebase and see how it goes with respect of other researchers' methods. Their algorithm in turn, is used as a baseline for other methods, creating a virtuous circle. This is only possible thanks to the simple, yet powerful idea of providing shared benchmarks, training and evaluation in a single place.

• Improved Modularity: Avalanche has been designed with modularity in mind. As you will learn more about Avalanche, you will realize we have sometimes forego simplicity in favor of modularity and reusability (we hate code replication as you do 🤪). However, we believe this will help us scale in the near future as we collaboratively bring this codebase into maturity.

• Increased Efficiency & Scalability: Full-stack researchers & data scientists know this, making your algorithm memory and computationally efficient is tough. Avalanche is already optimized for you, so that you can run your ImageNet continual learning experiment on your 8GB laptop (buy a cooling fan 💨) or even try it on embedded devices of your latest product!

But most of all, Avalanche, can help us standardize our field and work better together, more collaboratively, towards our shared goal of making machines learn over time like humans do.

Avalanche the first experiment of a End-to-end Library for reproducible continual learning research where you can find benchmarks, algorithms, evaluation utilities and much more in the same place.

Let's make it together 👫 a wonderful ride! 🎈

Avalanche offers significant support for training (with templates, strategies and plug-ins). Here you can find a list of examples related to the training and some strategies available in Avalanche (each strategy reproduces original paper results in the CL-Baselines repository:

• Joint-Training: this example shows how to take a stream of experiences and train simultaneously on all of them. This is useful to implement the "offline" or "multi-task" upper bound.

• Replay strategy: simple example on the usage of replay in Avalanche.

• AR1 strategy: this is a simple example on how to use the AR1 strategy.

• CoPE Strategy: this is a simple example on how to use the CoPE plugin. It's an example in the online data incremental setting, where both learning and evaluation is completely task-agnostic.

• Cumulative Strategy: how to define your own cumulative strategy based on the different Data Loaders made available in Avalanche.

• Deep SLDA: this is a simple example on how to use the Deep SLDA strategy.

• Early Stopping: this example shows how to use early stopping to dynamically stop the training procedure when the model converged instead of training for a fixed number of epochs.

• Object Detection: this example shows how to run object detection/segmentation tasks.

• Object Detection with Elvis: this example shows how to run object detection/segmentation tasks with a toy benchmark based on the LVIS dataset.

• Object Detection Training: set of examples showing how you can use Avalanche for distributed training of object detector.

• EWC on MNIST: this example tests EWC on Split MNIST and Permuted MNIST.

• LWF on MNIST: this example tests LWF on Permuted MNIST.

• GEM and A-GEM on MNIST: this example shows how to use GEM and A-GEM strategies on MNIST.

• Ex-Model Continual Learning: this example shows how to create a stream of pre-trained model from which to learn.

• Generative Replay: this is a simple example on how to implement generative replay in Avalanche.

• iCARL strategy: simple example to show how to use the iCARL strategy.

• LaMAML strategy: example on how to use a meta continual learning in Avalanche.

• RWalk strategy: example of the RWalk strategy usage.

• Online Naive: example to run a naive strategy in an online setting.

• Synaptic Intelligence: this is a simple example on how to use the Synaptic Intelligence Plugin.

• Continual Sequence Classification: sequence classification example using torchaudio and Speech Commands.

The AvalancheDataset is an implementation of the PyTorch Dataset class that comes with many useful out-of-the-box functionalities. For most users, the AvalancheDataset can be used as a plain PyTorch Dataset. For classification problems, AvalancheDataset return x, y, t elements (input, target, task label). However, the AvalancheDataset can be easily extended for any custom needs.

A serie of Mini How-Tos will guide you through the functionalities of the AvalancheDataset and its subclasses:

• AvalancheDatasets basics

• Advanced Transformations

Avalanche offers basic support for defining your own models or adapt existing PyTorch models with a particular emphasis on model adaptation over time.

You can find examples related to the models here:

• Using PyTorchCV pre-trained models: This example shows how to train models provided by pytorchcv with the rehearsal strategy.

• Use a Multi-Head model: This example trains a Multi-head model on Split MNIST with Elastich Weight Consolidation. Each experience has a different task label, which is used at test time to select the appropriate head.

Welcome to the "benchmarks" tutorial of the "From Zero to Hero" series. In this part we will present the functionalities offered by the Benchmarks module.

!pip install avalanche-lib==0.3.1

🎯 Nomenclature

First off, let's clarify a bit the nomenclature we are going to use, introducing the following terms: Datasets, Scenarios, Benchmarks and Generators.

• By Dataset we mean a collection of examples that can be used for training or testing purposes but not already organized to be processed as a stream of batches or tasks. Since Avalanche is based on Pytorch, our Datasets are torch.utils.Datasets objects.

• By Scenario we mean a particular setting, i.e. specificities about the continual stream of data, a continual learning algorithm will face.

• By Benchmark we mean a well-defined and carefully thought combination of a scenario with one or multiple datasets that we can use to asses our continual learning algorithms.

• By Generator we mean a function that given a specific scenario and a dataset can generate a Benchmark.

📚 The Benchmarks Module

The bechmarks module offers 3 types of utils:

• Datasets: all the Pytorch datasets plus additional ones prepared by our community and particularly interesting for continual learning.

• Classic Benchmarks: classic benchmarks used in CL litterature ready to be used with great flexibility.

• Benchmarks Generators: a set of functions you can use to create your own benchmark starting from any kind of data and scenario. In particular, we distinguish two type of generators: Specific and Generic. The first ones will let you create a benchmark based on a clear scenarios and Pytorch dataset(s); the latters, instead, are more generic and flexible, both in terms of scenario definition then in terms of type of data they can manage.

  • Specific:

    • nc_benchmark: given one or multiple datasets it creates a benchmark instance based on scenarios where New Classes (NC) are encountered over time. Notable scenarios that can be created using this utility include Class-Incremental, Task-Incremental and Task-Agnostic scenarios.

    • ni_benchmark: it creates a benchmark instance based on scenarios where New Instances (NI), i.e. new examples of the same classes are encountered over time. Notable scenarios that can be created using this utility include Domain-Incremental scenarios.

  • Generic:

    • filelist_benchmark: It creates a benchmark instance given a list of filelists.

    • paths_benchmark: It creates a benchmark instance given a list of file paths and class labels.

    • tensors_benchmark: It creates a benchmark instance given a list of tensors.

    • dataset_benchmark: It creates a benchmark instance given a list of pytorch datasets.

But let's see how we can use this module in practice!

🖼️ Datasets

Let's start with the Datasets. As we previously hinted, in Avalanche you'll find all the standard Pytorch Datasets available in the torchvision package as well as a few others that are useful for continual learning but not already officially available within the Pytorch ecosystem.

import torch
import torchvision
from avalanche.benchmarks.datasets import MNIST, FashionMNIST, KMNIST, EMNIST, \
QMNIST, FakeData, CocoCaptions, CocoDetection, LSUN, ImageNet, CIFAR10, \
CIFAR100, STL10, SVHN, PhotoTour, SBU, Flickr8k, Flickr30k, VOCDetection, \
VOCSegmentation, Cityscapes, SBDataset, USPS, HMDB51, UCF101, \
CelebA, CORe50Dataset, TinyImagenet, CUB200, OpenLORIS

# As we would simply do with any Pytorch dataset we can create the train and 
# test sets from it. We could use any of the above imported Datasets, but let's
# just try to use the standard MNIST.
train_MNIST = MNIST(
    './data/mnist', train=True, download=True, transform=torchvision.transforms.ToTensor()
)
test_MNIST = MNIST(
    './data/mnist', train=False, download=True, transform=torchvision.transforms.ToTensor()
)

# Given these two sets we can simply iterate them to get the examples one by one
for i, example in enumerate(train_MNIST):
    pass
print("Num. examples processed: {}".format(i))

# or use a Pytorch DataLoader
train_loader = torch.utils.data.DataLoader(
    train_MNIST, batch_size=32, shuffle=True
)
for i, (x, y) in enumerate(train_loader):
    pass
print("Num. mini-batch processed: {}".format(i))

Of course also the basic utilities ImageFolder and DatasetFolder can be used. These are two classes that you can use to create a Pytorch Dataset directly from your files (following a particular structure). You can read more about these in the Pytorch official documentation here.

We also provide an additional FilelistDataset and AvalancheDataset classes. The former to construct a dataset from a filelist (caffe style) pointing to files anywhere on the disk. The latter to augment the basic Pytorch Dataset functionalities with an extention to better deal with a stack of transformations to be used during train and test.

from avalanche.benchmarks.utils import ImageFolder, DatasetFolder, FilelistDataset, AvalancheDataset

🛠️ Benchmarks Basics

The Avalanche benchmarks (instances of the Scenario class), contains several attributes that characterize the benchmark. However, the most important ones are the train and test streams.

In Avalanche we often suppose to have access to these two parallel stream of data (even though some benchmarks may not provide such feature, but contain just a unique test set).

Each of these streams are iterable, indexable and sliceable objects that are composed of unique experiences. Experiences are batch of data (or "tasks") that can be provided with or without a specific task label.

Efficiency

It is worth mentioning that all the data belonging to a stream are not loaded into the RAM beforehand. Avalanche actually loads the data when a specific mini-batches are requested at training/test time based on the policy defined by each Dataset implementation.

This means that memory requirements are very low, while the speed is guaranteed by a multi-processing data loading system based on the one defined in Pytorch.

Scenarios

So, as we have seen, each scenario object in Avalanche has several useful attributes that characterizes the benchmark, including the two important train and test streams. Let's check what you can get from a scenario object more in details:

from avalanche.benchmarks.classic import SplitMNIST
split_mnist = SplitMNIST(n_experiences=5, seed=1)

# Original train/test sets
print('--- Original datasets:')
print(split_mnist.original_train_dataset)
print(split_mnist.original_test_dataset)

# A list describing which training patterns are assigned to each experience.
# Patterns are identified by their id w.r.t. the dataset found in the
# original_train_dataset field.
print('--- Train patterns assignment:')
print(split_mnist.train_exps_patterns_assignment)

# A list describing which test patterns are assigned to each experience.
# Patterns are identified by their id w.r.t. the dataset found in the
# original_test_dataset field
print('--- Test patterns assignment:')
print(split_mnist.test_exps_patterns_assignment)

# the task label of each experience.
print('--- Task labels:')
print(split_mnist.task_labels)

# train and test streams
print('--- Streams:')
print(split_mnist.train_stream)
print(split_mnist.test_stream)

# A list that, for each experience (identified by its index/ID),
# stores a set of the (optionally remapped) IDs of classes of patterns
# assigned to that experience.
print('--- Classes in each experience:')
print(split_mnist.original_classes_in_exp)

Train and Test Streams

The train and test streams can be used for training and testing purposes, respectively. This is what you can do with these streams:

# each stream has a name: "train" or "test"
train_stream = split_mnist.train_stream
print(train_stream.name)

# we have access to the scenario from which the stream was taken
train_stream.benchmark

# we can slice and reorder the stream as we like!
substream = train_stream[0]
substream = train_stream[0:2]
substream = train_stream[0,2,1]

len(substream)

Experiences

Each stream can in turn be treated as an iterator that produces a unique experience, containing all the useful data regarding a batch or task in the continual stream our algorithms will face. Check out how can you use these experiences below:

# we get the first experience
experience = train_stream[0]

# task label and dataset are the main attributes
t_label = experience.task_label
dataset = experience.dataset

# but you can recover additional info
experience.current_experience
experience.classes_in_this_experience
experience.classes_seen_so_far
experience.previous_classes
experience.future_classes
experience.origin_stream
experience.benchmark

# As always, we can iterate over it normally or with a pytorch
# data loader.
# For instance, we can use tqdm to add a progress bar.
from tqdm import tqdm
for i, data in enumerate(tqdm(dataset)):
  pass
print("\nNumber of examples:", i + 1)
print("Task Label:", t_label)

🏛️ Classic Benchmarks

Now that we know how our benchmarks work in general through scenarios, streams and experiences objects, in this section we are going to explore common benchmarks already available for you with one line of code yet flexible enough to allow proper tuning based on your needs:

from avalanche.benchmarks.classic import CORe50, SplitTinyImageNet, \
SplitCIFAR10, SplitCIFAR100, SplitCIFAR110, SplitMNIST, RotatedMNIST, \
PermutedMNIST, SplitCUB200, SplitImageNet

# creating PermutedMNIST (Task-Incremental)
perm_mnist = PermutedMNIST(
    n_experiences=2,
    seed=1234,
)

Many of the classic benchmarks will download the original datasets they are based on automatically and put it under the "~/.avalanche/data" directory.

How to Use the Benchmarks

Let's see now how we can use the classic benchmark or the ones that you can create through the generators (see next section). For example, let's try out the classic PermutedMNIST benchmark (Task-Incremental scenario).

# creating the benchmark instance (scenario object)
perm_mnist = PermutedMNIST(
  n_experiences=3,
  seed=1234,
)

# recovering the train and test streams
train_stream = perm_mnist.train_stream
test_stream = perm_mnist.test_stream

# iterating over the train stream
for experience in train_stream:
  print("Start of task ", experience.task_label)
  print('Classes in this task:', experience.classes_in_this_experience)

  # The current Pytorch training set can be easily recovered through the
  # experience
  current_training_set = experience.dataset
  # ...as well as the task_label
  print('Task {}'.format(experience.task_label))
  print('This task contains', len(current_training_set), 'training examples')

  # we can recover the corresponding test experience in the test stream
  current_test_set = test_stream[experience.current_experience].dataset
  print('This task contains', len(current_test_set), 'test examples')

🐣 Benchmarks Generators

What if we want to create a new benchmark that is not present in the "Classic" ones? Well, in that case Avalanche offer a number of utilites that you can use to create your own benchmark with maximum flexibility: the benchmarks generators!

Specific Generators

The specific scenario generators are useful when starting from one or multiple Pytorch datasets you want to create a "New Instances" or "New Classes" benchmark: i.e. it supports the easy and flexible creation of a Domain-Incremental, Class-Incremental or Task-Incremental scenarios among others.

For the New Classes scenario you can use the following function:

• nc_benchmark

for the New Instances:

• ni_benchmark

from avalanche.benchmarks.generators import nc_benchmark, ni_benchmark

Let's start by creating the MNIST dataset object as we would normally do in Pytorch:

from torchvision.transforms import Compose, ToTensor, Normalize, RandomCrop
train_transform = Compose([
    RandomCrop(28, padding=4),
    ToTensor(),
    Normalize((0.1307,), (0.3081,))
])

test_transform = Compose([
    ToTensor(),
    Normalize((0.1307,), (0.3081,))
])

mnist_train = MNIST(
    './data/mnist', train=True, download=True, transform=train_transform
)
mnist_test = MNIST(
    './data/mnist', train=False, download=True, transform=test_transform
)

Then we can, for example, create a new benchmark based on MNIST and the classic Domain-Incremental scenario:

scenario = ni_benchmark(
    mnist_train, mnist_test, n_experiences=10, shuffle=True, seed=1234,
    balance_experiences=True
)

train_stream = scenario.train_stream

for experience in train_stream:
    t = experience.task_label
    exp_id = experience.current_experience
    training_dataset = experience.dataset
    print('Task {} batch {} -> train'.format(t, exp_id))
    print('This batch contains', len(training_dataset), 'patterns')

Or, we can create a benchmark based on MNIST and the Class-Incremental (what's commonly referred to as "Split-MNIST" benchmark):

scenario = nc_benchmark(
    mnist_train, mnist_test, n_experiences=10, shuffle=True, seed=1234,
    task_labels=False
)

train_stream = scenario.train_stream

for experience in train_stream:
    t = experience.task_label
    exp_id = experience.current_experience
    training_dataset = experience.dataset
    print('Task {} batch {} -> train'.format(t, exp_id))
    print('This batch contains', len(training_dataset), 'patterns')

Generic Generators

Finally, if you cannot create your ideal benchmark since it does not fit well in the aforementioned new classes or new instances scenarios, you can always use our generic generators:

• filelist_benchmark

• paths_benchmark

• dataset_benchmark

• tensors_benchmark

from avalanche.benchmarks.generators import filelist_benchmark, dataset_benchmark, \
                                            tensors_benchmark, paths_benchmark

Let's start with the filelist_benchmark utility. This function is particularly useful when it is important to preserve a particular order of the patterns to be processed (for example if they are frames of a video), or in general if we have data scattered around our drive and we want to create a sequence of batches/tasks providing only a txt file containing the list of their paths.

For Avalanche we follow the same format of the Caffe filelists ("path class_label"):

/path/to/a/file.jpg 0 /path/to/another/file.jpg 0 ... /path/to/another/file.jpg M /path/to/another/file.jpg M ... /path/to/another/file.jpg N /path/to/another/file.jpg N

So let's download the classic "Cats vs Dogs" dataset as an example:

!wget -N --no-check-certificate \
    https://storage.googleapis.com/mledu-datasets/cats_and_dogs_filtered.zip
!unzip -q -o cats_and_dogs_filtered.zip

You can now see in the content directory on colab the image we downloaded. We are now going to create the filelists and then use the filelist_benchmark function to create our benchmark:

import os
# let's create the filelists since we don't have it
dirpath = "cats_and_dogs_filtered/train"

for filelist, rel_dir, t_label in zip(
        ["train_filelist_00.txt", "train_filelist_01.txt"],
        ["cats", "dogs"],
        [0, 1]):
    # First, obtain the list of files
    filenames_list = os.listdir(os.path.join(dirpath, rel_dir))

    # Create the text file containing the filelist
    # Filelists must be in Caffe-style, which means
    # that they must define path in the format:
    #
    # relative_path_img1 class_label_first_img
    # relative_path_img2 class_label_second_img
    # ...
    #
    # For instance:
    # cat/cat_0.png 1
    # dog/dog_54.png 0
    # cat/cat_3.png 1
    # ...
    # 
    # Paths are relative to a root path
    # (specified when calling filelist_benchmark)
    with open(filelist, "w") as wf:
        for name in filenames_list:
            wf.write(
                "{} {}\n".format(os.path.join(rel_dir, name), t_label)
            )

# Here we create a GenericCLScenario ready to be iterated
generic_scenario = filelist_benchmark(
   dirpath,  
   ["train_filelist_00.txt", "train_filelist_01.txt"],
   ["train_filelist_00.txt"],
   task_labels=[0, 0],
   complete_test_set_only=True,
   train_transform=ToTensor(),
   eval_transform=ToTensor()
)

In the previous cell we created a benchmark instance starting from file lists. However, paths_benchmark is a better choice if you already have the list of paths directly loaded in memory:

train_experiences = []
for rel_dir, label in zip(
        ["cats", "dogs"],
        [0, 1]):
    # First, obtain the list of files
    filenames_list = os.listdir(os.path.join(dirpath, rel_dir))

    # Don't create a file list: instead, we create a list of 
    # paths + class labels
    experience_paths = []
    for name in filenames_list:
      instance_tuple = (os.path.join(dirpath, rel_dir, name), label)
      experience_paths.append(instance_tuple)
    train_experiences.append(experience_paths)

# Here we create a GenericCLScenario ready to be iterated
generic_scenario = paths_benchmark(
   train_experiences,
   [train_experiences[0]],  # Single test set
   task_labels=[0, 0],
   complete_test_set_only=True,
   train_transform=ToTensor(),
   eval_transform=ToTensor()
)

Let us see how we can use the dataset_benchmark utility, where we can use several PyTorch datasets as different batches or tasks. This utility expectes a list of datasets for the train, test (and other custom) streams. Each dataset will be used to create an experience:

train_cifar10 = CIFAR10(
    './data/cifar10', train=True, download=True
)
test_cifar10 = CIFAR10(
    './data/cifar10', train=False, download=True
)

generic_scenario = dataset_benchmark(
    [train_MNIST, train_cifar10],
    [test_MNIST, test_cifar10]
)

Adding task labels can be achieved by wrapping each datasets using AvalancheDataset. Apart from task labels, AvalancheDataset allows for more control over transformations and offers an ever growing set of utilities (check the documentation for more details).

# Alternatively, task labels can also be a list (or tensor)
# containing the task label of each pattern
from avalanche.benchmarks.utils import make_classification_dataset

train_MNIST_task0 = make_classification_dataset(train_cifar10, task_labels=0)
test_MNIST_task0 = make_classification_dataset(test_cifar10, task_labels=0)

train_cifar10_task1 = make_classification_dataset(train_cifar10, task_labels=1)
test_cifar10_task1 = make_classification_dataset(test_cifar10, task_labels=1)

scenario_custom_task_labels = dataset_benchmark(
    [train_MNIST_task0, train_cifar10_task1],
    [test_MNIST_task0, test_cifar10_task1]
)

print('Without custom task labels:',
      generic_scenario.train_stream[1].task_label)

print('With custom task labels:',
      scenario_custom_task_labels.train_stream[1].task_label)

And finally, the tensors_benchmark generator:

pattern_shape = (3, 32, 32)

# Definition of training experiences
# Experience 1
experience_1_x = torch.zeros(100, *pattern_shape)
experience_1_y = torch.zeros(100, dtype=torch.long)

# Experience 2
experience_2_x = torch.zeros(80, *pattern_shape)
experience_2_y = torch.ones(80, dtype=torch.long)

# Test experience
# For this example we define a single test experience,
# but "tensors_benchmark" allows you to define even more than one!
test_x = torch.zeros(50, *pattern_shape)
test_y = torch.zeros(50, dtype=torch.long)

generic_scenario = tensors_benchmark(
    train_tensors=[(experience_1_x, experience_1_y), (experience_2_x, experience_2_y)],
    test_tensors=[(test_x, test_y)],
    task_labels=[0, 0],  # Task label of each train exp
    complete_test_set_only=True
)

This completes the "Benchmark" tutorial for the "From Zero to Hero" series. We hope you enjoyed it!

🤝 Run it on Google Colab

Welcome to the "Models" tutorial of the "From Zero to Hero" series. In this notebook we will talk about the features offered by the models Avalanche sub-module.

Support for pytorch Modules

Every continual learning experiment needs a model to train incrementally. You can use any torch.nn.Module, even pretrained models. The models sub-module provides the most commonly used architectures in the CL literature.

You can use any model provided in the Pytorch official ecosystem models as well as the ones provided by pytorchcv!

!pip install avalanche-lib==0.3.1

from avalanche.models import SimpleCNN
from avalanche.models import SimpleMLP
from avalanche.models import SimpleMLP_TinyImageNet
from avalanche.models import MobilenetV1

model = SimpleCNN()
print(model)

Dynamic Model Expansion

A continual learning model may change over time. As an example, a classifier may add new units for previously unseen classes, while progressive networks add a new set units after each experience. Avalanche provides DynamicModules to support these use cases. DynamicModules are torch.nn.Modules that provide an addition method, adaptation, that is used to update the model's architecture. The method takes a single argument, the data from the current experience.

For example, an IncrementalClassifier updates the number of output units:

from avalanche.benchmarks import SplitMNIST
from avalanche.models import IncrementalClassifier

benchmark = SplitMNIST(5, shuffle=False, class_ids_from_zero_in_each_exp=False)
model = IncrementalClassifier(in_features=784)

print(model)
for exp in benchmark.train_stream:
    model.adaptation(exp)
    print(model)

As you can see, after each call to the adaptation method, the model adds 2 new units to account for the new classes. Notice that no learning occurs at this point since the method only modifies the model's architecture.

Keep in mind that when you use Avalanche strategies you don't have to call the adaptation yourself. Avalanche strategies automatically call the model's adaptation and update the optimizer to include the new parameters.

Multi-Task models

Some models, such as multi-head classifiers, are designed to exploit task labels. In Avalanche, such models are implemented as MultiTaskModules. These are dynamic models (since they need to be updated whenever they encounter a new task) that have an additional task_labels argument in their forward method. task_labels is a tensor with a task id for each sample.

from avalanche.benchmarks import SplitMNIST
from avalanche.models import MultiHeadClassifier

benchmark = SplitMNIST(5, shuffle=False, return_task_id=True, class_ids_from_zero_in_each_exp=True)
model = MultiHeadClassifier(in_features=784)

print(model)
for exp in benchmark.train_stream:
    model.adaptation(exp)
    print(model)

When you use a MultiHeadClassifier, a new head is initialized whenever a new task is encountered. Avalanche strategies automatically recognize multi-task modules and provide task labels to them.

How to define a multi-task Module

If you want to define a custom multi-task module you need to override two methods: adaptation (if needed), and forward_single_task. The forward method of the base class will split the mini-batch by task-id and provide single task mini-batches to forward_single_task.

from avalanche.models import MultiTaskModule

class CustomMTModule(MultiTaskModule):
    def __init__(self, in_features, initial_out_features=2):
        super().__init__()

    def adaptation(self, dataset):
        super().adaptation(dataset)
        # your adaptation goes here

    def forward_single_task(self, x, task_label):
        # your forward goes here.
        # task_label is a single integer
        # the mini-batch is split by task-id inside the forward method.
        pass