Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Avalnche Features: Benchmarks, Strategies & Metrics
Avalanche is a framework in constant development. Thanks to the support of the ContinualAI community and its active members we plan to extend its features and improve its usability based on the demands of our research community! A the moment, Avalanche is in Beta (v0.3.1). We support a large number of Benchmarks, Strategies and Metrics, that makes it, we believe, the best tool out there for your continual learning research! 💪
You can find the full list of available features on the API documentation.
Do you think we are missing some important features? Please let us know! We deeply value your feedback!
You find a complete list of the features on the benchmarks API documentation.
Avalanche supports all the most popular computer vision datasets used in Continual Learning. Some of them are available in Torchvision, while other have been integrated by us. Most datasets can be automatically downloaded by Avalanche.
Toy datasets: MNIST, Fashion MNIST, KMNIST, EMNIST, QMNIST.
CIFAR: CIFAR10, CIFAR100.
ImageNet: TinyImagenet, MiniImagenet, Imagenet.
Others: EndlessCLDataset, CUB200, OpenLORIS, Stream-51, INATURALIST2018, Omniglot, CLEARImage, ...
All the major continual learning benchmarks are available and ready to use. Benchmarks split the datasets and create the train and test streams:
MNIST: SplitMNIST, RotatedMNIST, PermutedMNIST, SplitFashionMNIST.
CIFAR10: SplitCIFAR10, SplitCIFAR100, SplitCIFAR110.
CORe50: all the CORe50 scenarios are supported.
Others: SplitCUB200, CLStream51, CLEAR.
Avalanche provides Continual Learning algorithms (strategies). We are continuously expanding the library with new algorithms and making sure they can reproduce seminal papers results in the sibling project CL-Baselines.
Baselines: Naive, JointTraining, Cumulative.
Rehearsal: Replay with reservoir sampling and balanced buffers, GSS greedy, CoPE, Generative Replay.
Regularization: EWC, LwF, GEM, AGEM, CWR*, Synaptic Intelligence, MAS.
Architectural: Progressive Neural Networks, multi-head, incremental classifier.
Others: GDumb, iCaRL, AR1, Streaming LDA, LFL.
Avalanche uses and extends pytorch nn.Module
to define continual learning models:
support for nn.Module
s and torchvision
models.
Dynamic output heads for class-incremental scenarios and multi heads for task-incremental scenarios.
support for architectural strategies and dynamically expanding models such as progressive neural networks.
Avalanche provides continuous evaluation of CL strategies with a large set of Metrics. They are collected and logged automatically by the strategy during the training and evaluation loops.
Standard Performance Metrics: accuracy, loss, confusion (averaged over streams or experiences).
CL-Metrics: backward/forward transfer, forgetting.
Computational Resources: CPU and RAM usage, MAC, execution times.
and many more.
A Brief Introduction to Avalanche
Avalanche was born within ContinualAI with a clear goal in mind:
Pushing Continual Learning to the next level, providing a shared and collaborative library for fast prototyping, training and reproducible evaluation of continual learning algorithms.
As a powerful avalanche, a Continual Learning agent incrementally improves its knowledge and skills over time, building upon the previously acquired ones and learning how to interact with the external world.
We hope Avalanche may trigger the same positive reinforcement loop within our community, moving towards a more collaborative and inclusive way of doing research and helping us tackle bigger problems, faster and better, but together! 👪
Avalanche has several advantages:
Shared & Coherent Codebase: Aren't you tired of re-inventing the wheel in continual learning? We are. Re-producing paper results has always been daunting in machine learning and it is even more so in continual learning. Avalanche makes you stop re-write your (and other people) code all over again with a coherent and shared codebase that provides already all the utilities, benchmark, metrics and baselines you may need for your next great continual learning research project!
Errors Reduction: The more code we write, the more bugs we introduce in our code. This is the rule, not the exception. Avalanche, let you focus on what really matters: defining your CL solution. Benchmarks preparation to training, evaluation and comparison with other methods will be already there for you. This in turn, massively reduce the amount of errors introduced and the time needed to debug your code.
Faster Prototyping: As researchers or data scientists, we have dozens ideas every day and time is always too little to execute them. However, if we think about it, most of the time spent in bringing our ideas to life is consumed in installing software, preparing and cleaning our data, preparing the experiments code infrastructure and so on. Avalanche lets you focus just on the original algorithmic proposal, taking care of most of the rest!
Improved Reproducibility & Portability: One of the great features of Avalanche, is the possibility of reproducing experimental results easily and on any OS. Researchers can simply plug-in their algorithm into the codebase and see how it goes with respect of other researchers' methods. Their algorithm in turn, is used as a baseline for other methods, creating a virtuous circle. This is only possible thanks to the simple, yet powerful idea of providing shared benchmarks, training and evaluation in a single place.
Improved Modularity: Avalanche has been designed with modularity in mind. As you will learn more about Avalanche, you will realize we have sometimes forego simplicity in favor of modularity and reusability (we hate code replication as you do 🤪). However, we believe this will help us scale in the near future as we collaboratively bring this codebase into maturity.
Increased Efficiency & Scalability: Full-stack researchers & data scientists know this, making your algorithm memory and computationally efficient is tough. Avalanche is already optimized for you, so that you can run your ImageNet continual learning experiment on your 8GB laptop (buy a cooling fan 💨) or even try it on embedded devices of your latest product!
But most of all, Avalanche, can help us standardize our field and work better together, more collaboratively, towards our shared goal of making machines learn over time like humans do.
Avalanche the first experiment of a End-to-end Library for reproducible continual learning research where you can find benchmarks, algorithms, evaluation utilities and much more in the same place.
Let's make it together 👫 a wonderful ride! 🎈
First things first: let's start with a good model!
Welcome to the "Models" tutorial of the "From Zero to Hero" series. In this notebook we will talk about the features offered by the models
Avalanche sub-module.
Every continual learning experiment needs a model to train incrementally. You can use any torch.nn.Module
, even pretrained models. The models
sub-module provides the most commonly used architectures in the CL literature.
You can use any model provided in the Pytorch official ecosystem models as well as the ones provided by pytorchcv!
A continual learning model may change over time. As an example, a classifier may add new units for previously unseen classes, while progressive networks add a new set units after each experience. Avalanche provides DynamicModule
s to support these use cases. DynamicModule
s are torch.nn.Module
s that provide an addition method, adaptation
, that is used to update the model's architecture. The method takes a single argument, the data from the current experience.
For example, an IncrementalClassifier updates the number of output units:
As you can see, after each call to the adaptation
method, the model adds 2 new units to account for the new classes. Notice that no learning occurs at this point since the method only modifies the model's architecture.
Keep in mind that when you use Avalanche strategies you don't have to call the adaptation yourself. Avalanche strategies automatically call the model's adaptation and update the optimizer to include the new parameters.
Some models, such as multi-head classifiers, are designed to exploit task labels. In Avalanche, such models are implemented as MultiTaskModule
s. These are dynamic models (since they need to be updated whenever they encounter a new task) that have an additional task_labels
argument in their forward
method. task_labels
is a tensor with a task id for each sample.
When you use a MultiHeadClassifier
, a new head is initialized whenever a new task is encountered. Avalanche strategies automatically recognize multi-task modules and provide task labels to them.
If you want to define a custom multi-task module you need to override two methods: adaptation
(if needed), and forward_single_task
. The forward
method of the base class will split the mini-batch by task-id and provide single task mini-batches to forward_single_task
.
Alternatively, if you only want to convert a single-head model into a multi-head model, you can use the as_multitask
wrapper, which converts the model for you.
Whenever one or more dynamic modules are nested one inside the other, you must call the recursive_adaptation
method, and if they are nested inside a normal pytorch module (non dynamic), you can call the avalanche_model_adaptation
function. Avalanche strategies will by default adapt the models before training on each experience by calling avalanche_model_adaptation
Design Your Continual Learning Experiments
Welcome to the "Putting All Together" tutorial of the "From Zero to Hero" series. In this part we will summarize the major Avalanche features and how you can put them together for your continual learning experiments.
Here we report a complete example of the Avalanche usage:
Installing Avalanche has Never Been so Simple
Avalanche has been designed for extreme portability and usability. Indeed, it can be run on every OS and native python environment. 💻🍎🐧
you can install Avalanche with pip:
This will install the core version of Avalanche, without extra packages (e.g., object detection support, reinforcement learning support). To install all the extra packages run:
You can install also specific extra packages by specifying the appropriate code name within the square brackets. This is the complete list of options:
Avalanche will raise an error if you need one extra package and will suggest the appropriate package to install.
Note that in some alternatives to bash like zsh you may need to enclose `avalanche-lib[code]` into quotation marks ( " " ), since square brackets are used as special characters.
Warning: by installing the [all] and [extra] versions, the PyTorch version may be limited to <2.* due to the dependencies of those additional packages.
If you want, you can install Avalanche directly from the master branch (latest version) in a single command. Make sure to have pytorch already installed in your environment, then execute
To update avalanche to the latest version, uninstall the package with pip uninstall avalanche-lib
and then execute again the pip install command.
To help us to expand and improve Avalanche, you can install Avalanche in a fresh environment with the command
pip install -e ".[dev]"
This will install in editable mode, so that you can develop and modify the installed Avalanche package. It will also install the "extra" dev dependencies necessary to run tests and build the documentation.
You can run this chapter and play with it on Google Colaboratory: