Benchmarks

Welcome to the "benchmarks" tutorial of the "From Zero to Hero" series. In this part we will present the functionalities offered by the Benchmarks module.
1
!pip install git+https://github.com/ContinualAI/avalanche.git
Copied!
🎯 Nomenclature
First off, let's clarify a bit the nomenclature we are going to use, introducing the following terms: Datasets, Scenarios, Benchmarks and Generators.
By Dataset we mean a collection of examples that can be used for training or testing purposes but not already organized to be processed as a stream of batches or tasks. Since Avalanche is based on Pytorch, our Datasets are torch.utils.Datasets objects.
By Scenario we mean a particular setting, i.e. specificities about the continual stream of data, a continual learning algorithm will face.
By Benchmark we mean a well-defined and carefully thought combination of a scenario with one or multiple datasets that we can use to asses our continual learning algorithms.
By Generator we mean a function that given a specific scenario and a dataset can generate a Benchmark.
📚 The Benchmarks Module
The bechmarks module offers 3 types of utils:
Datasets: all the Pytorch datasets plus additional ones prepared by our community and particularly interesting for continual learning.
Classic Benchmarks: classic benchmarks used in CL litterature ready to be used with great flexibility.
Benchmarks Generators: a set of functions you can use to create your own benchmark starting from any kind of data and scenario. In particular, we distinguish two type of generators: Specific and Generic. The first ones will let you create a benchmark based on a clear scenarios and Pytorch dataset(s); the latters, instead, are more generic and flexible, both in terms of scenario definition then in terms of type of data they can manage.
Specific:
nc_benchmark: given one or multiple datasets it creates a benchmark instance based on scenarios where New Classes (NC) are encountered over time. Notable scenarios that can be created using this utility include Class-Incremental, Task-Incremental and Task-Agnostic scenarios.
ni_benchmark: it creates a benchmark instance based on scenarios where New Instances (NI), i.e. new examples of the same classes are encountered over time. Notable scenarios that can be created using this utility include Domain-Incremental scenarios.
Generic:
filelist_benchmark: It creates a benchmark instance given a list of filelists.
paths_benchmark: It creates a benchmark instance given a list of file paths and class labels.
tensors_benchmark: It creates a benchmark instance given a list of tensors.
dataset_benchmark: It creates a benchmark instance given a list of pytorch datasets.
But let's see how we can use this module in practice!
🖼️ Datasets
Let's start with the Datasets. As we previously hinted, in Avalanche you'll find all the standard Pytorch Datasets available in the torchvision package as well as a few others that are useful for continual learning but not already officially available within the Pytorch ecosystem.
1
import torch
2
import torchvision
3
from avalanche.benchmarks.datasets import MNIST, FashionMNIST, KMNIST, EMNIST, \
4
QMNIST, FakeData, CocoCaptions, CocoDetection, LSUN, ImageNet, CIFAR10, \
5
CIFAR100, STL10, SVHN, PhotoTour, SBU, Flickr8k, Flickr30k, VOCDetection, \
6
VOCSegmentation, Cityscapes, SBDataset, USPS, Kinetics400, HMDB51, UCF101, \
7
CelebA, CORe50Dataset, TinyImagenet, CUB200, OpenLORIS
8
​
9
# As we would simply do with any Pytorch dataset we can create the train and 
10
# test sets from it. We could use any of the above imported Datasets, but let's
11
# just try to use the standard MNIST.
12
train_MNIST = MNIST(
13
    './data/mnist', train=True, download=True, transform=torchvision.transforms.ToTensor()
14
)
15
test_MNIST = MNIST(
16
    './data/mnist', train=False, download=True, transform=torchvision.transforms.ToTensor()
17
)
18
​
19
# Given these two sets we can simply iterate them to get the examples one by one
20
for i, example in enumerate(train_MNIST):
21
    pass
22
print("Num. examples processed: {}".format(i))
23
​
24
# or use a Pytorch DataLoader
25
train_loader = torch.utils.data.DataLoader(
26
    train_MNIST, batch_size=32, shuffle=True
27
)
28
for i, (x, y) in enumerate(train_loader):
29
    pass
30
print("Num. mini-batch processed: {}".format(i))
Copied!
Of course also the basic utilities ImageFolder and DatasetFolder can be used. These are two classes that you can use to create a Pytorch Dataset directly from your files (following a particular structure). You can read more about these in the Pytorch official documentation here.
We also provide an additional FilelistDataset and AvalancheDataset classes. The former to construct a dataset from a filelist (caffe style) pointing to files anywhere on the disk. The latter to augment the basic Pytorch Dataset functionalities with an extention to better deal with a stack of transformations to be used during train and test.
1
from avalanche.benchmarks.utils import ImageFolder, DatasetFolder, FilelistDataset, AvalancheDataset
Copied!
🛠️ Benchmarks Basics
The Avalanche benchmarks (instances of the Scenario class), contains several attributes that characterize the benchmark. However, the most important ones are the train and test streams.
In Avalanche we often suppose to have access to these two parallel stream of data (even though some benchmarks may not provide such feature, but contain just a unique test set).
Each of these streams are iterable, indexable and sliceable objects that are composed of unique experiences. Experiences are batch of data (or "tasks") that can be provided with or without a specific task label.
Efficiency
It is worth mentioning that all the data belonging to a stream are not loaded into the RAM beforehand. Avalanche actually loads the data when a specific mini-batches are requested at training/test time based on the policy defined by each Dataset implementation.
This means that memory requirements are very low, while the speed is guaranteed by a multi-processing data loading system based on the one defined in Pytorch.
Scenarios
So, as we have seen, each scenario object in Avalanche has several useful attributes that characterizes the benchmark, including the two important train and test streams. Let's check what you can get from a scenario object more in details:
1
from avalanche.benchmarks.classic import SplitMNIST
2
split_mnist = SplitMNIST(n_experiences=5, seed=1)
3
​
4
# Original train/test sets
5
print('--- Original datasets:')
6
print(split_mnist.original_train_dataset)
7
print(split_mnist.original_test_dataset)
8
​
9
# A list describing which training patterns are assigned to each experience.
10
# Patterns are identified by their id w.r.t. the dataset found in the
11
# original_train_dataset field.
12
print('--- Train patterns assignment:')
13
print(split_mnist.train_exps_patterns_assignment)
14
​
15
# A list describing which test patterns are assigned to each experience.
16
# Patterns are identified by their id w.r.t. the dataset found in the
17
# original_test_dataset field
18
print('--- Test patterns assignment:')
19
print(split_mnist.test_exps_patterns_assignment)
20
​
21
# the task label of each experience.
22
print('--- Task labels:')
23
print(split_mnist.task_labels)
24
​
25
# train and test streams
26
print('--- Streams:')
27
print(split_mnist.train_stream)
28
print(split_mnist.test_stream)
29
​
30
# A list that, for each experience (identified by its index/ID),
31
# stores a set of the (optionally remapped) IDs of classes of patterns
32
# assigned to that experience.
33
print('--- Classes in each experience:')
34
split_mnist.classes_in_experience
Copied!
Train and Test Streams
The train and test streams can be used for training and testing purposes, respectively. This is what you can do with these streams:
1
# each stream has a name: "train" or "test"
2
train_stream = split_mnist.train_stream
3
print(train_stream.name)
4
​
5
# we have access to the scenario from which the stream was taken
6
train_stream.benchmark
7
​
8
# we can slice and reorder the stream as we like!
9
substream = train_stream[0]
10
substream = train_stream[0:2]
11
substream = train_stream[0,2,1]
12
​
13
len(substream)
Copied!
Experiences
Each stream can in turn be treated as an iterator that produces a unique experience, containing all the useful data regarding a batch or task in the continual stream our algorithms will face. Check out how can you use these experiences below:
1
# we get the first experience
2
experience = train_stream[0]
3
​
4
# task label and dataset are the main attributes
5
t_label = experience.task_label
6
dataset = experience.dataset
7
​
8
# but you can recover additional info
9
experience.current_experience
10
experience.classes_in_this_experience
11
experience.classes_seen_so_far
12
experience.previous_classes
13
experience.future_classes
14
experience.origin_stream
15
experience.benchmark
16
​
17
# As always, we can iterate over it normally or with a pytorch
18
# data loader.
19
# For instance, we can use tqdm to add a progress bar.
20
from tqdm import tqdm
21
for i, data in enumerate(tqdm(dataset)):
22
  pass
23
print("\nNumber of examples:", i + 1)
24
print("Task Label:", t_label)
Copied!
🏛️ Classic Benchmarks
Now that we know how our benchmarks work in general through scenarios, streams and experiences objects, in this section we are going to explore common benchmarks already available for you with one line of code yet flexible enough to allow proper tuning based on your needs:
1
from avalanche.benchmarks.classic import CORe50, SplitTinyImageNet, \
2
SplitCIFAR10, SplitCIFAR100, SplitCIFAR110, SplitMNIST, RotatedMNIST, \
3
PermutedMNIST, SplitCUB200, SplitImageNet
4
​
5
# creating PermutedMNIST (Task-Incremental)
6
perm_mnist = PermutedMNIST(
7
    n_experiences=2,
8
    seed=1234,
9
)
Copied!
Many of the classic benchmarks will download the original datasets they are based on automatically and put it under the "~/.avalanche/data" directory.
How to Use the Benchmarks
Let's see now how we can use the classic benchmark or the ones that you can create through the generators (see next section). For example, let's try out the classic PermutedMNIST benchmark (Task-Incremental scenario).
1
# creating the benchmark instance (scenario object)
2
perm_mnist = PermutedMNIST(
3
  n_experiences=3,
4
  seed=1234,
5
)
6
​
7
# recovering the train and test streams
8
train_stream = perm_mnist.train_stream
9
test_stream = perm_mnist.test_stream
10
​
11
# iterating over the train stream
12
for experience in train_stream:
13
  print("Start of task ", experience.task_label)
14
  print('Classes in this task:', experience.classes_in_this_experience)
15
​
16
  # The current Pytorch training set can be easily recovered through the
17
  # experience
18
  current_training_set = experience.dataset
19
  # ...as well as the task_label
20
  print('Task {}'.format(experience.task_label))
21
  print('This task contains', len(current_training_set), 'training examples')
22
​
23
  # we can recover the corresponding test experience in the test stream
24
  current_test_set = test_stream[experience.current_experience].dataset
25
  print('This task contains', len(current_test_set), 'test examples')
Copied!
🐣 Benchmarks Generators
What if we want to create a new benchmark that is not present in the "Classic" ones? Well, in that case Avalanche offer a number of utilites that you can use to create your own benchmark with maximum flexibility: the benchmarks generators!
Specific Generators
The specific scenario generators are useful when starting from one or multiple Pytorch datasets you want to create a "New Instances" or "New Classes" benchmark: i.e. it supports the easy and flexible creation of a Domain-Incremental, Class-Incremental or Task-Incremental scenarios among others.
For the New Classes scenario you can use the following function:
nc_benchmark
for the New Instances:
ni_benchmark
1
from avalanche.benchmarks.generators import nc_benchmark, ni_benchmark
Copied!
Let's start by creating the MNIST dataset object as we would normally do in Pytorch:
1
from torchvision.transforms import Compose, ToTensor, Normalize, RandomCrop
2
train_transform = Compose([
3
    RandomCrop(28, padding=4),
4
    ToTensor(),
5
    Normalize((0.1307,), (0.3081,))
6
])
7
​
8
test_transform = Compose([
9
    ToTensor(),
10
    Normalize((0.1307,), (0.3081,))
11
])
12
​
13
mnist_train = MNIST(
14
    './data/mnist', train=True, download=True, transform=train_transform
15
)
16
mnist_test = MNIST(
17
    './data/mnist', train=False, download=True, transform=test_transform
18
)
Copied!
Then we can, for example, create a new benchmark based on MNIST and the classic Domain-Incremental scenario:
1
scenario = ni_benchmark(
2
    mnist_train, mnist_test, n_experiences=10, shuffle=True, seed=1234,
3
    balance_experiences=True
4
)
5
​
6
train_stream = scenario.train_stream
7
​
8
for experience in train_stream:
9
    t = experience.task_label
10
    exp_id = experience.current_experience
11
    training_dataset = experience.dataset
12
    print('Task {} batch {} -> train'.format(t, exp_id))
13
    print('This batch contains', len(training_dataset), 'patterns')
Copied!
Or, we can create a benchmark based on MNIST and the Class-Incremental (what's commonly referred to as "Split-MNIST" benchmark):
1
scenario = nc_benchmark(
2
    mnist_train, mnist_test, n_experiences=10, shuffle=True, seed=1234,
3
    task_labels=False
4
)
5
​
6
train_stream = scenario.train_stream
7
​
8
for experience in train_stream:
9
    t = experience.task_label
10
    exp_id = experience.current_experience
11
    training_dataset = experience.dataset
12
    print('Task {} batch {} -> train'.format(t, exp_id))
13
    print('This batch contains', len(training_dataset), 'patterns')
Copied!
Generic Generators
Finally, if you cannot create your ideal benchmark since it does not fit well in the aforementioned new classes or new instances scenarios, you can always use our generic generators:
filelist_benchmark
paths_benchmark
dataset_benchmark
tensors_benchmark
1
from avalanche.benchmarks.generators import filelist_benchmark, dataset_benchmark, \
2
                                            tensors_benchmark, paths_benchmark
Copied!
Let's start with the filelist_benchmark utility. This function is particularly useful when it is important to preserve a particular order of the patterns to be processed (for example if they are frames of a video), or in general if we have data scattered around our drive and we want to create a sequence of batches/tasks providing only a txt file containing the list of their paths.
For Avalanche we follow the same format of the Caffe filelists ("path class_label"):
/path/to/a/file.jpg 0
/path/to/another/file.jpg 0
...
/path/to/another/file.jpg M
/path/to/another/file.jpg M
...
/path/to/another/file.jpg N
/path/to/another/file.jpg N
So let's download the classic "Cats vs Dogs" dataset as an example:
1
!wget -N --no-check-certificate \
2
    https://storage.googleapis.com/mledu-datasets/cats_and_dogs_filtered.zip
3
!unzip -q -o cats_and_dogs_filtered.zip
Copied!
You can now see in the content directory on colab the image we downloaded. We are now going to create the filelists and then use the filelist_benchmark function to create our benchmark:
1
import os
2
# let's create the filelists since we don't have it
3
dirpath = "cats_and_dogs_filtered/train"
4
​
5
for filelist, rel_dir, t_label in zip(
6
        ["train_filelist_00.txt", "train_filelist_01.txt"],
7
        ["cats", "dogs"],
8
        [0, 1]):
9
    # First, obtain the list of files
10
    filenames_list = os.listdir(os.path.join(dirpath, rel_dir))
11
​
12
    # Create the text file containing the filelist
13
    # Filelists must be in Caffe-style, which means
14
    # that they must define path in the format:
15
    #
16
    # relative_path_img1 class_label_first_img
17
    # relative_path_img2 class_label_second_img
18
    # ...
19
    #
20
    # For instance:
21
    # cat/cat_0.png 1
22
    # dog/dog_54.png 0
23
    # cat/cat_3.png 1
24
    # ...
25
    # 
26
    # Paths are relative to a root path
27
    # (specified when calling filelist_benchmark)
28
    with open(filelist, "w") as wf:
29
        for name in filenames_list:
30
            wf.write(
31
                "{} {}\n".format(os.path.join(rel_dir, name), t_label)
32
            )
33
​
34
# Here we create a GenericCLScenario ready to be iterated
35
generic_scenario = filelist_benchmark(
36
   dirpath,  
37
   ["train_filelist_00.txt", "train_filelist_01.txt"],
38
   ["train_filelist_00.txt"],
39
   task_labels=[0, 0],
40
   complete_test_set_only=True,
41
   train_transform=ToTensor(),
42
   eval_transform=ToTensor()
43
)
Copied!
In the previous cell we created a benchmark instance starting from file lists. However, paths_benchmark is a better choice if you already have the list of paths directly loaded in memory:
1
train_experiences = []
2
for rel_dir, label in zip(
3
        ["cats", "dogs"],
4
        [0, 1]):
5
    # First, obtain the list of files
6
    filenames_list = os.listdir(os.path.join(dirpath, rel_dir))
7
​
8
    # Don't create a file list: instead, we create a list of 
9
    # paths + class labels
10
    experience_paths = []
11
    for name in filenames_list:
12
      instance_tuple = (os.path.join(dirpath, rel_dir, name), label)
13
      experience_paths.append(instance_tuple)
14
    train_experiences.append(experience_paths)
15
​
16
# Here we create a GenericCLScenario ready to be iterated
17
generic_scenario = paths_benchmark(
18
   train_experiences,
19
   [train_experiences[0]],  # Single test set
20
   task_labels=[0, 0],
21
   complete_test_set_only=True,
22
   train_transform=ToTensor(),
23
   eval_transform=ToTensor()
24
)
Copied!
Let us see how we can use the dataset_benchmark utility, where we can use several PyTorch datasets as different batches or tasks. This utility expectes a list of datasets for the train, test (and other custom) streams. Each dataset will be used to create an experience:
1
train_cifar10 = CIFAR10(
2
    './data/cifar10', train=True, download=True
3
)
4
test_cifar10 = CIFAR10(
5
    './data/cifar10', train=False, download=True
6
)
7
​
8
generic_scenario = dataset_benchmark(
9
    [train_MNIST, train_cifar10],
10
    [test_MNIST, test_cifar10]
11
)
Copied!
Adding task labels can be achieved by wrapping each datasets using AvalancheDataset. Apart from task labels, AvalancheDataset allows for more control over transformations and offers an ever growing set of utilities (check the documentation for more details).
1
# Alternatively, task labels can also be a list (or tensor)
2
# containing the task label of each pattern
3
​
4
train_MNIST_task0 = AvalancheDataset(train_cifar10, task_labels=0)
5
test_MNIST_task0 = AvalancheDataset(test_cifar10, task_labels=0)
6
​
7
train_cifar10_task1 = AvalancheDataset(train_cifar10, task_labels=1)
8
test_cifar10_task1 = AvalancheDataset(test_cifar10, task_labels=1)
9
​
10
scenario_custom_task_labels = dataset_benchmark(
11
    [train_MNIST_task0, train_cifar10_task1],
12
    [test_MNIST_task0, test_cifar10_task1]
13
)
14
​
15
print('Without custom task labels:',
16
      generic_scenario.train_stream[1].task_label)
17
​
18
print('With custom task labels:',
19
      scenario_custom_task_labels.train_stream[1].task_label)
Copied!
And finally, the tensors_benchmark generator:
1
pattern_shape = (3, 32, 32)
2
​
3
# Definition of training experiences
4
# Experience 1
5
experience_1_x = torch.zeros(100, *pattern_shape)
6
experience_1_y = torch.zeros(100, dtype=torch.long)
7
​
8
# Experience 2
9
experience_2_x = torch.zeros(80, *pattern_shape)
10
experience_2_y = torch.ones(80, dtype=torch.long)
11
​
12
# Test experience
13
# For this example we define a single test experience,
14
# but "tensors_benchmark" allows you to define even more than one!
15
test_x = torch.zeros(50, *pattern_shape)
16
test_y = torch.zeros(50, dtype=torch.long)
17
​
18
generic_scenario = tensors_benchmark(
19
    train_tensors=[(experience_1_x, experience_1_y), (experience_2_x, experience_2_y)],
20
    test_tensors=[(test_x, test_y)],
21
    task_labels=[0, 0],  # Task label of each train exp
22
    complete_test_set_only=True
23
)
Copied!
This completes the "Benchmark" tutorial for the "From Zero to Hero" series. We hope you enjoyed it!
🤝 Run it on Google Colab
You can run this chapter and play with it on Google Colaboratory: ​
​​
From Zero to Hero Tutorial - Previous
Models
Next - From Zero to Hero Tutorial
Training
Last modified 9d ago
Export as PDF
Copy link
Contents
🎯 Nomenclature
📚 The Benchmarks Module
🖼️ Datasets
🛠️ Benchmarks Basics
🏛️ Classic Benchmarks
How to Use the Benchmarks
🐣 Benchmarks Generators
Specific Generators
Generic Generators
🤝 Run it on Google Colab