Creating AvalancheDatasets

The AvalancheDataset is an implementation of the PyTorch Dataset class which comes with many out-of-the-box functionalities. The AvalancheDataset (an its few subclass) are extensively used through the whole Avalanche library as the reference way to manipulate datasets:
The dataset carried by the experience.dataset field is always an AvalancheDataset.
Benchmark creation functions accept AvalancheDatasets to create benchmarks where a finer control over task labels is required.
Internally, benchmarks are created by manipulating AvalancheDatasets.
This first Mini How-To will guide through the main ways you can use to instantiate an AvalancheDataset while the other Mini How-Tos (complete list here) will show how to use its functionalities.
It is warmly recommended to run this page as a notebook using Colab (info at the bottom of this page).
Let's start by installing avalanche:
1
!pip install avalanche-lib
Copied!
AvalancheDataset vs PyTorch Dataset
This mini How-To will guide you through the main ways used to instantiate an AvalancheDataset.
First thing: the base class AvalancheDataset is a wrapper for existing datasets. Only two things must be considered when wrapping an existing dataset:
Apart from the x and y values, the resulting AvalancheDataset will also return a third value: the task label (which defaults to 0).
The wrapped dataset must contain a valid targets field.
The targets field is available is nearly all torchvision datasets. It must be a list containing the label for each data point (usually the y value). In this way, Avalanche can use that field when instantiating benchmarks like the "Class/Task-Incremental* and Domain-Incremental ones.
Avalanche exposes 4 classes of AvalancheDatasets which map exactly the 4 Dataset classes offered by PyTorch:
AvalancheDataset: the base class, which acts a wrapper to existing Dataset instances.
AvalancheTensorDataset: equivalent to PyTorch TesnsorDataset.
AvalancheSubset: equivalent to PyTorch Subset.
AvalancheConcatDataset: equivalent to PyTorch ConcatDataset.
🛠️ Create an AvalancheDataset
Given a dataset (like MNIST), an AvalancheDataset can be instantiated as follows:
1
from avalanche.benchmarks.utils import AvalancheDataset
2
from torchvision.datasets import MNIST
3
​
4
# Instantiate the MNIST train dataset from torchvision
5
mnist_dataset = MNIST('mnist_data', download=True)
6
​
7
# Create the AvalancheDataset
8
mnist_avalanche_dataset = AvalancheDataset(mnist_dataset)
Copied!
Just like any other Dataset, a data point can be obtained using the x, y = dataset[idx] syntax. When obtaining a data point from an AvalancheDataset, an additional third value (the task label) will be returned:
1
# Obtain the first instance from the original dataset
2
x, y = mnist_dataset[0]
3
print(f'x={x}, y={y}')
4
# Output: "x=<PIL.Image.Image image mode=L size=28x28 at 0x7FBEDFDB2430>, y=5"
5
​
6
# Obtain the first instance from the AvalancheDataset
7
x, y, t = mnist_avalanche_dataset[0]
8
print(f'x={x}, y={y}, t={t}')
9
# Output: "x=<PIL.Image.Image image mode=L size=28x28 at 0x7FBEEFD3A850>, y=5, t=0"
Copied!
Useful tip: if you are not sure if you are dealing with a PyTorch Dataset or an AvalancheDataset, or if you want to ignore task labels, you can use this syntax:
1
# You can use "x, y, *_" to manage both kinds of Datasets
2
x, y, *_ = mnist_dataset[0]  # OK
3
x, y, *_ = mnist_avalanche_dataset[0]  # OK
Copied!
The AvalancheTensorDataset
The PyTorch TensorDataset is one of the most useful Dataset classes as it can be used to quickly prototype the data loading part of your code.
A TensorDataset can be wrapped in an AvalancheDataset just like any Dataset, but this is not much convenient, as shown below:
1
import torch
2
from torch.utils.data import TensorDataset
3
​
4
​
5
# Create 10 instances described by 7 features 
6
x_data = torch.rand(10, 7)
7
​
8
# Create the class labels for the 10 instances
9
y_data = torch.randint(0, 5, (10,))
10
​
11
# Create the tensor dataset
12
tensor_dataset = TensorDataset(x_data, y_data)
13
​
14
# Wrap it in an AvalancheDataset
15
wrapped_tensor_dataset = AvalancheDataset(tensor_dataset)
16
​
17
# Obtain the first instance from the dataset
18
x, y, t = wrapped_tensor_dataset[0]
19
print(f'x={x}, y={y}, t={t}')
20
# Output: "x=tensor([0.6329, 0.8495, 0.1853, 0.7254, 0.7893, 0.8079, 0.1106]), y=4, t=0"
Copied!
Instead, it is recommended to use the AvalancheTensorDataset class to get the same result. In this way, you can just skip one intermediate step.
1
from avalanche.benchmarks.utils import AvalancheTensorDataset
2
​
3
# Create the tensor dataset
4
avl_tensor_dataset = AvalancheTensorDataset(x_data, y_data)
5
​
6
# Obtain the first instance from the AvalancheTensorDataset
7
x, y, t = avl_tensor_dataset[0]
8
print(f'x={x}, y={y}, t={t}')
9
# Output: "x=tensor([0.6329, 0.8495, 0.1853, 0.7254, 0.7893, 0.8079, 0.1106]), y=4, t=0"
Copied!
In both cases, AvalancheDataset will automatically populate its targets field by using the values from the second Tensor (which usually contains the Y values). This behaviour can be customized by passing a custom targets constructor parameter (by either passing a list of targets or the index of the Tensor to use).
The cell below shows the content of the target field of the dataset created in the cell above. Notice that the targets field has been filled with the content of the second Tensor (y_data).
1
# Check the targets field
2
print('y_data=', y_data)
3
 # Output: "y_data= tensor([4, 3, 3, 2, 0, 1, 3, 3, 3, 2])"
4
​
5
print('targets field=', avl_tensor_dataset.targets)
6
# Output: "targets field= [tensor(4), tensor(3), tensor(3), tensor(2), 
7
#          tensor(0), tensor(1), tensor(3), tensor(3), tensor(3), tensor(2)]"
Copied!
The AvalancheSubset and AvalancheConcatDataset classes
Avalanche offers the AvalancheSubset and AvalancheConcatDataset implementations that extend the functionalities of PyTorch Subset and ConcatDataset.
Regarding the subsetting operation, AvalancheSubset behaves in the same way the PyTorch Subset class does: both implementations accept a dataset and a list of indices as parameters. The resulting Subset is not a copy of the dataset, it's just a view. This is similar to creating a view of a NumPy array by passing a list of indexes using the numpy_array[list_of_indices] syntax. This can be used to both create a smaller dataset and to change the order of data points in the dataset.
Here we create a toy dataset in which each X and Y values are ints. We then obtain a subset of it by creating an AvalancheSubset:
1
from avalanche.benchmarks.utils import AvalancheSubset
2
​
3
# Define the X values of 10 instances (each instance is an int)
4
x_data_toy = [50, 51, 52, 53, 54, 55, 56, 57, 58, 59]
5
​
6
# Define the class labels for the 10 instances
7
y_data_toy = [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
8
​
9
# Create  the tensor dataset
10
# Note: AvalancheSubset can also be applied to PyTorch TensorDataset directly!
11
# However, note that PyTorch TensorDataset doesn't support Python lists...
12
# ... (it only supports Tensors) while AvalancheTensorDataset does.
13
toy_dataset = AvalancheTensorDataset(x_data_toy, y_data_toy) 
14
​
15
# Define the indices for the subset
16
# Here we want to obtain a subset containing only the data points...
17
# ... at indices 0, 5, 8, 2 (in this specific order)
18
subset_indices = [0, 5, 8, 2]
19
​
20
# Create the subset
21
avl_subset = AvalancheSubset(toy_dataset, indices=subset_indices)
22
print('The subset contains', len(avl_subset), 'instances.')
23
# Output: "The subset contains 4 instances."
24
​
25
# Obtain instances from the AvalancheSubset
26
for x, y, t in avl_subset:
27
    print(f'x={x}, y={y}, t={t}')
28
# Output:
29
# x=50, y=10, t=0
30
# x=55, y=15, t=0
31
# x=58, y=18, t=0
32
# x=52, y=12, t=0
Copied!
Concatenation is even simpler. Just like with PyTorch ConcatDataset, one can easily concatentate datasets with AvalancheConcatDataset.
Both AvalancheConcatDataset and PyTorch ConcatDataset accept a list of datasets to concatenate.
1
from avalanche.benchmarks.utils import AvalancheConcatDataset
2
​
3
# Define the 2 datasets to be concatenated
4
x_data_toy_1 = [50, 51, 52, 53, 54]
5
y_data_toy_1 = [10, 11, 12, 13, 14]
6
x_data_toy_2 = [60, 61, 62, 63, 64]
7
y_data_toy_2 = [20, 21, 22, 23, 24]
8
​
9
# Create the datasets
10
toy_dataset_1 = AvalancheTensorDataset(x_data_toy_1, y_data_toy_1) 
11
toy_dataset_2 = AvalancheTensorDataset(x_data_toy_2, y_data_toy_2) 
12
​
13
# Create the concat dataset
14
avl_concat = AvalancheConcatDataset([toy_dataset_1, toy_dataset_2])
15
print('The concat dataset contains', len(avl_concat), 'instances.')
16
# Output: "The concat dataset contains 10 instances."
17
​
18
# Obtain instances from the AvalancheConcatDataset
19
for x, y, t in avl_concat:
20
    print(f'x={x}, y={y}, t={t}')
21
# Output:
22
# x=51, y=11, t=0
23
# x=52, y=12, t=0
24
# x=53, y=13, t=0
25
# x=54, y=14, t=0
26
# x=60, y=20, t=0
27
# x=61, y=21, t=0
28
# x=62, y=22, t=0
29
# x=63, y=23, t=0
30
# x=64, y=24, t=0
Copied!
Dataset Creation wrap-up
This Mini How-To showed you how to create instances of AvalancheDataset (and its subclasses).
Other Mini How-Tos will guide you through the functionalities offered by AvalancheDataset. The list of Mini How-Tos can be found here.
🤝 Run it on Google Colab
You can run this chapter and play with it on Google Colaboratory by clicking here: ​
​​

Preamble: PyTorch Datasets

Advanced Transformations

Last modified 22d ago

Export as PDF

Copy link

Contents

AvalancheDataset vs PyTorch Dataset

🛠️ Create an AvalancheDataset

The AvalancheTensorDataset

The AvalancheSubset and AvalancheConcatDataset classes

Dataset Creation wrap-up

🤝 Run it on Google Colab