Welcome to the "Logging" tutorial of the "From Zero to Hero" series. In this part we will present the functionalities offered by the Avalanche
!pip install git+https://github.com/ContinualAI/avalanche.git
In the previous tutorial we have learned how to evaluate a continual learning algorithm in Avalanche, through different metrics that can be used off-the-shelf via the Evaluation Plugin or stand-alone. However, computing metrics and collecting results, may not be enough at times.
While running complex experiments with long waiting times, logging results over-time is fundamental to "babysit" your experiments in real-time, or even understand what went wrong in the aftermath.
This is why in Avalanche we decided to put a strong emphasis on logging and provide a number of loggers that can be used with any set of metrics!
Avalanche at the moment supports three main Loggers:
InteractiveLogger: This logger provides a nice progress bar and displays real-time metrics results in an interactive way (meant for
TextLogger: This logger, mostly intended for file logging, is the plain text version of the
InteractiveLogger. Keep in mind that it may be very verbose.
TensorboardLogger: It logs all the metrics on Tensorboard in real-time. Perfect for real-time plotting.
In order to keep track of when each metric value has been logged, we leverage a
global counter. You can see the
global counter reported in the x axis of the logged plots.
global counter is an ever-increasing value which starts from 0 and it is increased by one each time a training or evaluation iteration is performed (i.e. after each training or evaluation minibatch). The
global counter is updated automatically by the strategy. It should be reset by creating a new instance of the strategy.
from torch.optim import SGDfrom torch.nn import CrossEntropyLossfrom avalanche.benchmarks.classic import SplitMNISTfrom avalanche.evaluation.metrics import forgetting_metrics, \accuracy_metrics, loss_metrics, timing_metrics, cpu_usage_metrics, \confusion_matrix_metrics, disk_usage_metricsfrom avalanche.models import SimpleMLPfrom avalanche.logging import InteractiveLogger, TextLogger, TensorboardLoggerfrom avalanche.training.plugins import EvaluationPluginfrom avalanche.training.strategies import Naivescenario = SplitMNIST(n_experiences=5)# MODEL CREATIONmodel = SimpleMLP(num_classes=scenario.n_classes)# DEFINE THE EVALUATION PLUGIN and LOGGERS# The evaluation plugin manages the metrics computation.# It takes as argument a list of metrics, collectes their results and returns# them to the strategy it is attached to.# log to Tensorboardtb_logger = TensorboardLogger()# log to text filetext_logger = TextLogger(open('log.txt', 'a'))# print to stdoutinteractive_logger = InteractiveLogger()eval_plugin = EvaluationPlugin(accuracy_metrics(minibatch=True, epoch=True, experience=True, stream=True),loss_metrics(minibatch=True, epoch=True, experience=True, stream=True),timing_metrics(epoch=True, epoch_running=True),cpu_usage_metrics(experience=True),forgetting_metrics(experience=True, stream=True),confusion_matrix_metrics(num_classes=scenario.n_classes, save_image=False,stream=True),disk_usage_metrics(minibatch=True, epoch=True, experience=True, stream=True),loggers=[interactive_logger, text_logger, tb_logger])# CREATE THE STRATEGY INSTANCE (NAIVE)cl_strategy = Naive(model, SGD(model.parameters(), lr=0.001, momentum=0.9),CrossEntropyLoss(), train_mb_size=500, train_epochs=1, eval_mb_size=100,evaluator=eval_plugin)# TRAINING LOOPprint('Starting experiment...')results = for experience in scenario.train_stream:print("Start of experience: ", experience.current_experience)print("Current Classes: ", experience.classes_in_this_experience)# train returns a dictionary which contains all the metric valuesres = cl_strategy.train(experience)print('Training completed')print('Computing accuracy on the whole test set')# test also returns a dictionary which contains all the metric valuesresults.append(cl_strategy.eval(scenario.test_stream))
If the available loggers are not sufficient to suit your needs, you can always implement a custom logger by specializing the behaviors of the
StrategyLogger base class.
This completes the "Logging" tutorial for the "From Zero to Hero" series. We hope you enjoyed it!
You can run this chapter and play with it on Google Colaboratory: