How to do research and visualize progress

In this tutorial, we will show how to use BaaL for research ie. when we know the labels. We will introduce notions such as dataset management, MC-Dropout, BALD. If you need more documentation, be sure to check our Additional resources section below!

BaaL can be used on a variety of research domains:

  • Active Learning

  • Uncertainty estimation

  • Fairness, Accountability, Transparency and Ethics (FATE)

  • And more!

Today we will focus on a simple example with CIFAR10 and we will animate the progress of active learning!

Requirements

In addition to BaaL standard requirements, you will need to install:

  • MulticoreTSNE

  • Matplotlib

Additional resources

  • More info on the inner working of Active Learning Dataset here.

  • To know more about Bayesian deep learning please see our Literature review.

Let’s do this!

[1]:
# Let's start with a bunch of imports.

import random
from copy import deepcopy
from dataclasses import dataclass

import numpy as np
import torch
import torch.backends
from torch import optim
from torch.hub import load_state_dict_from_url
from torch.nn import CrossEntropyLoss
from torchvision import datasets
from torchvision import models
from torchvision.transforms import transforms
from tqdm.autonotebook import tqdm

from baal.active import get_heuristic, ActiveLearningDataset
from baal.active.active_loop import ActiveLearningLoop
from baal.bayesian.dropout import patch_module
from baal.modelwrapper import ModelWrapper

def vgg16(num_classes):
    model = models.vgg16(pretrained=False, num_classes=num_classes)
    weights = load_state_dict_from_url('https://download.pytorch.org/models/vgg16-397923af.pth')
    weights = {k: v for k, v in weights.items() if 'classifier.6' not in k}
    model.load_state_dict(weights, strict=False)
    return model

Dataset management and the pool

At many places in our library, we mention the pool, but what is it? The pool is simply the set of unlabelled examples, the example that the model has not deemed important enough to be labelled.

In BaaL, the object that manages this is called the baal.active.ActiveLearningDataset. You supply it with your dataset and by default, everything is unlabelled.

al_dataset = ActiveLearningDataset(your_dataset)

You can then start with the initial number of examples that are labelled randomly. Let’s label 100 examples.

al_dataset.label_randomly(100)

When iterating over al_dataset, you will only get the labelled examples. If you need to work on the pool, then you can call al_dataset.pool which would return it.

How can I disable data augmentation when iterating on the pool?

Disabling data augmentation when computing uncertainty is preferable, we wouldn’t want to disrupt the uncertainty estimation. Fortunately, BaaL can help you with that.

In ActiveLearningDataset, we can supply the attribute pool_specifics with a dictionary of what to modify in the pool. For example, my Dataset has an attribute transform which applies the data augmentation. I can modify with:

ActiveLearningDataset(your_dataset, pool_specifics:{'transform': test_transform}

where test_transform is the test version of transform without data augmentation.

[2]:
"""
We will make an adapter so that `pool_specifics` modifies the transform correctly.
Because the training set is now a torchdata.Subset, modifying the `transform` attribute is harder.
"""

import torch.utils.data as torchdata


class TransformAdapter(torchdata.Subset):

    @property
    def transform(self):
        if hasattr(self.dataset, 'transform'):
            return self.dataset.transform
        else:
            raise AttributeError()

    @transform.setter
    def transform(self, transform):
        if hasattr(self.dataset, 'transform'):
            self.dataset.transform = transform

Here we define our Experiment configuration, this can come from your favorite experiment manager like MLFlow. BaaL does not expect a particular format as all arguments are supplied.

[3]:


@dataclass
class ExperimentConfig:
    epoch: int = 20000//100
    batch_size: int = 32
    initial_pool: int = 512
    query_size: int = 100
    lr: float = 0.001
    heuristic: str = 'bald'
    iterations: int = 40
    training_duration: int = 10

Problem definition

We will perform active learning on a toy dataset, CIFAR-3 where we only keep dogs, cats and airplanes. This will make visualization easier.

[4]:
def get_datasets(initial_pool):
    """
    Let's create a subset of CIFAR10 named CIFAR3, so that we can visualize thing better.

    We will only select the classes airplane, cat and dog.

    Args:
        initial_pool: Amount of labels to start with.

    Returns:
        ActiveLearningDataset, Dataset, the training and test set.
    """
    # airplane, cat, dog
    classes_to_keep = [0, 3, 5]
    transform = transforms.Compose(
        [transforms.Resize((32, 32)),
         transforms.RandomHorizontalFlip(),
         transforms.RandomRotation(30),
         transforms.ToTensor(),
         transforms.Normalize(3 * [0.5], 3 * [0.5]), ])
    test_transform = transforms.Compose(
        [
            transforms.Resize((32, 32)),
            transforms.ToTensor(),
            transforms.Normalize(3 * [0.5], 3 * [0.5]),
        ]
    )
    train_ds = datasets.CIFAR10('.', train=True,
                                transform=transform, target_transform=None, download=True)

    train_mask = np.where([y in classes_to_keep for y in train_ds.targets])[0]
    train_ds = TransformAdapter(train_ds, train_mask)

    # In a real application, you will want a validation set here.
    test_set = datasets.CIFAR10('.', train=False,
                                transform=test_transform, target_transform=None, download=True)
    test_mask = np.where([y in classes_to_keep for y in test_set.targets])[0]
    test_set = TransformAdapter(test_set, test_mask)

    # Here we set `pool_specifics`, where we set the transform attribute for the pool.
    active_set = ActiveLearningDataset(train_ds, pool_specifics={'transform': test_transform})

    # We start labeling randomly.
    active_set.label_randomly(initial_pool)
    return active_set, test_set

Creating our experiment

We are now ready to instantiate all of our components:

  1. Our ActiveLearningDataset and a test dataset.

  2. Our heuristic (BALD)

  3. Our model and its criterion.

BaaL simplifies your experiments by providing ModelWrapper and ActiveLearningLoop.

  • ModelWrapper

  • Performs MC sampling efficiently

  • Training/testing loops

  • ActiveLearningLoop

  • Will make prediction on the pool and label the most uncertain examples.

[5]:
hyperparams = ExperimentConfig()
use_cuda = torch.cuda.is_available()
torch.backends.cudnn.benchmark = True
random.seed(1337)
torch.manual_seed(1337)
if not use_cuda:
    print("warning, the experiments would take ages to run on cpu")

# Get datasets
active_set, test_set = get_datasets(hyperparams.initial_pool)

# Get our model.
heuristic = get_heuristic(hyperparams.heuristic)
criterion = CrossEntropyLoss()
model = vgg16(num_classes=10)

# change dropout layer to MCDropout
model = patch_module(model)

if use_cuda:
    model.cuda()
optimizer = optim.SGD(model.parameters(), lr=hyperparams.lr, momentum=0.9)

# Wraps the model into a usable API.
model = ModelWrapper(model, criterion)

# for ActiveLearningLoop we use a smaller batchsize
# since we will stack predictions to perform MCDropout.
active_loop = ActiveLearningLoop(active_set,
                                 model.predict_on_dataset,
                                 heuristic,
                                 hyperparams.query_size,
                                 batch_size=1,
                                 iterations=hyperparams.iterations,
                                 use_cuda=use_cuda,
                                 verbose=False)

# We will reset the weights at each active learning step so we make a copy.
init_weights = deepcopy(model.state_dict())
Files already downloaded and verified
Files already downloaded and verified

What is an active learning loop

An active learning loop is the process of:

  1. Training

  2. Estimate uncertainty on the pool

  3. Label the most uncertain examples.

[6]:
labelling_progress = active_set._labelled.copy().astype(np.uint16)
for epoch in tqdm(range(hyperparams.epoch)):
    # Load the initial weights.
    model.load_state_dict(init_weights)

    # Train the model on the currently labelled dataset.
    _ = model.train_on_dataset(active_set, optimizer=optimizer, batch_size=hyperparams.batch_size,
                                 use_cuda=use_cuda, epoch=hyperparams.training_duration)

    # Get test NLL!
    model.test_on_dataset(test_set, hyperparams.batch_size, use_cuda,
                          average_predictions=hyperparams.iterations)
    metrics = model.metrics

    # We can now label the most uncertain samples according to our heuristic.
    should_continue = active_loop.step()
    # Keep track of progress
    labelling_progress += active_set._labelled.astype(np.uint16)
    if not should_continue:
            break

    test_loss = metrics['test_loss'].value
    logs = {
        "test_nll": test_loss,
        "epoch": epoch,
        "Next Training set size": len(active_set)
    }
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T15:26:02.888118Z [info     ] Starting training              dataset=512 epoch=10
/opt/conda/lib/python3.9/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 1, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(
/opt/conda/lib/python3.9/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T15:26:32.784181Z [info     ] Training complete              train_loss=0.42513248324394226
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T15:26:32.785716Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T15:26:42.387235Z [info     ] Evaluation complete            test_loss=0.5483408570289612
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T15:26:42.391419Z [info     ] Start Predict                  dataset=14488
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T15:28:48.686742Z [info     ] Starting training              dataset=612 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T15:29:18.082733Z [info     ] Training complete              train_loss=0.023272458463907242
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T15:29:18.084489Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T15:29:26.288090Z [info     ] Evaluation complete            test_loss=1.0466769933700562
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T15:29:26.292233Z [info     ] Start Predict                  dataset=14388
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T15:31:33.634294Z [info     ] Starting training              dataset=712 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T15:32:06.180206Z [info     ] Training complete              train_loss=0.020062478259205818
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T15:32:06.181728Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T15:32:14.669412Z [info     ] Evaluation complete            test_loss=1.234800934791565
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T15:32:14.673486Z [info     ] Start Predict                  dataset=14288
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T15:34:19.535935Z [info     ] Starting training              dataset=812 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T15:34:50.487032Z [info     ] Training complete              train_loss=0.0782688781619072
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T15:34:50.488655Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T15:34:58.867665Z [info     ] Evaluation complete            test_loss=0.9648405909538269
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T15:34:58.871468Z [info     ] Start Predict                  dataset=14188
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T15:37:04.037450Z [info     ] Starting training              dataset=912 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T15:37:37.071740Z [info     ] Training complete              train_loss=0.006023803725838661
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T15:37:37.073237Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T15:37:45.168673Z [info     ] Evaluation complete            test_loss=0.8979674577713013
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T15:37:45.173268Z [info     ] Start Predict                  dataset=14088
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T15:39:49.231978Z [info     ] Starting training              dataset=1012 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T15:40:22.371968Z [info     ] Training complete              train_loss=0.015347965992987156
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T15:40:22.373994Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T15:40:30.474273Z [info     ] Evaluation complete            test_loss=0.8754228353500366
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T15:40:30.478465Z [info     ] Start Predict                  dataset=13988
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T15:42:36.883231Z [info     ] Starting training              dataset=1112 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T15:43:12.986473Z [info     ] Training complete              train_loss=0.008938436396420002
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T15:43:12.988493Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T15:43:21.974697Z [info     ] Evaluation complete            test_loss=0.8416990041732788
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T15:43:21.979689Z [info     ] Start Predict                  dataset=13888
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T15:45:28.336859Z [info     ] Starting training              dataset=1212 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T15:46:06.676436Z [info     ] Training complete              train_loss=0.006746976636350155
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T15:46:06.678271Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T15:46:15.267152Z [info     ] Evaluation complete            test_loss=0.8873146772384644
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T15:46:15.271275Z [info     ] Start Predict                  dataset=13788
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T15:48:20.279303Z [info     ] Starting training              dataset=1312 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T15:48:59.773418Z [info     ] Training complete              train_loss=0.07147088646888733
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T15:48:59.775274Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T15:49:08.279979Z [info     ] Evaluation complete            test_loss=0.6853619813919067
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T15:49:08.284169Z [info     ] Start Predict                  dataset=13688
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T15:51:13.170250Z [info     ] Starting training              dataset=1412 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T15:51:53.273587Z [info     ] Training complete              train_loss=0.04359261691570282
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T15:51:53.275475Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T15:52:01.677753Z [info     ] Evaluation complete            test_loss=0.6789661645889282
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T15:52:01.681596Z [info     ] Start Predict                  dataset=13588
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T15:54:03.351257Z [info     ] Starting training              dataset=1512 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T15:54:44.773633Z [info     ] Training complete              train_loss=0.018604231998324394
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T15:54:44.776034Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T15:54:53.375403Z [info     ] Evaluation complete            test_loss=0.7287857532501221
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T15:54:53.380431Z [info     ] Start Predict                  dataset=13488
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T15:56:55.411753Z [info     ] Starting training              dataset=1612 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T15:57:37.783421Z [info     ] Training complete              train_loss=0.03406292200088501
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T15:57:37.785087Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T15:57:46.284654Z [info     ] Evaluation complete            test_loss=0.638004720211029
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T15:57:46.288731Z [info     ] Start Predict                  dataset=13388
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T15:59:43.650554Z [info     ] Starting training              dataset=1712 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T16:00:25.788374Z [info     ] Training complete              train_loss=0.06721857935190201
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T16:00:25.789980Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T16:00:34.270598Z [info     ] Evaluation complete            test_loss=0.6078959703445435
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T16:00:34.274553Z [info     ] Start Predict                  dataset=13288
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T16:02:29.738611Z [info     ] Starting training              dataset=1812 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T16:03:13.372754Z [info     ] Training complete              train_loss=0.08680642396211624
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T16:03:13.374516Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T16:03:21.776862Z [info     ] Evaluation complete            test_loss=0.647302508354187
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T16:03:21.780676Z [info     ] Start Predict                  dataset=13188
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T16:05:17.095705Z [info     ] Starting training              dataset=1912 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T16:06:01.375814Z [info     ] Training complete              train_loss=0.06293369829654694
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T16:06:01.377432Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T16:06:09.786188Z [info     ] Evaluation complete            test_loss=0.6817241311073303
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T16:06:09.789944Z [info     ] Start Predict                  dataset=13088
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T16:08:04.612386Z [info     ] Starting training              dataset=2012 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T16:08:49.982937Z [info     ] Training complete              train_loss=0.012322206981480122
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T16:08:49.984675Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T16:08:58.488292Z [info     ] Evaluation complete            test_loss=0.8289020657539368
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T16:08:58.566936Z [info     ] Start Predict                  dataset=12988
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T16:10:53.208150Z [info     ] Starting training              dataset=2112 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T16:11:40.481402Z [info     ] Training complete              train_loss=0.06632529199123383
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T16:11:40.483683Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T16:11:48.892025Z [info     ] Evaluation complete            test_loss=0.6253312826156616
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T16:11:48.896544Z [info     ] Start Predict                  dataset=12888
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T16:13:44.590686Z [info     ] Starting training              dataset=2212 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T16:14:31.585109Z [info     ] Training complete              train_loss=0.044168129563331604
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T16:14:31.587383Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T16:14:40.077616Z [info     ] Evaluation complete            test_loss=0.6782843470573425
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T16:14:40.081708Z [info     ] Start Predict                  dataset=12788
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T16:16:33.938059Z [info     ] Starting training              dataset=2312 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T16:17:21.669894Z [info     ] Training complete              train_loss=0.09880666434764862
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T16:17:21.671646Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T16:17:30.173757Z [info     ] Evaluation complete            test_loss=0.8312588930130005
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T16:17:30.177913Z [info     ] Start Predict                  dataset=12688
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T16:19:20.962003Z [info     ] Starting training              dataset=2412 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T16:20:10.175347Z [info     ] Training complete              train_loss=0.049882616847753525
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T16:20:10.177008Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T16:20:18.676877Z [info     ] Evaluation complete            test_loss=0.6991893649101257
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T16:20:18.681326Z [info     ] Start Predict                  dataset=12588
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T16:22:10.999132Z [info     ] Starting training              dataset=2512 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T16:23:00.569287Z [info     ] Training complete              train_loss=0.06398215144872665
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T16:23:00.571141Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T16:23:08.785616Z [info     ] Evaluation complete            test_loss=0.5477628111839294
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T16:23:08.789326Z [info     ] Start Predict                  dataset=12488
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T16:24:58.624758Z [info     ] Starting training              dataset=2612 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T16:25:49.383363Z [info     ] Training complete              train_loss=0.046333637088537216
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T16:25:49.385026Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T16:25:57.787980Z [info     ] Evaluation complete            test_loss=0.702488124370575
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T16:25:57.792133Z [info     ] Start Predict                  dataset=12388
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T16:27:47.319856Z [info     ] Starting training              dataset=2712 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T16:28:39.489135Z [info     ] Training complete              train_loss=0.08484052121639252
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T16:28:39.490987Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T16:28:47.771346Z [info     ] Evaluation complete            test_loss=0.5731009840965271
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T16:28:47.775165Z [info     ] Start Predict                  dataset=12288
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T16:30:36.802687Z [info     ] Starting training              dataset=2812 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T16:31:29.576259Z [info     ] Training complete              train_loss=0.09145867079496384
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T16:31:29.578028Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T16:31:37.886296Z [info     ] Evaluation complete            test_loss=0.5549673438072205
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T16:31:37.889800Z [info     ] Start Predict                  dataset=12188
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T16:33:26.398167Z [info     ] Starting training              dataset=2912 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T16:34:20.490228Z [info     ] Training complete              train_loss=0.02744719199836254
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T16:34:20.492170Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T16:34:29.067964Z [info     ] Evaluation complete            test_loss=0.660302996635437
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T16:34:29.072246Z [info     ] Start Predict                  dataset=12088
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T16:36:15.329656Z [info     ] Starting training              dataset=3012 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T16:37:09.966144Z [info     ] Training complete              train_loss=0.06737153232097626
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T16:37:09.968091Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T16:37:18.489054Z [info     ] Evaluation complete            test_loss=0.621569812297821
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T16:37:18.569881Z [info     ] Start Predict                  dataset=11988
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T16:39:04.909076Z [info     ] Starting training              dataset=3112 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T16:40:01.971488Z [info     ] Training complete              train_loss=0.05985158681869507
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T16:40:01.973623Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T16:40:10.988141Z [info     ] Evaluation complete            test_loss=0.6705137491226196
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T16:40:10.992964Z [info     ] Start Predict                  dataset=11888
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T16:42:03.615463Z [info     ] Starting training              dataset=3212 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T16:43:06.187299Z [info     ] Training complete              train_loss=0.06435515731573105
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T16:43:06.189039Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T16:43:14.974582Z [info     ] Evaluation complete            test_loss=0.6966602206230164
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T16:43:14.978799Z [info     ] Start Predict                  dataset=11788
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T16:45:01.591964Z [info     ] Starting training              dataset=3312 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T16:45:59.571342Z [info     ] Training complete              train_loss=0.05543559044599533
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T16:45:59.572948Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T16:46:07.784239Z [info     ] Evaluation complete            test_loss=0.6278331279754639
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T16:46:07.788230Z [info     ] Start Predict                  dataset=11688
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T16:47:52.487984Z [info     ] Starting training              dataset=3412 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T16:48:52.691644Z [info     ] Training complete              train_loss=0.07221610099077225
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T16:48:52.765384Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T16:49:01.280861Z [info     ] Evaluation complete            test_loss=0.6179820895195007
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T16:49:01.285400Z [info     ] Start Predict                  dataset=11588
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T16:50:44.974180Z [info     ] Starting training              dataset=3512 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T16:51:42.676914Z [info     ] Training complete              train_loss=0.039833199232816696
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T16:51:42.678530Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T16:51:51.080616Z [info     ] Evaluation complete            test_loss=0.6217177510261536
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T16:51:51.084222Z [info     ] Start Predict                  dataset=11488
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T16:53:31.595579Z [info     ] Starting training              dataset=3612 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T16:54:29.772139Z [info     ] Training complete              train_loss=0.03375746309757233
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T16:54:29.774393Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T16:54:37.979455Z [info     ] Evaluation complete            test_loss=0.6929616928100586
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T16:54:37.982888Z [info     ] Start Predict                  dataset=11388
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T16:56:16.550993Z [info     ] Starting training              dataset=3712 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T16:57:15.780976Z [info     ] Training complete              train_loss=0.04057781398296356
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T16:57:15.782596Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T16:57:23.974826Z [info     ] Evaluation complete            test_loss=0.7048872113227844
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T16:57:23.978437Z [info     ] Start Predict                  dataset=11288
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T16:59:04.231574Z [info     ] Starting training              dataset=3812 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T17:00:07.275913Z [info     ] Training complete              train_loss=0.07070793211460114
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T17:00:07.277627Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T17:00:15.779335Z [info     ] Evaluation complete            test_loss=0.5328732132911682
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T17:00:15.782809Z [info     ] Start Predict                  dataset=11188
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T17:01:54.765307Z [info     ] Starting training              dataset=3912 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T17:02:58.388761Z [info     ] Training complete              train_loss=0.12230982631444931
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T17:02:58.391212Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T17:03:06.876905Z [info     ] Evaluation complete            test_loss=0.5321411490440369
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T17:03:06.881740Z [info     ] Start Predict                  dataset=11088
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T17:04:44.268732Z [info     ] Starting training              dataset=4012 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T17:05:51.968480Z [info     ] Training complete              train_loss=0.044358428567647934
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T17:05:51.970839Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T17:06:00.487090Z [info     ] Evaluation complete            test_loss=0.8059483170509338
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T17:06:00.491321Z [info     ] Start Predict                  dataset=10988
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T17:07:37.287176Z [info     ] Starting training              dataset=4112 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T17:08:43.176013Z [info     ] Training complete              train_loss=0.1225663274526596
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T17:08:43.177778Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T17:08:51.476998Z [info     ] Evaluation complete            test_loss=0.4315877854824066
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T17:08:51.481084Z [info     ] Start Predict                  dataset=10888
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T17:10:29.940602Z [info     ] Starting training              dataset=4212 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T17:11:37.180729Z [info     ] Training complete              train_loss=0.10635881125926971
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T17:11:37.182797Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T17:11:45.678586Z [info     ] Evaluation complete            test_loss=0.5154958963394165
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T17:11:45.682656Z [info     ] Start Predict                  dataset=10788
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T17:13:21.169707Z [info     ] Starting training              dataset=4312 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T17:14:29.792534Z [info     ] Training complete              train_loss=0.053799740970134735
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T17:14:29.866135Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T17:14:38.169583Z [info     ] Evaluation complete            test_loss=0.6495267748832703
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T17:14:38.173309Z [info     ] Start Predict                  dataset=10688
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T17:16:11.983602Z [info     ] Starting training              dataset=4412 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T17:17:20.387089Z [info     ] Training complete              train_loss=0.018582936376333237
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T17:17:20.388534Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T17:17:28.788662Z [info     ] Evaluation complete            test_loss=0.7638018131256104
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T17:17:28.867651Z [info     ] Start Predict                  dataset=10588
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T17:19:02.263938Z [info     ] Starting training              dataset=4512 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T17:20:11.585626Z [info     ] Training complete              train_loss=0.047023482620716095
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T17:20:11.587573Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T17:20:19.881492Z [info     ] Evaluation complete            test_loss=0.49579280614852905
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T17:20:19.885845Z [info     ] Start Predict                  dataset=10488
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T17:21:54.652841Z [info     ] Starting training              dataset=4612 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T17:23:06.479348Z [info     ] Training complete              train_loss=0.06473588943481445
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T17:23:06.480989Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T17:23:14.966838Z [info     ] Evaluation complete            test_loss=0.6052875518798828
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T17:23:14.970796Z [info     ] Start Predict                  dataset=10388
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T17:24:47.734368Z [info     ] Starting training              dataset=4712 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T17:25:59.879686Z [info     ] Training complete              train_loss=0.04754060506820679
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T17:25:59.881753Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T17:26:08.370846Z [info     ] Evaluation complete            test_loss=0.6196796298027039
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T17:26:08.374766Z [info     ] Start Predict                  dataset=10288
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T17:27:38.912284Z [info     ] Starting training              dataset=4812 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T17:28:56.790148Z [info     ] Training complete              train_loss=0.13738520443439484
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T17:28:56.791933Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T17:29:05.372354Z [info     ] Evaluation complete            test_loss=0.417594313621521
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T17:29:05.376664Z [info     ] Start Predict                  dataset=10188
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T17:30:37.433346Z [info     ] Starting training              dataset=4912 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T17:31:52.394380Z [info     ] Training complete              train_loss=0.08568105101585388
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T17:31:52.396245Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T17:32:00.787110Z [info     ] Evaluation complete            test_loss=0.48676493763923645
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T17:32:00.791032Z [info     ] Start Predict                  dataset=10088
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T17:33:28.340993Z [info     ] Starting training              dataset=5012 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T17:34:43.667946Z [info     ] Training complete              train_loss=0.036210086196660995
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T17:34:43.670010Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T17:34:52.185600Z [info     ] Evaluation complete            test_loss=0.6417835354804993
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T17:34:52.189976Z [info     ] Start Predict                  dataset=9988
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T17:36:20.994496Z [info     ] Starting training              dataset=5112 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T17:37:36.585937Z [info     ] Training complete              train_loss=0.055751074105501175
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T17:37:36.587822Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T17:37:44.981738Z [info     ] Evaluation complete            test_loss=0.5641336441040039
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T17:37:44.986213Z [info     ] Start Predict                  dataset=9888
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T17:39:15.946557Z [info     ] Starting training              dataset=5212 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T17:40:33.074149Z [info     ] Training complete              train_loss=0.04473729059100151
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T17:40:33.075860Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T17:40:41.380948Z [info     ] Evaluation complete            test_loss=0.5987882614135742
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T17:40:41.384711Z [info     ] Start Predict                  dataset=9788
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T17:42:07.956223Z [info     ] Starting training              dataset=5312 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T17:43:26.188865Z [info     ] Training complete              train_loss=0.04242468625307083
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T17:43:26.190898Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T17:43:34.776365Z [info     ] Evaluation complete            test_loss=0.573499858379364
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T17:43:34.780809Z [info     ] Start Predict                  dataset=9688
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T17:45:02.444034Z [info     ] Starting training              dataset=5412 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T17:46:22.986454Z [info     ] Training complete              train_loss=0.05522795766592026
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T17:46:22.988322Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T17:46:31.871482Z [info     ] Evaluation complete            test_loss=0.5418797731399536
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T17:46:31.876347Z [info     ] Start Predict                  dataset=9588
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T17:47:57.958922Z [info     ] Starting training              dataset=5512 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T17:49:19.288858Z [info     ] Training complete              train_loss=0.05585930123925209
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T17:49:19.290700Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T17:49:27.887083Z [info     ] Evaluation complete            test_loss=0.5781568884849548
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T17:49:27.891558Z [info     ] Start Predict                  dataset=9488
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T17:50:51.759392Z [info     ] Starting training              dataset=5612 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T17:52:10.167108Z [info     ] Training complete              train_loss=0.08212323486804962
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T17:52:10.168639Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T17:52:18.489575Z [info     ] Evaluation complete            test_loss=0.5190374255180359
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T17:52:18.568302Z [info     ] Start Predict                  dataset=9388
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T17:53:42.390100Z [info     ] Starting training              dataset=5712 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T17:55:06.677886Z [info     ] Training complete              train_loss=0.08772403001785278
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T17:55:06.679728Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T17:55:15.179280Z [info     ] Evaluation complete            test_loss=0.4441128075122833
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T17:55:15.183333Z [info     ] Start Predict                  dataset=9288
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T17:56:40.256626Z [info     ] Starting training              dataset=5812 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T17:58:00.871617Z [info     ] Training complete              train_loss=0.05669952929019928
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T17:58:00.873122Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T17:58:09.284316Z [info     ] Evaluation complete            test_loss=0.5060445666313171
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T17:58:09.288729Z [info     ] Start Predict                  dataset=9188
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T17:59:28.919979Z [info     ] Starting training              dataset=5912 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T18:00:51.078877Z [info     ] Training complete              train_loss=0.07387827336788177
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T18:00:51.080834Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T18:00:59.669384Z [info     ] Evaluation complete            test_loss=0.5086865425109863
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T18:00:59.674395Z [info     ] Start Predict                  dataset=9088
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T18:02:19.394067Z [info     ] Starting training              dataset=6012 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T18:03:42.782050Z [info     ] Training complete              train_loss=0.08140388131141663
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T18:03:42.783808Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T18:03:51.468837Z [info     ] Evaluation complete            test_loss=0.432904988527298
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T18:03:51.473174Z [info     ] Start Predict                  dataset=8988
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T18:05:10.688109Z [info     ] Starting training              dataset=6112 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T18:06:36.281515Z [info     ] Training complete              train_loss=0.04091908782720566
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T18:06:36.283262Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T18:06:44.686541Z [info     ] Evaluation complete            test_loss=0.5736671686172485
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T18:06:44.690767Z [info     ] Start Predict                  dataset=8888
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T18:08:05.198629Z [info     ] Starting training              dataset=6212 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T18:09:30.569787Z [info     ] Training complete              train_loss=0.06559653580188751
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T18:09:30.571482Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T18:09:38.668353Z [info     ] Evaluation complete            test_loss=0.5123801827430725
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T18:09:38.672007Z [info     ] Start Predict                  dataset=8788
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T18:10:56.504718Z [info     ] Starting training              dataset=6312 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T18:12:24.391821Z [info     ] Training complete              train_loss=0.07125910371541977
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T18:12:24.393844Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T18:12:32.869289Z [info     ] Evaluation complete            test_loss=0.4685639441013336
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T18:12:32.873654Z [info     ] Start Predict                  dataset=8688
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T18:13:52.161712Z [info     ] Starting training              dataset=6412 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T18:15:19.480747Z [info     ] Training complete              train_loss=0.08134432137012482
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T18:15:19.482431Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T18:15:27.767816Z [info     ] Evaluation complete            test_loss=0.4484506845474243
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T18:15:27.771722Z [info     ] Start Predict                  dataset=8588
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T18:16:45.635661Z [info     ] Starting training              dataset=6512 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T18:18:13.391296Z [info     ] Training complete              train_loss=0.07426474988460541
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T18:18:13.392929Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T18:18:22.003691Z [info     ] Evaluation complete            test_loss=0.5092469453811646
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T18:18:22.007843Z [info     ] Start Predict                  dataset=8488
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T18:19:37.445279Z [info     ] Starting training              dataset=6612 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T18:21:05.781014Z [info     ] Training complete              train_loss=0.05355135723948479
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T18:21:05.782790Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T18:21:14.180182Z [info     ] Evaluation complete            test_loss=0.504236102104187
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T18:21:14.184058Z [info     ] Start Predict                  dataset=8388
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T18:22:27.520906Z [info     ] Starting training              dataset=6712 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T18:24:01.288497Z [info     ] Training complete              train_loss=0.06972303986549377
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T18:24:01.290234Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T18:24:09.581470Z [info     ] Evaluation complete            test_loss=0.505696177482605
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T18:24:09.585144Z [info     ] Start Predict                  dataset=8288
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T18:25:23.082250Z [info     ] Starting training              dataset=6812 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T18:26:51.691389Z [info     ] Training complete              train_loss=0.08558610081672668
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T18:26:51.693177Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T18:27:00.171906Z [info     ] Evaluation complete            test_loss=0.4490826725959778
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T18:27:00.175724Z [info     ] Start Predict                  dataset=8188
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T18:28:12.777510Z [info     ] Starting training              dataset=6912 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T18:29:44.587334Z [info     ] Training complete              train_loss=0.027112364768981934
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T18:29:44.589016Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T18:29:52.967716Z [info     ] Evaluation complete            test_loss=0.6299618482589722
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T18:29:52.971769Z [info     ] Start Predict                  dataset=8088
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T18:31:03.915833Z [info     ] Starting training              dataset=7012 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T18:32:36.978128Z [info     ] Training complete              train_loss=0.05840374156832695
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T18:32:36.979875Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T18:32:45.379434Z [info     ] Evaluation complete            test_loss=0.49078628420829773
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T18:32:45.384069Z [info     ] Start Predict                  dataset=7988
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T18:33:57.892025Z [info     ] Starting training              dataset=7112 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T18:35:31.374373Z [info     ] Training complete              train_loss=0.07398192584514618
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T18:35:31.375994Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T18:35:39.775096Z [info     ] Evaluation complete            test_loss=0.3967265784740448
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T18:35:39.779247Z [info     ] Start Predict                  dataset=7888
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T18:36:49.517415Z [info     ] Starting training              dataset=7212 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T18:38:24.174877Z [info     ] Training complete              train_loss=0.04612201824784279
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T18:38:24.177022Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T18:38:32.379501Z [info     ] Evaluation complete            test_loss=0.49809715151786804
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T18:38:32.382998Z [info     ] Start Predict                  dataset=7788
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T18:39:41.917608Z [info     ] Starting training              dataset=7312 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T18:41:17.174369Z [info     ] Training complete              train_loss=0.06543444097042084
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T18:41:17.176491Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T18:41:25.471666Z [info     ] Evaluation complete            test_loss=0.5225977897644043
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T18:41:25.475692Z [info     ] Start Predict                  dataset=7688
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T18:42:33.522329Z [info     ] Starting training              dataset=7412 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T18:44:09.293539Z [info     ] Training complete              train_loss=0.060686737298965454
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T18:44:09.366949Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T18:44:17.887026Z [info     ] Evaluation complete            test_loss=0.5125385522842407
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T18:44:17.890832Z [info     ] Start Predict                  dataset=7588
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T18:45:25.400968Z [info     ] Starting training              dataset=7512 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T18:47:02.366684Z [info     ] Training complete              train_loss=0.07019717246294022
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T18:47:02.368580Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T18:47:10.878960Z [info     ] Evaluation complete            test_loss=0.470739483833313
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T18:47:10.882850Z [info     ] Start Predict                  dataset=7488
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T18:48:16.235930Z [info     ] Starting training              dataset=7612 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T18:49:53.493500Z [info     ] Training complete              train_loss=0.04707063362002373
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T18:49:53.495700Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T18:50:01.788730Z [info     ] Evaluation complete            test_loss=0.5984643697738647
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T18:50:01.866775Z [info     ] Start Predict                  dataset=7388
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T18:51:07.901960Z [info     ] Starting training              dataset=7712 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T18:52:46.877393Z [info     ] Training complete              train_loss=0.06192445009946823
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T18:52:46.879872Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T18:52:55.176958Z [info     ] Evaluation complete            test_loss=0.5132700204849243
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T18:52:55.181403Z [info     ] Start Predict                  dataset=7288
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T18:54:00.257566Z [info     ] Starting training              dataset=7812 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T18:55:43.588375Z [info     ] Training complete              train_loss=0.03979147970676422
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T18:55:43.590052Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T18:55:51.875854Z [info     ] Evaluation complete            test_loss=0.5150150656700134
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T18:55:51.880159Z [info     ] Start Predict                  dataset=7188
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T18:56:56.536715Z [info     ] Starting training              dataset=7912 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T18:58:41.879868Z [info     ] Training complete              train_loss=0.06223804131150246
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T18:58:41.881963Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T18:58:50.468474Z [info     ] Evaluation complete            test_loss=0.5643148422241211
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T18:58:50.472745Z [info     ] Start Predict                  dataset=7088
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T18:59:54.295632Z [info     ] Starting training              dataset=8012 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T19:01:35.476312Z [info     ] Training complete              train_loss=0.06737291067838669
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T19:01:35.478476Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T19:01:43.780950Z [info     ] Evaluation complete            test_loss=0.48747894167900085
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T19:01:43.785449Z [info     ] Start Predict                  dataset=6988
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T19:02:46.131753Z [info     ] Starting training              dataset=8112 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T19:04:29.880063Z [info     ] Training complete              train_loss=0.05925571545958519
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T19:04:29.881840Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T19:04:38.267654Z [info     ] Evaluation complete            test_loss=0.564688503742218
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T19:04:38.271766Z [info     ] Start Predict                  dataset=6888
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T19:05:39.800913Z [info     ] Starting training              dataset=8212 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T19:07:23.486186Z [info     ] Training complete              train_loss=0.040423326194286346
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T19:07:23.487988Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T19:07:31.880357Z [info     ] Evaluation complete            test_loss=0.5314301252365112
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T19:07:31.884041Z [info     ] Start Predict                  dataset=6788
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T19:08:32.589051Z [info     ] Starting training              dataset=8312 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T19:10:17.365494Z [info     ] Training complete              train_loss=0.06140904501080513
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T19:10:17.367863Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T19:10:25.972279Z [info     ] Evaluation complete            test_loss=0.49688518047332764
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T19:10:25.976035Z [info     ] Start Predict                  dataset=6688
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T19:11:25.835585Z [info     ] Starting training              dataset=8412 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T19:13:10.774985Z [info     ] Training complete              train_loss=0.050612445920705795
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T19:13:10.776734Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T19:13:19.167663Z [info     ] Evaluation complete            test_loss=0.5152626037597656
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T19:13:19.171962Z [info     ] Start Predict                  dataset=6588
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T19:14:18.243027Z [info     ] Starting training              dataset=8512 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T19:16:04.791515Z [info     ] Training complete              train_loss=0.033627718687057495
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T19:16:04.793285Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T19:16:13.288919Z [info     ] Evaluation complete            test_loss=0.4974973201751709
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T19:16:13.292998Z [info     ] Start Predict                  dataset=6488
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T19:17:11.209628Z [info     ] Starting training              dataset=8612 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T19:18:58.093512Z [info     ] Training complete              train_loss=0.04596696048974991
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T19:18:58.095249Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T19:19:06.484257Z [info     ] Evaluation complete            test_loss=0.6129364967346191
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T19:19:06.488960Z [info     ] Start Predict                  dataset=6388
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T19:20:04.209439Z [info     ] Starting training              dataset=8712 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T19:21:55.388171Z [info     ] Training complete              train_loss=0.049339789897203445
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T19:21:55.390263Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T19:22:04.078244Z [info     ] Evaluation complete            test_loss=0.5226636528968811
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T19:22:04.082582Z [info     ] Start Predict                  dataset=6288
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T19:23:01.440119Z [info     ] Starting training              dataset=8812 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T19:24:53.994417Z [info     ] Training complete              train_loss=0.061960719525814056
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T19:24:53.996345Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T19:25:02.372585Z [info     ] Evaluation complete            test_loss=0.5233380198478699
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T19:25:02.377398Z [info     ] Start Predict                  dataset=6188
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T19:25:58.792316Z [info     ] Starting training              dataset=8912 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T19:27:51.591211Z [info     ] Training complete              train_loss=0.04009644314646721
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T19:27:51.593404Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T19:28:00.069251Z [info     ] Evaluation complete            test_loss=0.5073189735412598
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T19:28:00.073255Z [info     ] Start Predict                  dataset=6088
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T19:28:55.614687Z [info     ] Starting training              dataset=9012 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T19:30:48.691304Z [info     ] Training complete              train_loss=0.02636721171438694
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T19:30:48.693153Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T19:30:57.194178Z [info     ] Evaluation complete            test_loss=0.4932842552661896
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T19:30:57.198696Z [info     ] Start Predict                  dataset=5988
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T19:31:52.014806Z [info     ] Starting training              dataset=9112 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T19:33:50.286149Z [info     ] Training complete              train_loss=0.038404107093811035
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T19:33:50.288005Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T19:33:58.867647Z [info     ] Evaluation complete            test_loss=0.4707241952419281
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T19:33:58.871875Z [info     ] Start Predict                  dataset=5888
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T19:34:52.597191Z [info     ] Starting training              dataset=9212 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T19:36:52.979111Z [info     ] Training complete              train_loss=0.03221401944756508
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T19:36:52.981332Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T19:37:01.382679Z [info     ] Evaluation complete            test_loss=0.6017580628395081
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T19:37:01.386797Z [info     ] Start Predict                  dataset=5788
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T19:37:57.489992Z [info     ] Starting training              dataset=9312 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T19:39:55.967846Z [info     ] Training complete              train_loss=0.04899067059159279
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T19:39:55.969679Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T19:40:04.272984Z [info     ] Evaluation complete            test_loss=0.4522852897644043
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T19:40:04.278449Z [info     ] Start Predict                  dataset=5688
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T19:40:57.436623Z [info     ] Starting training              dataset=9412 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T19:42:55.093364Z [info     ] Training complete              train_loss=0.0554390586912632
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T19:42:55.095567Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T19:43:03.678599Z [info     ] Evaluation complete            test_loss=0.5251901149749756
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T19:43:03.682851Z [info     ] Start Predict                  dataset=5588
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T19:43:54.653131Z [info     ] Starting training              dataset=9512 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T19:45:51.790112Z [info     ] Training complete              train_loss=0.04198170453310013
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T19:45:51.791730Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T19:46:00.284703Z [info     ] Evaluation complete            test_loss=0.4602336585521698
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T19:46:00.289764Z [info     ] Start Predict                  dataset=5488
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T19:46:50.358892Z [info     ] Starting training              dataset=9612 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T19:48:46.692007Z [info     ] Training complete              train_loss=0.044020820409059525
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T19:48:46.693602Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T19:48:55.168600Z [info     ] Evaluation complete            test_loss=0.508590579032898
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T19:48:55.172559Z [info     ] Start Predict                  dataset=5388
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T19:49:43.776890Z [info     ] Starting training              dataset=9712 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T19:51:42.176822Z [info     ] Training complete              train_loss=0.044634196907281876
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T19:51:42.179048Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T19:51:50.672765Z [info     ] Evaluation complete            test_loss=0.45551300048828125
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T19:51:50.676552Z [info     ] Start Predict                  dataset=5288
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T19:52:38.484667Z [info     ] Starting training              dataset=9812 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T19:54:36.177208Z [info     ] Training complete              train_loss=0.04296591877937317
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T19:54:36.178931Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T19:54:44.583521Z [info     ] Evaluation complete            test_loss=0.47437164187431335
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T19:54:44.587728Z [info     ] Start Predict                  dataset=5188
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T19:55:31.707641Z [info     ] Starting training              dataset=9912 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T19:57:30.279767Z [info     ] Training complete              train_loss=0.03641924262046814
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T19:57:30.281499Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T19:57:38.466727Z [info     ] Evaluation complete            test_loss=0.5371329188346863
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T19:57:38.470612Z [info     ] Start Predict                  dataset=5088
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T19:58:24.271875Z [info     ] Starting training              dataset=10012 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T20:00:25.895237Z [info     ] Training complete              train_loss=0.03702413663268089
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T20:00:25.897511Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T20:00:34.571676Z [info     ] Evaluation complete            test_loss=0.5401384234428406
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T20:00:34.576397Z [info     ] Start Predict                  dataset=4988
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T20:01:22.740200Z [info     ] Starting training              dataset=10112 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T20:03:27.695031Z [info     ] Training complete              train_loss=0.05812685936689377
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T20:03:27.698130Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T20:03:36.176022Z [info     ] Evaluation complete            test_loss=0.4845340847969055
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T20:03:36.179729Z [info     ] Start Predict                  dataset=4888
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T20:04:22.320485Z [info     ] Starting training              dataset=10212 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T20:06:27.779220Z [info     ] Training complete              train_loss=0.0416891947388649
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T20:06:27.781323Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T20:06:36.067846Z [info     ] Evaluation complete            test_loss=0.46326562762260437
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T20:06:36.072402Z [info     ] Start Predict                  dataset=4788
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T20:07:22.045382Z [info     ] Starting training              dataset=10312 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T20:09:30.292345Z [info     ] Training complete              train_loss=0.041010648012161255
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T20:09:30.294497Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T20:09:38.871867Z [info     ] Evaluation complete            test_loss=0.49109208583831787
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T20:09:38.876577Z [info     ] Start Predict                  dataset=4688
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T20:10:24.154535Z [info     ] Starting training              dataset=10412 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T20:12:33.415576Z [info     ] Training complete              train_loss=0.028930526226758957
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T20:12:33.417436Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T20:12:42.284002Z [info     ] Evaluation complete            test_loss=0.5107240676879883
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T20:12:42.288185Z [info     ] Start Predict                  dataset=4588
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T20:13:26.235085Z [info     ] Starting training              dataset=10512 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T20:15:35.385270Z [info     ] Training complete              train_loss=0.031576935201883316
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T20:15:35.387155Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T20:15:43.779605Z [info     ] Evaluation complete            test_loss=0.6074793338775635
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T20:15:43.783983Z [info     ] Start Predict                  dataset=4488
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T20:16:27.250669Z [info     ] Starting training              dataset=10612 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T20:18:38.077150Z [info     ] Training complete              train_loss=0.025566451251506805
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T20:18:38.079132Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T20:18:46.374243Z [info     ] Evaluation complete            test_loss=0.6829275488853455
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T20:18:46.378420Z [info     ] Start Predict                  dataset=4388
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T20:19:29.033297Z [info     ] Starting training              dataset=10712 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T20:21:40.186438Z [info     ] Training complete              train_loss=0.03265538439154625
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T20:21:40.188513Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T20:21:48.585218Z [info     ] Evaluation complete            test_loss=0.48324117064476013
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T20:21:48.589070Z [info     ] Start Predict                  dataset=4288
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T20:22:29.946839Z [info     ] Starting training              dataset=10812 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T20:24:41.668771Z [info     ] Training complete              train_loss=0.04195354878902435
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T20:24:41.670907Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T20:24:49.971122Z [info     ] Evaluation complete            test_loss=0.5226069688796997
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T20:24:49.975409Z [info     ] Start Predict                  dataset=4188
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T20:25:30.746312Z [info     ] Starting training              dataset=10912 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T20:27:43.090670Z [info     ] Training complete              train_loss=0.03698594123125076
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T20:27:43.092609Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T20:27:51.571422Z [info     ] Evaluation complete            test_loss=0.5863118767738342
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T20:27:51.576126Z [info     ] Start Predict                  dataset=4088
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T20:28:31.534919Z [info     ] Starting training              dataset=11012 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T20:30:44.186803Z [info     ] Training complete              train_loss=0.04117933288216591
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T20:30:44.188896Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T20:30:52.585989Z [info     ] Evaluation complete            test_loss=0.4863777160644531
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T20:30:52.589908Z [info     ] Start Predict                  dataset=3988
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T20:31:31.716610Z [info     ] Starting training              dataset=11112 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T20:33:46.383277Z [info     ] Training complete              train_loss=0.04627871885895729
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T20:33:46.385149Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T20:33:54.779086Z [info     ] Evaluation complete            test_loss=0.470214307308197
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T20:33:54.783126Z [info     ] Start Predict                  dataset=3888
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T20:34:33.190703Z [info     ] Starting training              dataset=11212 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T20:36:48.681686Z [info     ] Training complete              train_loss=0.04613247141242027
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T20:36:48.683987Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T20:36:57.278346Z [info     ] Evaluation complete            test_loss=0.4558008015155792
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T20:36:57.282786Z [info     ] Start Predict                  dataset=3788
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T20:37:34.291065Z [info     ] Starting training              dataset=11312 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T20:39:51.289119Z [info     ] Training complete              train_loss=0.02740650437772274
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T20:39:51.291195Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T20:39:59.882257Z [info     ] Evaluation complete            test_loss=0.5408679246902466
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T20:39:59.887662Z [info     ] Start Predict                  dataset=3688
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T20:40:35.740146Z [info     ] Starting training              dataset=11412 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T20:42:52.090300Z [info     ] Training complete              train_loss=0.0475490465760231
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T20:42:52.092186Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T20:43:00.380313Z [info     ] Evaluation complete            test_loss=0.48505839705467224
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T20:43:00.384850Z [info     ] Start Predict                  dataset=3588
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T20:43:35.409152Z [info     ] Starting training              dataset=11512 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T20:45:53.678636Z [info     ] Training complete              train_loss=0.029358908534049988
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T20:45:53.680683Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T20:46:01.986445Z [info     ] Evaluation complete            test_loss=0.5404146313667297
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T20:46:01.990899Z [info     ] Start Predict                  dataset=3488
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T20:46:36.192044Z [info     ] Starting training              dataset=11612 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T20:48:54.491902Z [info     ] Training complete              train_loss=0.03432118520140648
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T20:48:54.494055Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T20:49:03.187720Z [info     ] Evaluation complete            test_loss=0.5160902142524719
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T20:49:03.192422Z [info     ] Start Predict                  dataset=3388
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T20:49:36.627034Z [info     ] Starting training              dataset=11712 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T20:51:53.684736Z [info     ] Training complete              train_loss=0.025415150448679924
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T20:51:53.687300Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T20:52:02.180711Z [info     ] Evaluation complete            test_loss=0.4870961308479309
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T20:52:02.185758Z [info     ] Start Predict                  dataset=3288
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T20:52:34.039245Z [info     ] Starting training              dataset=11812 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T20:54:52.384529Z [info     ] Training complete              train_loss=0.04198145121335983
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T20:54:52.386512Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T20:55:00.588475Z [info     ] Evaluation complete            test_loss=0.4633568823337555
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T20:55:00.593256Z [info     ] Start Predict                  dataset=3188
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T20:55:32.526412Z [info     ] Starting training              dataset=11912 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T20:57:52.280876Z [info     ] Training complete              train_loss=0.027177348732948303
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T20:57:52.282884Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T20:58:00.669065Z [info     ] Evaluation complete            test_loss=0.5218877792358398
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T20:58:00.673801Z [info     ] Start Predict                  dataset=3088
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T20:58:31.200998Z [info     ] Starting training              dataset=12012 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T21:00:51.678552Z [info     ] Training complete              train_loss=0.03196889907121658
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T21:00:51.680520Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T21:00:59.969555Z [info     ] Evaluation complete            test_loss=0.4975127875804901
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T21:00:59.974549Z [info     ] Start Predict                  dataset=2988
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T21:01:29.609903Z [info     ] Starting training              dataset=12112 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T21:03:52.395673Z [info     ] Training complete              train_loss=0.029373254626989365
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T21:03:52.397885Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T21:04:00.787526Z [info     ] Evaluation complete            test_loss=0.48535236716270447
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T21:04:00.791903Z [info     ] Start Predict                  dataset=2888
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T21:04:29.892682Z [info     ] Starting training              dataset=12212 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T21:06:53.894096Z [info     ] Training complete              train_loss=0.029735658317804337
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T21:06:53.895831Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T21:07:02.080102Z [info     ] Evaluation complete            test_loss=0.5647792220115662
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T21:07:02.084360Z [info     ] Start Predict                  dataset=2788
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T21:07:30.166622Z [info     ] Starting training              dataset=12312 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T21:09:55.179011Z [info     ] Training complete              train_loss=0.0220402330160141
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T21:09:55.180688Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T21:10:03.477053Z [info     ] Evaluation complete            test_loss=0.5359561443328857
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T21:10:03.481581Z [info     ] Start Predict                  dataset=2688
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T21:10:30.528810Z [info     ] Starting training              dataset=12412 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T21:12:54.091861Z [info     ] Training complete              train_loss=0.03964528813958168
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T21:12:54.094102Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T21:13:02.369356Z [info     ] Evaluation complete            test_loss=0.48836764693260193
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T21:13:02.374057Z [info     ] Start Predict                  dataset=2588
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T21:13:27.496414Z [info     ] Starting training              dataset=12512 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T21:15:48.582187Z [info     ] Training complete              train_loss=0.04704689979553223
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T21:15:48.584036Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T21:15:56.668202Z [info     ] Evaluation complete            test_loss=0.44271594285964966
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T21:15:56.672079Z [info     ] Start Predict                  dataset=2488
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T21:16:21.327724Z [info     ] Starting training              dataset=12612 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T21:18:50.083526Z [info     ] Training complete              train_loss=0.0628628060221672
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T21:18:50.085875Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T21:18:58.770438Z [info     ] Evaluation complete            test_loss=0.503575325012207
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T21:18:58.775111Z [info     ] Start Predict                  dataset=2388
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T21:19:22.821996Z [info     ] Starting training              dataset=12712 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T21:21:51.483589Z [info     ] Training complete              train_loss=0.02907649055123329
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T21:21:51.485452Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T21:21:59.875917Z [info     ] Evaluation complete            test_loss=0.5075518488883972
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T21:21:59.879896Z [info     ] Start Predict                  dataset=2288
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T21:22:23.219614Z [info     ] Starting training              dataset=12812 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T21:24:51.385507Z [info     ] Training complete              train_loss=0.030792735517024994
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T21:24:51.387626Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T21:24:59.785684Z [info     ] Evaluation complete            test_loss=0.44357359409332275
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T21:24:59.790069Z [info     ] Start Predict                  dataset=2188
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T21:25:22.213913Z [info     ] Starting training              dataset=12912 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T21:27:50.994998Z [info     ] Training complete              train_loss=0.03515024855732918
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T21:27:50.996822Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T21:27:59.080001Z [info     ] Evaluation complete            test_loss=0.45534369349479675
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T21:27:59.084982Z [info     ] Start Predict                  dataset=2088
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T21:28:19.309633Z [info     ] Starting training              dataset=13012 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T21:30:47.379066Z [info     ] Training complete              train_loss=0.028724441304802895
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T21:30:47.381229Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T21:30:55.668478Z [info     ] Evaluation complete            test_loss=0.5947825908660889
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T21:30:55.672879Z [info     ] Start Predict                  dataset=1988
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T21:31:16.513428Z [info     ] Starting training              dataset=13112 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T21:33:46.087868Z [info     ] Training complete              train_loss=0.03632659092545509
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T21:33:46.090190Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T21:33:54.470903Z [info     ] Evaluation complete            test_loss=0.3916391432285309
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T21:33:54.475662Z [info     ] Start Predict                  dataset=1888
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T21:34:13.817268Z [info     ] Starting training              dataset=13212 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T21:36:44.087828Z [info     ] Training complete              train_loss=0.0293908528983593
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T21:36:44.089967Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T21:36:52.285403Z [info     ] Evaluation complete            test_loss=0.553371787071228
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T21:36:52.289324Z [info     ] Start Predict                  dataset=1788
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T21:37:10.997355Z [info     ] Starting training              dataset=13312 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T21:39:42.870680Z [info     ] Training complete              train_loss=0.024943392723798752
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T21:39:42.872809Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T21:39:51.376196Z [info     ] Evaluation complete            test_loss=0.5904661417007446
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T21:39:51.380310Z [info     ] Start Predict                  dataset=1688
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T21:40:09.211327Z [info     ] Starting training              dataset=13412 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T21:42:40.875515Z [info     ] Training complete              train_loss=0.03411614149808884
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T21:42:40.877663Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T21:42:49.181025Z [info     ] Evaluation complete            test_loss=0.5226052403450012
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T21:42:49.185080Z [info     ] Start Predict                  dataset=1588
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T21:43:06.385285Z [info     ] Starting training              dataset=13512 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T21:45:39.896472Z [info     ] Training complete              train_loss=0.02790781483054161
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T21:45:39.898469Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T21:45:48.768464Z [info     ] Evaluation complete            test_loss=0.48294728994369507
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T21:45:48.773007Z [info     ] Start Predict                  dataset=1488
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T21:46:05.101706Z [info     ] Starting training              dataset=13612 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T21:48:38.587002Z [info     ] Training complete              train_loss=0.021476466208696365
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T21:48:38.589038Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T21:48:46.967470Z [info     ] Evaluation complete            test_loss=0.6516388654708862
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T21:48:46.971723Z [info     ] Start Predict                  dataset=1388
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T21:49:02.400625Z [info     ] Starting training              dataset=13712 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T21:51:38.993724Z [info     ] Training complete              train_loss=0.02276446670293808
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T21:51:38.996209Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T21:51:47.371012Z [info     ] Evaluation complete            test_loss=0.5613048076629639
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T21:51:47.375167Z [info     ] Start Predict                  dataset=1288
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T21:52:01.682782Z [info     ] Starting training              dataset=13812 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T21:54:41.984078Z [info     ] Training complete              train_loss=0.028370166197419167
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T21:54:41.985861Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T21:54:50.184247Z [info     ] Evaluation complete            test_loss=0.4826107919216156
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T21:54:50.188377Z [info     ] Start Predict                  dataset=1188
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T21:55:03.498739Z [info     ] Starting training              dataset=13912 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T21:57:39.587512Z [info     ] Training complete              train_loss=0.022563118487596512
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T21:57:39.589138Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T21:57:48.077240Z [info     ] Evaluation complete            test_loss=0.5660622715950012
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T21:57:48.080857Z [info     ] Start Predict                  dataset=1088
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T21:58:00.385493Z [info     ] Starting training              dataset=14012 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T22:00:36.183661Z [info     ] Training complete              train_loss=0.019728390499949455
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T22:00:36.185580Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T22:00:44.479725Z [info     ] Evaluation complete            test_loss=0.5509332418441772
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T22:00:44.484148Z [info     ] Start Predict                  dataset=988
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T22:00:56.003275Z [info     ] Starting training              dataset=14112 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T22:03:32.995503Z [info     ] Training complete              train_loss=0.03207049146294594
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T22:03:32.997422Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T22:03:41.378439Z [info     ] Evaluation complete            test_loss=0.5133938193321228
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T22:03:41.382642Z [info     ] Start Predict                  dataset=888
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T22:03:52.104510Z [info     ] Starting training              dataset=14212 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T22:06:29.792679Z [info     ] Training complete              train_loss=0.037114620208740234
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T22:06:29.794621Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T22:06:38.077351Z [info     ] Evaluation complete            test_loss=0.5605881214141846
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T22:06:38.082232Z [info     ] Start Predict                  dataset=788
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T22:06:48.003399Z [info     ] Starting training              dataset=14312 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T22:09:27.080457Z [info     ] Training complete              train_loss=0.02423758991062641
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T22:09:27.082417Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T22:09:35.575955Z [info     ] Evaluation complete            test_loss=0.4343477487564087
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T22:09:35.580152Z [info     ] Start Predict                  dataset=688
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T22:09:44.492943Z [info     ] Starting training              dataset=14412 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T22:12:23.871556Z [info     ] Training complete              train_loss=0.025025462731719017
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T22:12:23.873606Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T22:12:32.375955Z [info     ] Evaluation complete            test_loss=0.49787238240242004
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T22:12:32.380195Z [info     ] Start Predict                  dataset=588
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T22:12:40.275824Z [info     ] Starting training              dataset=14512 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T22:15:20.281568Z [info     ] Training complete              train_loss=0.025414831936359406
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T22:15:20.283811Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T22:15:28.687757Z [info     ] Evaluation complete            test_loss=0.5783530473709106
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T22:15:28.691740Z [info     ] Start Predict                  dataset=488
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T22:15:35.699876Z [info     ] Starting training              dataset=14612 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T22:18:15.677656Z [info     ] Training complete              train_loss=0.03256714344024658
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T22:18:15.679679Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T22:18:23.788199Z [info     ] Evaluation complete            test_loss=0.4111860692501068
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T22:18:23.792636Z [info     ] Start Predict                  dataset=388
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T22:18:29.996618Z [info     ] Starting training              dataset=14712 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T22:21:10.987781Z [info     ] Training complete              train_loss=0.031116826459765434
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T22:21:10.989619Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T22:21:19.588062Z [info     ] Evaluation complete            test_loss=0.6319525837898254
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T22:21:19.592013Z [info     ] Start Predict                  dataset=288
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T22:21:24.766597Z [info     ] Starting training              dataset=14812 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T22:24:06.881760Z [info     ] Training complete              train_loss=0.029832642525434494
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T22:24:06.884590Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T22:24:15.278154Z [info     ] Evaluation complete            test_loss=0.47507792711257935
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T22:24:15.282500Z [info     ] Start Predict                  dataset=188
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T22:24:19.298270Z [info     ] Starting training              dataset=14912 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T22:27:02.087124Z [info     ] Training complete              train_loss=0.028897961601614952
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T22:27:02.089190Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T22:27:10.469021Z [info     ] Evaluation complete            test_loss=0.4525805115699768
[131537-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T22:27:10.473399Z [info     ] Start Predict                  dataset=88
[131537-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T22:27:13.881440Z [info     ] Starting training              dataset=15000 epoch=10
[131537-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T22:29:57.197467Z [info     ] Training complete              train_loss=0.03034781664609909
[131537-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T22:29:57.199424Z [info     ] Starting evaluating            dataset=3000
[131537-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T22:30:05.574946Z [info     ] Evaluation complete            test_loss=0.45382431149482727

We will now save our progress on disk.

[7]:
model_weight = model.state_dict()
dataset = active_set.state_dict()
torch.save({'model':model_weight, 'dataset':dataset, 'labelling_progress':labelling_progress},
           'checkpoint.pth')
print(model.state_dict().keys(), dataset.keys(), labelling_progress)
odict_keys(['features.0.weight', 'features.0.bias', 'features.2.weight', 'features.2.bias', 'features.5.weight', 'features.5.bias', 'features.7.weight', 'features.7.bias', 'features.10.weight', 'features.10.bias', 'features.12.weight', 'features.12.bias', 'features.14.weight', 'features.14.bias', 'features.17.weight', 'features.17.bias', 'features.19.weight', 'features.19.bias', 'features.21.weight', 'features.21.bias', 'features.24.weight', 'features.24.bias', 'features.26.weight', 'features.26.bias', 'features.28.weight', 'features.28.bias', 'classifier.0.weight', 'classifier.0.bias', 'classifier.3.weight', 'classifier.3.bias', 'classifier.6.weight', 'classifier.6.bias']) dict_keys(['labelled', 'random_state']) [103  89 135 ... 121  15  77]

Visualization

Now that our active learning experiment is completed, we can visualize it!

Get t-SNE features.

We will use MultiCoreTSNE to get a t-SNE representation of our dataset. This will allows us to visualize the progress.

[8]:
# modify our model to get features
from torch import nn
from torch.utils.data import DataLoader


# Make a feature extractor from our trained model.
class FeatureExtractor(nn.Module):
    def __init__(self, model):
        super().__init__()
        self.model = model
    def forward(self, x):
        return torch.flatten(self.model.features(x),1)


features = FeatureExtractor(model.model)
acc = []
for x,y in DataLoader(active_set._dataset, batch_size=10):
    acc.append((features(x.cuda()).detach().cpu().numpy(), y.detach().cpu().numpy()))

xs, ys = zip(*acc)
[9]:
from MulticoreTSNE import MulticoreTSNE as TSNE

# Compute t-SNE on the extracted features.
tsne = TSNE(n_jobs=4)
transformed = tsne.fit_transform(np.vstack(xs))
[10]:
labels = np.concatenate(ys)
labels.shape
[10]:
(15000,)

To make the animation, BaaL has baal.utils.plot_utils.make_animation_from_data which takes a set of features, their labels and the array containing the progress we created earlier.

[11]:
from baal.utils.plot_utils import make_animation_from_data

# Create frames to animate the process.
frames = make_animation_from_data(transformed, labels, labelling_progress, ["airplane", "cat", "dog"])

[12]:
from IPython.display import HTML
import matplotlib.pyplot as plt
from matplotlib import animation

def plot_images(img_list):
    def init():
        img.set_data(img_list[0])
        return (img,)

    def animate(i):
        img.set_data(img_list[i])
        return (img,)

    fig = plt.Figure(figsize=(10,10))
    ax = fig.gca()
    img = ax.imshow(img_list[0])
    anim = animation.FuncAnimation(fig, animate, init_func=init,
                                 frames=len(img_list), interval=60, blit=True)
    return anim

HTML(plot_images(frames).to_jshtml())
Animation size has reached 21159316 bytes, exceeding the limit of 20971520.0. If you're sure you want a larger animation embedded, set the animation.embed_limit rc parameter to a larger value (in MB). This and further frames will be dropped.
[12]: