How to use Deep ensembles in BaaL

Ensemble are one of the easiest form of Bayesian deep learning. The main drawback from this approach is the important amount of computational resources needed to perform it. In this notebook, we will present BaaL’s Ensemble API namely EnsembleModelWrapper.

This notebook is similar to our notebook on how to do research, we suggest you look at it first if you haven’t.

Additional resources

  • More info on the inner working of Active Learning Dataset here.

  • To know more about Bayesian deep learning please see our Literature review.

[1]:
import random
from copy import deepcopy
from dataclasses import dataclass

import numpy as np
import torch.backends
from torch import optim, nn
from torch.hub import load_state_dict_from_url
from torch.nn import CrossEntropyLoss
from torchvision import datasets
from torchvision import models
from torchvision.transforms import transforms
from tqdm.autonotebook import tqdm

from baal.active import get_heuristic, ActiveLearningDataset
from baal.active.active_loop import ActiveLearningLoop
from baal.ensemble import EnsembleModelWrapper

def vgg16(num_classes):
    model = models.vgg16(pretrained=False, num_classes=num_classes)
    weights = load_state_dict_from_url('https://download.pytorch.org/models/vgg16-397923af.pth')
    weights = {k: v for k, v in weights.items() if 'classifier.6' not in k}
    model.load_state_dict(weights, strict=False)
    return model

def weights_reset(m):
    if isinstance(m, nn.Linear):
        nn.init.normal_(m.weight, 0, 0.01)
        nn.init.constant_(m.bias, 0)
[2]:
@dataclass
class ExperimentConfig:
    epoch: int = 15000//256
    batch_size: int = 32
    initial_pool: int = 512
    query_size: int = 100
    lr: float = 0.001
    heuristic: str = 'bald'
    iterations: int = 5 # Set a low number here since each iteration will train a new model.
    training_duration: int = 10


def get_datasets(initial_pool):
    transform = transforms.Compose(
        [transforms.Resize((32, 32)),
         transforms.RandomHorizontalFlip(),
         transforms.RandomRotation(30),
         transforms.ToTensor(),
         transforms.Normalize(3 * [0.5], 3 * [0.5]), ])
    test_transform = transforms.Compose(
        [
            transforms.Resize((32, 32)),
            transforms.ToTensor(),
            transforms.Normalize(3 * [0.5], 3 * [0.5]),
        ]
    )
    train_ds = datasets.CIFAR10('.', train=True,
                                transform=transform, target_transform=None, download=True)

    # In a real application, you will want a validation set here.
    test_set = datasets.CIFAR10('.', train=False,
                                transform=test_transform, target_transform=None, download=True)

    # Here we set `pool_specifics`, where we set the transform attribute for the pool.
    active_set = ActiveLearningDataset(train_ds, pool_specifics={'transform': test_transform})

    # We start labeling randomly.
    active_set.label_randomly(initial_pool)
    return active_set, test_set
[3]:
hyperparams = ExperimentConfig()
use_cuda = torch.cuda.is_available()
torch.backends.cudnn.benchmark = True
random.seed(1337)
torch.manual_seed(1337)
if not use_cuda:
    print("warning, the experiments would take ages to run on cpu")

# Get datasets
active_set, test_set = get_datasets(hyperparams.initial_pool)

heuristic = get_heuristic(hyperparams.heuristic)
criterion = CrossEntropyLoss()
model = vgg16(num_classes=10)

if use_cuda:
    model.cuda()
optimizer = optim.SGD(model.parameters(), lr=hyperparams.lr, momentum=0.9)

# Wraps the model into a usable API.
model = EnsembleModelWrapper(model, criterion)

# for prediction we use a smaller batchsize
# since it is slower
active_loop = ActiveLearningLoop(active_set,
                                 model.predict_on_dataset,
                                 heuristic,
                                 hyperparams.query_size,
                                 batch_size=1,
                                 iterations=hyperparams.iterations,
                                 use_cuda=use_cuda,
                                 verbose=False)

# We will reset the weights at each active learning step.
init_weights = deepcopy(model.state_dict())
Files already downloaded and verified
Files already downloaded and verified

Presenting EnsembleModelWrapper

EnsembleModelWrapper is similar to ModelWrapper, but instead of training a single model, we will train multiple. Each model will start its training from a different set parameters.

EnsembleModelWrappe methods:

class EnsembleModelWrapper:
    def add_checkpoint(self):
        """
        Add a checkpoint to the list of weights used for inference.
        """

    def clear_checkpoints(self):
        """
        Clear the list of saved checkpoints.
        """

As you see in the next cell, we call both of these methods alternatively. We train N models, calling add_checkpoint, perform the active learning step and then restart by calling clear_checkpoints.

[4]:
labelling_progress = active_set._labelled.copy().astype(np.uint16)
report = []
for epoch in tqdm(range(hyperparams.epoch)):
    model.clear_checkpoints()
    # Load the initial weights.
    for model_iter in range(hyperparams.iterations):
        print(f"Training model {model_iter}")
        model.load_state_dict(init_weights)
        model.model.apply(weights_reset)
        _ = model.train_on_dataset(active_set, optimizer=optimizer, batch_size=hyperparams.batch_size,
                                 use_cuda=use_cuda, epoch=hyperparams.training_duration)
        model.add_checkpoint()



    # Get test NLL!
    model.test_on_dataset(test_set, hyperparams.batch_size, use_cuda,
                          average_predictions=hyperparams.iterations)
    metrics = model.metrics

    # We can now label the most uncertain samples according to our heuristic.
    should_continue = active_loop.step()
    # Keep track of progress
    labelling_progress += active_set._labelled.astype(np.uint16)
    if not should_continue:
            break

    test_loss = metrics['test_loss'].value
    logs = {
        "test_nll": test_loss,
        "epoch": epoch,
        "Next Training set size": len(active_set)
    }
    report.append(logs)
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T15:03:23.225157Z [info     ] Starting training              dataset=512 epoch=10
/opt/conda/lib/python3.9/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 1, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(
/opt/conda/lib/python3.9/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T15:03:54.272799Z [info     ] Training complete              train_loss=0.8364141583442688
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T15:03:54.281157Z [info     ] Starting training              dataset=512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T15:04:27.085055Z [info     ] Training complete              train_loss=0.8260515332221985
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T15:04:27.092947Z [info     ] Starting training              dataset=512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T15:05:00.277956Z [info     ] Training complete              train_loss=0.8478720188140869
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T15:05:00.286100Z [info     ] Starting training              dataset=512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T15:05:33.876247Z [info     ] Training complete              train_loss=0.8530490398406982
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T15:05:33.890366Z [info     ] Starting training              dataset=512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T15:06:08.077229Z [info     ] Training complete              train_loss=0.8666098117828369
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T15:06:08.084093Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T15:06:22.489413Z [info     ] Evaluation complete            test_loss=1.082019567489624
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T15:06:22.498875Z [info     ] Start Predict                  dataset=49488
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T15:23:19.746892Z [info     ] Starting training              dataset=612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T15:23:49.982128Z [info     ] Training complete              train_loss=0.8243563771247864
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T15:23:49.989835Z [info     ] Starting training              dataset=612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T15:24:21.172522Z [info     ] Training complete              train_loss=0.9930000305175781
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T15:24:21.180164Z [info     ] Starting training              dataset=612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T15:24:54.582503Z [info     ] Training complete              train_loss=0.9522870779037476
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T15:24:54.590289Z [info     ] Starting training              dataset=612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T15:25:27.677519Z [info     ] Training complete              train_loss=1.0974868535995483
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T15:25:27.684730Z [info     ] Starting training              dataset=612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T15:26:00.088513Z [info     ] Training complete              train_loss=1.154961109161377
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T15:26:00.169698Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T15:26:13.881201Z [info     ] Evaluation complete            test_loss=1.050412893295288
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T15:26:13.888829Z [info     ] Start Predict                  dataset=49388
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T15:43:24.920358Z [info     ] Starting training              dataset=712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T15:43:57.485136Z [info     ] Training complete              train_loss=0.932747483253479
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T15:43:57.492869Z [info     ] Starting training              dataset=712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T15:44:30.077726Z [info     ] Training complete              train_loss=0.918374240398407
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T15:44:30.085204Z [info     ] Starting training              dataset=712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T15:45:02.867968Z [info     ] Training complete              train_loss=0.9606548547744751
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T15:45:02.875076Z [info     ] Starting training              dataset=712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T15:45:35.279739Z [info     ] Training complete              train_loss=1.016992211341858
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T15:45:35.287374Z [info     ] Starting training              dataset=712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T15:46:07.184076Z [info     ] Training complete              train_loss=0.9882396459579468
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T15:46:07.189735Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T15:46:20.788120Z [info     ] Evaluation complete            test_loss=1.0184006690979004
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T15:46:20.795889Z [info     ] Start Predict                  dataset=49288
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T16:03:30.744977Z [info     ] Starting training              dataset=812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T16:04:04.781179Z [info     ] Training complete              train_loss=0.9256729483604431
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T16:04:04.788756Z [info     ] Starting training              dataset=812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T16:04:38.886644Z [info     ] Training complete              train_loss=0.8962534666061401
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T16:04:38.894324Z [info     ] Starting training              dataset=812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T16:05:12.584859Z [info     ] Training complete              train_loss=1.099798560142517
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T16:05:12.592090Z [info     ] Starting training              dataset=812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T16:05:46.968839Z [info     ] Training complete              train_loss=0.9000493288040161
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T16:05:46.976785Z [info     ] Starting training              dataset=812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T16:06:20.685078Z [info     ] Training complete              train_loss=0.9171968698501587
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T16:06:20.690647Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T16:06:34.579119Z [info     ] Evaluation complete            test_loss=0.9765419960021973
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T16:06:34.586519Z [info     ] Start Predict                  dataset=49188
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T16:23:45.107789Z [info     ] Starting training              dataset=912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T16:24:20.079465Z [info     ] Training complete              train_loss=0.8302789926528931
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T16:24:20.086495Z [info     ] Starting training              dataset=912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T16:24:54.285867Z [info     ] Training complete              train_loss=0.9873713254928589
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T16:24:54.295280Z [info     ] Starting training              dataset=912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T16:25:28.584150Z [info     ] Training complete              train_loss=0.8690028786659241
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T16:25:28.591403Z [info     ] Starting training              dataset=912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T16:26:02.984630Z [info     ] Training complete              train_loss=0.9912691712379456
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T16:26:02.991666Z [info     ] Starting training              dataset=912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T16:26:37.672282Z [info     ] Training complete              train_loss=0.9958629012107849
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T16:26:37.677315Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T16:26:51.273046Z [info     ] Evaluation complete            test_loss=0.9095031023025513
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T16:26:51.281388Z [info     ] Start Predict                  dataset=49088
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T16:43:41.295745Z [info     ] Starting training              dataset=1012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T16:44:16.976296Z [info     ] Training complete              train_loss=0.9254322052001953
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T16:44:16.983815Z [info     ] Starting training              dataset=1012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T16:44:52.478948Z [info     ] Training complete              train_loss=1.0186196565628052
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T16:44:52.485844Z [info     ] Starting training              dataset=1012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T16:45:27.668294Z [info     ] Training complete              train_loss=0.9414865970611572
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T16:45:27.675891Z [info     ] Starting training              dataset=1012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T16:46:03.285576Z [info     ] Training complete              train_loss=0.9331195950508118
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T16:46:03.292214Z [info     ] Starting training              dataset=1012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T16:46:39.089283Z [info     ] Training complete              train_loss=1.0527617931365967
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T16:46:39.165706Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T16:46:52.691071Z [info     ] Evaluation complete            test_loss=0.9129394888877869
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T16:46:52.769545Z [info     ] Start Predict                  dataset=48988
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T17:03:22.458586Z [info     ] Starting training              dataset=1112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T17:03:59.079071Z [info     ] Training complete              train_loss=0.9513149857521057
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T17:03:59.085982Z [info     ] Starting training              dataset=1112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T17:04:35.268902Z [info     ] Training complete              train_loss=1.043251633644104
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T17:04:35.276104Z [info     ] Starting training              dataset=1112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T17:05:11.276178Z [info     ] Training complete              train_loss=0.911968469619751
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T17:05:11.283077Z [info     ] Starting training              dataset=1112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T17:05:47.967951Z [info     ] Training complete              train_loss=0.9133453369140625
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T17:05:47.975760Z [info     ] Starting training              dataset=1112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T17:06:23.978351Z [info     ] Training complete              train_loss=0.9577516913414001
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T17:06:23.983301Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T17:06:36.879626Z [info     ] Evaluation complete            test_loss=0.8657751083374023
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T17:06:36.886557Z [info     ] Start Predict                  dataset=48888
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T17:23:02.721417Z [info     ] Starting training              dataset=1212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T17:23:39.677137Z [info     ] Training complete              train_loss=1.0441893339157104
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T17:23:39.685051Z [info     ] Starting training              dataset=1212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T17:24:16.871166Z [info     ] Training complete              train_loss=0.929194986820221
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T17:24:16.881245Z [info     ] Starting training              dataset=1212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T17:24:54.368581Z [info     ] Training complete              train_loss=1.0284929275512695
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T17:24:54.375269Z [info     ] Starting training              dataset=1212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T17:25:31.480339Z [info     ] Training complete              train_loss=0.9321802854537964
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T17:25:31.487068Z [info     ] Starting training              dataset=1212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T17:26:09.074131Z [info     ] Training complete              train_loss=0.973421573638916
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T17:26:09.080513Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T17:26:22.782218Z [info     ] Evaluation complete            test_loss=0.8525174260139465
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T17:26:22.793393Z [info     ] Start Predict                  dataset=48788
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T17:42:46.221663Z [info     ] Starting training              dataset=1312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T17:43:24.664875Z [info     ] Training complete              train_loss=0.9666503071784973
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T17:43:24.672229Z [info     ] Starting training              dataset=1312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T17:44:03.190809Z [info     ] Training complete              train_loss=1.002950668334961
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T17:44:03.269028Z [info     ] Starting training              dataset=1312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T17:44:41.075336Z [info     ] Training complete              train_loss=0.9888299107551575
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T17:44:41.082244Z [info     ] Starting training              dataset=1312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T17:45:19.077166Z [info     ] Training complete              train_loss=0.97415691614151
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T17:45:19.084448Z [info     ] Starting training              dataset=1312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T17:45:57.374593Z [info     ] Training complete              train_loss=1.0200046300888062
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T17:45:57.380081Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T17:46:10.685676Z [info     ] Evaluation complete            test_loss=0.8271186947822571
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T17:46:10.693032Z [info     ] Start Predict                  dataset=48688
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T18:02:32.635236Z [info     ] Starting training              dataset=1412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T18:03:12.272650Z [info     ] Training complete              train_loss=1.075010895729065
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T18:03:12.280246Z [info     ] Starting training              dataset=1412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T18:03:52.190764Z [info     ] Training complete              train_loss=0.992854654788971
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T18:03:52.198475Z [info     ] Starting training              dataset=1412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T18:04:32.369968Z [info     ] Training complete              train_loss=1.0081067085266113
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T18:04:32.376590Z [info     ] Starting training              dataset=1412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T18:05:12.665417Z [info     ] Training complete              train_loss=0.9663589596748352
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T18:05:12.672271Z [info     ] Starting training              dataset=1412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T18:05:52.280446Z [info     ] Training complete              train_loss=1.04006028175354
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T18:05:52.285840Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T18:06:05.572855Z [info     ] Evaluation complete            test_loss=0.804664671421051
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T18:06:05.580908Z [info     ] Start Predict                  dataset=48588
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T18:22:25.401360Z [info     ] Starting training              dataset=1512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T18:23:06.483224Z [info     ] Training complete              train_loss=0.9277119040489197
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T18:23:06.491979Z [info     ] Starting training              dataset=1512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T18:23:48.188278Z [info     ] Training complete              train_loss=1.0428746938705444
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T18:23:48.267931Z [info     ] Starting training              dataset=1512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T18:24:30.874456Z [info     ] Training complete              train_loss=0.9510531425476074
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T18:24:30.881489Z [info     ] Starting training              dataset=1512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T18:25:12.688178Z [info     ] Training complete              train_loss=0.9416515827178955
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T18:25:12.766617Z [info     ] Starting training              dataset=1512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T18:25:53.775859Z [info     ] Training complete              train_loss=0.9402214884757996
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T18:25:53.780544Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T18:26:06.993932Z [info     ] Evaluation complete            test_loss=0.7674832940101624
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T18:26:07.068616Z [info     ] Start Predict                  dataset=48488
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T18:42:29.514702Z [info     ] Starting training              dataset=1612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T18:43:12.192095Z [info     ] Training complete              train_loss=1.028326153755188
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T18:43:12.266413Z [info     ] Starting training              dataset=1612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T18:43:53.779097Z [info     ] Training complete              train_loss=0.9863273501396179
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T18:43:53.786243Z [info     ] Starting training              dataset=1612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T18:44:36.290856Z [info     ] Training complete              train_loss=0.9068755507469177
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T18:44:36.366920Z [info     ] Starting training              dataset=1612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T18:45:18.870138Z [info     ] Training complete              train_loss=1.023092269897461
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T18:45:18.877238Z [info     ] Starting training              dataset=1612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T18:46:00.383976Z [info     ] Training complete              train_loss=0.965911328792572
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T18:46:00.388895Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T18:46:13.684169Z [info     ] Evaluation complete            test_loss=0.7569476962089539
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T18:46:13.691282Z [info     ] Start Predict                  dataset=48388
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T19:02:30.984315Z [info     ] Starting training              dataset=1712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T19:03:15.674997Z [info     ] Training complete              train_loss=0.9350682497024536
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T19:03:15.681606Z [info     ] Starting training              dataset=1712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T19:03:58.887455Z [info     ] Training complete              train_loss=0.9201800227165222
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T19:03:58.894480Z [info     ] Starting training              dataset=1712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T19:04:41.578060Z [info     ] Training complete              train_loss=0.905670166015625
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T19:04:41.585305Z [info     ] Starting training              dataset=1712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T19:05:24.475178Z [info     ] Training complete              train_loss=0.9620814323425293
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T19:05:24.482448Z [info     ] Starting training              dataset=1712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T19:06:07.490241Z [info     ] Training complete              train_loss=0.9224438071250916
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T19:06:07.570348Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T19:06:20.696662Z [info     ] Evaluation complete            test_loss=0.7361646890640259
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T19:06:20.771211Z [info     ] Start Predict                  dataset=48288
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T19:22:41.163738Z [info     ] Starting training              dataset=1812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T19:23:25.570368Z [info     ] Training complete              train_loss=0.9820511937141418
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T19:23:25.579157Z [info     ] Starting training              dataset=1812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T19:24:10.075295Z [info     ] Training complete              train_loss=0.9375017881393433
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T19:24:10.083318Z [info     ] Starting training              dataset=1812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T19:24:54.880439Z [info     ] Training complete              train_loss=0.9257637858390808
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T19:24:54.888091Z [info     ] Starting training              dataset=1812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T19:25:39.078416Z [info     ] Training complete              train_loss=0.9971398711204529
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T19:25:39.086671Z [info     ] Starting training              dataset=1812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T19:26:23.472168Z [info     ] Training complete              train_loss=1.013749599456787
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T19:26:23.476976Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T19:26:36.574364Z [info     ] Evaluation complete            test_loss=0.756501317024231
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T19:26:36.581478Z [info     ] Start Predict                  dataset=48188
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T19:43:22.766592Z [info     ] Starting training              dataset=1912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T19:44:11.265697Z [info     ] Training complete              train_loss=0.9557903409004211
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T19:44:11.274063Z [info     ] Starting training              dataset=1912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T19:44:59.982236Z [info     ] Training complete              train_loss=0.9173471927642822
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T19:44:59.989619Z [info     ] Starting training              dataset=1912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T19:45:48.168328Z [info     ] Training complete              train_loss=1.0321499109268188
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T19:45:48.176667Z [info     ] Starting training              dataset=1912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T19:46:35.773009Z [info     ] Training complete              train_loss=0.9191875457763672
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T19:46:35.781286Z [info     ] Starting training              dataset=1912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T19:47:22.576191Z [info     ] Training complete              train_loss=0.9246641397476196
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T19:47:22.581554Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T19:47:36.582237Z [info     ] Evaluation complete            test_loss=0.7167291045188904
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T19:47:36.589409Z [info     ] Start Predict                  dataset=48088
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T20:04:31.552071Z [info     ] Starting training              dataset=2012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T20:05:20.382208Z [info     ] Training complete              train_loss=0.9722501039505005
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T20:05:20.390139Z [info     ] Starting training              dataset=2012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T20:06:09.175924Z [info     ] Training complete              train_loss=0.9446245431900024
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T20:06:09.183759Z [info     ] Starting training              dataset=2012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T20:06:58.075425Z [info     ] Training complete              train_loss=0.9757698774337769
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T20:06:58.083521Z [info     ] Starting training              dataset=2012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T20:07:46.880669Z [info     ] Training complete              train_loss=0.9409571886062622
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T20:07:46.888642Z [info     ] Starting training              dataset=2012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T20:08:34.876303Z [info     ] Training complete              train_loss=0.9852419495582581
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T20:08:34.881729Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T20:08:48.985392Z [info     ] Evaluation complete            test_loss=0.723760724067688
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T20:08:48.992518Z [info     ] Start Predict                  dataset=47988
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T20:25:34.747891Z [info     ] Starting training              dataset=2112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T20:26:23.872387Z [info     ] Training complete              train_loss=0.9171951413154602
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T20:26:23.880777Z [info     ] Starting training              dataset=2112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T20:27:13.165485Z [info     ] Training complete              train_loss=0.9799726009368896
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T20:27:13.173679Z [info     ] Starting training              dataset=2112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T20:28:03.283328Z [info     ] Training complete              train_loss=0.9386671185493469
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T20:28:03.290982Z [info     ] Starting training              dataset=2112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T20:28:52.777442Z [info     ] Training complete              train_loss=0.978201687335968
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T20:28:52.784738Z [info     ] Starting training              dataset=2112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T20:29:42.877656Z [info     ] Training complete              train_loss=0.9618527293205261
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T20:29:42.883872Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T20:29:57.578619Z [info     ] Evaluation complete            test_loss=0.7093442678451538
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T20:29:57.585998Z [info     ] Start Predict                  dataset=47888
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T20:46:40.178333Z [info     ] Starting training              dataset=2212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T20:47:31.289002Z [info     ] Training complete              train_loss=0.9138000011444092
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T20:47:31.296781Z [info     ] Starting training              dataset=2212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T20:48:21.980726Z [info     ] Training complete              train_loss=1.0114140510559082
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T20:48:21.987649Z [info     ] Starting training              dataset=2212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T20:49:12.391125Z [info     ] Training complete              train_loss=1.171674370765686
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T20:49:12.467107Z [info     ] Starting training              dataset=2212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T20:50:02.169114Z [info     ] Training complete              train_loss=0.9495107531547546
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T20:50:02.177071Z [info     ] Starting training              dataset=2212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T20:50:52.579582Z [info     ] Training complete              train_loss=1.0180065631866455
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T20:50:52.585367Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T20:51:06.175383Z [info     ] Evaluation complete            test_loss=0.7294108271598816
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T20:51:06.183712Z [info     ] Start Predict                  dataset=47788
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T21:07:22.539634Z [info     ] Starting training              dataset=2312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T21:08:16.280207Z [info     ] Training complete              train_loss=0.9690930247306824
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T21:08:16.286669Z [info     ] Starting training              dataset=2312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T21:09:09.184223Z [info     ] Training complete              train_loss=0.9622244834899902
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T21:09:09.191550Z [info     ] Starting training              dataset=2312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T21:10:02.666252Z [info     ] Training complete              train_loss=0.9211553335189819
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T21:10:02.674865Z [info     ] Starting training              dataset=2312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T21:10:57.070214Z [info     ] Training complete              train_loss=0.9597650766372681
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T21:10:57.077798Z [info     ] Starting training              dataset=2312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T21:11:50.182542Z [info     ] Training complete              train_loss=0.964903712272644
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T21:11:50.188292Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T21:12:03.866348Z [info     ] Evaluation complete            test_loss=0.6798441410064697
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T21:12:03.874067Z [info     ] Start Predict                  dataset=47688
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T21:28:11.883129Z [info     ] Starting training              dataset=2412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T21:29:03.879859Z [info     ] Training complete              train_loss=1.0068079233169556
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T21:29:03.887980Z [info     ] Starting training              dataset=2412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T21:29:56.381578Z [info     ] Training complete              train_loss=0.9520434141159058
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T21:29:56.390312Z [info     ] Starting training              dataset=2412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T21:30:48.191878Z [info     ] Training complete              train_loss=0.9582843780517578
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T21:30:48.266421Z [info     ] Starting training              dataset=2412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T21:31:40.281782Z [info     ] Training complete              train_loss=1.0118552446365356
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T21:31:40.288720Z [info     ] Starting training              dataset=2412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T21:32:32.970914Z [info     ] Training complete              train_loss=0.9314520955085754
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T21:32:32.976789Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T21:32:46.474366Z [info     ] Evaluation complete            test_loss=0.6736973524093628
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T21:32:46.480994Z [info     ] Start Predict                  dataset=47588
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T21:48:57.148416Z [info     ] Starting training              dataset=2512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T21:49:50.280161Z [info     ] Training complete              train_loss=0.9337089657783508
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T21:49:50.286868Z [info     ] Starting training              dataset=2512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T21:50:43.675634Z [info     ] Training complete              train_loss=1.005215048789978
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T21:50:43.686572Z [info     ] Starting training              dataset=2512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T21:51:37.178821Z [info     ] Training complete              train_loss=0.9765651226043701
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T21:51:37.185438Z [info     ] Starting training              dataset=2512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T21:52:30.772492Z [info     ] Training complete              train_loss=1.0099214315414429
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T21:52:30.780204Z [info     ] Starting training              dataset=2512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T21:53:25.086434Z [info     ] Training complete              train_loss=1.1066418886184692
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T21:53:25.092135Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T21:53:38.890457Z [info     ] Evaluation complete            test_loss=0.6832886934280396
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T21:53:38.968482Z [info     ] Start Predict                  dataset=47488
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T22:09:47.037387Z [info     ] Starting training              dataset=2612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T22:10:41.882671Z [info     ] Training complete              train_loss=0.9732304811477661
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T22:10:41.889490Z [info     ] Starting training              dataset=2612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T22:11:36.080223Z [info     ] Training complete              train_loss=0.9936309456825256
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T22:11:36.087166Z [info     ] Starting training              dataset=2612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T22:12:30.385642Z [info     ] Training complete              train_loss=0.8979130983352661
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T22:12:30.392645Z [info     ] Starting training              dataset=2612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T22:13:24.886768Z [info     ] Training complete              train_loss=1.0363106727600098
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T22:13:24.894615Z [info     ] Starting training              dataset=2612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T22:14:19.284788Z [info     ] Training complete              train_loss=0.9428572058677673
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T22:14:19.289613Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T22:14:32.774810Z [info     ] Evaluation complete            test_loss=0.6570330858230591
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T22:14:32.781457Z [info     ] Start Predict                  dataset=47388
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T22:30:41.520167Z [info     ] Starting training              dataset=2712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T22:31:38.892835Z [info     ] Training complete              train_loss=0.922064483165741
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T22:31:38.971116Z [info     ] Starting training              dataset=2712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T22:32:34.989046Z [info     ] Training complete              train_loss=0.8859795331954956
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T22:32:35.067765Z [info     ] Starting training              dataset=2712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T22:33:32.582960Z [info     ] Training complete              train_loss=0.9763253331184387
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T22:33:32.590187Z [info     ] Starting training              dataset=2712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T22:34:29.085416Z [info     ] Training complete              train_loss=0.9625152349472046
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T22:34:29.091940Z [info     ] Starting training              dataset=2712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T22:35:25.181361Z [info     ] Training complete              train_loss=0.9341091513633728
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T22:35:25.186428Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T22:35:39.483789Z [info     ] Evaluation complete            test_loss=0.6503035426139832
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T22:35:39.491229Z [info     ] Start Predict                  dataset=47288
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T22:51:37.149598Z [info     ] Starting training              dataset=2812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T22:52:33.470586Z [info     ] Training complete              train_loss=0.9524716734886169
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T22:52:33.477044Z [info     ] Starting training              dataset=2812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T22:53:29.469538Z [info     ] Training complete              train_loss=0.9347421526908875
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T22:53:29.476470Z [info     ] Starting training              dataset=2812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T22:54:25.470438Z [info     ] Training complete              train_loss=0.96184903383255
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T22:54:25.477112Z [info     ] Starting training              dataset=2812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T22:55:21.368596Z [info     ] Training complete              train_loss=0.9115388989448547
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T22:55:21.375550Z [info     ] Starting training              dataset=2812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T22:56:17.476897Z [info     ] Training complete              train_loss=0.9248518347740173
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T22:56:17.481429Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T22:56:30.870313Z [info     ] Evaluation complete            test_loss=0.6633090972900391
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T22:56:30.876905Z [info     ] Start Predict                  dataset=47188
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T23:12:24.288451Z [info     ] Starting training              dataset=2912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T23:13:21.378543Z [info     ] Training complete              train_loss=0.9456859230995178
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T23:13:21.385101Z [info     ] Starting training              dataset=2912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T23:14:18.189772Z [info     ] Training complete              train_loss=0.9629805684089661
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T23:14:18.267688Z [info     ] Starting training              dataset=2912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T23:15:15.173773Z [info     ] Training complete              train_loss=0.9584450125694275
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T23:15:15.180834Z [info     ] Starting training              dataset=2912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T23:16:12.666025Z [info     ] Training complete              train_loss=0.9231571555137634
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T23:16:12.672462Z [info     ] Starting training              dataset=2912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T23:17:10.073503Z [info     ] Training complete              train_loss=0.9232361912727356
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T23:17:10.078705Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T23:17:23.270843Z [info     ] Evaluation complete            test_loss=0.6433601975440979
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T23:17:23.277508Z [info     ] Start Predict                  dataset=47088
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T23:33:28.882364Z [info     ] Starting training              dataset=3012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T23:34:28.174089Z [info     ] Training complete              train_loss=0.9940481781959534
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T23:34:28.181205Z [info     ] Starting training              dataset=3012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T23:35:27.384836Z [info     ] Training complete              train_loss=0.9763245582580566
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T23:35:27.392383Z [info     ] Starting training              dataset=3012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T23:36:27.078643Z [info     ] Training complete              train_loss=0.9977985620498657
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T23:36:27.085664Z [info     ] Starting training              dataset=3012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T23:37:26.387570Z [info     ] Training complete              train_loss=0.9883565902709961
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T23:37:26.394578Z [info     ] Starting training              dataset=3012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T23:38:26.078454Z [info     ] Training complete              train_loss=1.0123181343078613
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T23:38:26.084095Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T23:38:39.683168Z [info     ] Evaluation complete            test_loss=0.6295325756072998
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T23:38:39.690456Z [info     ] Start Predict                  dataset=46988
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T23:54:29.465550Z [info     ] Starting training              dataset=3112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T23:55:29.272588Z [info     ] Training complete              train_loss=0.993098795413971
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T23:55:29.279219Z [info     ] Starting training              dataset=3112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T23:56:29.374525Z [info     ] Training complete              train_loss=0.982636570930481
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T23:56:29.381500Z [info     ] Starting training              dataset=3112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T23:57:28.969284Z [info     ] Training complete              train_loss=0.9553168416023254
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T23:57:28.978303Z [info     ] Starting training              dataset=3112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T23:58:28.386104Z [info     ] Training complete              train_loss=0.9859296679496765
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-28T23:58:28.392510Z [info     ] Starting training              dataset=3112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-28T23:59:28.085509Z [info     ] Training complete              train_loss=0.911634624004364
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-28T23:59:28.090740Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-28T23:59:41.378569Z [info     ] Evaluation complete            test_loss=0.6155526041984558
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-28T23:59:41.388091Z [info     ] Start Predict                  dataset=46888
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T00:15:27.000611Z [info     ] Starting training              dataset=3212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T00:16:27.485323Z [info     ] Training complete              train_loss=0.9690141081809998
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T00:16:27.492326Z [info     ] Starting training              dataset=3212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T00:17:27.971410Z [info     ] Training complete              train_loss=0.9362989664077759
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T00:17:27.978854Z [info     ] Starting training              dataset=3212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T00:18:28.374476Z [info     ] Training complete              train_loss=0.9801816940307617
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T00:18:28.381476Z [info     ] Starting training              dataset=3212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T00:19:28.688675Z [info     ] Training complete              train_loss=0.982334315776825
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T00:19:28.695882Z [info     ] Starting training              dataset=3212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T00:20:29.280950Z [info     ] Training complete              train_loss=1.0058739185333252
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T00:20:29.285715Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T00:20:42.576387Z [info     ] Evaluation complete            test_loss=0.624253511428833
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T00:20:42.582922Z [info     ] Start Predict                  dataset=46788
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T00:36:24.397328Z [info     ] Starting training              dataset=3312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T00:37:25.777847Z [info     ] Training complete              train_loss=0.9525146484375
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T00:37:25.785148Z [info     ] Starting training              dataset=3312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T00:38:27.670850Z [info     ] Training complete              train_loss=0.9617348909378052
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T00:38:27.678614Z [info     ] Starting training              dataset=3312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T00:39:29.474158Z [info     ] Training complete              train_loss=0.9088602066040039
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T00:39:29.480853Z [info     ] Starting training              dataset=3312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T00:40:31.282180Z [info     ] Training complete              train_loss=0.941899836063385
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T00:40:31.288671Z [info     ] Starting training              dataset=3312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T00:41:32.674235Z [info     ] Training complete              train_loss=0.9036476612091064
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T00:41:32.679012Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T00:41:45.880151Z [info     ] Evaluation complete            test_loss=0.6003412008285522
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T00:41:45.886711Z [info     ] Start Predict                  dataset=46688
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T00:57:25.233096Z [info     ] Starting training              dataset=3412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T00:58:27.771124Z [info     ] Training complete              train_loss=0.9652302861213684
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T00:58:27.778539Z [info     ] Starting training              dataset=3412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T00:59:29.682729Z [info     ] Training complete              train_loss=0.9133856892585754
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T00:59:29.689246Z [info     ] Starting training              dataset=3412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T01:00:32.276974Z [info     ] Training complete              train_loss=0.9120355248451233
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T01:00:32.285180Z [info     ] Starting training              dataset=3412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T01:01:34.571251Z [info     ] Training complete              train_loss=0.9987239241600037
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T01:01:34.579233Z [info     ] Starting training              dataset=3412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T01:02:36.971180Z [info     ] Training complete              train_loss=1.0511140823364258
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T01:02:36.976140Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T01:02:50.389403Z [info     ] Evaluation complete            test_loss=0.600139319896698
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T01:02:50.466974Z [info     ] Start Predict                  dataset=46588
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T01:18:50.039255Z [info     ] Starting training              dataset=3512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T01:19:55.272920Z [info     ] Training complete              train_loss=0.9255532026290894
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T01:19:55.280740Z [info     ] Starting training              dataset=3512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T01:21:00.783239Z [info     ] Training complete              train_loss=0.9507350325584412
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T01:21:00.791110Z [info     ] Starting training              dataset=3512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T01:22:06.082850Z [info     ] Training complete              train_loss=0.9777060151100159
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T01:22:06.090186Z [info     ] Starting training              dataset=3512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T01:23:11.572780Z [info     ] Training complete              train_loss=0.9614008069038391
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T01:23:11.581010Z [info     ] Starting training              dataset=3512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T01:24:17.388319Z [info     ] Training complete              train_loss=0.9385112524032593
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T01:24:17.464897Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T01:24:31.274227Z [info     ] Evaluation complete            test_loss=0.5854552388191223
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T01:24:31.281479Z [info     ] Start Predict                  dataset=46488
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T01:40:39.084449Z [info     ] Starting training              dataset=3612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T01:41:51.372209Z [info     ] Training complete              train_loss=0.9902898669242859
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T01:41:51.384139Z [info     ] Starting training              dataset=3612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T01:43:04.776776Z [info     ] Training complete              train_loss=0.9506107568740845
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T01:43:04.784669Z [info     ] Starting training              dataset=3612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T01:44:16.971692Z [info     ] Training complete              train_loss=0.9101556539535522
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T01:44:16.980149Z [info     ] Starting training              dataset=3612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T01:45:30.171964Z [info     ] Training complete              train_loss=0.9209990501403809
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T01:45:30.180936Z [info     ] Starting training              dataset=3612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T01:46:41.988783Z [info     ] Training complete              train_loss=0.9617601633071899
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T01:46:41.995001Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T01:46:56.588932Z [info     ] Evaluation complete            test_loss=0.588925838470459
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T01:46:56.596366Z [info     ] Start Predict                  dataset=46388
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T02:03:02.545742Z [info     ] Starting training              dataset=3712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T02:04:10.991458Z [info     ] Training complete              train_loss=0.902265727519989
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T02:04:11.070627Z [info     ] Starting training              dataset=3712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T02:05:20.275718Z [info     ] Training complete              train_loss=0.9274501204490662
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T02:05:20.283525Z [info     ] Starting training              dataset=3712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T02:06:30.384887Z [info     ] Training complete              train_loss=0.948907732963562
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T02:06:30.392616Z [info     ] Starting training              dataset=3712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T02:07:40.073581Z [info     ] Training complete              train_loss=0.9361139535903931
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T02:07:40.081202Z [info     ] Starting training              dataset=3712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T02:08:49.870122Z [info     ] Training complete              train_loss=0.9426257014274597
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T02:08:49.876067Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T02:09:03.779674Z [info     ] Evaluation complete            test_loss=0.5798165798187256
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T02:09:03.787281Z [info     ] Start Predict                  dataset=46288
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T02:25:00.390871Z [info     ] Starting training              dataset=3812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T02:26:08.266439Z [info     ] Training complete              train_loss=0.9778515100479126
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T02:26:08.273235Z [info     ] Starting training              dataset=3812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T02:27:16.871698Z [info     ] Training complete              train_loss=0.9607134461402893
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T02:27:16.879594Z [info     ] Starting training              dataset=3812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T02:28:25.190132Z [info     ] Training complete              train_loss=1.0153460502624512
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T02:28:25.268855Z [info     ] Starting training              dataset=3812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T02:29:34.683644Z [info     ] Training complete              train_loss=1.0046292543411255
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T02:29:34.692586Z [info     ] Starting training              dataset=3812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T02:30:44.576145Z [info     ] Training complete              train_loss=0.9653735756874084
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T02:30:44.582048Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T02:30:58.377427Z [info     ] Evaluation complete            test_loss=0.5952193140983582
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T02:30:58.384668Z [info     ] Start Predict                  dataset=46188
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T02:46:37.602564Z [info     ] Starting training              dataset=3912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T02:47:48.374654Z [info     ] Training complete              train_loss=0.9733763933181763
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T02:47:48.381969Z [info     ] Starting training              dataset=3912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T02:48:58.977107Z [info     ] Training complete              train_loss=0.9541704058647156
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T02:48:58.984005Z [info     ] Starting training              dataset=3912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T02:50:07.770611Z [info     ] Training complete              train_loss=0.8996275067329407
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T02:50:07.777522Z [info     ] Starting training              dataset=3912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T02:51:17.876229Z [info     ] Training complete              train_loss=0.9932214617729187
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T02:51:17.883210Z [info     ] Starting training              dataset=3912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T02:52:26.082136Z [info     ] Training complete              train_loss=1.0322301387786865
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T02:52:26.087786Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T02:52:39.383655Z [info     ] Evaluation complete            test_loss=0.5735118985176086
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T02:52:39.391237Z [info     ] Start Predict                  dataset=46088
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T03:08:19.372014Z [info     ] Starting training              dataset=4012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T03:09:30.573365Z [info     ] Training complete              train_loss=0.8765227794647217
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T03:09:30.580674Z [info     ] Starting training              dataset=4012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T03:10:40.077172Z [info     ] Training complete              train_loss=0.9590541124343872
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T03:10:40.084249Z [info     ] Starting training              dataset=4012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T03:11:51.185509Z [info     ] Training complete              train_loss=0.9163060784339905
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T03:11:51.192620Z [info     ] Starting training              dataset=4012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T03:13:01.870247Z [info     ] Training complete              train_loss=0.9452018737792969
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T03:13:01.878052Z [info     ] Starting training              dataset=4012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T03:14:12.270683Z [info     ] Training complete              train_loss=0.9404565095901489
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T03:14:12.276100Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T03:14:25.490276Z [info     ] Evaluation complete            test_loss=0.562659740447998
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T03:14:25.569134Z [info     ] Start Predict                  dataset=45988
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T03:30:03.646820Z [info     ] Starting training              dataset=4112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T03:31:13.466228Z [info     ] Training complete              train_loss=1.0031845569610596
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T03:31:13.472767Z [info     ] Starting training              dataset=4112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T03:32:23.173042Z [info     ] Training complete              train_loss=0.9533597230911255
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T03:32:23.179884Z [info     ] Starting training              dataset=4112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T03:33:33.078510Z [info     ] Training complete              train_loss=0.9691417217254639
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T03:33:33.084899Z [info     ] Starting training              dataset=4112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T03:34:42.771816Z [info     ] Training complete              train_loss=0.9820665717124939
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T03:34:42.778569Z [info     ] Starting training              dataset=4112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T03:35:52.866703Z [info     ] Training complete              train_loss=0.9796773195266724
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T03:35:52.871526Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T03:36:05.879288Z [info     ] Evaluation complete            test_loss=0.5647993087768555
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T03:36:05.885746Z [info     ] Start Predict                  dataset=45888
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T03:51:29.489449Z [info     ] Starting training              dataset=4212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T03:52:40.770569Z [info     ] Training complete              train_loss=0.936839759349823
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T03:52:40.778765Z [info     ] Starting training              dataset=4212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T03:55:02.783307Z [info     ] Training complete              train_loss=0.9478463530540466
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T03:55:02.789775Z [info     ] Starting training              dataset=4212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T03:56:14.090717Z [info     ] Training complete              train_loss=0.9612425565719604
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T03:56:14.168573Z [info     ] Starting training              dataset=4212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T03:57:24.880913Z [info     ] Training complete              train_loss=0.9398402571678162
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T03:57:24.885770Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T03:57:38.175341Z [info     ] Evaluation complete            test_loss=0.5565947890281677
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T03:57:38.182963Z [info     ] Start Predict                  dataset=45788
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T04:12:59.505991Z [info     ] Starting training              dataset=4312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T04:14:11.385840Z [info     ] Training complete              train_loss=1.0070544481277466
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T04:14:11.393299Z [info     ] Starting training              dataset=4312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T04:15:23.470047Z [info     ] Training complete              train_loss=0.9844921231269836
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T04:15:23.476889Z [info     ] Starting training              dataset=4312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T04:16:35.675671Z [info     ] Training complete              train_loss=0.9934103488922119
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T04:16:35.682764Z [info     ] Starting training              dataset=4312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T04:17:50.472318Z [info     ] Training complete              train_loss=0.9531852602958679
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T04:17:50.479302Z [info     ] Starting training              dataset=4312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T04:19:05.178550Z [info     ] Training complete              train_loss=0.9936967492103577
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T04:19:05.184399Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T04:19:18.580249Z [info     ] Evaluation complete            test_loss=0.5665599703788757
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T04:19:18.586943Z [info     ] Start Predict                  dataset=45688
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T04:34:51.766049Z [info     ] Starting training              dataset=4412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T04:36:06.384482Z [info     ] Training complete              train_loss=0.9465316534042358
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T04:36:06.391749Z [info     ] Starting training              dataset=4412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T04:37:21.270006Z [info     ] Training complete              train_loss=0.9687955379486084
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T04:37:21.277308Z [info     ] Starting training              dataset=4412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T04:38:34.181247Z [info     ] Training complete              train_loss=0.9473304152488708
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T04:38:34.188378Z [info     ] Starting training              dataset=4412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T04:39:48.682254Z [info     ] Training complete              train_loss=0.9860917329788208
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T04:39:48.688619Z [info     ] Starting training              dataset=4412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T04:41:01.381298Z [info     ] Training complete              train_loss=0.9456995129585266
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T04:41:01.386213Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T04:41:14.570893Z [info     ] Evaluation complete            test_loss=0.5559056997299194
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T04:41:14.578396Z [info     ] Start Predict                  dataset=45588
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T04:56:32.245143Z [info     ] Starting training              dataset=4512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T04:57:46.188021Z [info     ] Training complete              train_loss=0.9349108934402466
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T04:57:46.195296Z [info     ] Starting training              dataset=4512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T04:59:00.277858Z [info     ] Training complete              train_loss=0.974810779094696
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T04:59:00.284605Z [info     ] Starting training              dataset=4512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T05:00:14.582182Z [info     ] Training complete              train_loss=0.9310106635093689
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T05:00:14.589957Z [info     ] Starting training              dataset=4512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T05:01:29.273130Z [info     ] Training complete              train_loss=0.9381789565086365
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T05:01:29.279718Z [info     ] Starting training              dataset=4512 epoch=10
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T05:18:11.286794Z [info     ] Starting training              dataset=4612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T05:19:27.672247Z [info     ] Training complete              train_loss=0.9956920742988586
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T05:19:27.679533Z [info     ] Starting training              dataset=4612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T05:20:44.580321Z [info     ] Training complete              train_loss=0.9173332452774048
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T05:20:44.587566Z [info     ] Starting training              dataset=4612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T05:22:01.084456Z [info     ] Training complete              train_loss=0.9863068461418152
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T05:22:01.091486Z [info     ] Starting training              dataset=4612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T05:23:18.674201Z [info     ] Training complete              train_loss=0.9764572381973267
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T05:23:18.681107Z [info     ] Starting training              dataset=4612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T05:24:35.480255Z [info     ] Training complete              train_loss=1.0168014764785767
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T05:24:35.485128Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T05:24:48.874589Z [info     ] Evaluation complete            test_loss=0.5520719289779663
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T05:24:48.881463Z [info     ] Start Predict                  dataset=45388
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T05:40:04.304007Z [info     ] Starting training              dataset=4712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T05:41:21.974405Z [info     ] Training complete              train_loss=0.9784602522850037
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T05:41:21.981149Z [info     ] Starting training              dataset=4712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T05:42:39.381711Z [info     ] Training complete              train_loss=0.9848698973655701
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T05:42:39.388686Z [info     ] Starting training              dataset=4712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T05:43:57.482323Z [info     ] Training complete              train_loss=0.9804074168205261
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T05:43:57.489593Z [info     ] Starting training              dataset=4712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T05:45:14.773833Z [info     ] Training complete              train_loss=0.9591724276542664
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T05:45:14.781518Z [info     ] Starting training              dataset=4712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T05:46:33.281245Z [info     ] Training complete              train_loss=1.0082279443740845
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T05:46:33.286733Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T05:46:46.591746Z [info     ] Evaluation complete            test_loss=0.5615522861480713
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T05:46:46.665617Z [info     ] Start Predict                  dataset=45288
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T06:01:59.899100Z [info     ] Starting training              dataset=4812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T06:03:19.880195Z [info     ] Training complete              train_loss=0.9809603691101074
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T06:03:19.888224Z [info     ] Starting training              dataset=4812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T06:04:39.375615Z [info     ] Training complete              train_loss=0.973745584487915
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T06:04:39.383822Z [info     ] Starting training              dataset=4812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T06:05:58.485925Z [info     ] Training complete              train_loss=0.9743210673332214
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T06:05:58.493312Z [info     ] Starting training              dataset=4812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T06:07:17.076932Z [info     ] Training complete              train_loss=0.91501784324646
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T06:07:17.083588Z [info     ] Starting training              dataset=4812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T06:08:35.989119Z [info     ] Training complete              train_loss=0.9752476215362549
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T06:08:35.995117Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T06:08:49.377772Z [info     ] Evaluation complete            test_loss=0.5705881118774414
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T06:08:49.385668Z [info     ] Start Predict                  dataset=45188
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T06:23:59.766065Z [info     ] Starting training              dataset=4912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T06:25:20.285377Z [info     ] Training complete              train_loss=0.9525318741798401
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T06:25:20.293520Z [info     ] Starting training              dataset=4912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T06:26:39.891338Z [info     ] Training complete              train_loss=0.9308268427848816
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T06:26:39.970277Z [info     ] Starting training              dataset=4912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T06:27:59.877774Z [info     ] Training complete              train_loss=0.9380195140838623
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T06:27:59.885615Z [info     ] Starting training              dataset=4912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T06:29:19.772513Z [info     ] Training complete              train_loss=0.9800739288330078
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T06:29:19.780060Z [info     ] Starting training              dataset=4912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T06:30:40.389743Z [info     ] Training complete              train_loss=0.9535344243049622
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T06:30:40.395101Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T06:30:53.871705Z [info     ] Evaluation complete            test_loss=0.5377533435821533
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T06:30:53.879359Z [info     ] Start Predict                  dataset=45088
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T06:46:03.495772Z [info     ] Starting training              dataset=5012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T06:47:24.584999Z [info     ] Training complete              train_loss=0.9156500697135925
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T06:47:24.591922Z [info     ] Starting training              dataset=5012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T06:48:45.286743Z [info     ] Training complete              train_loss=0.9553552269935608
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T06:48:45.294576Z [info     ] Starting training              dataset=5012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T06:50:05.775591Z [info     ] Training complete              train_loss=0.9296905994415283
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T06:50:05.783333Z [info     ] Starting training              dataset=5012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T06:51:26.690279Z [info     ] Training complete              train_loss=0.9139729738235474
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T06:51:26.697377Z [info     ] Starting training              dataset=5012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T06:52:47.585409Z [info     ] Training complete              train_loss=0.9346900582313538
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T06:52:47.591531Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T06:53:00.973940Z [info     ] Evaluation complete            test_loss=0.5232065320014954
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T06:53:00.981762Z [info     ] Start Predict                  dataset=44988
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T07:08:07.516317Z [info     ] Starting training              dataset=5112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T07:09:28.877012Z [info     ] Training complete              train_loss=0.9632118940353394
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T07:09:28.884978Z [info     ] Starting training              dataset=5112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T07:10:50.481289Z [info     ] Training complete              train_loss=0.8998667597770691
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T07:10:50.489168Z [info     ] Starting training              dataset=5112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T07:12:12.371390Z [info     ] Training complete              train_loss=0.9703662991523743
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T07:12:12.378297Z [info     ] Starting training              dataset=5112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T07:13:34.783263Z [info     ] Training complete              train_loss=0.911615252494812
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T07:13:34.790932Z [info     ] Starting training              dataset=5112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T07:14:57.593475Z [info     ] Training complete              train_loss=0.9733651280403137
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T07:14:57.666507Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T07:15:10.972475Z [info     ] Evaluation complete            test_loss=0.5270566344261169
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T07:15:10.979908Z [info     ] Start Predict                  dataset=44888
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T07:30:15.747722Z [info     ] Starting training              dataset=5212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T07:31:38.673276Z [info     ] Training complete              train_loss=0.8865935802459717
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T07:31:38.680685Z [info     ] Starting training              dataset=5212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T07:33:01.991828Z [info     ] Training complete              train_loss=0.9334018230438232
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T07:33:02.072821Z [info     ] Starting training              dataset=5212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T07:34:24.690463Z [info     ] Training complete              train_loss=0.9533438682556152
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T07:34:24.765045Z [info     ] Starting training              dataset=5212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T07:35:48.084328Z [info     ] Training complete              train_loss=0.9691643118858337
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T07:35:48.091523Z [info     ] Starting training              dataset=5212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T07:37:10.381950Z [info     ] Training complete              train_loss=0.9655492305755615
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T07:37:10.387022Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T07:37:23.881587Z [info     ] Evaluation complete            test_loss=0.5173717141151428
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T07:37:23.888603Z [info     ] Start Predict                  dataset=44788
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T07:52:26.318401Z [info     ] Starting training              dataset=5312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T07:53:50.090727Z [info     ] Training complete              train_loss=0.9988188147544861
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T07:53:50.166640Z [info     ] Starting training              dataset=5312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T07:55:13.680111Z [info     ] Training complete              train_loss=0.9845055341720581
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T07:55:13.687638Z [info     ] Starting training              dataset=5312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T07:56:38.190458Z [info     ] Training complete              train_loss=0.9391801357269287
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T07:56:38.268577Z [info     ] Starting training              dataset=5312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T07:58:02.480110Z [info     ] Training complete              train_loss=0.9259323477745056
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T07:58:02.486874Z [info     ] Starting training              dataset=5312 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T07:59:26.789141Z [info     ] Training complete              train_loss=0.9599472284317017
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T07:59:26.794346Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T07:59:40.283603Z [info     ] Evaluation complete            test_loss=0.5252811908721924
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T07:59:40.292579Z [info     ] Start Predict                  dataset=44688
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T08:14:41.452168Z [info     ] Starting training              dataset=5412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T08:16:06.479870Z [info     ] Training complete              train_loss=0.9627469182014465
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T08:16:06.487092Z [info     ] Starting training              dataset=5412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T08:17:32.184454Z [info     ] Training complete              train_loss=0.9845385551452637
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T08:17:32.193680Z [info     ] Starting training              dataset=5412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T08:18:58.381604Z [info     ] Training complete              train_loss=0.9816064238548279
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T08:18:58.389140Z [info     ] Starting training              dataset=5412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T08:20:23.482114Z [info     ] Training complete              train_loss=0.9480637311935425
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T08:20:23.489218Z [info     ] Starting training              dataset=5412 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T08:21:48.982720Z [info     ] Training complete              train_loss=0.9441766142845154
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T08:21:48.988076Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T08:22:02.376927Z [info     ] Evaluation complete            test_loss=0.5122457146644592
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T08:22:02.384138Z [info     ] Start Predict                  dataset=44588
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T08:36:59.422635Z [info     ] Starting training              dataset=5512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T08:38:25.077782Z [info     ] Training complete              train_loss=0.9657859206199646
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T08:38:25.084857Z [info     ] Starting training              dataset=5512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T08:39:51.477342Z [info     ] Training complete              train_loss=0.9360918402671814
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T08:39:51.483954Z [info     ] Starting training              dataset=5512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T08:41:17.575167Z [info     ] Training complete              train_loss=0.938201904296875
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T08:41:17.582039Z [info     ] Starting training              dataset=5512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T08:42:44.375687Z [info     ] Training complete              train_loss=0.9420015811920166
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T08:42:44.383021Z [info     ] Starting training              dataset=5512 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T08:44:09.980322Z [info     ] Training complete              train_loss=0.9979342222213745
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T08:44:09.985577Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T08:44:23.583289Z [info     ] Evaluation complete            test_loss=0.5096003413200378
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T08:44:23.592039Z [info     ] Start Predict                  dataset=44488
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T08:59:20.005780Z [info     ] Starting training              dataset=5612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T09:00:47.983898Z [info     ] Training complete              train_loss=0.9347040057182312
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T09:00:47.992177Z [info     ] Starting training              dataset=5612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T09:02:14.675232Z [info     ] Training complete              train_loss=0.9421486258506775
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T09:02:14.682607Z [info     ] Starting training              dataset=5612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T09:03:42.471977Z [info     ] Training complete              train_loss=0.9339758157730103
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T09:03:42.478964Z [info     ] Starting training              dataset=5612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T09:05:09.588801Z [info     ] Training complete              train_loss=0.940785825252533
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T09:05:09.596157Z [info     ] Starting training              dataset=5612 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T09:06:36.388508Z [info     ] Training complete              train_loss=0.9172220230102539
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T09:06:36.394159Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T09:06:49.779906Z [info     ] Evaluation complete            test_loss=0.5034990906715393
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T09:06:49.787079Z [info     ] Start Predict                  dataset=44388
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T09:21:42.877712Z [info     ] Starting training              dataset=5712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T09:23:11.072422Z [info     ] Training complete              train_loss=0.9773575663566589
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T09:23:11.080054Z [info     ] Starting training              dataset=5712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T09:24:39.979758Z [info     ] Training complete              train_loss=0.9121693968772888
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T09:24:39.986819Z [info     ] Starting training              dataset=5712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T09:26:08.273716Z [info     ] Training complete              train_loss=0.9608463644981384
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T09:26:08.282000Z [info     ] Starting training              dataset=5712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T09:27:36.386442Z [info     ] Training complete              train_loss=0.9310780167579651
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T09:27:36.393392Z [info     ] Starting training              dataset=5712 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T09:29:04.675088Z [info     ] Training complete              train_loss=0.9387174844741821
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T09:29:04.681026Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T09:29:18.473080Z [info     ] Evaluation complete            test_loss=0.502855658531189
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T09:29:18.480180Z [info     ] Start Predict                  dataset=44288
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T09:44:11.984106Z [info     ] Starting training              dataset=5812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T09:45:41.684589Z [info     ] Training complete              train_loss=0.9386858940124512
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T09:45:41.691708Z [info     ] Starting training              dataset=5812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T09:47:11.889461Z [info     ] Training complete              train_loss=0.9746957421302795
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T09:47:11.896628Z [info     ] Starting training              dataset=5812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T09:48:41.878885Z [info     ] Training complete              train_loss=0.9614348411560059
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T09:48:41.885215Z [info     ] Starting training              dataset=5812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T09:50:10.674407Z [info     ] Training complete              train_loss=0.9561299681663513
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T09:50:10.680954Z [info     ] Starting training              dataset=5812 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T09:51:40.788730Z [info     ] Training complete              train_loss=0.9587898850440979
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T09:51:40.794217Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T09:51:54.684919Z [info     ] Evaluation complete            test_loss=0.506767749786377
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T09:51:54.692014Z [info     ] Start Predict                  dataset=44188
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T10:06:42.728074Z [info     ] Starting training              dataset=5912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T10:08:13.492879Z [info     ] Training complete              train_loss=0.9357155561447144
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T10:08:13.571886Z [info     ] Starting training              dataset=5912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T10:09:43.775531Z [info     ] Training complete              train_loss=0.928965151309967
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T10:09:43.783025Z [info     ] Starting training              dataset=5912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T10:11:15.290185Z [info     ] Training complete              train_loss=0.9106123447418213
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T10:11:15.366495Z [info     ] Starting training              dataset=5912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T10:12:46.477856Z [info     ] Training complete              train_loss=0.9481375813484192
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T10:12:46.484984Z [info     ] Starting training              dataset=5912 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T10:14:20.480515Z [info     ] Training complete              train_loss=0.9085630774497986
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T10:14:20.485748Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T10:14:34.189367Z [info     ] Evaluation complete            test_loss=0.4975428879261017
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T10:14:34.198983Z [info     ] Start Predict                  dataset=44088
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T10:29:23.082284Z [info     ] Starting training              dataset=6012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T10:30:54.879188Z [info     ] Training complete              train_loss=0.9179600477218628
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T10:30:54.885979Z [info     ] Starting training              dataset=6012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T10:32:27.188698Z [info     ] Training complete              train_loss=0.954762876033783
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T10:32:27.196154Z [info     ] Starting training              dataset=6012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T10:33:58.874241Z [info     ] Training complete              train_loss=0.9262583255767822
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T10:33:58.881235Z [info     ] Starting training              dataset=6012 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T10:35:30.088966Z [info     ] Training complete              train_loss=0.9067783951759338
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T10:35:30.096937Z [info     ] Starting training              dataset=6012 epoch=10
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T10:51:59.151384Z [info     ] Starting training              dataset=6112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T10:53:36.383207Z [info     ] Training complete              train_loss=0.9925647974014282
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T10:53:36.391280Z [info     ] Starting training              dataset=6112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T10:55:12.673108Z [info     ] Training complete              train_loss=0.9393655061721802
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T10:55:12.681112Z [info     ] Starting training              dataset=6112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T10:56:49.685065Z [info     ] Training complete              train_loss=0.9082939028739929
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T10:56:49.692182Z [info     ] Starting training              dataset=6112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T10:58:26.181962Z [info     ] Training complete              train_loss=0.9237620830535889
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T10:58:26.189559Z [info     ] Starting training              dataset=6112 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T11:00:00.983051Z [info     ] Training complete              train_loss=0.9448640942573547
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T11:00:00.988386Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T11:00:14.479037Z [info     ] Evaluation complete            test_loss=0.5105274319648743
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T11:00:14.485896Z [info     ] Start Predict                  dataset=43888
Training model 0
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T11:14:56.388463Z [info     ] Starting training              dataset=6212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T11:16:28.581044Z [info     ] Training complete              train_loss=0.9154177308082581
Training model 1
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T11:16:28.587949Z [info     ] Starting training              dataset=6212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T11:18:00.773198Z [info     ] Training complete              train_loss=0.9639526605606079
Training model 2
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T11:18:00.780174Z [info     ] Starting training              dataset=6212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T11:19:33.565889Z [info     ] Training complete              train_loss=1.0114784240722656
Training model 3
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T11:19:33.573952Z [info     ] Starting training              dataset=6212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T11:21:08.172122Z [info     ] Training complete              train_loss=0.932121217250824
Training model 4
[14095-MainThread] [baal.modelwrapper:train_on_dataset:109] 2021-07-29T11:21:08.179191Z [info     ] Starting training              dataset=6212 epoch=10
[14095-MainThread] [baal.modelwrapper:train_on_dataset:119] 2021-07-29T11:22:41.977505Z [info     ] Training complete              train_loss=0.9630128741264343
[14095-MainThread] [baal.modelwrapper:test_on_dataset:147] 2021-07-29T11:22:41.984938Z [info     ] Starting evaluating            dataset=10000
[14095-MainThread] [baal.modelwrapper:test_on_dataset:156] 2021-07-29T11:22:55.375432Z [info     ] Evaluation complete            test_loss=0.4921897053718567
[14095-MainThread] [baal.modelwrapper:predict_on_dataset_generator:241] 2021-07-29T11:22:55.382629Z [info     ] Start Predict                  dataset=43788
[5]:
import matplotlib.pyplot as plt

x = [v['test_nll'] for v in report]
y = [v['Next Training set size'] for v in report]
plt.plot(y, x)
[5]:
[<matplotlib.lines.Line2D at 0x7f1f102d6a90>]
../_images/notebooks_deep_ensemble_6_1.png