Using PyTorch and VISSL Losses¶
VISSL supports all PyTorch loss functions and also implements several loss functions that are specific to self-supervised approaches like DINO, SwAV, MoCo, PIRL, SimCLR, etc. Using any loss is very easy in VISSL and involves simply editing the configuration files to specify the loss name and the parameters of that loss. See all the VISSL custom losses here.
To use a certain loss, users need to simply set LOSS.name=<my_loss_name>
and set the parameter values that loss requires.
Using Cross Entropy Loss for Training and Testing¶
LOSS:
name: "CrossEntropyLoss"
# ----------------------------------------------------------------------------------- #
# Standard PyTorch Cross-Entropy Loss. Use the loss name exactly as in PyTorch.
# pass any variables that the loss takes.
# ----------------------------------------------------------------------------------- #
CrossEntropyLoss:
ignore_index: -1
Using SwAV loss for Training¶
LOSS:
name: swav_loss
swav_loss:
temperature: 0.1
use_double_precision: False
normalize_last_layer: True
num_iters: 3
epsilon: 0.05
crops_for_assign: [0, 1]
temp_hard_assignment_iters: 0
num_crops: 2 # automatically inferred from data transforms
num_prototypes: [3000] # automatically inferred from model HEAD settings
embedding_dim: 128 # automatically inferred from HEAD params
# for dumping the debugging info in case loss becomes NaN
output_dir: "" # automatically inferred and set to checkpoint dir
queue:
local_queue_length: 0 # automatically inferred to queue_length // world_size
queue_length: 0 # automatically adjusted to ensure queue_length % global batch size = 0
start_iter: 0
Using DINO loss for Training¶
LOSS:
name: dino_lodd
dino_loss:
momentum: 0.996
student_temp: 0.1
teacher_temp_min: 0.04
teacher_temp_max: 0.07
teacher_temp_warmup_iters: 37500 # 30 epochs
crops_for_teacher: [0, 1]
ema_center: 0.9
normalize_last_layer: true
output_dim: 65536 # automatically inferred from model HEAD settings