Hooks

Hooks are the helper functions that can be executed at several parts of a training process as described below:

  • on_start: These hooks are executed before the training starts.

  • on_phase_start: executed at the beginning of every epoch (including test, train epochs)

  • on_forward: executed after every forward pass

  • on_loss_and_meter: executed after loss and meters are calculateds

  • on_backward: executed after every backward pass of the model

  • on_update: executed after model parameters are updated by the optimizer

  • on_step: executed after one single training (or test) iteration finishes

  • on_phase_end: executed after the epoch (train or test) finishes

  • on_end: executed at the very end of training.

VISSL Hooks

Hooks are executed by inserting task.run_hooks(SSLClassyHookFunctions.<type>.name) at several steps of the training. VISSL currently supports the following hooks. To see comprehensive documentation on these hooks, pelase see the defaults.yaml.

This hook will log configured metric to Tensorboard. To enable this hook, set HOOKS.TENSORBOARD_SETUP.USE_TENSORBOARD=true and configure the tensorboard settings.

Performance Stats hook

This hook will log performance stats to the log.txt output file. To enable this hook, set HOOKS.PERF_STATS.MONITOR_PERF_STATS=true and configure the performance stats frequency and other settings.

Memory Summary hook

This hook will log cpu and gpu memory metrics to the log.txt output file. To enable this hook, set HOOKS.MEMORY_SUMMARY.PRINT_MEMORY_SUMMARY=true and configure the performance stats frequency and other settings.

Model Complexity Hook

This hook performs one single forward pass of the model on the synthetic input and computes the #FLOPs, #params and #activations in the model. To enable this hook, set HOOKS.MODEL_COMPLEXITY.COMPUTE_COMPLEXITY=true and configure it.

Monitor Activation Statistics

This hook reports several activation statistics, like mean and spread, to Tensorboard. To enable this hook, set HOOKS.MONITORING.MONITOR_ACTIVATION_STATISTICS=NUM_ITERS and configure the INPUT_SHAPE.

Profiling Hook

This hook reports comprehensive memory and runtime profiling metrics and visualizations. To enable this hook, set HOOKS.PROFILING.MEMORY_PROFILING.TRACK_BY_LAYER_MEMORY=true and/or HOOKS.PROFILING.RUNTIME_PROFILING.USE_PROFILER=true and configure the additional settings as desired.

Logging, checkpoint, training variable update hooks

These hooks are used by default in VISSL and perform operations like logging the training progress (loss, LR, eta etc) on stdout, save checkpoints etc.

Self-supervised Loss hooks

VISSL has hooks specific to self-supervised approaches like MoCo, SwAV etc. These hooks are handy in performing some intermediate operations required in self-supervision. For example: MoCoHook is called after every forward pass of the model and updates the momentum encoder network. Users don’t need to do anything special to use these hooks. If the user configuration file has the loss function for an approach, VISSL will automatically enable the hooks for the approach.