GitLab

Summary: Pull Request resolved: https://github.com/facebookresearch/d2go/pull/318

Reviewed By: mcimpoi

Differential Revision: D37501246

fbshipit-source-id: 6dbe5dcbaf7454f451d4a3bb3fa2d856cc87d5cc

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/299

This implements the first iteration of generalized distillation in D2 (https://github.com/facebookresearch/d2go/commit/87374efb134e539090e0b5c476809dc35bf6aedb)Go. The functionality is separated into the following:

=> Adding distillation functionality without user changing their meta architecture:

class DistillationModelingHook
* This is an implementation detail that we hide from the user.
* We use dynamic mixin to specify additional functionality to the user's model. In this way, the original (student) model retains all attributes but the mixin class will override the forward (and provide more functionality like teacher updates).
* Teacher build currently only supports loading a torchscript model, pytorch compatiblity in later diffs

=> Implementing distillation methods

class DistillationAlgorithm
* The user can use some default algorithm (e.g., LabelDistillation) or create their own. This is where we specify the overrided forward func of the model and any other distillation requirements (updating the weights of the teacher model).
* The basic LabelDistillation allows a user to use a teacher model during training to relabel the gt

=> User customization

class DistillationHelper
* This is what we expect the user to customize. As an example the user probably needs to write their own pseudo_labeler to take batched_inputs and relabel this with the teacher

Both DistillationHelper and DistillationAlgorithm use a registry so that users can add their customization in their own code and use these customizations by specifying in the config

Reviewed By: newstzpz

Differential Revision: D36708227

fbshipit-source-id: bc427c5d42d0c7ff4d839bf10782efac24dea107