Distillation techniques

[ClementMayer]

Facilitate "distillation" techniques (compression, anonymization): starting from an existing template then create a copy that has better properties (example: question of size, better generalization, ...)

[ClementMayer]

Based on existing work on data scrambling, the objective is to work on similar methods for models: starting from a teacher model, student models are created with the advantage of being more secure, smaller in size, and more efficient in some cases.
Reflection work is in progress on the subject.

[ClementMayer]

Update from last MAP committee (10/09/2020):

• [Mathieu] Distillation is to go from a base model to a “student” model, kind of a reduced and scrambled version. Provides privacy guarantees. Well described in literature. It happens at each iteration post-processing after obtaining gradients - implementable on Substra, as additional operations in learning algos (transparent thanks to Substra being agnostic to learning algos).
• [Eric] Are there existing libraries for that (like tensorflow-privacy and Opacus for DP)?
  • Mathieu has thought about the basics but no dev at this stage
  • [Romain] It is also interesting to identify what already exists. For example: https://nervanasystems.github.io/distiller/index.html

[SaboniAmine]

Here is a blogpost and associated notebooks which describes a keras implementation and benchmarks training scenarios.
@mattthieu could you please give some details about your ideas?

[RomainGoussault]

Closing stale issue.