Whitening transform that supports mean subtraction

[linzhiqiu]

I added a Whitening class that builds on the LinearTransformation class, but this one further supports pixel-wise zero centering using a mean vector computed offline, which is essential for ZCA whitening.

[fmassa]

Hi,

Thanks for the PR!

It looks like most of the code is taken from the LinearTransformation layer.
Can't we instead have a Normalize layer before the LinearTransformation in order to obtain the same results?

[linzhiqiu]

Thanks for the reply!

I think the Normalize layer is meant to normalize each channel separately - however, sometimes before we use ZCA we need a feature-wise normalization (i.e. pixel-wise). If you think this is better solved by a Normalize layer and a LinearTransformation layer, then we should probably modify the Normalize layer since right now it only says it supports channel-wise normalization.

[fmassa]

Indeed, currently Normalize only supports passing around the same values for the channel normalization.
Still, I think it's simpler to have the user write a basic Normalilze that performs pointwise subtraction, than have an almost duplicate implementation of LinearTransformation.

[linzhiqiu]

Maybe we could combine this mean subtraction feature into LinearTransformation and change it to a more suitable name? Unless pixel-wise mean subtraction is really uncommon for deep learning, otherwise they will need such a Transform which is currently not supported.

[fmassa]

Sorry for the delay in replying.

I would be ok with adding the addition operation into LinearTransformation, but then it shouldn't be called LinearTransformation anymore, but AffineTransformation.

What about the following:

• add an addition coefficient to LinearTransformation, and rename it to AffineTransformation
• make LinearTransformation call AffineTransformation, and deprecate LinearTransformation

Could you do those changes?

[fmassa]

Closing in favor of #793