Add normalized version of MVN entropy estimator and fix floating point issue #20
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
1. Add agument
normalized
Add an argument
normalized
to theget_h_mvn
function which returns the entropy of the normalized MVN distribution by normalizing such that its variance is 1 and the covariance matrix becomes equal to the Pearson correlation coefficients. Thus, the entropy becomes invariant under (some) linear transformation (scalar multiplication).calculates the entropy
H
and the normalized entropyH’
for two distributions a and b and a third is c=5a+10, etc.:Thus, the normalized entropy of a MVN random variable
X
with dimensiond
is equal toThis is also the maximum normalized entropy for a
d
-dimensional variable. It is lower if the components are correlated, e.g., in the case of rotated 2D MVN random variable (see table above).2. Fix floating point issue
The current implementation fails to calculate the entropy properly of highly correlated variables because of float resolution. I fixed this by returning
-inf
if the determinant of the Pearson correlation coefficients matrix equals 0 andnan
if the determinant is close to 0 (|det(…)|<10⁻¹³). The last three columns of above table demonstrate the new behaviour. The entropy of[a a+b/1e5]
is-7.99
, of[a a+b/1e9]
isnan
and of[a a]
is-inf
, indicating that the second one cannot be calculated.3. Speed-up of MVN entropy estimate for 1D variables
… by using the variance instead of the covariance matrix calculation