Entropy of Encrypted Data

Introduction

This is the final project for the course, "Math Methods for Physicists" in National Tsing Hua University. The goal of this project is to study the entropy of encrypted data.

In followings, I will introduce some basic concepts used in this project.

---

Details about important properties in information theory

Information

Information is a measure of the uncertainty of an outcome. It is related to the amount of data that is required to specify the outcome of an event. The more uncertain an outcome is, the more information is required to resolve uncertainty of the outcome.

The information is calculated using the Shannon information. Shannon information is defined as:

where $p(x_i)$ is the probability of the $i$-th symbol in the data.

Entropy

Entropy is a fundamental concept in information theory that quantifies the uncertainty or randomness associated with a random variable. It measures the average amount of information required to describe or encode an event or a set of outcomes.

The entropy is calculated using the Shannon entropy. Shannon entropy is defined as:

where $p(x_i)$ is the probability of the $i$-th symbol in the data.

Mutual Information

Mutual information is a fundamental concept in information theory and statistics that measures the amount of information that two random variables share. It provides a quantitative measure of the dependence or association between the variables, revealing how much knowing the value of one variable can reduce uncertainty about the other.

The mutual information is calculated using the Kullback-Leibler divergence. Kullback-Leibler divergence is defined as:

where $p(x_i)$ is the probability of the $i$-th symbol in the data, and $q(x_i)$ is the probability of the $i$-th symbol in the encrypted data.

And the mutual information is defined as: