Eric Holloway: Wrap up Experiment on Mutual Information

Alright @EricMH,

The problem is that your MI function always returns zero, because you have not used a good compression function. This leads to contradiction.

Experiment #3

Even your own experiment #3 fails, but I’m certain you are miscalculating I(X:Y) in this. On your code, this is what I get:

Prediction: I(X:Y) >= I_LZ(X:Y)
Experiment 3: X = Y = 1s
Average success: 0.0

Where you are using this.

I(X:Y) = log(new_str_len,2)*2-log(new_str_len,2)-1

However, that is not the right formula. I’d implement a fix, but in this specific case (given the inputs), we know that I(X:Y) should approximately equal zero. So certainly it is TRUE that I(X:Y) >= I_LZ(X:Y).

Experiment #5

I added a control for you. THe mutual information of an IID bit-string with itself should be the length of the string, much greater than zero. Here is the results.

Prediction I_LZ(Y:Y)>0
Experiment 5: iid Y
Average success: 0.0

It fails? Why? Because I_LZ always returns zero everytime. It cannot measure MI. It is not a valid measure of MI.

What is the Problem?

The compression algorithm used always returns the same value every time. In the case of these experiments, with a length of 1000, it always returns a compression size of 1001, which then makes the MI always equal zero, no matter what the input. This means it is not actually a compression algorithm. This is a consequence of the hack you added in here:

Essentially, every single bit string in the whole experiment is incompressible, the compression algorithm always just adds a bit, inflating every example. The situation is worse in how it computes MI. The algorithm cannot recognize that identical objects have MI = I. It is an invalid compression function over this data for this reason.

What is the Fix?

I substituted GZ compression for your “modified-LZ” compression, and reran the experiments. We can now pass the basic test. The algorithm can actually return MI > 0 when given identical objects.

Prediction I_GZ(Y:Y)>0 (CONTROL PASSED)
Experiment 5: iid Y
Average success: 1.0

However, now all your claims fail, because they were never empirically valid in the first place.

Prediction: I_LZ(X:Y) >= I_LZ(E(X):Y)
Experiment 1: iid X, E, Y (FAIL, NOT VALID EMPERICAL CLAIM)
Average success: 0.6

Prediction: I_LZ(X:Y) >= I_GZ(E(X):Y)
Experiment 2: iid E, X = Y = 1s (FAIL, NOT VALID EMPERICAL CLAIM)
Average success: 0.0

Prediction: I(X:Y) >= I_GZ(X:Y)
Experiment 3: X = Y = 1s (FAIL, NOT VALID EMPERICAL CLAIM)
Average success: 0.0

Prediction MI(X:Y)==0 (FAIL, NOT VALID CLAIM)
Experiment 4: iid X, Y
Average success: 0.0

You can see my code here: Holloway: Info Nongrowth - Replit

2 Likes