Alright @EricMH,
The problem is that your MI function always returns zero, because you have not used a good compression function. This leads to contradiction.
Experiment #3
Even your own experiment #3 fails, but I’m certain you are miscalculating I(X:Y) in this. On your code, this is what I get:
Prediction: I(X:Y) >= I_LZ(X:Y)
Experiment 3: X = Y = 1s
Average success: 0.0
Where you are using this.
I(X:Y) = log(new_str_len,2)*2-log(new_str_len,2)-1
However, that is not the right formula. I’d implement a fix, but in this specific case (given the inputs), we know that I(X:Y) should approximately equal zero. So certainly it is TRUE that I(X:Y) >= I_LZ(X:Y).
Experiment #5
I added a control for you. THe mutual information of an IID bit-string with itself should be the length of the string, much greater than zero. Here is the results.
Prediction I_LZ(Y:Y)>0
Experiment 5: iid Y
Average success: 0.0
It fails? Why? Because I_LZ always returns zero everytime. It cannot measure MI. It is not a valid measure of MI.
What is the Problem?
The compression algorithm used always returns the same value every time. In the case of these experiments, with a length of 1000, it always returns a compression size of 1001, which then makes the MI always equal zero, no matter what the input. This means it is not actually a compression algorithm. This is a consequence of the hack you added in here:
Essentially, every single bit string in the whole experiment is incompressible, the compression algorithm always just adds a bit, inflating every example. The situation is worse in how it computes MI. The algorithm cannot recognize that identical objects have MI = I. It is an invalid compression function over this data for this reason.
What is the Fix?
I substituted GZ compression for your “modified-LZ” compression, and reran the experiments. We can now pass the basic test. The algorithm can actually return MI > 0 when given identical objects.
Prediction I_GZ(Y:Y)>0 (CONTROL PASSED)
Experiment 5: iid Y
Average success: 1.0
However, now all your claims fail, because they were never empirically valid in the first place.
Prediction: I_LZ(X:Y) >= I_LZ(E(X):Y)
Experiment 1: iid X, E, Y (FAIL, NOT VALID EMPERICAL CLAIM)
Average success: 0.6Prediction: I_LZ(X:Y) >= I_GZ(E(X):Y)
Experiment 2: iid E, X = Y = 1s (FAIL, NOT VALID EMPERICAL CLAIM)
Average success: 0.0Prediction: I(X:Y) >= I_GZ(X:Y)
Experiment 3: X = Y = 1s (FAIL, NOT VALID EMPERICAL CLAIM)
Average success: 0.0Prediction MI(X:Y)==0 (FAIL, NOT VALID CLAIM)
Experiment 4: iid X, Y
Average success: 0.0
You can see my code here: Holloway: Info Nongrowth - Replit