Durston: Functional Information

[swamidass (S. Joshua Swamidass)]

@Kirk

1. Given: H = - K ∑ p ( i ) log p ( i ) which, according to Shannon is a measure of “information, choice and uncertainty.” I trust we are in agreement thus far.

2. H will always be positive. I expect we have agreement here as well.

3. If H (A) is the Shannon information of state A, and H (B) is the Shannon information of state B, then the difference in Shannon information between the two states is H (A) – H (B). It is at this point that I am not 100% sure we are in agreement. If not, I would like to understand why you think that the difference between H (A) and H (B) is not, in fact, the difference between the Shannon information of the two states.

That is all correct.

@Kirk

If H (A) is positive and H (B) is positive, and if H (A) > H (B), then H (A) – H (B) will be positive. This shows that there is an information gain between states A and B. However, if H (A) < H (B), then H (A) – H (B) will be negative. This shows that there is an information loss between states A and B.

This is very idiosyncratic use of the term information gain and loss. I think this is why you are confused. However, as I agreed with you before, Delta H can be negative or positive.

@Kirk

Tangential to (4) Re. the K-L information gain: I have been using the K-L information gain as defined here Information theory - Wikipedia You will note that it is defined as the difference in Shannon information between two probability distributions,

The wikipedia page you reference does NOT define KL as the difference in entropy between to probability distributions. Why would you keep saying that when you can just look at the page and see that it is defined differently? It is defined, just as I explained before.

@swamidass

KL is defined , however, by this equation:

D_{KL} ( Q || P) = \sum P(i) \ln \frac{P(i)}{Q(i)}

Which we can prove is equivalent to:

D_{KL} ( Q || P) = H(P,Q) - H(P)

Note that it is guaranteed that D_{KL} is greater than zero because the information content of two objects is always greater than or equal to the information content of either one of them alone.

H(P,Q) \geq H(P)

I hope it is very clear now that D_{KL} is not merely ΔH. As I’ve explained already, in some cases, for example where there is no correlation between P and Q and P is higher entropy than Q, then this might reduce down to delta H. However, it is a grand error to see this as generally true.

You can learn more about this from reading the page on KL divergence that uses the exact same definition in the wikipedia page you reference: Kullback–Leibler divergence - Wikipedia.

@Kirk

The Shannon information of that site, for P53 proteins that permit the cancerous cell to multiply is H (cancerous) = 4.2 bits. The change in Shannon information between the functional P53 state and the cancerous P53 state is H (func) – H (cancerous) = 0 bits – 4.2 bits = - 4.2 bits. We can conclude, therefore, that the change in state is negative, corresponding to a loss of Shannon information between the healthy cell and the cancerous cell.

That calculation is in error.

KL is always positive. Information was required to move from a non cancerous to a cancerous state. You are making a very profound math error here.

The right way to think of this is different. The normal P53 state is well defined. We need to know how to change it into a carcinogenic state. There is a certain amount of bits required to specify how to change it into a carcinogenic state. That information is all functional information and it will always be positive because KL is always positive. If you used the formula that I just gave you, it would come up with a different number, a positive number.

Computing the precise amount of information in this case is another question. It turns out the mathematical framework you have can’t compute KL when the ground state isn’t IID MaxEnt. In this case, the ground state is normal P53, which is most certainly not IID MaxEnt, so that is why you are getting an aberrant result. You thought KL = delta H, but it does not. It never has except in the boundary case you stumbled into that has essentially no relevance to biology.