Continuing the discussion from Explaining the Cancer Information Calculation:
Information Non Growth
Just to back a bit, it seemed worth explaining this. It is actually fairly straightforward. Let us say we have a deterministic program F, that works on arbitrary data X, to produce output Y. Let us say that H is a function that returns the information content of these things. Therefore, by definition:
F(X) = Y
What is the information content of Y? Well we know that If we have F and X we can compute Y. So what ever the information content we know it is less than or equal to:
H(X) + H(F)
Another way of putting it is that F(X) is equivalent way of representing Y, so we know that it can never take more than H(F) + H(X) bits to represent Y. Anything more than this and we know for a fact that we are using an inefficient compression. That is all the law of information non-growth is. We go a bit further if we know that F is a reversible function. Then we know that information is conserved as well. Information content never goes up or down.
Let us say that X and Y are random bit strings and F is a pseudorandom number generator that can be seeded by a bit string. Let us say we chose:
X to be the binary representation of the integer “12934824840”
F to be a function that returns the first billion bits of the random number generator seeded by X
Y is then F(X)
So, what is the information content of Y? Well, knowing how it was generated, this is really easy. It is just:
H(F) + H(X)
Which is perhaps surely much much less than 1 billion bits. That is what the law of information non growth tells us. What if we do not know what F and X are, and can only observe Y? Well, now we have no idea how much information there is here. We won’t see any patterns in the data, so we will incorrectly (but empirically) conclude that Y has a billion bits of information. This isn’t correct, but we have no way of knowing this.
In fact, we can never really know if we have the best compression. We can therefore never know for sure in examining Y alone if we are looking at a billion bits of information, or a compressible object that reduces down to the seed and the algorithm.
How this Maps Back
The information no-growth law applies where:
- We are in a deterministic world
- Taking into account the whole system.
- Have perfect knowledge of every detail of the system.
The conservation of information law applies if also:
- The system is reversible.
In a deterministic world of perfect knowledge, taking into account the whole system, the information no-growth theorom applies. However in the real world:
- we do not have perfect knowledge
- we only look at parts of the system, and
- we do not even know if physics is reversible or deterministic (Predictability Problems in Physics)
Basically, in practice, in empirical work, not one of the assumptions required for information no-growth law to apply.
We can see information grow in DNA because it is not the whole system, because we are not modeling the whole system, and we do not have full perfect knowledge of the whole system. This is just a context where the law of information non-growth to apply. In fact, in just about no empirical scenario does the law apply. It is a theoretical construct that requires omniscience and determinism to apply. It would be difficult to imagine one where it would apply.
That is what I told @EricMH in the beginning.