Pesky Implementation Details in Information Theory

Continuing the discussion from Swamidass: Computing the Functional Information in Cancer:

What About the Proofs?

I expect this will puzzle some people:

So, as surprising as this may be, it is a not at all relevant to what I’ve worked out here. The reason why is that this formalism depends on an theoretical understanding of mutual information that is uncomputable and unobservable in practice.

This is theoretically important in physics. For example, “reversibility” in physics logically implies “conservation of information.” Perfect knowledge of the state of a deterministic system in principle (and only in principle) allows us to reconstruct any point in the past or the future, but only in principle. This tells us all the information in a system must be same at all times. However, I’ve explained before that information=entropy. We observe entropy increasing. What gives? What resolves this contradiction?

As Susskind explains:

Professor Susskind then discusses the apparent contradiction between the second law of thermodynamics, and the reversibility of classical mechanics. If entropy always increases, reversibility is violated. The resolution of this conflict lies in the (lack of) precision of our observations. Undetectable differences in initial conditions lead to large changes in results. This is the foundation of chaos theory.

So measured entropy increases, even though true but unknowable information content (which is entropy) of the system stays the same. The 2nd law of thermodynamics is a real observation, even though information content, though unobservable, stays the same (comments @PdotdQ or @dga471?) .

That is a very wide gap between theory and observation. The gap here, as Susskind notes, is required to make sense of paradoxes in our use of the term “information” and “entropy.” This gap between theory and observation is exactly what Demski (for example) shows no indication of understanding. He talks about the conservation of information, but does not recognize two key things:

  1. Conservation of information only applies in a fully-reversible deterministic world, which may not be our reality. Quantum mechanics might inject information into the world in the form of quantum randomness.

  2. Minds are not an exception to the “conservation of information” rule, even in a deterministic physics.

  3. Whatever the case, the gap between unobservable and observable information content is so large as to render proofs in one domain almost entirely irrelevant to proofs in another. There is no way around this problem. That is what the uncomputability proof tells us.

Those Pesky Implementation Details

As @dga471 almost writes:

The only disagreement I have with this is that we already have discovered that the implementation issues are fundamentally intractable. In fact, we know this from information theory. All versions of the “conservation of information” proofs, such as those regularly put forward by @EricMH, are essentially philosophical constructs that do not connect with with any observables. The uncomputability proof actually proves that they cannot be reduced to observables.

With this in mind, that is how I constructed the impossible puzzle.

@EricMH task was to compute the information content of these sequences without knowing how they were generated: Eric Holloway: Algorithmic Specified Complexity. This is a provably impossible task because because it requires perfect knowledge to compute. That is why we say compression (and information) uncomputable.

The information content of these strings obeys the “conservation of information” proofs that @EricMH repeats: randomness + determinism can’t increase information. I computed a specified amount of information, and then used a deterministic process to scramble this information. I designed the scramble to be reversible (but only in principle). From this theorem, then, I know the information content of the strings.

However, I did not reveal how I scrambled this information or how to decrypt it, or how to know if the decryption was successful, or how the initial information was generated. Consequently, there is no obvious way to figure out what the information content of these strings are without the information I withheld.

Randomness is Information

In context, @EricMH appears to misunderstand what is meant by randomness in his proofs. In the context of his proofs, the “randomness” is the pre-existing “information” of full detailed state of the entire universe or a totally closed system. It does not apply at all to an open system. He is correct though that knowing this, determinism will not create or destroy any of that information. This follows logically from reversibility. Also, this is what Demski heavily relies upon in his argumentation, and also implied in in every argument for ID I’ve seen.

What this is saying is that global information + determinism can’t produce more information if a system is reversible. In contrast, information + more randomness/information can produce more information (as might occur with quantum mechanics if it is not deterministic)…

So what breaks determinism? Perhaps intelligence, if we have free will, can add information. Quantum randomness, also, can add information. Any thing that is not determinism can add information to reality. Regardless, all these conservation of information proofs only apply to a totally closed system any ways. Intrusions of information (by randomness or intelligence or natural processes) can easily increase information (or mutual information) in a subsystem, even in the idealized version of information, even in a deterministic world.

I hope that makes some sense. There is a lot of theory in these posts. I hope this isn’t loosing people. And I hope the physicists are suddenly realizing how information theory arises in their field.


This is a huge can of worms. While intuitively this seems to be true, in practice it is very difficult to prove. Even for the simplest system of modelling a gas of particles as a collection of hard spheres there is still some debate on where exactly irreversibility comes in. For an introduction to this rich problem, one can read about the Boltzmann-Grad limit. A good source is but this website might be too technical and boring for non-physicists.

I would be very careful with this statement. In quantum mechanics, the conservation of information is encompassed by the statement that (most of) quantum evolution is unitary. Unitarity has a specific mathematical meaning, but in broad strokes it is a restriction that quantum evolution keeps the sum of probabilities 1.

The amount of entropy in a quantum system is given by a quantum extension of the Shannon entropy called the von Neumann entropy, and one can show that unitary evolution does not change the von Neumann entropy. For for those who care, this is the quantum version of the classical Liouville theorem.

Whether ALL of quantum mechanical evolution is unitary is a subject of debate that is highly dependent on one’s favorite QM interpretation. In collapse theories like the Copenhagen interpretation, a wavefunction collapse is non-unitary. However, in decoherence theories (e.g. many-worlds) and hidden variable theories, all evolution is unitary and information is perfectly preserved. At least in decoherence theories, the appearance of non-unitarity and the irreversibility of the evolution results from quantum decoherence - i.e. the interaction of the system with the many hidden variables of the environment (e.g. the lab apparatus).

There is at least one more example of non-unitary evolution: the total evaporation of a black hole by Hawking radiation might be non-unitary - whether it actually is non-unitary is at the heart of the black hole information paradox. Nevertheless, I don’t think one would want to anchor one’s philosophy on black holes.


Notice the weasel word “might”, as I know that it is a can worms. It is, however, a loophole that an ID advocate MUST close if they are to make use of the Conservation of Information Law, as Dembski does. So your cautions here are bolstering my point.

I agree. Yet this is precisely where Dembski is philosophically resting the claim that intelligence is a unique source of CSI. In his works, this traces back to substance dualism. Minds are not subject to the laws of physics, and exist outside of them, but they affect the physical world. That effect Dembski is equating with injecting information into the physical world, analogous to God’s ability to inject information.

If the world really is reversibly deterministic, and minds exist outside the physical world, then this might be true. However, it still does not mean they can measure this information in any meaningful or useful way.

I’ll just add that for a host of reasons, most of not all realistic simulations are not reversible in practice. While it might be a philosophical truth of the world, it does not actually interact with observations in experiments or simulations.

In the previous post I was just commenting on the physics and am completely detached from the philosophy. I don’t want to be seen as if I’m trying to support ID :rofl:

From this paragraph I think I now understand what the philosophical stakes are. As someone who believes that the physical world is closed (i.e. physics is fully consistent), personally I would reject Dembski’s proposal.

Hmm, here I take the opposite stance: it does not matter if simulations are not reversible, but it is important to see in the real world where irreversibility actually comes from.

1 Like

I’m just insisting there is a gap between our best computational models of the world, and the pure mathematical models. Computation (practice) and math (theory) do not align in this case, and it matters.

So how did the Resurrection happen without violating closure?

Of course it matters scientifically, but I don’t think it matters for philosophical questions that rely on the reversibility of the actual world.

No idea.

Well if you are a Christian/Catholic that affirms the bodily Resurrection, and presumably the Miracles of Jesus, I think you are boxed in. At least at times, it would appear the universe is not totally closed.

I am not sure. As God presumably has control of the data on any Cauchy surfaces, if anything atomic rearrangement from a dead state to a living state that still follows the laws of physics are not beyond His abilities. For example, for this specific instance entropy can reduce drastically. This does not violate the 2nd law as it is an isolated case and the 2nd law is statistical.

Edit: I kept forgetting to quote the things I am replying to.

So how does He do this without being simultaneously outside the physical world and able to affect it? Is this not the definition of an open system?

Why would even matter or not anyways?

The consistency of the laws of physics just specify the evolution equation, but does not specify the initial conditions (I hesitate to use the word initial as the data can be specified at any Cauchy surface, but you get the gist). Note that I am not saying this is how God does it, this is just one way it could be done.

It looks like you misunderstand Levin’s proof. It addresses both determinism and randomness.

Rather than debate this, the experimental rebuttal of the proof and/or application is much more salient. As can be seen in the thread on cancer: Swamidass: Computing the Functional Information in Cancer. Perhaps focus on that.

Sure, but to reiterate:

is a category mistake.

You cannot empirically disprove something that has been mathematically proven. You can only empirically disprove mathematical conjectures. So, there is no hope to empirically disprove Levin’s proof. Let’s not use that language for clarity’s sake.

A mathematical proof is about an abstract model. It isn’t about reality.

An experimental rebuttal could demonstrate that the abstract model is an inadequate model of the real world problem. That does not disprove what has been mathematically proven, but it does question the relevance of that mathematical proof.


In that case you have proven it is irrelevant, not proven it is false. Two different things.

Fine. I’ve proven that it is irrelevant or false. Choose your poison.

1 Like

You’ve only proven it is irrelevant. Impossible to empirically prove math facts are false. Playing with words like this diminishes trust.

Why are you objecting? I don’t get it. It all comes down to precisely what we mean by MI. The proof about the abstract and immeasurable MI is true, the attempt to equate it with the measurable MI is false, and that means the proofs about the abstract MI are irrelevant.

It seems we likely agree on this, and that is all I am saying.

1 Like