I’e not been able to find in any book on information anyone taking the position that you have, that Shannon gave us an equation for entropy, another equation for information, and that the formulas are identical. I don’t take this to mean that you are wrong or mistaken. But I do take it as a reason to remain undecided.
1 Like
The three messages have exactly 3 bits of information. There are 8 possible states that these messages can be in . log 8 (base 2) = 3 So there is 3 bits of entropy in the three messages.
I’ll try to explain later, but the three messages do not each have 3 bits of information. Assume they do each have 3 bits of information, is the information additive? So from the three messages you get 9 bits of information? If you receive the same message twice is that 6 bits of information or still only 3 bits of information?
Mung,
The message is binary. The message contains 3 bits of information. There are 8 possible messages (000, 001, 010, 011, 100, 101, 110, and 111). As such the number of states is 8. Log 8 base 2 = 3. There is 3 bits of entropy (information) in the message. I can’t get more basic than this.
So the message is different then the information?
You could though recognize that you are not describing the message but rather the probability distribution. Could you not make the exact same point and the exact same calculation with no reference at all to any message?
Given eight possible outcomes or trials of an experiment where each of the outcomes is equally likely how many binary yes/no questions would you need to ask to identify a specific outcome?
You could flip a coin 3 times and record the sequences of heads/tails and ask someone to guess the sequence. There are eight possible sequences. 1/2 * 1/2 * 1/2 = 1/8. Not surprising then that 3 = log(8) in base two. 
But that is a feature of the distribution and not of some message. Now if you wanted to be able to encode any of the possible outcomes and transmit in binary pulses of 0 and 1 now you know wht the “size” of your message needs to be. But again, that is a result of the distribution, and not because the message contains 3 bits of information.
You tried explaining. Did you try listening and did you try to see how you might be mistaken?
Do you at least agree that we are talking about a measure on a probability distribution?
I don’t listen to undergraduates. 
Got to have a least a Master’s for me to listen.
1 Like
Mung, what is your precise, mathematical definition of “message”?
It seems to me that you are hung up over extra-mathematical connotations of what “message” means.
I don’t need one in order to understand entropy. And that is my point. We can dispense with talk of messages and still be talking about entropy and Shannon’s measure.
I think the issue here is fundamentally that of semantics.
You’re using the word “information” in the different sense than Josh does. Going to a dictionary, you are using definition 1:
the communication or reception of knowledge or intelligence
Whereas Josh is using definition 2d:
a quantitative measure of the content of information
specifically: a numerical quantity that measures the uncertainty in the outcome of an experiment to be performed
“Information theory” only applies to definition 2d.
The same applies to the word “entropy”:
Definition 3 is the way we use entropy in common parlance:
Definition 4 is what we mean when doing statistical mechanics:
statistical mechanics: a factor or quantity that is a function of the physical state of a mechanical system and is equal to the logarithm of the probability for the occurrence of the particular molecular arrangement in that state
1 Like
Exactly. We should start applying warning labels to textbooks.
To be clear though, there is an information theory take on:
It just applies in an unexpected way. “Entropy” is a much better word for it.
And popular science books that claim to be about “decoding the universe” etc.
Note the lack of mention of messages. I say we can discuss Shannon’s formula for H aka entropy without reference to messages and in doing so perhaps it will help bring clarity.
I agree that it would be clearer if anyone dropped use of the word “message” as well as using the word “information” in a common parlance manner (i.e. Definition 1). The same goes for “meaning” and “purposes”. However, it seems to me that some ID advocates are not doing this very rigorously.
1 Like
It seems very fair to say they are emphasizing the (false) equivalence between the technical definition of information (entropy) and that of messages/language/meaning/purpose. That is why it was so explosive in ID circles when I started that thread at BIoLogos “Information = Entropy.”
2 Likes
I might be here all day …
1 Like
To be sure this word information in communication theory relates not so much to what you do say, as to what you could say. That is, information is a measure of one’s freedom of choice when one selects a message. … Note that it is misleading (although often convenient) to say that one or the other message conveys unit information. The concept of information applies not to the individual messages (as the concept of meaning would), but rather to the situation as a whole, the unit of information indicating that in this situation one has an amount of freedom of choice, in selecting a message, which is convenient to regard as a standard or unit amount.
- Shannon and Weaver. The Mathematical Theory of Information. p. 8-9
1 Like