[From Bill Powers (2009.09.30.1442 MDT)]
Martin Taylor 2009.09.30.11.14 –
MT: May I requote from my
paragraph you quoted above? “Only a communication engineer would
consider the uncertainty of the character by character reception as being
related to the information transmitted, and the reason for that is that
the link for which the engineer is responsible must carry messages of
arbitrary types.” Isn’t that what Shannon says in your quote (which
I had not seen before)?
BP: Yes, and Shannon precedes that by saying
"…I wish to review briefly what we mean by redundancy. In
communication engineering, we regard information perhaps a little
differently than some of the rest of you do. In particular we are not at
all interested in semantics or the meaning implications of information.
Information for the communication engineer is something he transmits from
one point to another as it is given to him, and it may not have any
meaning at all. It might, for example, be a random sequence of digits, or
it might be information for a guided missile or a television signal.
…
For communication work, we abstract all properties of the messages except
the statistical properties which turn out to be very important."
Page 123.
Clearly, Shannon represented himself as an example of “we”
communication engineers. His presentation is titled “The redundancy
of English.” The publication is titled "Cybernetics: circular
causal and feedback mechanisms in biological and social systems.
Transactions of the seventh conference, March 23-24, 1950, New York, NY.
Edited by Heinz von Foerster, assistant editors Margaret Meade and Hans
Lukas Teuber. Publisher: The Josiah Macy, Jr. Foundation, 565 Park
Avenue, New York, Copyright 1951. Sorry I didn’t give the full
particulars before.
MT: My reference to Shannon is
to the book that popularized information analysis (Shannon and Weaver,
“The Mathematical Theory of Communication” U of Illinois Press,
1949).BP earlier: I think your definition of information as reduction of
psychological uncertainty in the receiver of the message is
impracticalShannon’s definition, not mine. You added the word
“psychological”.
Yes, because you are talking about reduction of subjective uncertainty,
not calculable uncertainty in a given message. Shannon says very plainly
that he is a communications engineer who is not concerned with
meaning.
MT: Shannon deals
throughout the book with using observation of what was received to reduce
the receiver’s uncertainty as to what might have been transmitted.
BP: When Shannon speaks of the receiver and transmitter, he is talking
about the electronic devices, not the human being.
MT: That reduction is the
information transmitted. I think it’s made most explicit with the diagram
on page 41 and the text in that neighbourhood.BP earlier:, because to
calculate that you would have to know every possible meaning of
all possible messages and I believe that is impossible to know.
MT: Why do you make those two
assertions? To me, they both come out of thin air.
They come out of my imperfect understanding of what is meant by reduction
of uncertainty. What I have understood is that to calculate the
information content of a message you must first determine the number of
possible messages, so you can calculate the number of bits required to
distinguish one message from all others. Shannon, in his article on
English, does this in terms of alphabets and statistical distributions of
characters used in messages, plus distributions of digrams and trigrams
and so on. He calculates the redundancy in this way, to see what the
actual minimum number of bits is for sending an unambiguous message using
English letters.
As a receiver, you can’t
ever know the meaning intended by the sender, though you can use
“The Test” to make a pretty good guess after enough
back-and-forth interactions. As a sender, you can’t ever know whether the
receiver has gathered the meaning you intended to convey, although you
can make a pretty good guess by applying “The Test” over a
series of interactions. But as a receiver you CAN know something about
what range of meanings you anticipate the sender might be wanting you to
perceive, and you usually do. In most circumstances, one has a reasonably
restricted range of messages one might expect.
Here you are using the term “receiver” in a way not intended by
Shannon, as far as I can see. His receiver is the box with dials and
lights out of which comes the wire going to a printer. Your receiver is
the person, the recipient, to whom the printed output is handed. These
are two completely different things, and the universe of possible
meanings is very different from the universe of possible messages (in
part because the messages themselves, once received, become experiences
that can be meanings: The meaning of “Yours of the 21st inst”
is a particular message). You propose that we can know something about
the range of possible meanings the person reading the message can get
from it, but I doubt, and indeed flatly reject, the idea that this is how
we get meanings from messages. Certainly, the meaning we end up with is
influenced by the meaning we expect to get, based on the general context
of the communications that are going on. But this is hardly a
quantifiable judgment and it’s not arrived at in the same way we compute
the information content of string of characters. You can’t go through the
list of all possible meanings to determine what either you or another
person mean. Often you may see that there are several possible meanings
and have to pause to decide which one is intended (or find that you can’t
decide), but the fact that several meanings come to mind by no means is
the same as the number of possible meanings that exists.
If you were to accept my definition of meanings in the sense used here,
meanings would be simply the perceptions that are indicated by the
messages you get. The number of possible meanings, therefore, would be
the total number of memories of experiences at all levels that you have
associated with words or other symbols. I have no idea, and I doubt that
anyone else does, either, how many of those there are. It must be an
exceedingly large number. The larger the number, of course, the less is
the information contained in any one message, because it takes a large
number of bits to distinguish one element of a large set from all the
others. But that is moot, because I see no possibility of determining the
number of bits required to isolate one meaning, or even determining the
meaning of “one bit” in that context.
Take your own example. You
hypothesised that the listener quite clearly heard and understood
"Meet me " and later in the message “but leave your car at
the station and walk to the corner of” and “and West”.
I merely assumed that those strings of characters were transmitted
without error, while some of the others were distorted because of
missing, extra, or flipped bits in the transmitted codes. I assume the
recipient of the message (to distinguish that agent from the electronic
“receiver” that accepts the transmission) could find meanings
to go with all the strings that had meanings for the person, except those
strings that did not have any associations. The latter would result
result from extreme distortions like those I included in the message and
which I trust you had no trouble distinguishing from meaningful (for you)
strings. Of course strings like “Bring Jane with you” might
also lose meaning (Jane who? I don’t know any Jane I might bring) if you
don’t happen to have a relevant experience with the referent of that
string. On the other hand, a communication engineer would probably find
the most meaning in the garbled strings, because they indicate errors
that the engineer is supposed to prevent.
MT: " The listener also can
perceive roughly the length of the missing bits. Now it’s extremely
unlikely that the first gap contained a dissertation on Hawking radiation
from black holes, and quite likely that it contained a time of day.
Moreover, it is quite likely that the time of day in question is on the
same day, and at a time both parties are expected to be free of other
obligation. The probability distribution of messages that the receiver
might expect is quite a bit less wide than “the meaning of all
possible messages”. The same is true of the second gap, where the
receiver almost certainly have an uncertainty distribution for the
meaning that ranges only over a small number of street corners, and does
not have a high probability for issues concerning solar
energy.
BP: These are all very fuzzy and iffy guesses based on imagination more
than data. You assume that the message will never have any extraneous
references in it, like “Meet me outside the store that has solar
panels in its window.” You have very little ability to narrow the
possible meanings that might occur in any message. And it’s the number of
meanings that MIGHT occur that are in the denominator when you calculate
information content.
MT: On the other hand, if the
listener was in the middle of listening to a discussion of Hawking
radiation, the words “Meet me”, though clearly spoken, might
not have been immediately or easily understood, since their meaning would
not have been anywhere in the high-likelihood region of meanings within
the lecture.
BP: They might if Hawking says “you will know what a tidal force in
a black hole is if you meet me on my way into it.” You simply can’t
anticipate all the meanings that might crop up in a message.
BP: The message “I don’t
know” has a meaning that’s entirely dependent on what messages
precede and follow it and that is practically an infinite universe of
possibilities. The preceding message might have been “what’s the
density of tungsten?” or “What was John’s answer?”
or “What’s a good crossword clue for
ignorance?”MT: Yes, of course. I don’t know what point you are trying to make, here.
Everything that happened prior to the message constrains the expectations
for what might be in the garbled gaps, does it not? In what I said above,
I assumed the situation to be that the two people had expected to meet,
but had not arranged when and where. The situation always constrains
one’s expectation of what might be said. I’m sure you have experienced
occasions when someone has said something that comes “out of left
field”, and have not immediately understood it, perhaps to the
extent of asking “Could you say that again” or “What do
you mean”.
BP: I guess you just don’t see what I’m talking about. Every word we use
in communicating has more than one meaning even inside a single person;
all I’m trying to get across is that I don’t see any reasonable way to
compute the information content in a message in terms of the meanings
recipients get from the message, compared with the total number of
possible meanings. In my opinion, you’re overstraining the information
metaphor when you take it out of the literal, quantitative engineering
context Shannon talked about. Perhaps you have some dazzling succinct
argument that will change my mind about that, but I haven’t seen it so
far.
MT: We verge on another
discussion of what is meant by “meaning”. It’s not a discussion
in which I have either interest or time to contribute much. But in
this context, I take “meaning” to be “influence on a
perception, whether that perception be controlled or not”.
BP: Influence of what on what perception? What perception does the word
“red” influence? I think it makes more sense to say that the
meaning of a word (itself a perception) is some other perception that is
indicated (perhaps through memory association) by the word serving as a
label or pointer.
I should think you would be vitally interested in trying to figure out
what the word “meaning” means, since everything else you say
depends on that.
Best,
Bill P.