Definitions of "information"

CSGnet Archive CSG1996
1

This is probably a futile exercise, but I'd like to point out
that there are as many conceptions of "information" as there
are epistemologies. One can find information theories with
only formal assumptions (without metaphysical content, e.g.
Shannon & Weaver), one can find "information theories" with
"realist" assumptions (Dretske), and one can find information
theories that are "pragmatist" or "operationalist" (Ashby).

Information is a vague, ill-defined term these days (about like
"computation"), unless someone takes the pains to define it.
There really is no consensus on what the term commonly means.

I think Ashby's notion of "reduction of uncertainty" is clearest,
(and I imagine that Martin's conception is probably pretty close
to Ashby's) and there is no question that it DOES NOT describe
matters in terms of "what's really out there" or "what the
signal was before noise was added to it". Thus it is very different
from some of the information theoretic "examples" that have been
offered (Marken (960628.1330)).

Consider a set of 5 sensor states with the past average frequencies
of outcomes observed over the last 100 measurements being the following:

B: .2
C: .2
D. .39
E: .01

One wants to assess the perceived state of the world (for
whatever reason, including in a control loop)..
One puts the sensor in its reference state and allows it to
interact with the outside world. One has no way of knowing
anything about the world beyond the sensor. The sensor
registers a reading, either A or B or C or D or E. Before
the reading, one is uncertain of the outcome of the
measurement, what the output state will be; after the
measurement, the uncertainty has been reduced. The
informational content of the measurement is the degree
to which information has been reduced from before the
measurement (t0) to afterwards (t1):

Information gain (bits) = U(t0) - U(t1)
where U (in bits) = sum ( p(i) * log2 (p(i))), over all states i
p(i) being the probability of getting a given reading i
                 given the observer's model of the outcome
          i.e. the observer's expectations of what will happen
                                                                                p(i=outcome state) = 1 after the measurement is made
                                                                                                     and the outcome is known

                               A very simple model would be to assume that the
                 probability of a specific outcome is the same as
                                      the observed frequency over the last 100 outcomes, above);

Information and uncertainty here are joint properties of the observer,
the observer's model and the observer's interactions with the world
via the sensor. They are not properties of the outcome by itself
nor of any "hidden variables". Information here is relative to a set
of expectations, "relative to a model".

One needs to distinguish when others are using terms colloquially
or when these same words are being used in a more specific sense.
It's almost never productive to impute to others (over their
strenuous objections) meanings of terms that they do not intend.
This makes it much harder to understand what they are trying to
convey, even if the concepts involved are relatively simple.

Peter Cariani

···

A: .2

2

[From Rick Marken (960630.1330)]

Peter Cariani (960630) --

[Martin's definition of information] is very different from
some of the information theoretic "examples" that have been
offered (Marken (960628.1330)).

I guess you just can't trust what you read in books these days; even
if the books were written in the 1970s;-)

One puts the sensor in its reference state and allows it to
interact with the outside world.

What is the "reference state" of a sensor?

The sensor registers a reading, either A or B or C or D or E.
Before the reading, one is uncertain of the outcome of the
measurement, what the output state will be; after the measurement,
the uncertainty has been reduced.

This sounds like what I had in my diagram, just with some change in the labels
-- actually, with just two changes in labels.

                                          Source
x.1 |
x.2 v
x.3 ------>Sensory---C(x.i)-->Neural--->N[C(x.i)] --->Brain---->Perception
. Encoder Channel Decoder or
. Action
. x.i'
x.n
^ ^
> >
"Possible "The Entity
Sensory Whose
States" Uncertainty
                                                             Is Being Reduced"

First, the x.i are now the possible states (outputs, readings, measurements)
of the sensor (A,B,..E). And the "perception or action" end of this channel
is the entity whose understainly is reduced "after the measurement" of
sensory state. Left unsaid in your post is the fact that the state of the
sensory measurement (whether it is A, B, C D or E) has to be transmitted
to the entity whose uncertainty is being reduced (not to be confused with
The Artist Formerly Know as Prince). This transmission process is all the
steps in my diagram between the x.i (your sensory measures) and "perception
or action" (the entity whose uncertainty is being reduced).

Even if you assume that transmission is perfect -- that the entity
whose uncertainty is being reduced always knows exactly what sensory
measure occurs -- there must be a transmission process that communicates
the value of the sensor measure to the entity whose uncertainty is being
reduced.

It's almost never productive to impute to others (over their
strenuous objections) meanings of terms that they do not >intend.

That's why I presented the diagram of "information carried by the perceptual
signal". I think I could understand your point better if you (and/or Martin)
would just explain what is wrong with my diagram (above) of "information
carried by the perceptual signal".

Thanks

Rick

3

[From Peter Cariani (960701)]

Richard Marken wrote:

[From Rick Marken (960630.1330)]

Peter Cariani (960630) --
>[Martin's definition of information] is very different from
>some of the information theoretic "examples" that have been
>offered (Marken (960628.1330)).

I guess you just can't trust what you read in books these days; even
if the books were written in the 1970s;-)

Good help is so hard to find as they say. :slight_smile:
Philosophy has largely been on a downhill (platonic-realist) slide
since the late 20's, but maybe it's finally beginning to bottom out.....
Many of the '70's books on information theory,
written after the complete funding triumph of
symbolic AI and model theoretic semantics over cybernetics, embodied
realist assumptions (e.g. Dretske); At one point in the mid-1980's
there were actually people seriously arguing that "measurements" (observations,
percepts, empirical data) were not even needed to do science (!), that
it could all be done in computer simulations.......

A few pockets of sanity, however, persisted.
Ashby's operationalist line of thinking was developed by Klir and others
in the form of general systems theory
and various entropy minimization strategies.
(how good is a model? how much uncertainty reduction does a particular
model afford?)

>One puts the sensor in its reference state and allows it to
>interact with the outside world.
What is the "reference state" of a sensor?

In the most primitive case, one has a sensor (like a pH meter) that one
resets after each discrete measurement, and the "reset state" is the
reference state. The sensor transits to one of several possible output
states and stays there until the result is recorded and the sensor is
reset. I know PCT assumes continuously-updated perceptual signals that
have continuous ranges of values, but for the sake of simplicity, it's
just easier to discuss discrete observations......(one doesn't need
all the methodological/explanatory apparatus for deciding when one signal value is
"different" from another one, or what the respective probabilities of
different outcomes are -- there are ways of converting these concepts to
the continuous case, but for these purposes, it just adds alot of complication).

>The sensor registers a reading, either A or B or C or D or E.
>Before the reading, one is uncertain of the outcome of the
>measurement, what the output state will be; after the measurement,
>the uncertainty has been reduced.

This sounds like what I had in my diagram, just with some change in the labels
-- actually, with just two changes in labels.

                                          Source
x.1 |
x.2 v
x.3 ------>Sensory---C(x.i)-->Neural--->N[C(x.i)] --->Brain---->Perception
. Encoder Channel Decoder or
. Action
. x.i'
x.n
^ ^
> >
"Possible "The Entity
Sensory Whose
States" Uncertainty
                                                             Is Being Reduced"

First, the x.i are now the possible states (outputs, readings, measurements)
of the sensor (A,B,..E). And the "perception or action" end of this channel
is the entity whose understainly is reduced "after the measurement" of
sensory state. Left unsaid in your post is the fact that the state of the
sensory measurement (whether it is A, B, C D or E) has to be transmitted
to the entity whose uncertainty is being reduced (not to be confused with
The Artist Formerly Know as Prince). This transmission process is all the
steps in my diagram between the x.i (your sensory measures) and "perception
or action" (the entity whose uncertainty is being reduced).

Even if you assume that transmission is perfect -- that the entity
whose uncertainty is being reduced always knows exactly what sensory
measure occurs -- there must be a transmission process that communicates
the value of the sensor measure to the entity whose uncertainty is being
reduced.

I don't assume that the observer has access to any signal outside of what reaches
the "entity whose uncertainty is being reduced" (let's call this the observer).
All of the noise that is in the environment and in the sensor/transmission channel
is rolled into the outcome state that the observer observes.
Your example is one of transmitting a (known) signal through a noisy channel,
and this assumes that the original signal
is known by someone, an external observer looking at the system. Ashby's analysis
of uncertainty reduction, however, is only from the perspective of the signal receiver
and its "model" of what its sensors will register. They are very different perspectives
that deal with different aspects of information
(signal transmission vs. uncertainty reduction/interaction with the external world).

>It's almost never productive to impute to others (over their
>strenuous objections) meanings of terms that they do not >intend.

That's why I presented the diagram of "information carried by the perceptual
signal". I think I could understand your point better if you (and/or Martin)
would just explain what is wrong with my diagram (above) of "information
carried by the perceptual signal".

Yes, the diagram helps. Thanks. I agree that if one takes the signal + noisy channel
perspective of information and sensing, then there can be the assumption that the
"information" is somehow present in the signal. In perception, this kind of explanation
entails a "realist" perspective, that one has access to those properties of the
environment that are encoded in the signal. Realism comes in there because the nature of
and exact form of the signal are assumed to be given.
In contrast, a pragmatist (nonrealist) account of signal transmission would
say that another observer is needed, one that has sensors that register
the signal before it went through the noisy channel, and that the information lost in
the channel is a joint property of the differences between the two observers and their
observations ("mutual information"). Here the nature and form of the signal are not
taken for granted, but are observed quantities. (Does this make sense?)

So, I (think I) understand some of your objections to the use of the term "information",
(in a realist context) because it means something very different to you than it does
to me (and probably Martin as well, although he may very well have a conception
that is orthogonal to each of the two perspectives that we've discussed.)
And we might all agree that a realist-based accounts of "information"
are largely incoherent and fatuous in their assumptions, except in that tiny subset
of systems (formal systems or "toy worlds") where one can define exactly what
the environment is, and one "knows" exactly what the signal is.

Anyway, this is how I understand "information" in the context of sensing/measurement, and it
does not involve any assumptions that "information about something else" resides in the
signal. (For me, this is what is most attractive about the framework.)

Peter
peter@epl.meei.harvard.edu