[From Bill Powers (960701.0630 MDT)]
Peter Cariani (960630) --
Consider a set of 5 sensor states with the past average
frequencies of outcomes observed over the last 100 measurements
being the following:
B: .2
C: .2
D. .39
E: .01
One wants to assess the perceived state of the world (for
whatever reason, including in a control loop).. One puts the
sensor in its reference state and allows it to interact with the
outside world. One has no way of knowing anything about the world
beyond the sensor. The sensor registers a reading, either A or B
or C or D or E. Before the reading, one is uncertain of the
outcome of the measurement, what the output state will be; after
the measurement, the uncertainty has been reduced. The
informational content of the measurement is the degree to which
information [WTP: uncertainty, I presume] has been reduced from
before the measurement (t0) to afterwards (t1):
I don't know what you mean by putting the sensor "in its reference
state." Does this have something to do with your interpretation of the
meaning of "reference signal?"
Some rambling follows:
I'd like to see an interpretation in terms more familiar to me. For
example, suppose the sensor is a force transducer. The output is a
signal with a magnitude that can range from 0 to 100: this signal is the
"reading" of the sensor. The past 100 readings, let's say, give values
that can be put into bins, with the following frequencies:
0 to 10 0.0
10 to 20 0.0
20 to 30 0.0
30 to 40 0.10
40 to 50 0.80
50 to 60 0.10
60 to 70 0.0
70 to 80 0.0
90 to 100 0.0
Now, before the reading, what is our uncertainty about the reading? Is
it 30 to 60, or 5 to 95? In my understanding, it would be 5 to 95,
because the totality of experience tells us that there have been values
over this entire range in the past. Over the past 100 readings, we have
found a mean value of 45 plus or minus 10, but I take this to mean that
we are inferring that the external world is in a state corresponding to
a signal of 45 units, plus or minus 10. The uncertainty that I see is
the "plus or minus ten" -- that is, given a mean reading, we know that
there could be a distribution of external states that would yield the
same reading, or conversely, if the environment remained the same, that
successive 100-element samples might give mean values that vary plus or
minus 10 units (well, I know that's not the variability of the mean, but
let it go).
Now we have a new sample of 100 entries:
0 to 10 0.10
10 to 20 0.80
20 to 30 0.10
30 to 40 0.0
40 to 50 0.0
50 to 60 0.0
60 to 70 0.0
70 to 80 0.0
90 to 100 0.0
The uncertainty in the signal is still, as I would see it, plus or minus
ten units around the mean of 15 units. But we now have a meta-sample of
two entries: 45 and 15 units, the two means. If we interpret each mean
as the "true" representation of a state of the environment, we have two
inferred states of the environment to be uncertain about: which state
will it be in the next time we take 100 samples? Over 100 sets of 100
samples each, we might come up with this table:
Mean value Frequency
00 to 10 0.0
10 to 20 0.5
20 to 30 0.0
30 to 40 0.0
40 to 50 0.5
50 to 60 0.0
60 to 70 0.0
70 to 80 0.0
90 to 100 0.0
Now we have an uncertainty not about the measurement but about the
environment. Or so we assume. We assume that the measurement error is
plus or minus 10, and that which is being measured has a 50% probability
of being in one of two states.
But how do we know that the plus or minus 10 is a measurement error, and
not a natural variability of the environment about its two mean states?
How do we know that the sensor doesn't have two states, with a
distribution about each one, for a perfectly constant environment? It
seems to me that we have a great deal more to be uncertain about than
just the distribution of the observed values.
Furthermore, we now have two mean values, but inevitably we obtained
them in a specific order: 45, 15. What will the next mean value be? For
all we know, the next values would be 45, 15, 45, 15, ..., or 40, 18,
35, 22, 28, 26, 25, 25, 25, 25 ...
In other words, the "environmental" variations might prove to be
perfectly regular, describable by a deterministic function such as a
square wave or a damped oscillation. In that case, speaking of the
"probability" that the environment will be in a specific state would
imply a falsehood -- that we are dealing with a random variable. The
only "probability" of which we would then speak would concern the
measurements -- how far from the ideal measurement the actual
measurements would vary on each sample. Likewise, our "uncertainty"
would then pertain only to the measurement error, not to the predicted
behavior of the means.
Suppose that we found a table of meta-values like this:
Mean value Frequency
(100 each)
0 to 10 0.1
10 to 20 0.1
20 to 30 0.1
30 to 40 0.1
40 to 50 0.1
50 to 60 0.1
60 to 70 0.1
70 to 80 0.1
90 to 100 0.1
Now the environment would look completely random, according to the usual
assumptions. The chances of finding any mean value are equal. But for
our pressure transducer, the actual pressure might be doing this:
* * *
* * *
* * *
* * * ===> etc.
* * *
* * *
* * *
* * *
Or it could be going in repeated descending ramps, or in any other form
of change that left the probabilities equal. If we knew that this was
going on, we wouldn't be uncertain at all about the next value of the
mean. But if we assumed that we were dealing with a random variable, we
would be way off in our expectations, and we would be quite needlessly
uncertain.
Now take this another step. Suppose the probabilities were not equal,
and also that we could see that the pressure measurements, in 100-sample
sets, looked like this:
* * * * *
* * * * *
* * * * *
* * * * * *
* * * * * *
···
A: .2
*
I hope you can't see any pattern in that. The assumption would be that
this is some sort of smoothed random disturbance, so the usual treatment
would now try to predict the next mean from the local slopes and the
probabilities. An unknown part of this waveform would come from
measurement errors, so we would be doubly uncertain: uncertain as to the
mean pressure represented by each point, and uncertain about what the
next mean would be.
But suppose that we now noticed a motor driving a cam which is
compressing a spring to create the pressure we're measuring. The cam is
cut so its radius is almost exactly in the same form as the succession
of pressure readings above. Now, suddenly, our main source of
uncertainty is gone. What we were seeing as an irregular unpredictable
variation of the pressure is now seen as a perfectly predictable
consequence of the rotation of the motor driving the cam. It doesn't
matter that the distribution of cam radii with angle would fit some
random distribution; in fact the whole process in the environment is
completely determinate and the application of statistical methods would
be unnecessary.
In fact, the initial uncertainty in the measurements could now be
reduced by observing the relationship between cam radius and angle.
There would be some measurement uncertainty in that, but now that we
have an independent measure of the pressure (using an environmental
model) we could predict the pressure and compare that prediction with
the measurement. This would give us an estimate of the measurement
uncertainty as opposed to the uncertainty in the thing measured. Model-
dependent, of course.
This last point I'm trying to make is somewhat subtle. When we see a
variation that we can't explain, the natural thing to do is to assume
that it's random: that its state at one moment predicts its state at the
next moment less than perfectly (because, in fact, we can't predict it
perfectly). But as we model the environment, we keep finding
explanations for variations, which are dependent on perfectly regular
processes elsewhere in the environment. Once we know those other
processes, we can reduce the uncertainty in our expectations for the
first process.
What if we assume that the _only_ uncertainties that matter in
perception are those associated with the perceiving apparatus itself?
Suppose we assume that all physical processes outside us are perfectly
regular, once the causes of variations are known. Then we are justified
in proposing regular models of the environment, and treating all random
variations as either imperfections in the model or noise in the
receiving apparatus.
The noise in the receiving apparatus is, I would maintain, quite small
in the mid-range of most perceptions. It is, of course, relatively
larger at the lower threshold of detection, but if we consider the whole
possible range of variation of a perception, the true noise is a small
fraction, like a few percent, of the range. That is why the world we
experience is, by and large, noise-free. Our main uncertainties about
the world of experience come not from its randomness, but from our
ignorance. As we develop better and better models, the apparent
unpredictability of the world becomes less and less, approaching the
noise level of direct experience -- essentially zero.
-----------------------------------------------------------------------
Best,
Bill P.