[From Bill Powers (930418.1500 MDT)]
Martin Taylor, Allen Randall, other IT phreaques:
In the latest Simcon simulation for testing Martin's proposition
about a relation between bandwidth and maximum loop gain, some
interesting developments have cropped up.
Martin asked me if I had done a fourier transform on the
variables. I hadn't, but I happen to have MathCad 3.0 for Windows
(a $79 bargain), and Mathcad can import ASCII data files, and
Simcon puts out ASCII data files, so what the hey.
You will recall that this simulation puts a combination of pulses
together to produce a disturbance, and smooths the result through
an amplifier with a time constant of 1 sec. The input function of
the control system receives the sum of disturbance and output,
and puts them through another amplifier with a time constant of 1
sec. So the bandwidth of the disturbance and that of the input
function are exactly equal. The output function is just a
multiplier of 100, with no time constant -- the delay around the
loop, with all the blocks grouped, is 0.01 sec. The loop gain of
the control system is 100 (Martin's algebra had predicted that
the maximum usable loop gain would be 0). Control is excellent;
the range of the disturbance is about 20 units, while the
variations in the perceptual signal have a range of about 0.2
units (reflecting the loop gain of 100).
I produced four plots: perceptual signal, its fourier transform,
disturbance, and its fourier transform. The fourier transform
plots are identical to the naked eye, so the bandwidths are in
fact identical. FYI, Martin.
But even more interesting, the disturbance and perceptual signal
plots were ALSO identical. Of course there was a scale difference
of a factor of 100, but MathCad automatically scales the plots to
fill the graph frame, and it was immediately apparent that every
detail of the disturbance was reproduced in the perceptual
signal, minified.
Now this result will not in general be true; it arises from the
fact that there is no time function or nonlinearity between the
comparator and the effect of the output on the controlled
quantity, and there is only a negigible transport lag. But it
brings up a point that so far nobody has thought about: the
effect on information content of scaling signals up and down.
In this case, we have excellent control, and in the simulation
the noise level is insignificant, even when the disturbance is
scaled down by a factor of 100. Because of this, the waveforms of
the disturbance, the perceptual signal, and the output signal
differ only in scale and sign: their forms are essentially the
same. The same plot would serve to represent any of the three,
with no important difference except the numbers along the y axis.
If I were to hand any of these plots to an IT expert, that expert
could calculate the information rate represented by the plots.
Absent any visible noise, that information rate would be the same
for all three of them.
So we have one case which, save for one unanswered question,
shows a control system with high loop gain keeping a perceptual
signal almost constant despite a large disturbance, with the same
information content in all the critical variables and signals.
All the information in the disturbance is present in the
perceptual signal, and all the information in the perceptual
signal is present in the output (give or take very minor
differences).
The unanswered question, of course, is how to handle information
content in signals that have the same form but very different
amplitudes -- here, different by a factor of 100. In the above, I
have assumed that the scaling factor makes no difference in the
information content. But behind that assumption lies some
missing, and possibly incorrect, reasoning.
In order for two signals of different amplitudes to have the same
information content, they must be exact scale replicas. And this
scaling must include not only the amplitude range D, but the size
of the minimum step in amplitude, r. To preserve the information
content, log(D/r), BOTH D AND r MUST BE SCALED BY THE SAME
AMOUNT. This may provide the missing piece in the puzzle.
Suppose we set up a situation in which the disturbance alone
affects the perceptual signal, and in which the maximum amplitude
of the disturbance would just produce the maximum possible amount
of perceptual signal. Call this maximum 1000 units. One thousand
units of disturbing variable, in physical units, produces 1000
units of perceptual signal, in nervous system units (NSU).
Furthermore, let's say that the resolution of the perceptual
signal is 1 unit. So D/r for the perceptual signal is 1000, and
its binary log is about 10. The channel capacity of the
perceptual signal is 10 bits per time-slice, which we will leave
indefinite.
Now connect the output of the control system modeled above, with
a loop gain of 100. This immediately reduces the fluctuations of
the perceptual signal from 1000 units (the disturbance's range)
to 10 units. The waveform of the perceptual signal is still
identical to that of the disturbance, except for the scale
factor, but now the resolution of 1 unit is 10 percent of the
actual range of the perceptual signal instead of 0.1 percent.
If we take that resolution as the size of r, and for D use the
actual range of fluctuations of the signal, the information rate
in the perceptual signal is now blog(10/1) instead of
blog(1000/1). The rate has dropped from 10 bits per time slice to
a little over 3 bits per time slice. The loss of a little under 7
bits of information per time slice reflects exactly the loop gain
of 100.
In the simulation, the time-slices are 0.01 sec apart. Very
roughly speaking, this means that the input time constant has the
effect of integrating over 100 of these time slices.
Let's suppose that the output of this system, driven by the 3-
bit-per-second perceptual signal via the comparator and simple
linear output function, also contains 3 bits per second. But the
effect of this output on the perceptual signal has to pass
through a time constant of 1 second, so its effects in opposing
the disturbance effects are cumulative over about 100 time
slices. Intuitively, this says that about 7 bits of information
should be accumulated through the time constant, adding back what
was lost because of the feedback and the scale reduction.
This is a far from rigorous analysis; it's just a sketch of the
logic. Now it's the turn of the IT gurus. If you can work out the
exact relationships here, you can come up with a RIGOROUS
application of IT to control systems, for at least this one
special case. I think you may be able to show that there IS a law
of conservation of information, only it is a law of conservation
of information rate times time, not just information rate.
I know that this is like asking Einstein to add up columns of
numbers, but I think if you want information theory to be
accepted as real by control theorists, you're going to have to
work out the actual exact computations for this case,
establishing once and for all an "acceptance theorem." Perhaps
all these relationships are already implicit in your high-flown
generalizations, but if you want to communicate you'll get down
and get your hands dirty with an actual application.
Once you have this special case worked out, the next step is to
vary the model somewhat. Put the integrator, for example, in the
output function instead. Play with different combinations of
disturbance and input function bandwidths (other than equal
bandwidths). Pretty soon, I predict, you will see the way to the
general theorem, and we will then be able to stop arguing. You
will be able to set up a Simcon simulation, predict the
information rates at each point in the system, and show that
these are in fact the results. Doing this would not only get rid
of the IT/PCT controversy, it would be good for your think-tank
souls.
Best,
Bill P.
I checked the simulation