[From Rick Marken (961127.1220)]
Bruce Abbott (961127.1245 EST) --
You didn't answer any of my questions.
An oversight?
No. I just didn't see the point and I didn't have much to say, anyway. I
thought my answer to your question about the problem with your learning model
was pretty good, though. Did you understand it?
Anyway, since you ask, here are my answers to the only other questions I
could find in your post:
How do you select remembered reference specifications for perceptions?
I don't know.
How were you able to imagine what the perceptual consequences of controlling
these perceptions would be for another person? How were you able to turn
these into reference specifications for exactly the right control systems?
I don't know.
Tell me about some of the empirical observations you see as offering
difficulties for HPCT as now constructed.
Any empirical observation that differs from the predictions of HPCT would
cause difficulties for HPCT. There are, thus, an infinite number of
observations that, were they made, would create difficulties for HPCT. For
example, someone might observe an organism controlling a variable that it
cannot perceive. That would create a BIG problem for HPCT.
Bill Powers (961126.1200 MST) --
How do we objectively evaluate the goodness of fit between a model's
behavior and real behavior?
How about the standard error of estimate? It measures deviation of model
behavior from real behavior in normalized (standard devation of the real
behavior) terms.
Best
Rick
[
From Bill Powers (961127.1440 MST)]
Rick Marken (961127.1220)--
How do we objectively evaluate the goodness of fit between a model's
behavior and real behavior?
How about the standard error of estimate? It measures deviation of model
behavior from real behavior in normalized (standard devation of the real
behavior) terms.
OK, how do you calculate that? I'm being very lazy; I want a cookbook
procedure that has some kind of official recognition.
Best,
Bill P.
From Mr. Remi Cote
281196.0911 EST
In my thesis:
To calculate z score I use the se of a sample of prediction
from a sample of employee.
n=68
I compute the mean and se of their prediction, and then I compare
the prediction of the model with the mean of prediction
from amployee.
So I calculate the distance between the two prediction using
z score.
Also I use correlation, to measure similarity of prediction
over a range of 20 set of parameter.
That is I compare 20 prediction of the model with 20 prediction
of employee (mean of 68 prediction for each set of parameter)
I also think of a way to account for variability. In another
calculus I pair each of 68 prediction whith the same model
prediction, and I do it for the 20 set of parameter.
So I calculate a correlation between 1360 pair of prediction.
The result are slightly different.
Also to compare to model, I compare the mean of their z score
with a student t-test...
Any comment?
Salut
[From Bill Powers (961128.0930 MST)]
Mr. Remi Cote 281196.0911 EST
Thanks for your suggestions. I think my problem may be a little different
from yours, because in a tracking experiment, the model predicts complete
runs of data (1800 points to 3600 points per run) for each subject -- for
example, it predicts where the control handle will be for every 1/30 to 1/60
second during a 1-minute experimental run. Typical RMS differences between
the model's stimulated handle position and those of the subject are 5% of
the range of movement of the handle, meaning that a 10% error of prediction
for a single point would occur by chance for about 1 in 20 points -- if
successive handle positions were independent.
Anyhow, I don't know what a Students's t-test is! In physics and electronics
these advanced statistical techniques are hardly ever used.
ยทยทยท
-----------------------------------------------------------------------------
Bruce Abbott (961128.1025 EST) --
RMS error basically tells you how far the the data are from the model's
predictions, on average, in standard deviation units.
But isn't the important thing the size of the RMS error in proportion to the
size of the measure being compared? If weight is 300 grams and the RMS
prediction error is 6 grams, the prediction is off by 2% of the weight,
which I think is pretty good. But if the weight is only 30 grams, the same
RMS error is 20% of the weight, which I think is pretty bad.
1. the prediction tracks the data perfectly but with a constant offset;
this constant offset would appear as a difference between the means of
the predicted and actual values.
2. the means of the two waveforms vary according to the same pattern and
have the same mean, but differing amplitudes.
3. the two waveforms have identical patterns, amplitudes, and means, but
differ in phase.
4. the two wave forms have identical amplitudes and means, and nearly
identical waveforms except for a very low-frequency component
(i.e., a slow drift).
5. the two wave forms have identical means and amplitudes but different
waveforms.
A useful summary of error sources, all of which have application to our
modeling of rat data. When we adjust parameters, we're trying to minimize
errors of the various types, by various methods. The difference between
means would be adjusted by adding an adjustable constant to the model; the
difference in amplitudes, by adjusting a gain factor; the difference in
phases by adjusting either an integral lag (as we do) or a time-shift (which
we probably ought to try) or both; the slow drifts by adding a
time-dependent term to the model. If the waveforms are different, of course,
this probably means we have a basically wrong model, since the model's
output waveform emerges from its independent variables and its organization.
Right now we're using the sum of three squared-error measures as the basis
for optimizing the model: weight, home-cage intake, and experimental-cage
intake. This is probably far from the best approach. I suppose what we ought
to do is to calculate the effect on the various error measures of adjusting
each parameter of the model, and construct appropriate error criteria for
each dimension of adjustment. I wish I knew how to do that. I'm sure someone
knows -- this can't be a new problem. My brute-force method of sampling the
entire range of possibilities in 50 steps takes a couple of hours even
running under DOS instead of Windows, and that's with only three dimensions
(125000 runs of the model). Adding a couple more adjustable parameters would
mean that a single evaluation would take 5000 hours, a good part of a year!
This is really the problem behind the problem as I first presented it.
Considering my ignorance in this area, it's really amazing that we're
getting as close as we are.
Best,
Bill P.