[From Bruce Abbott (961127.1800 EST)]
Bill Powers (961127.1440 MST) --
Rick Marken (961127.1220)--
How do we objectively evaluate the goodness of fit between a model's
behavior and real behavior?
How about the standard error of estimate? It measures deviation of model
behavior from real behavior in normalized (standard devation of the real
behavior) terms.
OK, how do you calculate that? I'm being very lazy; I want a cookbook
procedure that has some kind of official recognition.
Bill, believe it or not, this is just another name for RMS error -- the
measure of goodness of fit we've been using all along. But it does seem to
have the properties you're looking for.
Regards,
Bruce
[From Bill Powers (961128.0130 MST]
Bruce Abbott (961127.1800 EST) --
OK, how do you calculate that? I'm being very lazy; I want a cookbook
procedure that has some kind of official recognition.
Bill, believe it or not, this is just another name for RMS error -- the
measure of goodness of fit we've been using all along. But it does seem to
have the properties you're looking for.
As my Dad once said upon opening his pay envelope, "What a disappointment --
just what I expected."
RMS error has to be compared with something, doesn't it? What about RMS
error divided by mean value? This would give us the error as a fraction of
the mean observed value. Of course that wouldn't work too well if the mean
observed value were close to zero.
In electronics, signal-to-noise ratio is used, which is calculated as the
peak-to-peak value of the signal divided by the RMS noise level. If the
signal is unipolar, the peak value is used. For judging the significance of
errors of prediction, the RMS noise level can be multiplied by some fraction
to give the standard deviation, and any given excursion of the signal away
from its modeled value can be judged as meaningful according to the number
of standard deviations of departure from the model. This gives you a way to
judge the chances of the deviation being real, or due to random fluctutations.
I don't know what to make of the fact that there's no standard way of
handling this problem, if that's really true. Perhaps it makes a statement
about the kind of research that's usually done. Or maybe there just isn't
any one-size-fits-all way of judging goodness of fit of a model to the data.
Best,
Bill P.