[From Bruce Abbott (961127.1800 EST)]

Bill Powers (961127.1440 MST) --

Rick Marken (961127.1220)--

How do we objectively evaluate the goodness of fit between a model's

behavior and real behavior?

How about the standard error of estimate? It measures deviation of model

behavior from real behavior in normalized (standard devation of the real

behavior) terms.

OK, how do you calculate that? I'm being very lazy; I want a cookbook

procedure that has some kind of official recognition.

Bill, believe it or not, this is just another name for RMS error -- the

measure of goodness of fit we've been using all along. But it does seem to

have the properties you're looking for.

Regards,

Bruce

[From Bill Powers (961128.0130 MST]

Bruce Abbott (961127.1800 EST) --

OK, how do you calculate that? I'm being very lazy; I want a cookbook

procedure that has some kind of official recognition.

Bill, believe it or not, this is just another name for RMS error -- the

measure of goodness of fit we've been using all along. But it does seem to

have the properties you're looking for.

As my Dad once said upon opening his pay envelope, "What a disappointment --

just what I expected."

RMS error has to be compared with something, doesn't it? What about RMS

error divided by mean value? This would give us the error as a fraction of

the mean observed value. Of course that wouldn't work too well if the mean

observed value were close to zero.

In electronics, signal-to-noise ratio is used, which is calculated as the

peak-to-peak value of the signal divided by the RMS noise level. If the

signal is unipolar, the peak value is used. For judging the significance of

errors of prediction, the RMS noise level can be multiplied by some fraction

to give the standard deviation, and any given excursion of the signal away

from its modeled value can be judged as meaningful according to the number

of standard deviations of departure from the model. This gives you a way to

judge the chances of the deviation being real, or due to random fluctutations.

I don't know what to make of the fact that there's no standard way of

handling this problem, if that's really true. Perhaps it makes a statement

about the kind of research that's usually done. Or maybe there just isn't

any one-size-fits-all way of judging goodness of fit of a model to the data.

Best,

Bill P.