Hi, Richard -- (copies to Goldstein and CSGnet)
I feel a great reluctance to use the kinds of tests that psychologists have devised over the years, tests that David Goldstein feels might be useful in evaluating outcomes of therapy. I don't want to be dogmatic about this, so I'm looking (still) for some consensual way to evaluate this approach.
Here is a way of putting the problem I see:
Given: two tests, one measuring performance in some task and the other purporting to show whether a person has certain characteristics or traits. The question is, what are the chances that the first test will correctly indicate whether the person has a given characteristic as shown by the second test?
This implies that there are characteristics (like weight) that a person has or doesn't have regardless of our ability to measure them with the second test. The second test, we can say, produces results that are disturbed by unknown factors which cause the measurements to vary with some standard deviation. We can stipulate that without the disturbances, the test would measure accurately.
We can also stipulate that the first test is similar: if it were not for unknown randomly varying factors, the first test would always yield the same (correct) measure of performance. I know this assumption is unwarranted, but let's make it anyway since it seems to be a popular one.
Under these assumptions, the question now concerns the translation of a correlation between the first test and the second (given knowledge of the standard deviations) into a probability that the person actually has the characteristic in question. I suppose this can be cast in terms of the proportions of false positives or false negatives. I introduce the correlation topic because that is the figure that is usually obtainable from articles in the literature.
I know that you've already analyzed the case for screening people for traits or predicted performance on the basis of tests. I'm sure that paper is on my computer or in an archive somewhere -- I haven't found it yet. But maybe I'm asking a slightly different question now.
The context of my question is evaluation of therapy outcomes. This gives the question two meanings.
1. If one gives a patient a test before therapy and it indicates that the person has a condition like depression, what are the chances that the person actually has that disorder, under the assumptions above (i.e., there is actually a condition called depression that the person has or doesn't have, and so on).
2. If a before-and-after test is given, what are the chances that the condition actually changed in the direction implied by the two tests? I suppose that has to be put in the form of the probability that there is a finding of change when in reality there was a change in the opposite direction, or no change.
If the standard deviations were close to zero, these questions would answer themselves: the correct result would be the one we measure. But with correlations less than 1.0, the chances of a favorable result become less. My suspicion is that in the range of correlations usually found in the psychological literature, the favorable answer has less than a 50% probability. For example, suppose the probability is 70 percent that a person who shows as depressed on a performance test is actually depressed. And suppose the probability that a test which measures depression by other means has a 70 percent probability of being correct. In that case, the chance that person who is described as depressive by the first test will measure as depressive on tghe second is 50% -- a coin toss would do as well as this combination of tests. And, of course, if antidepressants are prescribed, half of the time the main effects on the patient will be the side-effects.
If my suspicions about these matters are correct, there are very serious implications for psychology which I don't need to spell out. I predict enormous resistance even to finding out the truth of this matter, but what would be new about that?
Best.
Bill P.