Martin Taylor 2007.07.21.15.52]
[From Bill Powers (2007.07.21.1010 MDT)]
Martin Taylor 2007.07.21.11.08 --
In other words, you would like to get more information about each individual, but would discard the information you are actually able to get, on the grounds that it is obtained from people "like" the person of interest, rather than from that person's self.
No, not on those grounds: on the grounds that it is more likely to be false for a given individual than true.
Perhaps, but less so than if you didn't have the group data.
I would prefer high-quality facts about each individual to low-quality facts, where "quality" is simply the probability that the fact will prove to be true of the individual being studied.
Wouldn't everybody? You aren't unique in wanting to act in a well known environment rather than in a fog. How do you clear the fog is the question.
An example would be the height of the individual. I would prefer actual measures of the individual's height, and discard measures of people "like" that individual in regards other than height.
What if you can't get hold of the person to see whether he would fit in your box? Wouldn't you rather select a pygmy bushman than a Masai warrior if you need a short person and all you know is tribal background?
I would rather know a person's actual grades in math courses than the grade point average of people who had similar SAT scores.
Unfortunately, those scores are in the future at the time you are making your choice. Too bad.
But would the result of interviews establish the "fact" you want to know about the individual (presumably the degree to wich they will benefit from the education you offer, or the success of the medical intervention, or ...)?
It would come a lot closer than an SAT score would. As to medical interventions, I would prefer an MRI scan to a survey of people of similar physical traits who show similar symptoms of a brain tumor.
You ae switching the situation. And in all cases you mention so far, you are asserting the possibility of getting information from other, and better sources, which everyone will agree is a good idea. The issue is that you want to NOT get information from one particular kind of source, even when it is available to you, and you seem to be asserting that rejecting the information from group statistics is justified because to use that inforamtion is WORSE than making a pure guess.
Let's consider the SAT - GPA scatterplot, which we all acknowledge shows a pretty poor correlation, reliable though it may be. MSAT accounts for only 22% of the variance in GPA. If we define the median GPA as the pass-fail point, 25 will pass and 25 fail. Suppose we choose 25 at random from the 50 candidates in the scatterplot. Different choices of them will result in different numbers of failures, but on average, 12 or 13 will fail. However, if we use the MSAT, and choose those that are above the median in the scatterplot you posted, we find that 16 pass and 9 fail (using the GPA of all of them to obtain the median). Of the ones rejected, 9 would have passed and 16 would have failed.
I think if you were an admissions officer, and GPA your criterion for success, you would prefer to thing that you would have 16 passes and 9 failures rather than 12 or 13 passes and 13 or 12 failures in the 25 you select. It's not a big improvement, but is an improvement.
You're offering all the standard defenses of statistics that I've heard all my life. They all sound like attempts to defend doing something that one intends to go on doing no matter what, even if it's wrong.
If you've heard all these so-called "defences" (I'd call them explanations), I'm wondering even more bwilderedly why you are defending so strongly the position that it is wrong to use what information you can when asessing a situation.
Back to Martin:
No, your interview would would not establish the "fact" you want to know. All you would do by conducting the interview is to make it more probable that you would select those who would be most likely to benefit (two levels of chance, here).
That would depend on what the interview is about. I would be trying to see how much the person wants to attend college, and why, and how much work the person is willing to contract to do, and whether the person's word is any good.
As I said: "All you would do by conducting the interview is to make it more probable that you would select those who would be most likely to benefit". Are you disagreeing with me? It sounds as though you are just giving examples to support my statement.
Who is there who doesn't deserve a chance to try, even if the end result is not something dazzling? And who is to say that a person who does poorly in high school will not get his act together in college? Not me, for certain! And who has the nerve to tell someone he doesn't deserve an education just because he isn't smart? Who decides what a "benefit" is? Would not a person with a middling to poor intellectual history benefit more than a person who already does intellectual gymnastics with ease?
All most laudable sentiments, with which I think few would disagree. However, admissions officers aren't in the pleasant position of being able to offer places to everyone who wants. They MUST tell some applicants "Sorry, we don't expect you to benefit as much as these others who we have accepted, and because we accepted them, we don't have room for you."
You say you don't want to be the one to select. Fine. I'd hate the job, too. But if I were in a position where I had to make a choice (about anything), I'd want as much information as I could reasonably get, recognizing that acting to get that information is likely to conflict with the requirement to act to control some other perceptions, and that if I don't make the choice before time T, I might as well not choose at all. I use what I can get, and so do real-life admissions officers.
With a high likelihood of misclassifying the person and wrongly treating the person. How high? That can be calculated, but nobody wants to touch that calculation with a 10-foot yardstick, apparently.
I answered that one a few times already. But you follow the Nelson principle of putting your telescope to your blind eye when there is something you don't want to see.
If you can show me what the test results have to be to lower the probability of misjudging an indivdual to an acceptable level, I will pay attention.
ANY correlation is better than none. ANY information is better than none. What you need in order to get the probability down to an "acceptable level" depends on the situation. If you have to make your decision NOW, then you guess based on what you know NOW. If you can wait, you can try to get more information. If the only acceptable level is zero error, you have to take an infinite time before you make your choice (Shannon).
Of course we then have to agree on what is acceptable. Richard Kennaway did that analysis once, and the results were so shocking that people with a vested interest in statistics as it is done immediately leaped on him and then abandoned the subject as quickly as possible.
Not my understanding of what happened.
There is always a way to get more information if you really want it.
Aye, there's the rub!
That stops all the discussion, right there. You can always wait, you can always devote more resources, if you really want it. But in the real world what you say simply is not true. You can't always wait, and you don't always have access to more resources. Substitute "is always" with "may well be" and I'll agree with you.
Sixty percent correct may not be much better than 50%, but it IS better. Causality doesn't enter into it; the correlation helps you make the choice that is more likely to turn out well.
I think that scheme is a lot like the one so many amateur gamblers come up with. If they just double their bets every time they lose, they will eventually come out ahead.
Nonsense. If you win 60% of the time with an even payoff win or lose, you win big-time by doing that, if you don't lose all your capital early on in the game. The random-walk problem of a long losing run that exhausts your capital very soon has a low probability of occurring within any reasonable finite time.
And anyway, the choice is not between 50% and 60% right for an individual; it goes the other way for many statistical tests. Look at that paper I did for Hershberger's collection, showing that there was an upward trend in a dataset where the same variables were related in the direction opposite to the trend, for every individual in the set.
I wondered why you hadn't brought up earlier that example of why effects for individuals occasionally don't go the same way as the correlations for the group would suggest. I had thought of mentioning it myself as a reason to be guarded. This as a perfectly valid argument for looking carefully at the implications of any correlation. You could make the same argument if the group correlation is 0.95. The effect still might go the other way within any individual. It ties in with the issue of confusing correlation with causality.
You do not have enough capital to survive being wrong 40% of the time. It wouldn't help the gambler to know that a certain strategy would increase his chances of winning from 50% to 60%.
It would. That's what the house bets on.
But if 40% of my patients die from the medication, there's little solace in knowing that 50% might have died without it. I would go on looking for a better medication, rather than stopping to practice medicine before I knew what I was doing.
Not knowing of a beter medication, then, I presume you would withold the medication and callously accept a 50% death rate, rather than using the medication you have available and reducing the death rate to 40%. I'd call that criminal negligence.
But you do a disservice to the individuals you want to serve if you arbitrarily discard relevant information on the grounds that probabilities are not certainties.
I have just the opposite feeling: I do a disservice if I use a treatment based on bad data on the grounds that on the average it provides an improvement -- but knowing that for any individual, it will most probably be useless or harmful.
"On the average it provides an improvement, but for any individual it will MOST PROBABLY be useless or harmful". Think about that statement for a moment.
I can think of statistical distributions for which this statement could be true, but they are pathological and very, very, rare in real-life situations. Whay take such situations as the norm, and base your actions on the supposition that they are likely to occur, before you have evidence (easily obtained) as to whether the situation of interest has this pathological form?
I write a lot of words, but the gist is this: some information is better than none, and good information is better than poor. If you have time and resources to get good information on something that matters to you, go for it. If you have to act now, don't, on ideological grounds, discard information that you do have.
Martin