out on a limb, part 2

[From Jeff Vancouver 940721.1808]

I hesitate to post this given that my previous post is my main concern, but I
forgot to ask Marken where "the Blind men and the elephant" was published.

But while here, I wanted to rebute the psychology is useless and
dangerous notion. In my field we develop tests of cognitive ability and other
predictors of job performance. Focusing on job samples, which rarely have
adverse impact (where scores differ depending on race or sex), these tests
can predict job performance at around .30 to .60, depending on the job
mostly. The alternative, doing nothing, would correlate .00 with job
performance, or the job interview (before we improved it) .11. The
differences might seem trivial to those who look for correlations in the
upper 90s, but the difference can save a company hundreds of thousands of
dollars (we have data on this). From the individuals stand point it will
improve the fit between
their skills and their job, which usually make the individual happier and
more secure. Without the work of psychologists, organizations would be
less productive and individuals would wander from job to job looking for a
good match.

There is work in psychology that is dangerous and other work that is useless.
My colleague next door does it (just kidding). But there is also useful
and helpful work. May we be the ones who decide which is which.

Later

Jeff

<[Bill Leach 940722.00:26 EST(EDT)]

[From Jeff Vancouver 940721.1808]

But while here, I wanted to rebute the psychology is useless and
dangerous notion. ...

There is work in psychology that is dangerous and other work that is
useless. My colleague next door does it (just kidding). But there is
also useful and helpful work.

Sounds like your work may indeed be useful. You work with data that does
not explain (or try to explain) how people function but rather with data
that is statistical and correctly so. There is no doubt the occassional
error where your data is not correct for a particular individual but like
the "mortality tables" your data "is the best we can do.

PCT is probably not at a point where is could be reliably or economically
applied to such a task (even if there were enough PCTers to try to do
so).

The quality of your data would however benefit from a deep understanding
of PCT. A serious problem with a great deal of the data collected in the
behavioural sciences is that critical information needed to relate the
data to people based upon how people actually function is not taken.

-bill

From Tom Bourbon [940722.1105]

[From Jeff Vancouver 940721.1808]
I hesitate to post this given that my previous post is my main concern, but I
forgot to ask Marken where "the Blind men and the elephant" was published.

It is in our ghetto journal, _Closed Loop_, 1993, vol. 3, no. 1 -- the same
issue that contains "Models and their worlds," by Bill Powers and me.
Copies are available from Mary Powers and I have several extras.

But while here, I wanted to rebute the psychology is useless and
dangerous notion.

Jeff, I'll start my reply by going to the conclusion of your post.

There is work in psychology that is dangerous and other work that is useless.
My colleague next door does it (just kidding). But there is also useful
and helpful work. May we be the ones who decide which is which.

I've stopped using the label in most settings, but I am also one of "we;" I
am -- gulp -- a psychologist.

I am still a card-carrying member of the American Psychological
Association, the American Psychological Society (a charter member), and the
Psychonomic Society (than which there is no purer group of experimentalists
;-). I taught psychology courses, undergrad and graduate, for over 28
years. I speak from _inside_ psychology.

What I am about to say is not "merely" a PCT issue, but an issue that should
concern all behavioral (life, medical, cognitive, etc) scientists, no matter
their theoretical stripes.

In my field we develop tests of cognitive ability and other
predictors of job performance. Focusing on job samples, which rarely have
adverse impact (where scores differ depending on race or sex),

Does this mean scores that lead to wrong predictions, but do not differ
according to the race or sex of the test taker, rarely have "adverse impact?"

these tests
can predict job performance at around .30 to .60, depending on the job
mostly. The alternative, doing nothing, would correlate .00 with job
performance, or the job interview (before we improved it) .11.

In my reply, I am assuming that the scores you report are correlation
coefficients, calculated from scores on the tests compared with measures of
performance on the job. If my assumption is wrong, never mind what I say
next. :slight_smile: I know, Jeff, that you already know many of the things I am
about to say, but there is a diverse audience looking over our shoulders and
I want to be sure the same vocabulary is available to everyone. And please
don't think I am trying to impugn your motives or your character.

A common interpretation of a correlation coefficient (r) is in terms of the
percentage of the variance it "accounts for" in the relationship between the
two variables (for example, between test scores and measures of job
performance). A common estimate of the variance accounted for is the value
of r-squared. In your example, r-squared ranges from .3^2 = .09, 0r 9%, to
.6^2 = .36 or 36%. Now it is certainly true that either of those
correlations and variances accounted for is greater than zero. But how well
do they work, and does the use of the tests really have no adverse impact on
people? From whose perspective is the effectiveness or adverse impact of
the tests decided?

Concerning the effectiveness of the tests, there are other informative ways
to interpret the relationships "captured" by correlation coefficients. One
of those ways is to calculate k, the "coefficient of alienation," which is a
coefficient of inefficiency or failure of the corrleation as a predictor.
The calculation is simple: k = square root( 1 - r-squared).

In your example, for r = .3, k = .954; for r = .6, k = .800.

What do those numbers mean? First, if you did not know the correlation
between test scores and job performance, but you did know the means of both
sets of scores, then given a particular person's score on the test, your
best estimate of the person's job performance would be the mean score on the
scale of job performance. This is always the relationship between predictor
and predicted scores, _if_ you do not know the corelation, or if the
correlation is zero. (When r = 0, k = 1.00 -- maximal alienation or
"uselessness" as a predictor.) Any non-zero correlation should reduce the
coefficient of alienation, indicating that the correlation improves your
ability to predict performance from the test. But the gain in predictive
ability is low, until the correlation coefficient is very large.

For example, your correlation of .3 leaves the chance of a failure in your
prediction of job performance 95.4% as great as it was when you did not have
the test. And for r = .6, the chance of failed predictions is 80% as great
as before the test.

I grant, right up front, that even a 4.6% success rate is non-zero and that
it might appear to be of some use to an employer, but what about the
people to whom the test is applied and whose lives are thereby affected?
The race, gender, age, height, and sock size of those individual persons are
irrelevant; the fact is that many more people will be harmed by the
application of such a test than will be helped, unless, of course, the
people we are talking about are the employer or people on the employer's
"team."

The
differences might seem trivial to those who look for correlations in the
upper 90s,

For now, let's just say they are demonstrably small and they are very poor,
as predictors.

but the difference can save a company hundreds of thousands of
dollars (we have data on this).

I grant you that.

From the individuals stand point it will
improve the fit between
their skills and their job, which usually make the individual happier and
more secure.

From the stand point of _which_ individuals? Certainly not from that

of the 95.4% to 80% of people who are misclassified by such a test.
The misclassifications will include both the very large numbers of people
who are denied a job and the ones who get the "wrong" job.

Without the work of psychologists, organizations would be
less productive and individuals would wander from job to job looking for a
good match.

I think both of those claims can be challenged, especially the one that
seems to portray humanity as lost and wandering in the wilderness, but for
the intervention of psychologists. Most of the psychologists I have known
seemed pretty badly lost, themselves. :wink:

I am _not_ saying the quest for efficiency and fairness is wrong. I _am_
saying that the psychological "knowledge" brought to that quest is poor and
that its application is dangerous, if and when the application is to
individuals.

ยทยทยท

=======

<[Bill Leach 940722.00:26 EST(EDT)]

[From Jeff Vancouver 940721.1808]

But while here, I wanted to rebute the psychology is useless and
dangerous notion. ...

There is work in psychology that is dangerous and other work that is
useless. My colleague next door does it (just kidding). But there is
also useful and helpful work.

Sounds like your work may indeed be useful. You work with data that does
not explain (or try to explain) how people function but rather with data
that is statistical and correctly so.

Ah, Bill, the tests _do_ allegedly say something about how people function;
individuals are the ones who are given jobs, or denied jobs, on the
psychologist's assumption that a single person's test score says something
about how that particular person would function in a specific job. The test
may be _administered_ to people en masse, and scored in bulk on a
computerized system, but its application and effects come down one
misclassified person at a time.

There is no doubt the occassional
error where your data is not correct for a particular individual

Occasional error? From 80% tp 95.4% errors don't rank as "occasional" in my
book. As a psychologist, I am ashamed that we use such lousy and dangerous
predictors. But then, as a psychologist, I'm a pretty lousy representative
of the field.

but like

the "mortality tables" your data "is the best we can do.

And "the best we can do" is not good enough, in my private opinion.

By the way, the mortality tables do represent descriptive statistics of high
quality -- we have pretty good data, from large numbers of cases, on the
proportions of people in various age groups who die during a given period of
time and on whether those proportions are stable or changing. We have
nothing near that quality with regard to the validity of screening tests.

PCT is probably not at a point where is could be reliably or economically
applied to such a task (even if there were enough PCTers to try to do
so).

As I said up top, this is not a PCT issue, but an issue of the adequacy of
data and predictions in behavioral science. I'll crawl even farther out
on my limb: The predictive power of psychological assessment "instruments"
will never improve very much, so long as the causal model behind
psychological research and test construction is lineal and so long as
psychologists continue to mis-apply statistical procedures, making
statements and predictions about _individuals_, when they have used
statistical procedures that (when used properly) only allow you to speak
about groups.

Later,

Tom