[Martin Taylor 2009.01.05.13.47]
[From Bill Powers (2009.01.05.0942 MST)]
Martin Taylor 2009.01.04.14.13 –
In my language,
“expectations” would be the prior probability distribution over
the hypotheses, whereas “background knowledge” is what leads to
those expectations.
OK. Both are established prior to testing the hypothesis.
I’m not keen on revising
the
words conventionally used, much as you were not keen on finding a new
word to represent an internal signal called “perception” in
PCT. “Probability” as necessarily subjective is as closely
related to “probability” in everyday parlance as are the
technical and everyday use of “perception”.
I’m not sure of that. Don’t quantum physicists typically speak as if
the
uncertainties involved in quantum phenomena are really there in nature,
rather than in the observer’s mind? Or do you exclude that from
“everyday” use? Don’t gamblers usually speak of probabilities
as if they were really there in the cards or the dice? Is a 30% chance
of
showers tomorrow spoken of as if it’s in the forecaster’s perceptions
or the viewer’s?
Lots of people talk about probabilities as though they were really out
there. That’s true of many perceptions, isn’t it? We don’t know whether
our perceptions match real reality for probabilities any more than they
do for anything else. Maybe they do, maybe they don’t. When a
forecaster says there’s a 30% chance of showers tomorrow, that
presumably is what he believes is the limit of his understanding and
what he wants us to believe if we base our estimate only on his word.
If he knew all of the physical conditions that applied to producing the
weather, his words would say exactly when and where each shower would
occur, wouldn’t they?
When gamblers think of probabilities as being “in the cards”, they are
working from a blend of models and past experience in developing their
subjective probabiities, are they not?
As for the quantum physicists, isn’t there an argument even within the
fraternity about the “reality” of the probabilities? Even there, there
are two sorts of probability to be considered: the probability that X
will be observed under condition Y, and the model of quantum physics
that asserts an intrinsic probability to lurk within the systems of the
very small. The first is a subjective probability, whose value may
involve the second if the person believes that model.
“Belief”, to me,
has a somewhat different connotation. Maybe it’s because I’ve used the
terms for 50 years, but “prior” and “posterior”
probabilities seem to have a nice precise connotation (limits between 0
and 1.0, the total of all mutually exclusive probabilities summing to
1.0, and so forth), as does “likelihood” (likelihoods don’t
necessarily sum to 1.0). Perhaps you can map “belief” onto
probability by normalizing. I don’t know.
OK, but when you start defining belief and likelihood so they can be
calculated from a set of definitions, it seems to me that this makes
them
something other than subjective. Is a Gaussian probability distribution
subjective? Or is it Gaussian for everybody, whether they believe it is
or not?
This is a bit like saying that 2 + 2 = 4 is a subjective opinion. It
is, but in most people’s minds it ties in with so much else that the
whole web of relationships is consistent enough to make it seem like an
objective reality. You are talking here about the subjective or
objective reality of mathematical operations. A Gaussian probability
distribution is a mathematical statement. Whether it applies in any
situation is another question. Whether mathematical operations exist
outside the mind is yet another, one that I don’t want to contemplate.
Let me use an example I like. Does the question “What is the
probability that the next stranger I see will be wearing brown shoes?”
make any sense to you? Is there such a thing as an “objective
probability” answer to that question? If so, what is the population of
“next stranger I see”, and how do I find the probability that a random
member of that population wears brown shoes? There is a definite
subjective probability answer, different perhaps for you and for me,
but it’s not a silly question if it is asked about your subjective
probability.
One of the definitions of probability as a mathematical concept is that
it is limited to a range of zero and unity. It would be quite possible
to define a related concept that ranged from zero to infinity, or from
minus to plus infinity. The zero to unity range is easy to work with,
and people usually don’t have much trouble with concepts that can map
onto it, such as 50-50 or “ten to one”. If I say I’d give ten to one on
something happening, I wouldn’t object if someone told me that this
meant I had assigned it a probability of 0.91.
Probability has some requirements if it is to be a useful concept, such
as that if certainty is taken to be probability 1.0, then if a set of
possibilities is exhaustive in that one and only one of them must be
true, then their probabilities must sum to 1.0 exactly. That must be
just as true for subjective probability as for “objective” probability.
You can’t be certain that they sky will be cloudless and overcast when
you first look out tomorrow, but you can say there’s a 50-50 chance of
either (assuming that there are no possible intermediate cloudiness
states). It’s convenient also to arrange the probability scale
mathematically so that if B never happens except if A has happened and
when A happens be happed p(B|A) of the time, the probability of B
happening will be p(B)p(B|A). That’s a normal intuition about
probability, so why not scale it so that works? The others of what
Jaynes calls “Desiderata” are of the same kind. They give expression to
what a lot of people think of as being natural properties of something
one would call “probability”.
So, it’s not true that “belief” and “likelihood” can be calculated from
a set of definitions, so much as that the properties of probability as
a mathematical concept are devised so as to agree with what most people
would think of as natural properties of probability, and those
properties in turn make it possible to do the calculations.
What you can say
about the
“White Swan” problem is that if the conditions remain the same,
the likelihood that the next swan you see will be white increases the
more swans you have seen without seeing a non-white one.
I think we can now say this in a way that doesn’t imply the objective
existence of some “liklihood” of seeing a white swan.
Sorry. I’m so used to thinking only of subjective probabilities that I
tend to omit the word “subjective”, A likelihood is the
probability of the hypothesis given the data,
… which means “the degree to which you will believe the hypothesis
given that you believe what the data appear to tell you?” The
shorthand way of putting it suggests strongly that there is an actual
probability of the hypothesis independent of the observer’s belief in
it,
and that the data have meaning independent of the observer’s
interpretation.
I think I accept your first sentence, but I don’t see how your second
relates to it. Maybe the second episode message will help.
I
meant only that the stage manager, following the rules of the script,
is
holding the black swan aside for the proper time, at which time it will
appear. The black swan already exists although we haven’t seen it yet.
“Whim” was the wrong word. Perhaps, like a superenergetic
cosmic ray, it will appear during only one observing period out of a
million. But appear it will, and it will show that not all cosmic rays
are “normal,” even though a given observer has seen only normal
ones in his lifetime. Its existence is not uncertain; only the time
when
it will be observed is.
Oh, but its existence IS uncertain to the person counting swans, even
if it is not to the stage manager. That uncertainty is why no amount of
counting swans that are white will ever lead to a probability 1.0 for
the hypothesis “All swans are white”, using the kind of analysis I
illustrated…
In effect, the
Bayesian
system is a model which is used to generate expectations about the
relative degrees of belief a person will arrive at on the basis of
successive subjective observations. While it creates a plausible
proposal, experiments have shown that no one model can explain
everyone’s
way of adjusting expectations on the basis of successive
observations.
True. But I think you misstate what the Bayesian system is. It isn’t a
model of what people do, but a model of the best they could
do.
Yes, I was really trying to make that point. It is, as you say, an
idealized method, invented by someone and adopted by others but not a
“natural” aspect inherent in the brain’s organization.
Correct. Neither is the Ideal Observer of psychoacoustics inherent in
someone’s brain. It’s the best that any mechanism, biological or
designed, could do under the specified circumstances. It doesn’t matter
how the organism or the mechanism works.
On the other hand, in a
lot of
situations, when something affects the performance of the ideal in some
way, people very often are affected in the same way.
In psychoacoustics, for
example, the ability of a person to detect a signal (say “yes”
when the signal exists and “no” when it doesn’t) as measured by
d’ has a well defined ideal value for any particular defined signal and
noise, provided that the prior information available to the ideal and
to
the human is the same.
But if you say “the same prior information” you’re simply
assuming a human being who “correctly” interprets prior
observations in the “correct” way, so you’re just comparing two
ideal computations, one made by you and one made by the subject.
No I’m not. I’m saying that no matter how the subject does it, the
result can never be better than the ideal, any more than a mechanical
engine can get more than an ideal engine out of a given temperature
drop. Ideal is ideal, a limiting possibility.
How did
the subject do it? You don’t know. Maybe the ones who do it best have
figured out some of the principles of Bayesian logic.
As my own subject in such experiments, I don’t know how the subject
does it. For sure, it’s not done with any conscious computation! You
just learn to hear something that turns out to be what you are supposed
to be hearing.
A well trained observer
may get within 6 db of the ideal, and a highly trained one within 3 db.
What kind of db are those? If you mean 10 Log(x)[base 10], 3 db is a
factor of two in the measure of x, and 6 db is a factor of 4, neither
of
which is very close.
Close is in the eye of the observe. 3 db is, for many people, about the
minimum difference that will allow them to say something is louder or
softer than something else, though trained subjects under good
conditions can probably do 1 db. What the ideal can do depends on the
length of the listening interval and on the information it has about
such things as the signal frequency. When you are measuring energy, 3db
is indeed a doubling of the energy. The point is that it takes a great
deal of training (like two months of 3 hours a day) for people to get
that close to the ideal when detecting a signal, and it takes some
training to come to within 6 db. When you are comparing with the ideal,
3 db is reasonably used to mean “half as precise” or the appropriate
analogue in other domains.
The problem with experiments like these is that they
create low-probability perceptions, and lead to interpreting
the
results as if they apply to all perceptions – most of which are very
high-probability perceptions. An accuracy of 3 db may be astonishingly
(or expectedly) good for a very noisy observation, but it’s terrible
for
an observation made under normal conditions.
I think you completely misunderstand. If the ideal finds a detection
easy, so will the human. If the ideal finds it difficult, so will the
human. In both cases, the human seems usually to be about the same
number of db worse than the ideal, the actual number depending on the
human’s listening ability. I have not the slightest idea what “an
accuracy of 3 db” could mean. Maybe you could explain that usage.
Sherlock Holmes can make
wrong
deductions from the given data, but for the same number of wrong
deductions he will make more correct ones than will most people. Even
he,
however, cannot do better than a Bayesian analyst working with the same
data and hypothesis universe.
That is true, to the extent that both live in a universe where it is
difficult to know what the data are.
No. It’s simply true. No qualification. No exemptions.
So, even if people don’t
actually behave in an ideal way, it’s not a bad thing to understand the
mathematically ideal limit to what they could do.
We part company when you assume that it is difficult most of the time
to
behave in an ideal way because of large uncertainties of observation.
Again, you completely misunderstand. It is you who insert “large
uncertainties of observation” into the discussion. Get rid of them. It
doesn’t matter in the least how large the uncertainties are. The ideal
is the ideal, and no human or machine can do better.
I will readily admit that Bayesian statistics are
appropriate in
cases where low probabilities are the norm. But I don’t admit that such
cases are predominant or even important in ordinary behavior – more
than
occasionally.
Could you explain how the mathematical relations depend on the
probabilities being low?
Personally, I would not
be
very surprised if it turned out that the peripheral neural structures
had
evolved to function in a near ideal way (within, say, 3db) when it
comes
to relating low-level perceptions to states of the external
environment.
(That speculation was at the base of my comments on neural spike timing
effects last month).
I think that normally we get a LOT closer than 3 db.
Demonstration, or reference, using the appropriate ideal performance
for the situation?
You couldn’t hum the
right note when the choir director blows into the pitch pipe if you
could
perceive the difference in pitch only within 3 db, a factor of 2. Most
people can get within a fraction of a half-tone, which is a fraction of
the twelfth root of 2 of an octave, or a fraction of 1/4 db.
I’m not sure what the ideal is for this example, but it has to depend
on the length of the sample and its SNR. To make the argument, you
would have to know the ideal for the experimental situation, and then
see if people could get within twice the ideal discriminable interval
(which I think would be a reasonable interpretation of a 3 db
difference, even though energy really doesn’t apply here).
If a
carpenter could measure distances only within 3 db, the house he is
building would be a random pile of sticks. Nobody would survive driving
a
car.
Oh, what ARE you talking about? What is the ideal for the carpenter
situation? What is it for the car driver? To reword those two
questions: What is the best that the best possible machine could do in
their situation? Then, “Do they come within being half as precise as
that ideal (3 db)?”
This is another way to see my reluctance to adopt your statistical
approach to perception. I think you are considering only a small part
of
the whole dynamic range of perception;
You seem to want to insist on this, for reasons unclear to me, rather
in the way Rick wants to insist that I want to extract individual
control parameters from group survey data. It’s as though for you there
is some threshold. A quantity less than epsilon is identically zero, it
seems, where epsilon is defined in whatever way suits your whim of the
moment. I don’t think of zero that way. The analyses work without
artificial thresholds. The interesting question is whether people work
with thresholds.
Anyway, the only claim I make is that no biological or mechanical
entity can outperform a properly conceived ideal mechanism, and that in
the situations where comparisons have been possible, people’s sensory
systems seem usually to mimic the behaviour of the ideal, but
consistently fall a little short – how short depends on training in
the situation.
···
As an aside, you might like the story of a couple of experiments done
by W. P. Tanner (Spike) in the late 50s when he was introducing signal
detection theory to psychologists, from radar engineering. In one
experiment, he was measuring frequency discrimination, simply whether a
tone moved up or down in pitch (so I imagine he had computed the ideal
observer for this condition). All his subjects except two learned to do
this quite well in one or two sessions. He assumed the two were not
tome deaf, because their native language was a tonal one, in which
pitch shifts made differences in meaning. They could not converse
properly in their native language without controlling their perceived
pitch shifts. Spike therefore assumed that their sensory systems could
discriminate up from down, but at some level they were not interpreting
it the same way when the tone shifts were divorced from voice. He kept
them in the experiment for many days (always letting them know after
each response what was the correct answer). Eventually, there came a
day for each of them when they cottoned on to what they were supposed
to be listening for, at which point their answers shifted from pure
chance to 100% correct within minutes. You have to learn to perceive
something consciously, even if you normally control it quite precisely.
The second experiment has something of the same flavour. In this one.
Tanner designed a binaural sound presentation that was unlike anything
anyone ever heard in the natural world. In each ear he played a ramp
tone lasting (I think) 100 msec – in other words, a “bip”. In one ear
the ramp rose, and in the other it dropped, so that the total energy
level was constant throughout the duration of the bip, but the apparent
placement swept from one side to the other. In this blip he placed a
short pulse. during this pulse, the energy level dropped to zero in one
ear, but went up to the same total energy level in the other ear. The
pulse might come early or late in the “bip”. Nobody could discriminate
whether it came early or late when they started the experiment, and
most people spent many days before they were able to discriminate the
difference. When they did, their scores went from chance to 100% over a
couple of sessions, and the difference between early and late was a
simple obvious perception. One subject took 44 days before this
happened, but it did happen for each of his subjects.
I draw no moral from this, but I thought it might interest you because
of the way people went from being quite unable to discriminate
differences that in the one experiment they must have been using and in
the other they had never heard before, to being able to discriminate
those differences as easily as discriminating red from blue.
The second episode of the Bayesian discussion has been a bit delayed
because I want to include some computed curves. I had bought a program
called iMathGeo to do this, but it had a bug, which the author has
apparently fixed this morning.
Martin
