Combined auditory-visual experimentation

CSGnet Archive CSG2004
1

[From Jim Beardsley (2004.12.21.1000 EST -0500)]
The following abstract from a NewScientist feed piqued my interest and now compels me into curiosity:
���
Ears play visual tricks on us

Our ears, it seems, can trick our eyes into seeing things that aren’t really there.

Fumiko Maeda of the California Institute of Technology in Pasadena and her colleagues asked volunteers to watch a shimmying computer image and judge whether the image was moving up or down. The image was accompanied by a tone whose pitch was either ascending or descending. The volunteers were more likely to perceive the image as moving up when they heard the ascending pitch and vice versa, regardless of its actual direction of movement. And when they were made to listen to white noise while viewing the image, their answers followed no pattern at all (Current Biology, vol 14, p R990).
���

This sounds like a topic that might have been covered on CS!
Gnet in some detail already, perhaps even through experimentation by one or more here, even if not pursued with PCT in mind. I searched CSGnet archive and found several loosely related threads but not to the extent I’m curious.

Firstly, if anyone remembers a closely related thread, please refer me.

In any case, my predictable (‘neophytical’) curiosity involves what hypotheses have been and yet could be developed about combined auditory-visual perceptions and how to create useful experiments from which theories of human (or non-human?) control and perception might be refined.

For instance, do I remember Martin and others discussing how the perception of optical illusions can be influenced by introducing an auditory component into the overall perception? In anyone’s “humble opinion”, do existing theories (and model/s?) remain too simplistic to approach such “complex” perceptions with potentially effective experimentation?

Jim !
Beardsley

2

[From Rick Marken (2004.12.21.0930)]

Jim Beardsley (2004.12.21.1000 EST -0500)

The following abstract from a NewScientist feed piqued my interest and now
compels me into curiosity:

���
Ears play visual tricks on us

Our ears, it seems, can trick our eyes into seeing things that aren't really
there.

Fumiko Maeda of the California Institute of Technology in Pasadena and her
colleagues asked volunteers to watch a shimmying computer image and judge
whether the image was moving up or down...

This sounds like a topic that might have been covered on CS! Gnet in some
detail already

Based on the abstract, it sounds like people are asked to control for
matching their responses (up vs down) to the up or down movement of a visual
display while a tone with ascending or descending pitch is played. Since the
visual movement is ambiguous, the subjects apparently control for matching
their response to the visual and/or the auditory event, depending on which
is least ambiguous. Since the least ambiguous is always the auditory event,
the response is usually matched to that. This gives the illusion that the
auditory is influencing the visual when, in fact, there is probably no
effect of one on the other at all.

Firstly, if anyone remembers a closely related thread, please refer me.

Sorry, I don't recall any.

In any case, my predictable ('neophytical') curiosity involves what hypotheses
have been and yet could be developed about combined auditory-visual
perceptions and how to create useful experiments from which theories of human
(or non-human?) control and perception might be refined.

In my "Hierarchical behavior of perception" paper (in _More Mind Readings_
-- surely you have a copy, it's highly recommended by none other than Bruce
Gregory) I discuss experiments that suggest that certain types of
perceptions -- transitions, sequences, programs -- are the same, in terms of
the level in the perceptual hierarchy at which they are controlled --
whether they are auditory or visual. This is consistent with the PCT idea
that perceptions above the level of sensation are modality independent. So a
sequence is a sequence whether the elements are visual - A, B, C, D -- or
auditory (the letters as pronounced verbally) or tactile (a sequence of
presses of four different keys, say).

I think it would be interesting to see whether people can control for
perceptions that are made of elements from multiple modalities. A sequence
with auditory and visual elements, for example. Obviously, people can
perceive and control such multi-modal perceptions. What might be interesting
is to see whether control of, say, a sequence where all the elements are
visual is any better or worse than control of the same sequence where some
elements are auditory and some are visual.

Best

Rick

···

--
Richard S. Marken
MindReadings.com
Home: 310 474 0313
Cell: 310 729 1400

--------------------

This email message is for the sole use of the intended recipient(s) and
may contain privileged information. Any unauthorized review, use,
disclosure or distribution is prohibited. If you are not the intended
recipient, please contact the sender by reply email and destroy all copies
of the original message.

3

[From Bill Powers (2004.12.22.0810 MST)]

Jim Beardsley (2004.12.21.1000 EST -0500)–

Your summary wasn’t clear about whether the visual image was actually
moving up or down, or just jiggling randomly. I assume it was just
jiggling, so without any other bias it wouldn’t seem to be
moving.

The subject of this experiment seems to be “synesthesia,” the
confusion of one sensory modality with another.

We could make a PCT experiment out of this fairly easily. Instead of
having the visual image just jiggle, we could make it actually move
slowly up or down, and tell the subject to stop its motion by using the
mouse. The mouse would alter the speed at which a sound increased or
decreased its pitch. If the rising or falling pitch perception actually
affects the visual perception of motion (fourth order in my scheme), this
should be possible.

To let the experiment go on for a while, the increasing or decreasing
pitch could be in the form of an “endless octave,” a sound that
seems to increase or decrease in pitch continually without getting higher
or lower. This clever demonstration appeared about thirty years ago
(actually, 1975 by Google search)-- I used to have an Endless Octave
vinyl record. Do a Google search on “Endless Octave” to find
discussions of it. For some reason my computer doesn’t play the demo, but
the web pages claim it’s there. You’d want to program a continuous
version of the demo instead of using discrete notes (A, B, C,
etc.).

And of course the same experiment could be tried the other way around –
visual motion affecting auditory perception of pitch.

My experiments with the “waterfall illusion” about 15 years ago
showed that if you spend too much time experiencing an illusion (as while
tweaking a program for measuring it) you stop having it. I think you just
go on reorganizing when presented with perceptual problems until all
contradictions are removed. We normally stop when perceptions are good
enough for practical purposes. The same thing happened to the woman, a
graduate student, for whom I was writing the program, or that’s what I
remember. Dick Robertson, can you recall? She was a student of
yours.

Best.

Bill P.

···

The following abstract from a
NewScientist feed piqued my interest and now compels me into curiosity:
»»»
Ears play visual tricks on us

Our ears, it seems, can trick our eyes into seeing things that aren’t
really there.

Fumiko Maeda of the California Institute of Technology in Pasadena and
her colleagues asked volunteers to watch a shimmying computer image and
judge whether the image was moving up or down. The image was accompanied
by a tone whose pitch was either ascending or descending. The volunteers
were more likely to perceive the image as moving up when they heard the
ascending pitch and vice versa, regardless of its actual direction of
movement. And when they were made to listen to white noise while viewing
the image, their answers followed no pattern at all (Current Biology, vol
14, p R990).

«««

This sounds like a topic that might have been covered on CS! Gnet in some
detail already, perhaps even through experimentation by one or more here,
even if not pursued with PCT in mind. I searched CSGnet archive and found
several loosely related threads but not to the extent I’m
curious.

Firstly, if anyone remembers a closely related thread, please refer me.

In any case, my predictable (‘neophytical’) curiosity involves what
hypotheses have been and yet could be developed about combined
auditory-visual perceptions and how to create useful experiments from
which theories of human (or non-human?) control and perception might be
refined.

For instance, do I remember Martin and others discussing how the
perception of optical illusions can be influenced by introducing an
auditory component into the overall perception? In anyone’s
“humble opinion”, do existing theories (and model/s?) remain
too simplistic to approach such “complex” perceptions with
potentially effective experimentation?

Jim ! Beardsley

4

[From Rick Marken (2004.12.22.0930)]

Bill Powers (2004.12.22.0810 MST)

Jim Beardsley (2004.12.21.1000 EST -0500)--

Your summary wasn't clear about whether the visual image was actually moving
up or down, or just jiggling randomly. I assume it was just jiggling, so
without any other bias it wouldn't seem to be moving.

It was actually two superimposed gratings moving in opposite directions. So
the visual image was completely ambiguous. The subjects were given a "forced
choice" to say up or down movement. So they were forced to make the wrong
reply about the visual image, since "both" was not an option.

We could make a PCT experiment out of this fairly easily. Instead of having
the visual image just jiggle, we could make it actually move slowly up or
down, and tell the subject to stop its motion by using the mouse. The mouse
would alter the speed at which a sound increased or decreased its pitch. If
the rising or falling pitch perception actually affects the visual perception
of motion (fourth order in my scheme), this should be possible.

Excellent idea!!

To let the experiment go on for a while, the increasing or decreasing pitch
could be in the form of an "endless octave," a sound that seems to increase or
decrease in pitch continually without getting higher or lower. This clever
demonstration appeared about thirty years ago (actually, 1975 by Google
search)

Without even looking at Google I can tell you that that illusion was
invented by Roger Shepard at Stanford. I love that illusion; I programmed a
version of it myself once. I think there is a glissando version of it, which
is what you would need for the experiment you suggest.

You'd want to program a continuous version of the demo instead of using
discrete notes (A, B, C, etc.).

Ah. You noticed it, too.

The illusion was called the "auditory barber pole", by the way, because of
the way it was created with the octave harmonics increasing under a spectral
envelope. The visual movement in your experiment could be a visual barber
pole (which is what one of the gratings in the forced choice experiment
looked like, I imagine) with the auditory barber pole played at the same
time.

Best regards

Rick

···

--
Richard S. Marken
MindReadings.com
Home: 310 474 0313
Cell: 310 729 1400

--------------------

This email message is for the sole use of the intended recipient(s) and
may contain privileged information. Any unauthorized review, use,
disclosure or distribution is prohibited. If you are not the intended
recipient, please contact the sender by reply email and destroy all copies
of the original message.

5

[From Bill Powers (2004.12.22.1630 MST)]
Rick Marken (2004.12.22.0930) --

The visual movement in your experiment could be a visual barber
pole (which is what one of the gratings in the forced choice experiment
looked like, I imagine) with the auditory barber pole played at the same
time.

What you mean, "your experiment?" I thought Jim Beardsley was volunteering
to do this one. Or you.

Best,

Bill P.
'

6

[From Rick Marken (2004.12.22.1540)]

Bill Powers (2004.12.22.1630 MST)]

What you mean, "your experiment?" I thought Jim Beardsley was volunteering
to do this one. Or you.

I meant that you thought of it and that, therefore, you should get credit
for the idea when Jim finishes doing it and writing it up;-)

Best

Rick

···

--
Richard S. Marken
MindReadings.com
Home: 310 474 0313
Cell: 310 729 1400

--------------------

This email message is for the sole use of the intended recipient(s) and
may contain privileged information. Any unauthorized review, use,
disclosure or distribution is prohibited. If you are not the intended
recipient, please contact the sender by reply email and destroy all copies
of the original message.

7

[From Jim Beardsley (2004.12.23.1430 EST -0500)]

Rick Marken (2004.12.22.1540) ==

> Bill Powers (2004.12.22.1630 MST)

> What you mean, "your experiment?" I thought Jim Beardsley was
> volunteering to do this one. Or you.

I meant that you thought of it and that, therefore, you should get
credit for the idea when Jim finishes doing it and writing it up :wink:

It's wonderful to see everyones' senses of humor remain in prime condition! :smiley:

If that weren't the case, I'd hate to be compelled to parrot anyone in memorial by resurrecting a certain phrase about "giant leaps"! 8-?

I shooer git a big kick out of Other People's Perceptions though, which often
seems to correlate closely with Other People's Problems.

Someday I will "attempt" legitimate experimental work, but for now my home is overflowing too much with Experiencing Mental Work!

:-jim

8

[Martin Taylor 2004.12.24.14.07]

[From Rick Marken (2004.12.22.0930)]

> Bill Powers (2004.12.22.0810 MST)

.....

To let the experiment go on for a while, the increasing or decreasing pitch
could be in the form of an "endless octave," a sound that seems to
increase or
decrease in pitch continually without getting higher or lower. This clever
demonstration appeared about thirty years ago (actually, 1975 by Google
search)

Without even looking at Google I can tell you that that illusion was
invented by Roger Shepard at Stanford. I love that illusion; I programmed a
version of it myself once. I think there is a glissando version of it, which
is what you would need for the experiment you suggest.

I have in frontof me a vinyl disk (Decca DL 710180) called "The Voice
of the Compute" (ISBM 77-751696, which I guess means it was made in
1977). It's from Max Mathews's group at Bell Labs (as it was then).
One of its tracks is the Shepard octaves in discrete, 12-tone form
(the Escher continuously ascending staircase picture is used to
illustrate it). More interesting is Jean-Claude Risset's "Computer
Suite from 'Little Boy'", one movement of which is called "Fall" and
depicts the dropping of the A-Bomb on Hiroshima. It uses the Sheppard
glissando to represent the ever-falling bomb that never lands, as
dreamed by the pilot.

Just thought you might be interested.

···

----------------------------------------------

More to the point in this thread, I think, is the McGurk effect. If
you look at a display of somebody saying something like, say "mat"
while the audio is actually of the person saying "cat", what you hear
will probably not be "cat". It might be "mat" but more probably it
will be something else, such as "bat" or "pat". Vision changes your
perception of the audio. I don't think you can call that synesthesia,
and I don't know if there are situations in which the reverse is true
-- sound affecting what you see.

The McGurk effect also works when there's no conflict. In noisy
situations, one's perception of speech is much more accurate if one
can see the face of the talker than if one only hears (or only sees,
which isn't much good anyway).

Martin

9

[From Bjorn Simonsen (2004.12.28,11:05 EST)]

From Jim Beardsley (2004.12.21.1000 EST -0500)

In any case, my predictable ('neophytical') curiosity involves what

hypotheses

have been and yet could be developed about combined auditory-visual
perceptions and how to create useful experiments from which theories of
human (or non-human?) control and perception might be refined.

Maybe I am beside what you think upon. But I think your question also has
something with consciousness/awareness to do. And because we don't know what
consciousness/awareness is, it is difficult to explain the promiscuity
between sight and sound. Neither can anybody answer why electrical impulses
apprehend as sight and the exact the same type of impulses also apprehend as
sound another place in the brain.
I have heard a "theory" saying that we learn to discriminate sight and sound
through experiencing the differences. And this "theory" is supported by the
fact that synesthesia occurs most often among children having not learned to
discriminate between senses.
Synesthesia is also triggered off among grown up people with psychotic
disturbances as schizophrenia or if people take hallucinogenic drugs. And if
Fumiko Maeda's experiment confirms synesthesia, it is a common phenomenon.

[Martin Taylor 2004.12.24.14.07]

...Vision changes your perception of the audio. I don't
think you can call that synesthesia, and I don't know
if there are situations in which the reverse is true
-- sound affecting what you see.

If synesthesia is an influence of one sense on another sense, I think we can
call it synesthesia (?). And I think it is not unusual that people
experience different colours when they hear certain tones.
I also know that physiotherapists use music to make more dynamics when they
direct group-training.

Synaesthesia can even occur when one of the senses no longer functions
properly, e.g. a person who can see colours when words are spoken can still
see the colours if he becomes blind in later life.

You mentioned the the Sheppard glissando. I think that is an example where
our imagination (no perception from the environment) is influenced by tones.
When I am looking at a certain movie, it happens that I imagine that a door
will be opened because of certain music.

Bjorn