Concerning multiplexing

[From Bill Powers (960515.0500 MDT)]

I am very tardy in acknowledging and passing on some shocking news that
Gary Cziko told me about a week ago: Donald T. Campbell has died.

Those who have read the dedication of my '73 book know that Don Campbell
was a primary source of encouragement and practical help to me. He
persisted until he got me to write and finish the book and he helped me
find a publisher for it. He was a true scientist. He did not care what
you thought or said or what your status in life was; if you were doing
science he granted you the unconditional right to be heard and
respected, quite independently of what he himself believed. I have never
known anyone with such a clear vision of the great art through which
humanity explores the world. I am acutely sad about his sudden and
permanent absence.

Peter Cariani [960514.2000) --

Your reply to Ellery Lanier was helpful to me, too. It reminded me of
one aspect of your approach that had slipped my mind, your concept of
"multiplexing." I can see why this concept would be useful in building a
model of the type you want to build, but of course that sort of goal-
driven postulate is to be considered most carefully; one has to ask just
how badly one wants a model that is heterarchical, anarchic, etc. -- and
why.

For some reason your explanation, this time, helped me to see a way in
which our concepts are basically similar (despite the above). You are
seeing multiplexing as analogous to time-sharing or parallel frequency
channels, in which a single signal can carry multiple channels of
information. I would like to suggest that this is an unnecessarily
literal transfer of technology from electronics to neuroscience.
However, we are talking about the same phenomena and in some ways
proposing similar ways for the nervous system to deal with it.

Your description makes it seem as though there is some mechanism which
systematically creates independent codings of various aspects of the
basic input data and superimposes them in a temporal pattern of some
sort. That is what a multiplexer would do. The basic idea of
multiplexing assumes that the information to be sent in the multiplexed
channels is separate to begin with, and is then combined in some orderly
way that provides for later sorting out of the various messages when
they reach their destinations. At the destinations, presumably, the
appropriate inverse operations would be carried out, to demultiplex the
combined messages as appropriate to their various destinations.

It is, of course, quite possible to design a system like this. But to
design it, you have to set up a specific multiplexing-demultiplexing
algorithm, and a coding system, so the compression processes at the
source correspond properly to the decompression processes at the
destination(s) and the decoding corresponds to the coding. In other
words, there has to be a kind of master design behind the whole process,
which is set up to assure that it will work. And you have to start with
a set of separate messages.

Suppose, however, that there are no separate messages in the environment
waiting to be transmitted into the nervous system: no frequencies, no
pitches, no timbres, no phonemes, and so forth. Suppose there is simply
a total physical process going on, which is sampled by the sensory
receptors of an organism. What we end up with is a set of input signals
that, just as you suppose, carry all the information that the organism
can obtain about the world outside it. But there is no multiplexer,
because there are no separate messages already sorted out in the
environment and waiting to be transmitted to destinations in the brain.

Instead, what we have is a set of neural signals, emitted by sensors, in
which there is an enormous potential for finding order. That there is
order to be found is a basic postulate of science, but the order does
not spontaneously reveal itself. It must be constructed and extracted.

It's instructive to compare systems with different capacities for
finding order. When I say the word "telescope," my ears and my cat's
ears both generate a fairly similar set of neural signals. If we
examined those signals immediately after they are generated, we would
find complex waveforms. But somehow I hear the word "telescope," and my
cat does not (I hope that my point will not be obscured by factual
quibbles about what cats actually hear).

The point is that in the basic neural signals is all the information
that any nervous system could possibly extract, even a nervous system
far more competent than the human one. But we do not extract all that
information, nor do we all extract the _same_ information. What we
extract depends, to be sure, on what is there, but it depends even more
critically on the mechanisms we possess for extraction.

In my model, perceptual signals are generated by computing functions
that receive sets of input signals and produce output signals that vary
as a function of the input variations. The meaning of the output signals
is determined by the form of the computing function. For example, a
computing function that sums its inputs create a signal that is
invariant with respect to the amplitude of the individual inputs, as
long as those inputs vary in a way that leave their sum constant. A
computing function that is tuned to respond maximally to variations in
its input at a specific frequency creates an output that is invariant
with respect to which inputs are varying, as long as the result is a net
variation at the specific frequency. This could also be viewed as a
function that reports the presence of a predominant interval between
input events.

I see I'm going to have some trouble with this next part, but here goes:

Now, what is the difference between saying that the input signals
contain multiplexed channels carrying pitch, intensity, and timbre
information which is then demultiplexed by a receiving system, and
saying that there is an input function which computes a perceptual
signal representing pitch, intensity, or timbre on the basis of the
input signal? There are some very sharp differences, but in either case
we are saying that a perceptual signal results from extracting a certain
kind of information from the input data.

In the Cariani system, the same basic input data, multiplexed, can be
passed to any systems in the brain, so that many systems could
demultiplex the same subchannel, for different purposes. Is this
fundamentally different from saying that the first system which
constructs a perceptual signal out of the basic input also passes that
signal on to higher systems? There is a practical difference, of course:
in my system, the "demultiplexing" is done only once, with the resulting
signal being available as inputs to other systems. But in terms of
information available to higher systems, there is no difference. The
same information gets to the higher systems.

The basic difference between our schemes is that I do not assume pre-
existing information channels in the signals reaching any system. That
is, I do not assume that the world as represented at any level was
sorted out into "messages" before entering the nervous system. I do not
assume any _given_ partitioning of the external world into familiar
concepts. The multiplexing concept assumes that there are specific
identifiable kind of information already sorted into the various
subchannels, so that the partitioning is objectively established prior
to perception. I think that some strong arguments can be mounted against
this idea.

I see that I'm going to have to try again later. There's a very clear
idea lurking in here somewhere, having to do with alternate ways of
interpreting the same input information, and with the fact that
different kinds of information are the business of different levels of
interpretation. But perhaps if the idea were really clear, I would be
able to state it. This calls for a little nap to finish out the night.

ยทยทยท

----------------------------------------------------------------------
Best to all,

Bill P.

[From Peter Cariani (960515.1100)]

[From Bill Powers (960515.0500 MDT)]
Your reply to Ellery Lanier was helpful to me, too. It reminded me of
one aspect of your approach that had slipped my mind, your concept of
"multiplexing." I can see why this concept would be useful in building a
model of the type you want to build, but of course that sort of goal-
driven postulate is to be considered most carefully; one has to ask just
how badly one wants a model that is heterarchical, anarchic, etc. -- and
why.

For some reason your explanation, this time, helped me to see a way in
which our concepts are basically similar (despite the above). You are
seeing multiplexing as analogous to time-sharing or parallel frequency
channels, in which a single signal can carry multiple channels of
information. I would like to suggest that this is an unnecessarily
literal transfer of technology from electronics to neuroscience.
However, we are talking about the same phenomena and in some ways
proposing similar ways for the nervous system to deal with it.

I'm glad it made some sense to people. The ideas really didn't come
from electronics (although I find it's easiest to explain them that
way -- radio is familiar enough to all of us), but from theoretical
biology. Among other things, my doctoral work concerned problems
of sensory evolution and the emergence of new perceptual primitives.
Here an evolving population of organisms can evolve sensors that
make new distinctions on the environment that are not just logical
combinations of pre--existing ones (I was/am concerned with the
problem of "closed world assumptions" in conventional AI systems --
how can these systems be "creative" in a fundamental sense if they
can only operate within fixed, finite notational systems that we
provide them?) If one has the means of evolving new sensors and
also of evolving new effectors, then it is possible for a population
of these adaptive elements to evolve new means of inter-element
signalling that are orthogonal to pre-existing ones. This allows
for the effective dimensionality of the signalling system (# independent
signals) to increase over time. I couldn't find anyone interested in
working on robotic devices with adaptively constructed sensors or
effectors, nor could I find anyone interested in making human sensory
prostheses adaptive (let the user search the parameter space). I
was thinking about what this kind of "emergent signalling" would
mean for neural networks, and had heard about the notion of temporal coding
(I was a student of Jerry Lettvin as a freshman at MIT), so I began to
think about the consequences of these ideas for the brain. I was
extremely lucky to come in contact with Nelson Kiang and Bertrand
Delgutte, who are auditory neurophysiologists with deep, long-term
interests in the nature of auditory neural representations and the
relations between them (e.g. rate-place vs. temporal codes). My ideas
about temporal patterns in spike trains and periodicity analysis
have mainly developed through the recording and analysis of spike
trains (making various kinds of histograms that tell you different
kinds of things about what is going on), so my understanding of
neural signals has come more from analysis in the time domain and
in temporal correlation than it has from operating in the frequency domain.
This is both an advantage (I see things that nobody who was trained
as a signal-processing engineer would ever in their right mind
think of looking for) and a disadvantage (it can be very hard
to understand the conventional assumptions if you have never
held them, so communication can be difficult).
So, no, the ideas really haven't come from electronics
(if anything, they've come from a very small and obscure nook of
theoretical biology that is concerned with epistemology and emergence
and what "information" and "codes" mean in biological contexts).
At the same time, I always try to ground whatever hypothesis I
have in the physiology, and I'm always on the watch for hard,
compelling evidence that would force a change in my mental models.
As much as from anywhere else, these ideas have been heavily shaped
by what I've seen in the auditory system -- for example, multiplexing
of periodicities is everywhere in the auditory nerve once one thinks
to look at the spike time patterns as signal vehicles.
One has to be imaginative, to think about what could be, and
at the same time it is essential to be very "critical",
to be willing to rule things out. One also must be always aware
of the pervasive blinders that hinder much of neuroscience,
and the limitations of published physiological data and the conclusions
that are drawn from them.

Your description makes it seem as though there is some mechanism which
systematically creates independent codings of various aspects of the
basic input data and superimposes them in a temporal pattern of some
sort. That is what a multiplexer would do. The basic idea of
multiplexing assumes that the information to be sent in the multiplexed
channels is separate to begin with, and is then combined in some orderly
way that provides for later sorting out of the various messages when
they reach their destinations. At the destinations, presumably, the
appropriate inverse operations would be carried out, to demultiplex the
combined messages as appropriate to their various destinations.

I think in each sensory system information pertaining to that modality is
combined in some way or another. Temporal multiplexing is automatically
set up by either 1) arrays of receptors that follow the temporal
structure of the signal, e.g. cochlear hair cells, mechanical receptors
in the skin) or 2) early lateral inhibition systems that set up
particular time patterns/sequences of disharges depending upon the
stimulus. These lateral inhibition systems have a common organization
(e.g. olfactory bulb, retina; Sventogothai, Shepard), and one sees them
for stimuli that does not have temporal structure that the receptors
can follow in its fine structure (e.g. chemical stimuli, light). Even
in vision, if you drift an image across a retina, the retinal ganglion
cells (and many cells in primary visual cortex) will "lock" to the
temporal structure of the contrast gradients (edges) as they are
presented at their corresponding place at the retina. So, the idea
of temporal multiplexing in sensory systems is very natural if one
thinks of the time patterns as a very phylogenetically-primitive
way of encoding stimuli.

It is, of course, quite possible to design a system like this. But to
design it, you have to set up a specific multiplexing-demultiplexing
algorithm, and a coding system, so the compression processes at the
source correspond properly to the decompression processes at the
destination(s) and the decoding corresponds to the coding. In other
words, there has to be a kind of master design behind the whole process,
which is set up to assure that it will work. And you have to start with
a set of separate messages.

In the auditory nerve, every stimulus component below 5kHz produces its
own periodicities in the neural output, so these do start out as "separate
signals". I have the feeling that it's much easier to evolve a means of
recognizing perceptual forms if the forms themselves are encoded one
way (temporal patterns) and the intensities of various parts of the
stimuli are encoded in another way (e.g. by rates or by variance of
latencies, or by variances of temporal patterns). This is an open
question. I've thought much more about temporal codes and about
possible neural architectures for processing them than the vast
majority of computational neuroscientists, so the possibilities are
much clearer in my mind, and the solutions seem much "simpler" than
they would have seemed to me a few years ago. One can also argue the
other way, that many, many stimulus properties affect the discharge rate
of each neuron, so that different kinds of stimulus information
(e.g. frequency, intensity, location in auditory space, amplitude
dynamics, etc.) are in effect "multiplexed" in the firing rate of each neuron
(this is what we observe in the auditory system), and that the higher
centers face a very complicated (and perhaps impossible) task of
simultanously disambiguating all of this multiplexed information. Add to
this multiple perceptual objects (sounds, visual forms) and the problem
gets much, much harder (how does one decide which neural rates
should be included with each object?)

Instead, what we have is a set of neural signals, emitted by sensors, in
which there is an enormous potential for finding order. That there is
order to be found is a basic postulate of science, but the order does
not spontaneously reveal itself. It must be constructed and extracted.

I'm reluctant to always be "passing the buck to higher centers" that
are capable of arbitrarily-complex pattern recognitions. I can see simple
ways of using time-structure to obviate the need to do these
complicated operations (interval representations for pitch do this
and there are simple ways of doing "auditory scene analysis" in the
time domain). This simplifies the problems that higher centers face,
and I think it brings the problem of form perception back into the
realm of tractability. There may be very elegant ways of extracting
perceptual order in the time domain (we need to investigate them).

In my model, perceptual signals are generated by computing functions
that receive sets of input signals and produce output signals that vary
as a function of the input variations. The meaning of the output signals
is determined by the form of the computing function. For example, a
computing function that sums its inputs create a signal that is
invariant with respect to the amplitude of the individual inputs, as
long as those inputs vary in a way that leave their sum constant. A
computing function that is tuned to respond maximally to variations in
its input at a specific frequency creates an output that is invariant
with respect to which inputs are varying, as long as the result is a net
variation at the specific frequency. This could also be viewed as a
function that reports the presence of a predominant interval between
input events.

I agree.

I see I'm going to have some trouble with this next part, but here goes:
Now, what is the difference between saying that the input signals
contain multiplexed channels carrying pitch, intensity, and timbre
information which is then demultiplexed by a receiving system, and
saying that there is an input function which computes a perceptual
signal representing pitch, intensity, or timbre on the basis of the
input signal? There are some very sharp differences, but in either case
we are saying that a perceptual signal results from extracting a certain
kind of information from the input data.

Yes. (Just in terms of the language, the input signals carry information
that "subserves pitch perception and discrimination", or they carry
"pitch-related" information, but pitch, timbre, and loudness are
not properties of the signals per se, but are perceptual properties of
the perceiver. The percepts are informational distinctions, discriminations
that the perceiver makes. For this reason I don't talk in terms of
"perceptual signals", but rather signals underlying a particular percept.
But I know what you mean.)

In the Cariani system, the same basic input data, multiplexed, can be
passed to any systems in the brain, so that many systems could
demultiplex the same subchannel, for different purposes. Is this
fundamentally different from saying that the first system which
constructs a perceptual signal out of the basic input also passes that
signal on to higher systems? There is a practical difference, of course:
in my system, the "demultiplexing" is done only once, with the resulting
signal being available as inputs to other systems. But in terms of
information available to higher systems, there is no difference. The
same information gets to the higher systems.

It is similar to your system, except that all subsequent recipients have
access to <all> aspects of the signal. This means that a lower level
processor need not determine in advance what might be relevant to a higher level
one.

The basic difference between our schemes is that I do not assume pre-
existing information channels in the signals reaching any system. That
is, I do not assume that the world as represented at any level was
sorted out into "messages" before entering the nervous system.

I don't assume that either. Why would you think this? I use the term
"message" to mean a signal that has some semantic content (it's linked
either to the state-of-the-world or to the state of some other
neuron) and some pragmatic consequence for the system.

I do not
assume any _given_ partitioning of the external world into familiar
concepts. The multiplexing concept assumes that there are specific
identifiable kind of information already sorted into the various
subchannels, so that the partitioning is objectively established prior
to perception. I think that some strong arguments can be mounted against
this idea.

I don't assume that either. Why would you think this? No partitioning of
the world exists prior to perception (except by some other external observer).
"Multiplexing" entails nothing of the sort; it just means that different
aspects of the external world can be embedded in different aspects of
the neural signals that are caused by the interaction of receptors with
that world. Nothing more, nothing less.

Peter Cariani