Modeling and higher levels

[Michael_Fehling]

[Michael Fehling 930914 10:28 AM PDT]

In re Bill Powers 930914.0830 MDT --

Bill,

A couple of quick remarks about modeling after your last message.

  Behaviorists (and other logical positivists in psychology) established the
prejudice that, first, we collect data, and then we build models to explain
it. In fact, it can't go that way. We first need som idea of _what_data_to_
_collect_. That, in my work, is the first and foremost reason for building a
model. It forces us to be _rigorous_ rather than _metaphorical_ about our
theoretical claims.

  For those of us who study living systems, another aspect of this deserves
mention. A model helps the experimentalist cope with formal complexity.
E.g., PCT is really a very simple thesis. That is what gives it its extreme
elegance. The problem is in really trusting to our intuitions about its
implications in real-world living systems. These implications involve the
interactions among vast numbers and kinds of PCT components. The facts and
subleties of complexity (cf., ideas from the Santa Fe institute) readily
defeat our ability to intuit the theory's higher level claims. Building a
model to rigorously explore higher-order interactions implied by a theory is
essential.

  On this last note, let me remind you of a bit of history. Back in the math
modeling days, the "one-element" model was used to predict some features of
verbal learning experiments. However, the real contribution of this model
was not in the first- or second-order facts that it described. (These, after
all, were the datat that the modeler's were driven by in formulation.) In
fact, the model was sufficiently rigorous that one could derive predictions
about relatively complex _contingent_sequence_effects_ that, prior to model
analysis, no-one had thought to look for. They wre found.

  If you think about it, I'm describing how models get used in well
established areas of the physical sciences. There is always a tension between
the modelers who are saying "my model says you should look over here" and
experimentalists who say "be sure your modeling assumptions make sense." By
the way, I occasionally teach a course on history of science. "In the good
old days of 19th century physics," the steps being taken by modelers were, in
fact, quite _large_. The aforementioned tension was just as real then as it
is today.

  As for your "anchor points in observation" of higher level phenomena, I am
suggesting the competence methodology as playing that role. In particular,
competencies give _invariants_ of higher order behavior that are approximated
in performance. (I hope you'll try chewing on this idea a bit. It's a theme
I've been introducing in my recent messages.) So, we have empirical guidance
for higher level modeling.

  These are the more important reasons that I suggested building a PCT version
of a UTC. It's not because I think anyone already has the answers. Rather,
it's because we need to do this to ask the questions. I know that your
remarks weren't aimed at me. But, you've given me an opportunity to expose
another behaviorist fallacy--the Popperian fallacy that the world will tell us
before we make any contribution of our own. In fact, doesn't this violate
PCT? I urge that we think of modeling as a device for exploring the
implications of our theories (as best we know those theories at a given time).

  Finally, in re wheels again, I fully understand the role of a static
analysis of control. But, as you showed in BCP, it's necessary but not
sufficient. Part of the job of control organization must inevitably be in
getting the time constants right. Dynamics must be modeld to understand this.
Statics _assumes_ stability. Reorganization must _achieve_ it.

- michael -

[Marken_Richard_S1]

[From Rick Marken (930914.1230)]

Michael Fehling (930914 10:28 AM PDT) --

In re Bill Powers 930914.0830 MDT --

We first need som idea of _what_data_to_
_collect_. That, in my work, is the first and foremost reason for building a
model. It forces us to be _rigorous_ rather than _metaphorical_ about our
theoretical claims.

So after all this modelling, you must know what data to collect, right?
Could you tell us what it is?

A model helps the experimentalist cope with formal complexity.
E.g., PCT is really a very simple thesis. That is what gives it its extreme
elegance. The problem is in really trusting to our intuitions about its
implications in real-world living systems.

Why trust to intuitions? Why not fire up the model and see how its behavior
compares to that of the real thing -- a living system?

Building a
model to rigorously explore higher-order interactions implied by a theory is
essential.

In fact, the model was sufficiently rigorous that one could derive

predictions

about relatively complex _contingent_sequence_effects_ that, prior to model
analysis, no-one had thought to look for. They wre found.

They were found by looking at behavior, no?

As for your "anchor points in observation" of higher level phenomena, I am
suggesting the competence methodology as playing that role.

The only problem with your "competence methodology" as an approach to
providing "anchor points in observation" is that it seems to be rather
non-observational. The methodology (as described in your book review,
p.323) is represented by the following set of questions:

1. What does the possessor of a competence need to accomplish to
have successfully completed the task?
2. What are the varying conditions under which the problem solver must
possess and exercise the competence?
3. What are the essential cognitive processes that are required?
4. How are the processes organized to cope with i) varations in
task requirements or ii) task conditions.

Let's look at the application of these questions to a commonly demonstrated
competence -- the ability to drive to work. We know (without looking
at any driver) what the possessor of this competence needs to accomplish
in order to be considered successful (the driver must end up with the car
at work). So Step 1 of the methodology is accomplished without any
observation at all.

We can list some of the varying conditions under which the driver must
exercise this competence -- variations in tire pressure, amount of gas
in the tank, road conditions, etc. Again, this Step (2) can be done
without ever observing a driver. Observation won't help much because we
could never observe all the conditions with which a driver might
have to deal (who would have imagined that I'd have to deal with a
closed 405 due to an "urban unrest" condition).

I have no idea what observation is implied by determining "the essential
cognitive processes" required (Step 3). It seems like this could be
determined by trying to model what the driver actually does. If so, then
step 3 does imply that we should actually observe the behavior of a driver
-- but we are not told what to observe. After observing whatever it is
we observe then we can build a model that behaves like the driver. Then
we keep removing components from the model until the model no longer
behaves like the driver. Just before the model stops acting like the
driver it contains (one set of) "the essential cognitive processes"
required to drive to work.

Step 4 requires completion of (my version of) step 3 (observation and
modelling) -- once you have a model driver you can look at the model
and see how its organization makes it able to cope with variations in
task requirements (the feedback funciton in PCT) and conditions
(disturbances in PCT). But, again, this step has no specific requirements
for observing what a person does in order to drive to work.

In fact, all the steps of the "competence methodology" can be carried out
in the comfort of one's arm chair (especially if that arm chair is
sitting near a desk top computer).

I think that what Bill Powers means by "anchor points in observation" is
observations of a competence as it is carried out -- with the observations
informed, of course, by the PCT model of behavior. The PCT model does
influence what we take to be relevent "observations". The relevent
aspects of competence to be observed are controlled variables. The
"anchor points in observation" to be obtained from the driver is
evidence of the variables the driver is controlling, and the re-
lationship between these variables. The methodology for making these
observations is called "the test for controlled variables". The test
for controlled variables has not been applied much to competence at
controlling higher level variables. Application of the test to higher
level variables is not that easy -- partly becuase the variables being
controlled are difficult to quantify. But it can be done.

Bill Powers has just had a paper published on the control of
higher-order (cognitive) variables (could you post the reference
Bill?). Also, I have started some very simple studies which
involve control of fairly high order variables -- sequences and
programs. My studies show that these variables can be controlled
and that the ability to control such variables depends on a person's
ability to perceive (not produce) them.

So we are starting to creep up (tortoise-like) to the study of
"higher level" variables -- but we (I, anyway) want to be sure
that we have good observations (especially because the models
of these levels will depend very much on words representing the
outputs of perceptual functions -- since we cannot yet build a
perceptual function that produces an output that indicates, say, the
degree to which a particular program is occuring). I think its too
easy to build models of higher level functions that seem to match
observations simply because the observations and the model are so
course at this level. For example, it's not hard to get a model
to solve math problems -- producing the same answer as that observed
in people -- even though the model is generating outputs in response
to inputs while the subject is actually controlling inputs. Bill
Powers' paper on control of cognitive variables shows one way to
make observations of "cognitive competence" in a way that
discriminates output generation from input control.

Happy observing.

Rick

[Michael_Fehling]

[Michael Fehling 930914 2:27 PM PDT]

In re Rick Marken (930914.1230 --

RIck,

You may have lost the context of the exchange that you, Bill, and I were
having when you ask, "... after all this modelling, you must know what data to
collect, right? Could you tell us what it is?" I was defending why we need
to build a PCT version of a UTC. SInce that hasn't been done, your question
must go unanswered.

  Since my own modeling work hasn't been grounded in PCT, my answers are of
little consequence to you. However, let me offer you one for which I've
achieved a bit of notoriety. In my agent model behavior is the result of a
tradeoff between pure reaction (based on prior experience) and deliberation
(e.g., formulating a new course of action, deciding among multiple
pre-existing candidate courses, etc.). This higher-level problem of balancing
reaction and deliberation puts my model between the extremes of traditional
logic-based systems (that always seem to deliberate, even when their is no
time) and more recent reactive approaches (e.g., Agre and Chapman). Two
things are clear in understanding how people intelligently manage to achieve a
balance--(a) they rely upon hueristics that are rationally built up from prior
experience, and (b) no matter what else is happening, the agent must be
_interruptable_ and have already learned capacities for servicing these
critical events. This model has identified some empirical issues. Regarding
(a), we need to learn more about how problem-solvers actually do build up
context-sensitive rules to frame their problems and otherwise manage their
problem-solving efforts. The work of John Payne comes closest to experimental
data of the kind that our model calls for. Regarding (b), we need empirical
data that tells us how and unde what conditions problem-solvers switch
attention (and then revert back again later) when the "interrupting" task has
been addressed. The classical work on selective attention is inadequate since
it addresses mostly switching. Of course, this is only a sketch of the more
detailed version of my response to your challenge.

  My comment about intuitions was poorly worded, so it misled you. I wanted
to say that _unless_ we use a (cognitive simulation) to explore PCT's
implications we are stuck with relying upon intuitions that may be very
mistaken regarding the implications of the theories basic principles. So,
indeed I am calling for building a PCT-UTC and so we can then "fire up the
model and see how its behavior compares to that of the real thing -- a living
system?"

  You respond to my example of the one-element model's "discovery" of sequence
effects with a rhetorical question: "They were found by looking at behavior,
no?" You seem to miss my point. Of course the sequence effects were
experimentally investigated. In fact, they were always present, sometimes
even recorded in the data of experiments that were conducted before the
one-element model was developed by Bower, Trabasso, and others. However,
until the model had been so formulated, no-one had the good sense to look for
these effects, let alone to understand their significance. A model's ability
to help define the research agenda is one of its major contributions when
carefully used.

  I hope that you will rethink your remarks about determining the competence
for driving to work without observation. Certainly, some aspects of the
analysis may not involve observations of the driver (arguably step 1) but they
do require observations, we call these _task_analysis_. However, from here on
out, you must either collect some data or make (as you seem to have done)
sweeping assumptions about the abilities (perceptual and otherwise) of the
agent who will do the driving. E.g., even in step 2, "the "varying conditions
about driving" must, in general, include soem idea of the other things that
the driver might _want_ to be doing at the same time and (as per step 3) her
ability to do them. Etc.

  Besides, so what if you don't buy my claims that the steps entail
commitments to facts about human capabilities? Then it's all task analysis.
This is still an anchor. Understanding what (perceived) task requirements are
operative provide a tremendous framework in terms of which to ask useful
questions about human performance. A clear example is the work of Tversky,
Kahneman, etc. on the so-called "cognitive illusions" in decision making.

  Allow me once more to emphasize The bottom line: I've already encountered a
large quantity of very interesting but unfounded speculation about the
implications of PCT. For the most part, my intuitions align with yours,
Bill's, and other who have so speculated. But, the model is interesting
precisely because it seems to have such implications. Part of the job in
checking them out is empirical, but an equally crucial part is the modelers'
responsibility to rigorously and unambiguously clarify what the theory
stipulates from what it does not. So, one aspect of modeling is its value in
adding clarity to a theory. (E.g., consider how much a UTC of PCT would help
us to clarify Bill's wonderful remarks about conflict in BCP. Until this done
his ideas will be classiified as "mere speculation." I think they deserive
more attention than that.)

  Moreover, as per my previous message, one aspect of the cognitive revolution
to which I thoroughly subscribe is its rejection of the positivist idea that
the world will reveal itself to you. There's no free lunch in science,
either. If you don't build an empirical study around carefully crafted
questions, you won't get useful results. The literature on S-R studies of
verbal learning is a clear illustration of what happens when this maxim is
violated. So, a model's second main advantage is as a thoughtful, hard-headed
excercise in _question_asking_. In contrast, you seem to think of a model
largely as a device for question answering.

  Happy modeling.

- michael -

[William_T_Powers2]

[From Bill Powers (930914.0830 MDT)]

Michael Fehling (930913.1039 PDT) --

... if you really want to make progress on the higher levels, I
think you will need to address logical/structural _facts_ of
cogntion/connation/affect, too. The challenge (as I've already
said) is to ground structural in (control) process accounts.

I agree completely. We need experimental data about higher level
processes. Unfortunately, a lot of what we have about
cognition/conation/affect is hypothetical data, not
observational. That is, someone puts together a logical scheme
that seems to be a plausible picture of how the system might
work, but it's a take-it-or-leave-it proposition; either you like
it or you don't. What data there are (not that I know much about
this, I'm just reporting impressions) are in the form of positive
instances of a proposed generalization, without much attention
paid to a search for counterexamples, and with a rather
dismissive attitude toward such counterexamples as are brought up
by critics.

Computer models haven't helped much with this situation. It's too
easy to confuse making a model of an actual existing system with
designing something that will accomplish a particular end.
Sometimes, of course, when you can't think of ANY model that
would do what the real system does, you just have to try to
design one as a starting point. But given a starting point, the
task is then to look for differences between the designed model
and the real system, so you can correct the design to conform
better with the real system. There seem to be lots of people who
fall in love with their designs and start defending them, instead
of testing them to see where they need changes.

You can think of S-R and S-O-R psychology as a prime example of
this. Way back somewhere, somebody had an idea for a model of
behavior in which sensory inputs operate motor outputs. This is a
plausible model -- it just happens to be wrong. If the people who
adopted this model had viewed it with skepticism, they would
quickly have found many circumstances under which it simply
doesn't behave like the real system. In fact they did find many
such circumstances, but instead of going back to the drawing
board they launched a huge effort to defend the model against the
facts. One way was simply to deny the inconvenient facts:
behavior may look purposive, but because the S-R model, of
course, completely explains all observable behavior, purpose is
an unneccessary concept, or an epiphenomenon, or an intervening
variable, and anyway doesn't exist. Another way was to use
statistics. There might be many actual cases in which inputs fail
to predict outputs correctly and in which outputs appear to
change spontaneously, but if you average a lot of cases together
you can hide the counterexamples and average the spontaneous
behaviors to zero or at least a constant term.

In modern times, the equivalent, I think, is found in
mathematical models. It's much too easy to get fascinated with a
beautiful mathematical scheme, to the point where you feel it
just HAS to have something significant to say about nature.
People point to mathematical proofs as if they were equivalent to
experimental proofs. They aren't. All they show is that your
system is SELF-consistent; they say nothing about whether systems
of that kind actually exist in nature. I think that a lot of
scientific approaches have worked themselves far out onto
mathematical limbs, getting farther and farther from supporting
data and paying attention only to whether the tip of the limb is
in the right place. They forget that there are many other ways of
getting to the same place: you can't define a curve with one
point.

In the good old days of 19th century physics, the steps taken by
theorists were very small. Every premise, before being adopted
into the overall model, had to be matched up with an experimental
observation. And every deduction also had to be matched with
observation -- there was none of this spinning out of
mathematical derivations to build long shaky bridges arching over
the void from one small isolated fact to another small isolated
fact. The result was that you could point to any term or constant
in a mathematical representation, and ask "What does THAT mean?"
and be shown the experimental data to which that term or constant
referred.

In the PCT model there are no elements that are assumed merely to
get from one step of the reasoning to the next. In the equations
we use for our simple models, every variable and constant and
every function is intended to correspond to an actual physical
process, something at least in principle observable and
measurable. There is at least some sort of experimental evidence
for every element of the model and every part of the system
equations. What's holding up investigations of higher-level
control processes is the lack of anchor points in observation. We
have to study what people DO at these higher levels before we can
even begin to set up models like those we have for the lower
systems.

The best data we have about higher level processes may be in the
papers people write about them. If you just ignore what the paper
says and look at it as an example of a human brain carrying out
certain operations, you might be able to find evidence for the
real cognitive processes that are going on. The real ones are not
in what the paper says (which is just a product of cognitive
processes), but in the processes that had to occur in order for
the paper to have been written. If you see a mathematical
derivation, don't ask if it's right or wrong (it's probably
mathematically right if it got published); ask what a person has
to do in order to produce such a thing. This is the track that
Newell, Simon, and Shaw got onto back in the early days, and for
a while they got somewhere. I don't know what has happened to
that naturalistic approach since then; it seems to have more or
less disappeared.

Not much of this was aimed at you, Michael; your post just
touched on things I've been gradually working out over a long
time, and when the plug was pulled, out it came.

···

-----------------------

I've been doing a little test of PCT claims with some of the
psychologists here at Stanford. Actually, they are very
comfortable with the idea that the causal role must be the
_interaction_ of disturbances, purposes, or current
perceptions. The problem is _not_ in the theoretical ideas
that psychologists have. After all, most of them are just as
anxious to correct S-R behaviorisms faults as you are. Rather,
the problem lies, as I said about Newell, in their inability to
see that, if the theories need to be entirely different, then
the empirical methods must change, too.

Interesting. We might find it very instructive to see how you
presented these ideas to them. Perhaps we've overlooked some very
powerful line of argument. You might find it instructive, too, to
see how you will present them a year or two from now.
----------------------------
My point about the wheel example was not intended to make the
discussion veer off into what wheels can be used for. It was to
point out that all functions in a control loop are in operation
all of the time, converting their inputs into outputs
continuously and concurrently. Outputs are changing WHILE error
signals are changing WHILE reference signals and perceptions are
changing, WHILE the actions are affecting the perceptions via the
environment, WHILE disturances are acting on the controlled
variable. This concurrency means that the system behaves in a way
very different from the behavior you would deduce through a
sequential analysis. The concurrency simply can't be understood
by following events around the loop, nor can correct predictions
of behavior be made. The system of equations must be solved
SIMULTANEOUSLY.
---------------------------------------------------------------
Best,

Bill P.