You guys are too smart for me. I’m trying to understand what you are talking about, though.

## ···

-----Original Message-----

From: Martin Taylor mmt-csg@MMTAYLOR.NET

To: CSGNET@LISTSERV.ILLINOIS.EDU

Sent: Tue, Aug 18, 2009 4:55 pm

Subject: Re: Entropy & Reorganization

[Martin Taylor 2009.08.18.14.41]

(Gavin Ritz,

2009.08.17.15.36NZT)

Hi

there Rick

and Bill and others.

Can

one or all of you

comment on the relationship bet

ween entropy and reorganisation?

Possibly

the relationship

between entropy and limits, and statistical entropy or/and

informational entropy

(if one dares).

I imagine I’m the one who is expected to answer this, but it’s a

question about which I have never thought. It’s certainly an

interesting question and answering it might be worthwhile in developing

ideas about reorganization. But, as you might be aware, my mind is

largely elsewhere than CSGnet and PCT theory at the moment. Maybe next

year.

On the other hand, I will point out (as I do every few years) that the

essence of control is the export of entropy from the controlled

variable to the environment in the face of environmental interactions

that tend to increase the entropy of the controlled variable. In other

words, every control system is a refrigerator.

Thinking about it from

that point of view, and thinking of reorganization as control, with the

perceptual hierarchy as its controlled variable, one is led to expect

that the essence of reorganization is reduction of the entropy of the

perceptual control hierarchy – in other words the imposition of

organization and order such as conflict-reducing modularity.

That’s very vague, but that probably would be the direction I would

start in thinking about addressing your question seriously.

As for the relation between statistical and informational entropy, I

can repost a message first posted in about 1993 and in edited form in

1996. In the light of recent threads, I draw attention to the last

couple of paragraphs (but you have to have read at least some of the

earlier material in order to understand them).

Martin

=======Reposted Message========

Control, Perception, and Entropy–a tutorial

The following takes off from an article in Physics Today, Sept 1993,

pp32-38 “Boltzmann’s entropy and time’s arrow” by Joel L. Lebowitz.

Boltzmann’s view of entropy is one of three classical views, the

others being those of Clausius and of Gibbs. Clausius entropy applies

only to macroscopic systems in equilibrium, and Gibbs entropy to a

statistical ensemble of microstates corresponding to a specific

macrostate, whereas Boltzmann entropy applies to any describable

system. Boltzmann entropy agrees with the other two when they ar

e

applicable, and is therefore a more general concept of entropy than

the others, which I will ignore in the rest of this posting.

I will give an introduction to the Boltzmann view of entropy as I

understand it from the Lebowitz article, and then suggest how this

implies that a control system is a cooling system, and why the energy

received from or supplied to the environment by the perceiving side of

the control system must be small compared to the energy involved in

the disturbance and the output side of the control system. I shall

also argue that there is a natural relationship between Boltzmann

entropy and Shannon information, beyond the formal similarity of the

equations that describe them.

**

Where does the concept of “entropy” apply?**

Entropy is a measure of disorder or lack of structure. Many people

understand that entropy tends to increase over time, but are not aware

that this is true only in a closed system. In an open system that can

import energy from and export energy to its environment, entropy can

increase or decrease, and structure can be maintained indefinitely. An

increase in entropy means that the system becomes less ordered or

structured. Any system has a maximum entropy, beyond which it cannot

increase. In common language, in a system at maximum entropy,

everything is random. Wherever there is structure, entropy is less

than it might be for that system.

Entropy is DEFINED over a DELIMITED system, not a closed one. A closed

system is one in which the elements do not interact with anything

outside the system, whereas a delimited system consists of a well

defined set of elements, which may interact in any way with the world

outside the system provided they do not lose their identity. A

delimited system may be open or closed.

It’s all perception, so the interests of the observer do matter, in

agreement with the comment in a message that induced me to write the

original posting from which this is derived: “To me, order/disorder is

a purely subjective concept.” It is, and moreover, it depends on the

perceptual functions used to observe the system. This is why I

commend the Lebowitz paper as a way of seeing how physical entropy

might relate to communicative information.

Entropy is a measure that has a value at an instant in time, and it

can be defined over limited parts of structures that are intimately

intertwined with other parts of the same structure, such as the

electrons in a metal, or the radiation field in a gas. If there are

two equal-sized defined systems of the same degree of disorder, the

entropy of the two lumped together is twice the entropy of one.

Entropy can decrease only if the system exports some to the world

outside, increasing the ordering of the system at the expense of

disordering the world outside. This can be done only if there is a

non-equilibrium energy flow through the open system under

consideration, from one part of the world outside to a different part

of the world outside. A living control system is just such an open

system, getting its energy ultimately from the sun and depositing its

entropy in the form of less organized waste products. The business of

a living control system is to create and maintain structure–to keep

entropy less than its maximum within the living body–and it can do

this for as long as it can maintain an energy flow through itself.

A living system maintains its entropy at a more or less constant

level, lower than would be the case for its components if it were not

alive. When a living organism dies, its components decay and are

scattered around the world, raising their entropy to a level consistent

with that of the world as a whole.

**A describable system: 2 balls in a box**

A describable system, which any physical systems is, can be described

in terms of a phase space. The dimensions of the phase space are

those variables on which the value makes a difference to the system’s

aspects of interest.

One way of looking at a system constructed of simple elastically

interacting balls (an idealized gas, in other words) is to record the

location and velocity of each ball in space, a 6-dimensional vector.

This 6-vector can be represented by=2

0a point in a 6-D space, which is

the phase space for that ball. For a gas of N balls, the phase space

has 6N dimensions.

If the ball constitutes a “closed system” in the sense of no energy

transfer, the velocity components of this vector will not change as

time goes on, at least until the ball bounces off a wall of the closed

container, if one exists. If there are walls, any such bounces do not

cause the ball to gain or lose energy. All that happens is that the

velocity vectors reverse their sign with respect to the orientation of

the box wall, and the point moves to a new point in the velocity

subspace at the same distance from the origin. The ball’s energy is

determined by its location in the 3-D velocity subspace–in fact by

the ball’s radial distance from the origin in that subspace, which

does not change in the bounce, by definition of “closed system”.

In a gravitational field, or if the ball were magnetic and the box

permeated by a non-uniform magnetic field, the energy would also

depend on the location. For a given energy, the radius of the sphere

in the velocity subspace would depend on the location of the phase

point in the 3-D position subspace. The ball would slow or speed up

depending on where in the box it happened to be. But we will ignore

such complications for the time being.

Trivial, so far.

Now let’s add another ball makin

g a 2-molecule ideal gas. The phase

space now has 12 dimensions. Ignoring position-dependent fields, the

total energy in the system is still represented by the radial distance

of the point from the origin in the (now 6-D) velocity subspace.

What happens when the balls meet? They bounce off each other, losing

no energy overall. Each of the six velocity components changes,

though. The system moves to a new point in the 12-D subspace. When

the balls bounce off each other, the system as a whole is still

closed, and no energy is gained or lost in the collision. The

position of the point in the 6-D velocity subspace is changed to a new

position, at the same distance from the origin. This new position will

not change until the balls bounce off each other or the wall again.

In the bounce, with high likelihood, vector components much larger

than average will be reduced, and components much smaller than average

will be enhanced, so the location of the point in phase space is most

likely to be found not very close to the axes of the velocity

subspace. More typically, the individual velocity vector lengths will

be distributed around some intermediate value.

Can there be order and disorder in such a 2-ball system? It depends

on how you choose to look at the system. There might be something

special about a state in which one ball was stationary at in the&nb

sp;

centre of the box, all the energy being concentrated in the other

ball. In the phase space, such a situation would be represented by a

point that lies in a particular 3-D subspace of the whole 6-D space.

There might be something special about a state in which the two balls

travelled together as a pair, in which case the three velocity vectors

for one ball would be defined by those of the other ball. Measure one,

and you know them both. Or there might be something special about a

state in which the balls exactly mirrored each other’s motion (as they

would if they were equal gravitational masses in outer space). The

particular subspace is defined by the observer, not by the momentary

behaviour of the balls. Each particular tightly defined subspace is

untypical, in Boltzmann’s sense, and in an everyday psychological

sense.

Even though any precise position for the system’s phase point is as

probable as any other position in the phase space, almost all of the

equiprobable points do not lie in or very close to any predefined

position. It is much like a bridge hand in which the deal gives each

player 13 cards of one suit. That hand is no less probable than any

other, but we see it as untypical because we have previously labelled

the cards so that we perceive certain relations to exist among them.

An observer is likely to ponder the possibility that the dealer might

h

ave cheated.

It is more probable that a bridge hand will be “typical” in having

each player receive cards of at least three suits. There are 24

different hands in which the four players all hold 13 cards of one

suit, but many thousands of hands in which each player has cards of at

least three suits. The hand is “typical” because it is a member of the

large class rather than of the small class. For the two-ball system,

it is most probable (typical) that its phase point will be in a region

of the phase space that represents both balls as moving. A state in

which one ball is nearly stopped and the other carries all the energy

can happen, but it is not typical.

Another aspect of typicality in the ideal gas (which refers to the

problem of time-asymmetry) is that if the phase point defined by (L,

V) is typical, then the phase point defined by (L, -V) is also typical

(L and V are the location and velocity vectors defining the position

of the point in the location and velocity subspaces). A typical

situation does not become less typical if all of the balls bounce off

the wall. The same applies if any subset of the V vector components

has a sign reversal.

Replacing the phase point (L, V) by (L, -V) is to reverse time. A

brief snapshot of the gas described by the two different phase points

would show no characteristic difference between them. The differe

nce

is in the detail of which ball or molecule is going in which

direction, but this ordinarily does not matter. What does matter is

that if the gas is in a small atypical region of the phase space,

collisions between the balls are more likely to move the phase point

into a typical region than the reverse. It is more likely that when

bridge hands are shuffled, the deal after a pure- suit deal will be

distributed at least three suits in each hand than that the reverse

will happen.

Similarly, if the gas is in an atypical region of the phase space at

time t, it will most probably be in a more typical region at time

t+delta t, and have come from a less typical region at time t-delta t.

If all the velocity vectors were replaced by their inverses, the gas

would revert to its less typical prior state. This is most improbable,

even in the two-ball “gas.” Although the states (L, V) as observed,

and its mirror image (L, -V) are of identical typicality, the two

microstates are not equally likely to occur in practice. One changes

over time from less typical to more typical states, and the other

becomes less and less typical over time. However, a casual observer

shown a brief snapshot of the two could not tell the difference

between them.

**More complex: 2 balls in a swarm of others**

Notice that none of the above description requires that there be a box

confining the balls.&n

bsp; If there is no box, then there will be only one

bounce of the two balls against each other, or none, and the situation

is less interesting.

Let’s remove the box, but add a lot more balls that individually do

not interest us. Call them “nondescript” because we will not describe

them within the phase space. They may, however, interact with the two

balls that interest us. We will still look only at the original two

balls, and the position of their phase point in their 12-D phase

space. Now the balls can encounter each other OR any of the

nondescript other balls. The phase space description of the two balls

is unchanged, but the behaviour of the phase point is different.

If the two balls bounce off one another, the point moves to another

position at the same distance from the origin in the velocity

subspace. But if they bounce off a nondescript ball, the radius of

the shell in which they live may change. If the nondescript ball was

moving very fast, the total velocity of the interesting ball will

probably be increased in the collision. The phase-point of the 2-ball

system will move further from the origin in the 6-D velocity

subspace. The open 2-ball system will have gained energy.

Conversely, if the nondescript ball happened to be moving very slowly,

the interesting ball is likely to lose speed and the nondescript ball

to gain i

t.

**Temperature**

Temperature is proportional to the energy in a system. In our 2-ball

system, the total energy is equal to the sum of the energies

associated with each of the velocity vectors, or 0.5*sum(vi^2) where*

vi is the velocity on axis i and the balls are assumed to be of unit

mass. The temperature, T, of the system is not affected by how many

balls are in the gas, and so we can write T = kmean(0.5*vi^2).

Another way of saying this is that the energy per degree of freedom in

the system is kT/2. The proportionality constant k is known as

“Boltzmann’s constant.”

In typical regions of the phase space, the velocity vectors are

distributed more or less evenly around some intermediate value, very

few being either very large or very small. When one of the interesting

balls collides with a ball from the environment, and it gains energy

from the collision, this increases the temperature of the system of

interesting balls, and decreases the temperature of the system of

environmental balls. When an interesting ball collides with a slower

environmental ball, conversely, the temperature of the interesting

system is lowered and that of the environment raised.

The radius of the sphere centred on the zero-velocity origin of the

velocity subspace is sqrt(sum(vi^2)), which means that the temperature

of the system is proportional to the square of the length of the

veloc

ity component of the phase vector, divided by the dimensionality

of that part of the phase space.

**

Typical and atypical description states: the measurement of entropy**

Now let’s add a lot of interesting balls, bringing the set to size N.

The phase space for this set now has 6N dimensions. Again, we don’t

care whether there are other uninteresting balls with which they

interact. The phase space has typical and untypical regions, defined

by the observer (exactly as any CEV is defined by the perceptual

functions of the observer). One thing to note is as before: if a

point is in a typical region, it will almost never move into an

untypical region as a consequence of reversing some or all of its

components in the velocity subspace.

To see that the typicality of a region of the phase space depends on

the observer, imagine an observer looking at the phase space plot

through the N-dimensional equivalent of a scrambled fibre-optic pipe.

In a fibre-optic pipe, a large number of glass fibres are placed

together so that at each end of the pipe their ends form a single

plane onto which a pattern can be focused. If the alignment of the

fibres were the same at both ends, a pattern focused onto one end of

the pipe would be seen glowing on the surface at the other end of the

pipe. But if the fibres are scrambled, what is seen at the other end

is a random hodge-podge of light and dark. There is, however, a very

specific configuration of what looks like random dots that, when

entered at the front end, emerges as a straight line at the back end.

Patterns of random dots are “typical” of dot patterns in general, in

that almost all such patterns of dots look alike; but a straight line

pattern of dots is untypical, there being only a relatively few ways

that the dots can be so arranged. A random procedure for locating the

dots could produce a straight line, but it would be most unlikely to

do so. A “random” pattern at one end of the pipe that produces a

straight line at the other is a most untypical random pattern, and the

observer that saw the line would so assess it.

Returning to the system of balls, the observer’s definition of what

constitutes a particular “atypical” structure is like the scrambling

of the fibres in the fibre-optic pipe. The pipe may be straight

through, and the observer may define a “natural” atypical

configuration, such as that half the balls have zero velocity and are

arranged in a repeated regular pattern of locations (a crystal), while

the rest of the balls have high velocity and are not specifically

related on location. Or the observer may define some arbitrary pattern

that another observer would see as quite typical.

Each point in the phase space defines a possible “microstate” of the

space. Any volume of the phase space

defines a “macrostate.” Any

configuration with a phase point that the observer saw as conforming

to the arbitrary pattern would belong to the macrostate for the

pattern, and any point that did not so conform would belong to the

larger “typical” macrostate of the phase space.

Boltzmann’s entropy is concerned with the volume of the phase space

that might be considered typical of the present state of the system.

For example, in the 2-ball case in which one ball was motionless or

nearly so, it would presumably not matter which of the balls was

stopped and which was moving, so the region of phase space typical of

the “one-stopped” state would at least include two thin slices near

the two 3-D subspaces of the 6-D velocity space which represent one of

the balls as stopped. In the Physics Today article, the “one-stopped”

state would be an example of a “macrostate,” whereas the specific

phase point associated with one such condition would be a

“microstate.”

To be accurate, it should be noted that macrostates do not normally

have discrete boundaries. It would be more proper to treat a

microstate not as being “inside” or “outside” a specified macrostate,

but as having a specified fuzzy set membership in the macrostate. The

“volume” of the macrostate would then be the integral over the phase

space of the membership function of the microstates. But we will

continue to treat this integral as20if it were the volume of a

well-defined region of the phase space.

The value of Boltzmann’s entropy is proportional to the log of the

size of the subspace typical of the current state. It does not depend

on whether the system is open or closed (whether there are

“nondescript” balls, in the ideal gas example). It does depend on the

system being delimited, (e.g., in the ideal gas example, knowing which

balls are being described). If the phase space is continuous, there

may not be a natural scale of measurement for the volume of the

typicality space, but if it is discrete, the natural unit is the

volume of one discrete cell. However since the value of the entropy is

a logarithmic function of the size of the typicality region, changing

the scale unit only adds or subtracts a fixed quantity to all

entropies measured using the unit. It makes no difference when we

consider changes in the entropy of a defined system, as is usually the

case.

If the system is closed, the region of phase space available to it is

limited to a shell of constant radius in the velocity subspace. That

shell has a volume, the log of which which represents the maximum

entropy possible for the system (in a high-dimensional system,

nontypical subspaces have a vanishingly small total contribution to

the volume).

If the system is open, the shell to which its phase point is confined

can ex

pand or contract, depending on whether energy is on balance

transferred into or out of the system. As the shell radius changes,

so does the maximum entropy possible for the system, and if (as is

highly probable) the system is in a state of near-maximum entropy, its

entropy will increase as it gains energy (gets hotter).

Notice that nowhere in the definition of the Boltzmann entropy does

the construct “probability” appear; it appears only consequentially.

At least in the ideal gas, any point in phase space is as likely as

any other to be occupied. Hence the probability of a point being in

any particular macrostate is proportional to the size of the region

typical of that macrostate relative to the total phase space volume.

In systems with attractor dynamics, this statement does not hold, and

probability and entropy go their separate ways. A low-entropy

condition is highly probable in a system with attractor dynamics–or

in the environment of a control system.

As Lebowitz’s article points out, Boltzmann’s entropy is numerically

consistent with other definitions of entropy if the situation is

appropriate for the application of the other definitions, such as

Gibbs, which depends on probabilities, or Clausius, which applies to

equilibrium systems.

Time’s arrow, as described in the Physics Today article, depends on

the fact that it is easier to get out of a tiny volume of the phase

space

into a larger volume than to find a way from an arbitrary point

in the large volume back into the tiny volume. Taking that to the

extreme, for a specific example, suppose that there were a macrostate

for which the typical region consisted of just one microstate (L, V).

Then, if after some time the system had evolved to a microstate (L’,

V’) in a larger macrostate and all the V vector components were

reversed exactly, the system would eventually evolve back to the

original microstate. But if any of the reversals were inexact, or if

some of the components were left alone, the system would not return to

the original microstate. The incomplete reversal would leave the

system in the same evolved macrostate, but in a microstate outside the

original small volume. Entropy (almost) always increases in a closed

system.

Entropy does not always increase in an open system. I, being outside

the open system of concern, can select the balls of interest and

deliberately place them in any state I desire, in location and

velocity. I can “control” them, at least insofar as I can perceive

their exact locations and velocities. I can have them move all in

parallel at the same velocity (a flow regime), or stand on top of one

another, or whatever. These structures have microstates with

relatively small typicality regions. I, the observer-manipulator, have

reduced the entropy of the system of i

nterest, putting the phase point

in a non-typical subspace of the descriptive space I have defined by

my perceptual functions. I can keep the structure in such a

low-entropy state, provided I can continue to observe and correct its

departures from this non-typical region under the influence of the

“nondescript” parts of the universe. I can control my perception of

it.

I argue that Boltzmann entropy can be applied to any descriptive

space, and in particular it can become Shannon entropy or uncertainty

under appropriate conditions. Why is this so? Consider my control of

the balls of the open system; I define a state in which I wish them to

be. I can force them into this state only as accurately as I can

perceive them. The region “typical” of my perception consists of all

those microstates I cannot distinguish. The less uncertain I am about

the location and velocity of all the balls, the smaller this

typicality region and the lower the entropy of the ball system as I

see it.