liberation from the bell: power to outliers

6
© 2014 Afif Say Liberation from the bell: power to outliers - P a g e | 1 Liberation from the bell: power to outliers Afif Say, MD Give me the fruitful error any time, full of seeds, bursting with its own corrections. You can keep your sterile truth for yourself. Vilfredo Pareto As our understanding of the complexity of the nature is less filtered by our yearning of idealist harmony, we are shifting away from a Gaussian view of events towards a Paretian world. Comfy but boring world of mediocrity We all know Gaussian Distribution, or Normal Distribution, with its bell shaped curve and symmetric slopes. It is used to present probability distribution of events. It has been widely utilized in natural and social sciences. The main take on Gaussian distribution is that there is a stable mean. As you add more samples the mean should stay relatively stable. Added to that there is a stable, definable variance, within which the majority of events tend to occur. Knowing this stable mean and variance, you can more or less define the characteristics of the population. This way we can understand what is supposed to happen or what had been supposed to happen; filter out the noise or extremes; focus on the beautiful balance and harmony of natural events, social structures, drug effects, consumer behavior, stock market, employee performance; you name it. Armed with statistical functions and spreadsheets we can churn raw data to spit out normalcy at the other end. When things go chaotic we can close our eyes, shun the extremes, and safely huddle around the mean in between the 2 standard deviation borders. Cutting your toes to fit into the shoe you chose There are some characteristics of normal distribution which we need to pay close attention. Normal distribution assumes total random and independent samples: like shoe sizes or height of a population, which are independent values. As the curve slides down towards infinity on each side the tails get thinner coming close to zero. This means on each side extremes are negligible or excluded in forced curve fitting. Also a fragment of a normal curve doesn’t show similarity to the rest of the curve – i.e. lacks scale-invariance. Smaller sample size doesn’t represent the population. Main characteristics of Normal Distribution

Upload: afif-say

Post on 21-Apr-2017

216 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Liberation From the Bell: power to outliers

© 2014 Afif Say Liberation from the bell: power to outliers - P a g e | 1

Liberation from the bell: power to outliers Afif Say, MD

Give me the fruitful error any time, full of seeds, bursting with its

own corrections. You can keep your sterile truth for yourself.

Vilfredo Pareto

As our understanding of the complexity of the nature is less filtered by our yearning of idealist harmony, we are

shifting away from a Gaussian view of events towards a Paretian world.

Comfy but boring world of mediocrity

We all know Gaussian Distribution, or Normal Distribution, with its bell shaped curve and symmetric slopes. It is

used to present probability distribution of events. It has been widely utilized in natural and social sciences. The

main take on Gaussian distribution is that there is a stable mean. As you add more samples the mean should stay

relatively stable. Added to that there is a stable, definable variance, within which the majority of events tend to

occur. Knowing this stable mean and variance, you can more or less define the characteristics of the population.

This way we can understand what is supposed to happen or what had been supposed to happen; filter out the noise

or extremes; focus on the beautiful balance and harmony of natural events, social structures, drug effects,

consumer behavior, stock market, employee performance; you name it. Armed with statistical functions and

spreadsheets we can churn raw data to spit out normalcy at the other end. When things go chaotic we can close

our eyes, shun the extremes, and safely huddle around the mean in between the 2 standard deviation borders.

Cutting your toes to fit into the shoe you chose

There are some characteristics of normal distribution which we need to pay close attention. Normal distribution

assumes total random and independent samples: like shoe sizes or height of a population, which are independent

values. As the curve slides down towards infinity on each side the tails get thinner coming close to zero. This

means on each side extremes are negligible or excluded in forced curve fitting. Also a fragment of a normal curve

doesn’t show similarity to the rest of the curve – i.e. lacks scale-invariance. Smaller sample size doesn’t represent

the population.

Main characteristics of Normal Distribution

Page 2: Liberation From the Bell: power to outliers

© 2014 Afif Say Liberation from the bell: power to outliers - P a g e | 2

Dreams of a predictable, harmonious, and linear universe

Beyond its obvious realm of complete random and totally independent events with low impact and insignificant

outliers, utilizing forced normal distribution seems like a product of an idealistic view of the universe seeking

balance and equilibrium with predictable outcomes. Just like the view of a universe revolving around the Earth, a

divine harmony. Even extremes happen, eventually everything tends to go back to a state of equilibrium, to

‘normalcy’, to robustness. From politics to policy making, to product design, to social engineering the whole past

century reflected this idealized, simplified view of the world for the experts, managers, leaders to define, design,

scale around this myth of normalcy.

As our understanding of the chaotic and complex behavior of natural or social systems improve, we realize that

most of the presumed independent events are actually interdependent. Weather patterns, earthquakes, epidemics,

social unrests, elections, consumer behavior, employee productivity, and so on. Can we say that each sample we

pull is independent of their siblings? While assumptions of independency go away, the bell tolls for the normal

curve. Then cometh the butterfly effect, power of the individual sample: a Paretian world, where outliers are

welcomed.

Middle Eastern bazaar not a suburban shopping mall

Pareto distribution, defined by Vilfredo Pareto, is a power law distribution and nothing like the normal

distribution. It is not symmetric. It doesn’t have a stable or well-behaved mean or variance. So these values

become meaningless. It has a long thick tail, allowing the presence of more extremes. It assumes that samples are

interdependent. Furthermore, like a fractal, for most parts it has scale-invariance, parts of curve show similarity to

the whole (self-similarity and self-affinity). As Mandelbrot explained fractals:

“Cauliflowers exemplify a second area of great simplicity, that of shapes which appear more or

less the same as you look at them up close or from far away, as you zoom in and zoom out.”

Systems that show a Paretian Distribution have interdependent, interactive, and/or self-organizing elements that

disallow linearity. The causal impacts on the system may not produce effects that are proportional: butterfly effect

on weather systems for example (proportionality of cause and effect). Likewise we cannot calculate the impact of

multiple causes acting on the system by summing them up (superposition).

Lacking stable or well-behaved mean and independent elements means that these systems do not show a trend

toward equilibrium, as the interdependent and self-organizing behavior of the elements do not allow this to

happen.

Main characteristics of Power Distribution

Page 3: Liberation From the Bell: power to outliers

© 2014 Afif Say Liberation from the bell: power to outliers - P a g e | 3

Systems that have Paretian distribution, such as most natural systems, social systems, and markets, do not allow

us to develop frameworks or models to predict their behavior like Gaussian systems that show linearity and

tendency for equilibrium.

Extremes happen

What about the long thick tail of the Pareto distribution? Fatter and longer tail end of the Pareto distribution

allows more extreme values to be accepted in comparison to Gaussian distribution. Pareto distribution accepts

higher probability of very strong earthquakes, tsunamis, market crashes, or social uprisings. It does not exclude

them as outliers.

The significance of including extremes rather than sacrificing them to the gods of normal curve fitting as outliers

is the understanding and studying the low-probability/high-impact phenomena in these systems. Probability of

events like the plague epidemic in Europe in the 14th Century, the tsunami that destroyed Fukushima, or the sub-

prime mortgage crash, now appears in our data set rather than being cut off.

Comfort zone

We have a tendency to seek equilibrium, stability, and predictability when observing life. That is why it surprises

us every time it snows or rains too much. We are also trained to seek normalcy, means, and averages; we try to fit

our truth within the borders of variance. The Gaussian world is comforting: market crashes are very rare, big

natural disasters happen once in a million years and generally somewhere else, and the products are for ‘normal’

people. We conveniently disregard the interdependency and interactivity of elements in a system. We see every

move, strategy, and project as a ‘controlled experiment’. We expect outcomes that are predictable and

measurable. We ignore, disregard, or shun the extremes or unlikely outcomes – if they happen, we justify that

they are aberrations. Also we think that if we analyze a few elements individually we can make theories to

explain the whole. All of these keep us in our comfort zone, the zone that fits in between plus/minus two standard

deviation marks on the normal distribution.

During industrialization we tried to create a world that can be standardized, measured and predicted. We built

companies, hired people, measured their performance, and expected consumer behavior, all forcing them to fit

into the normal curve within the 2 standard deviation zone. Taylorism, Total Quality Management, and Six Sigma

clergy still expect us to march with their mechanical unison tempo. But as the world globalizes, the internet, a

virtual universe - which is complex by design - connects people and thus exponentially increases the interactions,

normal curves of mediocrity started shaking.

Comfort Zone within ±2S

Page 4: Liberation From the Bell: power to outliers

© 2014 Afif Say Liberation from the bell: power to outliers - P a g e | 4

Wilderness of Pareto

Actual universe is complex or chaotic in behavior. Extremes happen all the time (by universe time, not our blink

of existence). The second law of thermodynamics tells us that the universe doesn’t tend towards equilibrium or

balance, actually it prefers chaos. Quantum mechanics throws away our dreams of certainty. Chaos is neither

something Kronos ended, nor a boogeyman in the mouths of totalitarian politicians anymore. Evolution is not a

subject limited to biology books. The evolutionary mechanisms are being used to explain the dynamic changes in

social and natural systems these days. Linear causality and superposition are not valid for evolutionary

mechanisms. Nature and society, and related systems are all complex, messy, and unpredictable, where extremes

can happen, individual elements can eventually make big impact, and the systems may not settle in a preset

equilibrium.

Surviving outside the box

If we cannot know anything for certain, cannot predict outcomes, or cannot see what is normal, how do we live?

Actually our brains can help us survive. The human brain is the product of millions of years of evolution. It is not

a machine that performs mechanical logic operations as philosophers and scientists tried to see in the past

centuries. Like the natural processes that developed it, our brain is messy and complex - and chaotic sometimes,

as in epilepsy. Human brain is very capable of recognizing patterns and pattern violations. Gestalt theorists

recognized this in the early 20th century. We can perceive things holistically rather than analyzing individual parts

first. As Gary Klein’s work (1998) showed that we make decisions by first-fit pattern matching based on our

direct and indirect experiences. We learn more from our failures than our successes as avoiding failure is a much

more potent drive in evolution than aiming for success.

To survive in Pareto world you pay attention to extremes, especially low-probability, high impact extremes.

Einstein was a low-probability/high impact human being, so was Hitler, they were outliers in the Gaussian world.

To survive the unpredictable world, where individual extremes can have high impact you know that you cannot

build a perfect 14 Richter earthquake proof building. You don’t build for robustness, you build for resilience:

shorter buildings, better evacuation, post-earthquake survival, less dense cities, and etc.

To survive the wilderness you need to be aware of the emerging

patterns and recognize them as they do. In the African savannas I

learned to listen and watch the birds and antelope scan the

environment much more efficiently than us. Their alarm calls and

behavior or recognizing tiny brown flickers above the grass line may

save your life. Our brains are very good at recognizing these weak

signals and processing them. We constantly try to make sense of our

environment.

Planning in the power law universe isn’t creating a concrete vision

and determining all the necessary steps to take us there. Outside the

comfort zone, you decide on a direction, you pay attention to the

journey. You try things, and see what patterns emerge. It is like

sailing: you set a direction, trim your sails and see how the boat

handles. You don’t look for perfection but something that works. Not

failsafe but safe-to-fail. If it doesn’t work, you stop doing it and try

something else. This is not trial-and-error however. You make

strategic decisions based on experience and active scanning, and

continue to gain more experience. If the wind changes you trim again.

You tack when it is necessary. If the wind gets stronger you reef, or

you go off course to avoid nasty storms. During sailing you are ‘off

During sailing you are off track most of the time

Page 5: Liberation From the Bell: power to outliers

© 2014 Afif Say Liberation from the bell: power to outliers - P a g e | 5

track’ most of the time. Instead of planning and making decisions once and then executing, you make strategic

decisions all the time with constant input from the environment.

In the complex system of nature and society elements constantly interact with and modify each other and the

system itself. Causality is not linear. Performance measurement by forced normal curve fitting kills the team

performance. Building companies, products, policies that satisfy the normal curve, hiring people that ‘fit’ the

general employee profile kill the resilience and adaptive capacity of the organization. Main drive of evolution lies

in the outlier individuals, and in Pareto world outliers are welcomed.

References & Further Reading

Andriani, Pierpaolo, and Bil McKelvey “Beyond Gaussian averages: redirecting international business and

management research toward extreme events and power laws,” Journal of International Business Studies

(2007) 38:1212–1230.

Andriani, Pierpaolo, and Bil McKelvey “Perspective--From Gaussian to Paretian Thinking: Causes and

Implications of Power Laws in Organizations,” Organization Science, (2006) 20(6):1053–1071.

Kauffman, Stuart A. The Origins of Order: Self-organization and Selection in Evolution. New York: Oxford

UP, 1993.

Klein, Gary A. Sources of Power: How People Make Decisions. Cambridge, MA: MIT, 1998.

Luccio, Riccardo "Gestalt Psychology and Cognitive Psychology." Humana.Mente Journal of Philosophical

Studies (2011) 17: 95-128. Accessed 4 May, 2014.

Mandelbrot, Benoît, Nassim Taleb "A focus on the exceptions that prove the rule". Financial Times (23 March

2006). Accessed 4 May 2014.

Mandelbrot, Benoît, and Richard L. Hudson. The (mis)behavior of Markets: A Fractal View of Financial

Turbulence. New York: Basic (2008).

Mckelvey, Bill, and Pierpaolo Andriani. "Why Gaussian statistics are mostly wrong for strategic organization."

Strategic Organization (2005) 3(2): 219-228.

O’Boyle Jr., Ernest and Herman Aguinis. "The best and the rest: revisiting the norm of normality of individual

performance." Personnel Psychology (2012) 65.1: 79-119.

Prigogine, Ilya., and Isabelle Stengers. Order out of chaos: man's new dialogue with nature. New York, N.Y.:

Bantam, 1984. Print.

Snowden, David. "Stories from the Frontier." E:CO (2006) 8(1): 85-88. Accessed 5 May 2014.

Taleb, Nassim Nicholas. The black swan: the impact of the highly improbable. New York: Random House,

2007.

Page 6: Liberation From the Bell: power to outliers

© 2014 Afif Say Liberation from the bell: power to outliers - P a g e | 6

Afif Say is a seasoned leader, manager, and strategist with experience across multiple

organizational environments including start-ups, social enterprises, consulting firms, and

large corporations. He uses his expertise in applied information and communication

technologies to create strategies for organizational knowledge, culture and development.

He in addition to the USA he lived and worked in Africa, Middle East, and Eastern

Europe.

He is also an accomplished photographer, avid traveler, and a trained medical doctor.

Follow on twitter: @afifsay