About your graduate studies, Part II:Knowledge, Science, Predictive

Models, and the Magic Spell

Seppo KarrilaMarch 2015

PSU Surat Thani

Executive summary

• This is a discussion of the philosophy and principles behind modern natural sciences

• As a graduate student your task is to perform and report a scientific study, so you need to know how science is done

• Predictive models are emphasized because– They give testable predictions, which allow application

of the scientific method– Using training, validation and test data is a simple

trusted approach to doing the job correctly

Theory of knowledge

• Has been studied for a very long time, called “epistemology”

• Such theories have been unable to reach practical usefulness– The question “how do we know what we know, and do

we really know?” is important– Time travel would also be important… – Companies are not struggling to find and hire

epistemologists

What is “understanding”?• In common language it often just means to sympathize. “I feel for you, I

understand.”• In science there are different levels of understanding

– Forming concepts and nomenclature, and general rules: an apple falls if you let it go

– Naming the ghost behind the action: gravitation. Being able to name the ghost gives a “feeling of understanding it”. • Why does the apple fall – gravitation! This is nonsense and word magic, just another

name that sounds more learned than “it falls because it falls”.

– Quantitative understanding: how fast does it fall, how hard does it hit. Can be calculated accurately with classical mechanics and ITS EQUATIONS FOR GRAVITATION. Now we are getting somewhere! (Some went to the moon, satellites for GPS and communications.)

– Application models based on experiments: when the apple hits the floor, will it break, how much is it damaged, does this affect its price or shelf-life, how should we package apples… These are clearly things you can’t calculate accurately from classical mechanics, you have to do experiments AND MAKE MODELS.

Stages of development• A “young” scientific discipline

– Names and documents things– Creates taxonomies– Is descriptive

• Becomes more quantitative with time, a “teenager”– Finds general rules that are equations, like “conservation of

mass”, thermodynamics of equilibrium• Strives to replace experiments with predictive

computations, becoming an “adult mature discipline”– This transition is currently happening in chemistry…

• In other words a somethingology tries to advance to a somethingonomy, from qualitative to quantitative

“Predictive” is the keyword!• There are theories that explain everything afterwards, but

predict nothing– Pyramidology predicts history accurately but future always poorly– There is a BIG difference between a regression fit and a predictive

equation. Ability to match what is known does not equal ability to predict the unknown.

• Engineering design is based on sufficiently accurate predictions. Design equations may be mandated in official standards. – You can’t know the strength of a concrete mix accurately, so engineers

use safety margins in designs– Still a lot of computations are useful and used in mechanical design,

inaccuracies don’t make a model useless

Training, validation and test data• Here is the correct way to fit predictive models to

experimental data.

• The data are split to three sets, or generated at different times for each of the three– All candidate models are fit to the training data– The validation data is “predicted” with every type of model, and the

best model type is chosen• Now it can be fit to training + validation data

– The chosen model is tested for performance• It has not “seen” the test data before, if you test with previously seen data

you are looking at “fit”, not “prediction ability” !

– Now you can pass on the model, and make some claim about its prediction accuracy

How to make pyramidology look good

• Make predictions about the starting year in January. • Check predictions next January

– Ups, all wrong again• Quickly update the model

– Publish how the new model fits history– Never discuss results from testing old predictions with new data, erase the old

models quickly• Effect: the public model is always an untested fit, there is never

embarrassing numeric evidence about predictiveness (actually complete lack of it)– This game is played all the time. Various companies have forecasters who don’t

want to look incompetent. So does the government. The forecasters want to keep their jobs and maintain credibility, and the purpose of the forecasts is not to be right but to influence decisions. Once the decisions are made, why look back…

Intermediate summary• In the natural sciences, the highest level of understanding is a

quantitative predictive model– Then you can predict effects of choices or actions on future function and

performance, this is called “engineering design”

• Such models must be based on reproducible experiments, the models must be validated, and their accuracies known. The models can only use validated reproducible characterizations whose accuracy is also known. – Characterizing material properties of cement mixes, or steel, or rubber, or plastics

– check the official standards.

• Fundamental theories are not yet good enough– You only learn EQUILIBRIUM thermodynamics– Even “viscosity” becomes difficult with polymeric liquids, so don’t expect accurate

flow calculations in a complicated geometry– Flow mixing reaction rates …

How does science progress?• According to Karl Popper

– there is an accepted “scientific paradigm” that scientist use, until enough evidence accumulates to refine or update it, and that is a “paradigm shift” (like classical mechanics relativity, continuum quanta) • Negative evidence = false predictions ! Without them there will be no

paradigm shift

– Various fields like sociology would not fit his definition of science at all, since they seldom predict anything. There are no testable hypotheses, but an ideology may “explain” anything in past history.

– Thomas Kuhn became popular because he essentially proposed that science is whatever scientists do. This made sociologists and various others happy again, they loved Kuhn.

– Popper’s views are most appropriate in the natural sciences and engineering that have developed to a quantitative stage.

The scientific method!

• Make a hypothesis or conjecture• Design experiments that could show it is false– In fact, the hypothesis should predict something that

is extremely unlikely without it being correct, so when that happens it is strong support to the hypothesis

• If the hypothesis survives the tests, keep it. If it makes useful predictions, others will adopt it and teach it as current wisdom.– It becomes part of the current scientific paradigm

The big point• Science looks for general truths that summarize

experiences in a useful way, and that allow making predictions.– Instead of learning every sentence you use, you learn grammar.

Summarization and general rules enable learning and doing your own things instead of just copying and repeating.

– Measurements may be needed, but doing measurements is not the same a doing science.

– The most valuable hypotheses have a wide generality. Very specific and restricted hypotheses are “doing the right motions” but amount to little in scientific learning. They provide a data point, not a useful summary. However, they are the necessary less glorious grassroots of science, and that is where most scientists live. It is the big things that are historic with fame.

Other points to note• If the hypothesis is such that nothing can falsify it,

then it only predicts things that would happen anyway (or nothing at all)– It cannot usefully predict anything new and surprising

• But the testing is based on predictions! – A theory that predicts nothing does not enable any

decision, design, or action. These are the only things that give value to scientific theories.

– This is why “theories” that predict nothing and can’t be tested are pseudoscience.

About “mistakes”• Progress of science depends on being wrong

– Falsification of old paradigm is the only way to a new paradigm

• If you are very afraid of mistakes, you can’t do anything– But it hurts less to learn from mistakes of others– Admit mistakes quickly, correct them quickly before they can

take effect, avoid repeating them. It is not honorable to hide a mistake.

– A man who never made a mistake has done nothing• However, don’t study mistakes, study successes

– First, this way there is less to study– Second, repeating a success is better than repeating a mistake

You can never prove that something is true – you can prove something false

• Mathematics is based on axioms, assumed truths. Its proofs are valid IF the axioms are valid. In other words the absolute truths of mathematics are confined within mathematics.– Still, logic and mathematics are the tools of clear thinking. We

want equations, and quantitative predictions!• However many red roses you see, it does not prove that all

roses are red.– It only takes ONE white rose to prove they are not ALL red– This is why a hypothesis must be FALSIFIABLE, that is the best we

can do. You cannot have a PROVABLE experimental hypothesis.

Null hypothesis

• Statistical testing is based on the same ideas.

• You make a NULL HYPOTHESIS “some roses are not red”, that means falsification of the actual hypothesis “all roses are red”

• You show that the null hypothesis is very unlikely to be true

• That supports your actual hypothesis as “significant”, meaning that we can keep it for now

Back to the magic spell…

• Your scientific story is “good” if you have– Issue, significance, approach, results, conclusions– The question must have importance, the conclusions state

effects of your results on theory or on practice!– It is really good if the results are unexpected and surprising

• A technically useful engineering result usually includes a quantitative model– How do you make a model? Where can you start? How is

“science” done in practice?

You are not Newton

• We don’t expect you to come up with new general principles of fundamental science

• But recall that our science is limited, we need experimental models– You have equilibrium thermodynamics, but kinetic

reactions, flows, multiphase materials– Rheology of a polymeric liquid is difficult enough

• Now add solids, perhaps nanomaterials, reactive species, electromagnetic fields, …

– Or just examine if an apple breaks when it hits the floor

For modeling

• Define concepts included in the model– Some need to link to reality through direct

measurements• Prefer physical measurements over industry technical

standards, the former will stay as they are today

– Others can be computed intermediate variables• These may come from physics, physical chemistry, etc.• Dimensionless groups! • You should use these in statistical modeling

The problem with indirect measurements

• Does “happiness” relate to “wealth”? – A sociology problem where no concept can be directly measured.

Does wealth include your future inheritance from grandfather, possibly winning lottery, or your relative who works for PTT where you may get a good job? A rich girlfriend? How do you measure “happiness”? Are you feeling 6.3 happy or 8.5 happy, on average this week?

– By adjusting definitions, you can get anything you want. The result is theories that live as long as the professor who started them, and who sits on committees of the National Research Fund while alive. So you better agree with his theory while he still lives, if you want to do related research. Oh, he is also the Editor or on board of all relevant journals…

– Stick with the real sciences, we do things better. I like Karl Popper.

Assume you got a topic from your advisor

• Were you given a hypothesis, or do you need to come up with one?

• Learn the basics, read review articles, find out about techniques used in experimental determinations– For example, nanomaterials are modern and difficult

exactly because they are too small to “see”. Some might also have nasty effects that are slow and delayed, like asbestos fibers, so take precautions. Don’t rush into exciting new things unless you can also measure and detect.

Hypothesis from review of literature

• Are there important gaps in knowledge? – Turn these into hypotheses that seem reasonable

• Can they be studied with YOUR available equipment and techniques? – Can you estimate or guess sizes of effects?

• What are the factors affecting results?– Which ones can you manipulate– Which ones can you observe/measure– Which ones can you limit or select, essentially

defining the scope of your study

Decisions and actions, once you have your hypothesis

• Select scope, manipulated and observed variables– Do you have the technical skills?– Do you need to design and construct devices?

• Do you need preliminary or “pilot” experiments? – Instead of gambling a long-term plan with uncertainties, can you quickly

check for some effects or phenomena?• Design of experiments, after selecting a minimal set of main

variables– Can be complicated, better to stay with some standard design (e.g.

Plackett-Burman), otherwise consult a statistician– If task is to optimize something like yield, check out response surfaces and

Box-Behnken design• Statistical analysis of results

– Significant effects AND their effect sizes !

About effect size

• Opening your car windows changes aerodynamics, probably for the worse– The top speed may go down by 3 km/h if you open

the windows– If in experiments you would repeatedly find this is

so (using GPS to measure speed), then this effect is statistically significant

– However, the effect size of 3 km/h is marginal, you don’t need to care. If it dropped the top speed by 40 km/h, that would be practically important.

A key observation

• Statistically significant is not the same as significant!– Statistically significant means, it is likely there is

some detectable difference. A detectable difference can be marginal in size.

– Significant Breakthrough Discovery: with a small effort and at a low cost, you get a large effect (on yield, quality, production rate, …)

– This is why you should pay attention to effect size!

The arts and sciences cherish novelty• But it is not enough. Pay attention to significance!– If I paint with a banana, it is a novelty but nobody will

buy my painting– If I multiply two 50-digit numbers and subtract 1234567,

nobody else ever did that calculation with the same numbers: it is a novelty!• No insight, nothing of interest, can’t be published

– In descriptive sciences, things are published just to document• Asteroid number 123456 photographed through telescope – it

goes to a database as an entry, but nobody really cares. This is their grassroot science, a data point.

Art, science and engineering, a caricature

• When something is done for the first time, it can be art or science

• When it is done repeatedly and efficiently, it can be plagiarism, forgery, or engineering research

• In science, if you know the result you should not do it, because you are looking to expand knowledge

• In engineering, if you don’t know it works, you should not do it, because most likely it will not work.

Engineering and science are intermixed

• The exploration exploitation dilemma in learning– The caricature was about extremes in exploration and exploitation– If you only exploit existing knowledge, that may be convenient and

predictable but you learn nothing new– If you only explore, you are wandering aimlessly and not

productive– How much of time or budget should be exploration? – Some companies have been very good at leaving explorations to

others, then quickly doing a technically reliable good job once a new technology has been demonstrated. This is a cost advantage.

– Similar opportunities abound in science, where you can transfer techniques established in another field to your field. You will not be “the first”, but if you are quick you can be the first in a specific context. So keep your eyes open, read widely.

In STEM you should use precise language

• You drive by a field and see a black bull in profile view.

• Layman’s statement: ”See, they have black bulls here!”

• Scientific statement: “In this region there is at least one bull that is black on at least one side.”

On evaluating your own work, or that of others

• Use the magic spell– Issue, significance, approach, results, conclusions– If any of these are missing, thumbs down

• Do this also when you are planning your work– What kind of results do you expect? – How can the results affect theory or practice?

Research proposal

• Surprise:– Issue, significance, approach, expected results, expected

effects (conclusions)

• Now approach needs to include– An experimental design

• How many samples, what measurements, what experiments, how many replications

– Time – how long would it take?– Budget – how much would it cost?– Risks – what can go wrong?

Dealing with risk

• Can you prevent it from happening?– Vaccination may prevent infection– Do you need to check for impurities of raw materials,

sterilize samples, remove suspended solids before optical measurements or chromatography, de-aerate liquids, …

– Start with small doses of an additive, see if trouble arises. Increase dose if all goes well. Large dose first is risky.

• What will you do if the bad thing happens?– Go to doctor or hospital when you get sick– Do you have alternatives or backups, for sources of

samples, for determination techniques, …

Conclusions• Most likely you will run experiments and do some statistical

analysis of the results– Planning of experiments should be based on a hypothesis, the

most “ignorant” just assuming that some factor has an effect• After literature research, you should have a hypothesis and

an experimental approach, convert these to a detailed research proposal

• When reporting results, or checking results of others, statistical significance is worth little, effect size is worth a lot

• The Magic Spell gives structure to any communication, or analysis of communication. If an item in it is missing, then the proposal/presentation/manuscript is incomplete.

One final note and warning on science vs. pseudoscience

• We have not defined science, but it clearly seeks knowledge and understanding that are useful and predictive– The best we can do is accept a useful hypothesis until it has been

falsified. Useful ones predict something, so they can always be tested. • We can identify much of pseudoscience from this already

– Either the hypothesis can’t be falsified ever, by anything– or its proponents bitterly oppose any such test that could falsify it,

their interest is not truth and knowledge but politics• But… everybody wants to have the clout of science. If you point

a finger at pseudoscience, many people will be upset.– It suffices that you know the truth, don’t waste your time in an

argument. Just keep it between you, me, and Sir Karl Popper. And that guy in the toothpaste commercial who wears a white lab coat, he knows, too.