Colloquium 1: Data presentation and basic of

descriptive statistics Dr. Paola Grosso

SNE research group

p.grosso@uva.nl paola.grosso@os3.nl (preferred!)

Roadmap •  Colloquium1: Wednesday Sep. 25

–  Collecting data –  Presenting data

–  Descriptive statistics

–  Basic probability theory

•  Colloquium 2: Wednesday Oct. 02 –  Probability distributions (cont)

–  Parameter estimation –  Confidence intervals, limits, significance

–  Hypothesis testing

Instructions for use

•  Time: morning slot:10.15am- 12.15pm afternoon slot:13.45-15.15 •  You will not get a grade, but you will have to do some ‘work’!

Introduction

Why should you pay attention?

We are going to talk about “Data presentation, analysis and basic statistics”.

Your idea is?

Our motivation

1.  An essential component of scientific research; 2.  A must-have skill (!) of any master student and researcher

(… but useful also in commercial/industry/business settings); 3.  It will help to communicate more effectively your results

(incidentally, it also means higher grades during RPs).

How to conduct a scientific project

q  Research your topic q  Make a hypothesis. q  Write down your procedure.

•  Control sample •  Variables

q  Assemble your Materials. q  Conduct the experiment. q  Repeat the experiment. q  Analyze your results. q  Draw a Conclusion.

This is the focus of this colloquia

The structure of a scientific paper Experimental process Section of Paper

What did I do in a nutshell?  Abstract

What is the problem? Introduction

How did I solve the problem? Materials and Methods

What did I find out? Results

What does it mean? Discussion (and Conclusions)

 Who helped me out? Acknowledgments (optional)

Whose work did I refer to? Literature Cited

Extra Information Appendices (optional)

Collecting data

Terminology Sampling

Data types

Basic terminology •  Population = the collection of items under investigation •  Sample = a representative subset of the population, used in the

experiments

•  Variable = the attribute that varies in each experiment •  Observation = the value of a variable during taken during one of the

experiments.

Quick test Estimate the proportion of a population given a sample.

The FNWI has N students: you interview n students on whether they use public transport to

come to the Science Park; a students answer yes. Can you estimate the number of students who travel by public

transport? And the percentage? N=1345 n=142 a=63

The problem of bias

Sampling •  Non-probability sampling:

some elements of the population have no chance of selection, or where the probability of selection can't be accurately determined.

–  Accidental (or convenience) Sampling; –  Quota Sampling; –  Purposive Sampling.

•  Probability sampling: every unit in the population has a chance (greater than zero) of being

selected in the sample, and this probability can be accurately determined.

–  Simple random sample –  Systematic random sample –  Stratified random sample

Variables Qualitative variables, cannot be assigned a numerical value. Quantitative variables, can be assigned a numerical value. •  Discrete data

values are distinct and separate, i.e. they can be counted •  Categorical data

values can be sorted according to category. •  Nominal data

values can be assigned a code in the form of a number, where the numbers are simply labels

•  Ordinal data values can be ranked or have a rating scale attached

•  Continuous data Values may take on any value within a finite or infinite interval

The attribute that varies in each experiment.

Quick test Discrete or continuous?

–  The number of suitcases lost by an airline. –  The height of apple trees. –  The number of apples produced. –  The number of green M&M's in a bag. –  The time it takes for a hard disk to fail. –  The annual production of cauliflower by weight.

Presenting the data

Tables Charts Graphs

Of course not everybody is a believer: “As the Chinese say, 1001 words is worth more than a picture” John McCartey

Shared folder I created a shared folder on Google Docs:

–  http://goo.gl/6j9y8n •  All the documents and forms used today are there.

–  Shared with you as read-only or editable.

Hands-on #1 How many countries have you ever lived in (or visited for at least

one full day ) in Europe and in the rest of the world?

…. Let’s collect the data in the Google Docs form… http://goo.gl/C1MgVb

Now you have 15minutes time to “present” these results.

Work group of two. Put the result online in:

http://goo.gl/T54Vlj

Frequency tables Friends Frequency Relative

Frequency Percentage (%)

Cumulative (less than)

Cumulative (greater than)

0-50 6 6/20 30% 6 20

51-100 4 4/20 20% 10 14

101-150 2 2/20 10% 12 10

151-200 4 4/20 20% 16 8

201-250 1 1/20 5% 17 4

251-300 3 3/20 15% 20 3

•  A way to summarize data. •  It records how often each value of the variable occurs. How do you build it?

–  Identify lower and upper limits –  Number of classes and width –  Segment data in classes –  Each value should fit in one (and no more) than one class: classes are mutually

exclusive

Histograms •  The graphical representation of a frequency table; •  Summarizes categorical, nominal and ordinal data; •  Display bar vertically or horizontally, where the area is proportional to

the frequency of the observations falling into that class.

Useful when dealing with large data sets; Show outliers and gaps in the data set.

Building an histogram

Add values

Add title (or caption in document)

Add axis legends

Pie charts Suitable to represent categorical data; Used to show percentages; Areas are proportional to value of category.

Caution: • You should never use a pie chart to show historical data over time; • Also do not use for the data in the frequency distribution.

Line charts Are commonly used to show changes in data over time; Can show trends or changes well.

Year RP2 thesis Students

2004/2005 9 17

2005/2006 7 14

2006/2007 8 15

2007/2008 11 13

2008/2009 10 17

2009/2010 11 21

2010/2011 10 13

2011/2012 14 18

Dependent vs. independent variables

•  N.b= the terms are used differently in statistics than in mathematics!

•  In statistics, the dependent variable is the event studied and expected to change whenever the independent variable is altered.

•  The ultimate goal of every research or scientific analysis is to find relations between variables.

Scatter plots •  Displays values for two

variables for a set of data; •  The independent variable is

plotted on the horizontal axis, the dependent variable on the vertical axis;

•  It allows to determine correlation –  Positive (bottom left -> top

right) –  Negative (top left -> bottom

right) –  Null

with a trend line ‘drawn’ on the data.

… and more Arrhenius plot

Bland-Altman plot

Bode plot

Lineweaver–Burk plot

Forest plot

Funnel plot

Nyquist plot Nichols plot

Galbraith plot

Recurrence plot

Q-Q plot

Star plot

Shmoo plot

Stemplot

Violin plot

Ternary plot

Statistics packages

Graphics and statistics tools Plenty of tools to use to plot and do statistical analysis. Just some

you could use: •  gnuplot •  ROOT •  Excel •  ???

R We will use the open-source statistical computer program R. Make installation yourself;

$> apt-get install r-base-core

Run R as: $> R

You find documentation at: http://www.r-project.org/ (project website) http://www.statmethods.net/index.html (instruction site)

Descriptive statistics

Median, mean and mode Variance and standard deviation

Basic concepts of distribution Correlation

Linear regression

Median, mean and mode To estimate the centre of a set of observations, to convey a ‘one-liner’

information about your measurements, you often talk of average. Let’s be precise. Given a set of measurements:

{ x1, x2, …, xN} •  The median is the middle number in the ordered data set; below and

above the median there is an equal number of observations.

•  The (arithmetic) mean is the sum of the observations divided by the number of observations. :

•  The mode is the most frequently occurring value in the data set.

∑=

=N

iixN

x1

1

Lunch assignment Look at the (fictitious!) monthly salary distribution of fresh OS3 graduates: http://goo.gl/ub54HP Using R:

–  Plot the data (save the plot(s) in PDF/JPEG)

–  Perform basic descriptive operations (mean,median and std) (save in a file the commands you used)

–  Write two/three conclusions (one liners on what you have learned)

OS3 graduates Monthly salary (gross in €)

Final grade

Grad_1 1250 6.5

Grad_2 2200 7.0

Grad_3 2345 8.5

Grad_4 6700 7.0

Grad_5 15000 7.0

Grad_6 3300 6.0

Grad_7 2230 8.0

Grad_8 1750 7.0

Grad_9 1900 6.0

Grad _10 1750 7.0

Grad_11 2100 8.0

Grad_12 2050 7.5

Grad_13 2400 6.0

Grad_14 4200 7.0

Grad_15 3100 7.5

Lunch

Quartiles and percentiles Quartiles: Q1, Q2 and Q3 divide the sample of observations into four groups:

25% of data points ≤ Q1; 50% of data points ≤ Q2; (Q2 is the median); 75% of data points ≤ Q3.

The semi-inter-quartile range (SIQR) , or quartile deviation, is: The 5-number summary: (min_value, Q1, Q2 , Q3 and max_value) Percentiles: The values that divide the data sample in 100 equal parts.

€

SIQR =Q3 −Q12

Box and whisker plot

It uses the 5-number summary. Length of the box is the IQR (Q3-Q1). Outliers are points more than 1.5 IQR from the end of the box: >= Q3+1.5IQR <= Q1-1.5IQR Whiskers extend to farthest points that are not outliers.

You can produce the above with boxplot in R based on the salary data. Can you?

Dispersion and variability The mean represents the ‘central tendency’ of the data set. But alone it does not really gives us an idea of how the data is distributed.

We want to have indications of the data variability. •  The range is the difference between the highest and lowest values in

a set of data.  It is the crudest measure of dispersion. •  The variance V(x) of x expresses how much x is liable to vary from its

mean value:

•  The standard deviation is the square root of the variance:

€

V (x) =1N

(xii∑ − x)2

= x 2 − x2

€

sx = V (x) =1N

(xi − x)2i∑ = x 2 − x 2

Different definitions of the Std

•  Presumably our data was taken from a parent distributions which has mean µ and S.F. σ

sx =1N

(xi − x )2

i∑ is the S.D. of the data sample

x – mean of our sample µ – mean of our parent dist

σ – S.D. of our parent dist s – S.D. of our sample

Beware Notational Confusion!

x

s σ

µ

Data Sample Parent Distribution

(from which data sample was drawn)

Different definitions of the Std •  Which definition of σ you use, sdata or σparent, is matter of preference,

but be clear which one you mean! In addition, you can get an unbiased estimate of σparent from a given data sample using

€

ˆ σ parent =1

N −1(x − x )2

i∑ = sdata

NN −1

x

sdata σparent

µ

Data Sample Parent Distribution (from which data sample was drawn)

Correlation and regression

Correlation Correlation offers a predictive relationship that can be exploited in practice; it determines the extent to which values of the two variables are "proportional" to each

other. . Proportional means linearly related; that is, the correlation is high if it can be

"summarized" by a straight line (sloped upwards or downwards); This line is called the regression line or least squares line, because it is determined

such that the sum of the squared distances of all the data points from the line is the lowest possible.

Covariance and Pearson’s correlation factor

Given 2 variables x,y and a dataset consisting of pairs of numbers: { (x1,y1), (x2,y2), … (xN,yN) }

Dependencies between x and y are described by the sample

covariance: The sample correlation coefficient is defined as:

yxxyyyxx

yyxxN

yxi

ii

−=

−−=

−−= ∑))((

))((1),cov(

(has dimension D(x)D(y))

€

r =cov(x,y)sxsy

∈ [−1,+1] (is dimensionless)

Visualization of correlation r = 0 r = 0.1 r = 0.5

r = -0.7 r = -0.9 r = 0.99

Correlation & covariance in >2 variables

•  Concept of covariance, correlation is easily extended to arbitrary number of variables

•  takes the form of a n x n symmetric matrix

This is called the covariance matrix, or error matrix Similarly the correlation matrix becomes

)()()()()()( ),cov( jijiji xxxxxx −=

Vij = cov(x(i), x( j) )

)()(

)()( ),cov(

ji

jiij

xxσσ

ρ = jiijijV σσρ=

Careful with correlation coefficients!

•  Correlation does not imply cause •  Correlation is a measure of linear

relation only •  Misleading influence of a third variable •  Spurious correlation of a part with the

whole •  Combination of unlike population •  Inference to an unlike population

Least-square regression The goal is to fit a line to (xi,yi):

such that the vertical distances εi

(the error on yi) are minimized. The resulting equation and

coefficients are: €

yi = a + bxi + εi

€

ˆ y = a + bx

b =(xi − x )∑ (yi − y )

(xi − x )2∑=

cov(x, y)sx

2 = rsy

sx

a = y − bx

€

εi = yi − ˆ y i

Note, the correlation coefficient here

Hands-on #2 Read the Food_consumption from the Google Docs area: http://goo.gl/L76URs Work in groups of two. Store the file into the R memory in variable aStudy. What is the correlation coefficient? Can you fit a least regression line to the data?

Inference statistics

Basic probability theory Probability distributions Parameter estimation

Confidence intervals, limits, significance Hypothesis testing

Descriptive and inference statistics

So far we talked about descriptive statistics: –  straightforward presentation of facts; –  modeling decisions made by a data analyst have had minimal influence.

Now we focus on inference statistics: –  it makes propositions about populations, using data drawn from the

population of interest via some form of random sampling.

Possible propositions/conclusions are: •  an estimate •  a confidence interval •  a credible interval

You will use probability distributions

From the observations we compute statistics that we use to estimate population parameters, which index the probability density, from which we can compute the probability of a future observation from that density.

Probability theory

Defining probability Probability as:

Long term expectation of occurrence (frequency) Measure of belief, measure of plausibility given incomplete prior knowledge

Given a trial,

a process which, when repeated, generates a set of results or observations; Given an outcome S,

the result of carrying out a trial; Given a event E,

a set of one or more of the possible outcomes of a trial:

€

P(E) =n(E)n(S)

which is the relative frequency for the event E

More precisely…. We define as the sample space, the set of possible outcomes.

The probability value f(x) of an outcome x is such that: An event E is any subset of the sample space , and the

probability of en event E is:

€

Ω

€

f (x)∈ 0,1[ ] for all x ∈ Ω

€

f (x)x∈Ω∑ =1

€

Ω

€

f (E) = f (x)x∈E∑

Compound events Event Probability A P(A) not A P(A’) =1- P(A)

A or B

A and B

A given B

€

P(A∪ B) = P(A) + P(B) − P(A∩ B) = P(A) + P(B) if A and B are mutually exclusive

€

P(A∩ B) = P(AB)P(B) = P(A)P(B) if A and B are independent

€

P(AB) =P(A∩ B)P(B)

Quick test

•  In a two-child family, the older child is a boy. What is the probability that the second child is also a boy?

•  In a two-child family, one child is a boy. What is the probability that the other child is also a boy?

Two Children Problem (from Gardener 1959)

Probability distributions

Probability density functions A random variable (or stochastic variable) X is a measurable function that maps a

probability space Ω into a observation space S. A probability density function (pdf) of an random variable is a function that

describes the relative likelihood for this random variable to occur at a given point in the observation space.

€

X :Ω→ S

PDFs Distributions for discrete random variable:

•  Binomial distribution •  Poisson distribution •  Bernoulli distribution •  Geometric distribution •  Negative binomial distribution •  Discrete uniform distribution Identifies the probability of each value of a random variable.

Distributions for continuous random variable:

•  Gaussian distribution •  Continuous uniform distribution •  Beta distribution •  Gamma distribution

Identifies the probability of the value falling within a particular interval

Mean and variances of random variables

Discrete random variable R The mean µ is: The variance is: The standard deviation is:

Continuous random variable X. The mean µ is: The variance is: The standard deviation is: €

E(R) = r = µ = prrall r∑

€

Var(R) =σ 2 = pr(r −µ)2

all r∑

€

σ = V[R]

E(X) = µ = x f (x) −∞

+∞

∫ dx

€

Var(X) =σ 2 = (x −µ)2 f (x)dx−∞

+∞

∫

€

σ = V[X]

Can you see the commonalities? What is the probability of …

•  A family with four children to have three girls?

•  Getting five time heads in tossing a coin 100 times?

•  Founding five defective NICs in a shipment of 5000 NICs?

•  You getting all the correct answers to 25 questions on statistics at the end of this colloquium?

The binomial distribution A discrete random variable R follows the binomial distribution if:

•  There is a fixed number of trials n; •  Only two outcomes (success or failure), are possible at each trial; •  The trials are independent; •  There is a constant probability p of success at each trial; •  The random variable r is the number of successes in n trials.

€

P(R = r) = pr (1− p)n−r n!r!(n − r)!

Probability of a specific outcome

Number of equivalent permutations for that

outcome

Hands-on #3 You are conducting a simple experiment, that is you are drawing

marbles from a bowl. 1.  Bowl with marbles, a fraction p = 0.5 are black, others are white 2. You draw n=4 marbles from bowl, put marble back after each drawing

What is probability for getting no black marble? Can you get a plot for this out of R? (Hint: look at dbinom)

CDF The Cumulative distribution function (CDF) gives you: •  The probability that a random variable X with a given PDF will

be found at a value less than or equal to x.

•  It is the "area so far" function of the probability distribution.

Hands-on #4 Five percent of the switches produced by a company are defective

or do not operate.  What is the probability that out of thirty switches you have to install, one will be defective?

And the probability that at most one is defective? Hint: look at dbinom and pbinom

Properties of the binomial distribution

•  Mean:

•  Variance:

r = n ⋅ p

V (r) = np(1− p) ⇒ σ = np(1− p)

Related distributions •  Bernoulli distribution

–  We only have one trial (n=1) •  Geometric distribution

–  The variable is the number of trials until success •  Negative binomial distribution

–  The variable is the number of trials until we obtain r successes, i.e the number of failures.

Hands-on #5 You sell network switches. Today you have to sell at least 3, and

you have time to visit at most 12 clients. (Make a realistic assumption on your persuasion power, ie the

probability of selling a switch to a client) Use R to calculate the probability of being done after 4 clients?

(Hint; use dnbinom)

See you next week