introduction to machine...

Machine Learning for Language Technology 2015

PreliminariesUnderstanding and Preprocessing Data

Marina Santinisantinim@stp.lingfil.uu.se

Department of Linguistics and PhilologyUppsala University, Uppsala, Sweden

Autumn 2015Lecture 2: Preliminaries 1

Acknowledgements

• Weka Slides (teaching material*), Wikipedia, MathIsFun and other websites.

* http://www.cs.waikato.ac.nz/ml/weka/book.html

Lecture 2: Preliminaries 2

Outline

– Raw Data and Feature Representation:

• Concepts, instances, attributes

– Digression 1: Pills of Statistics

• Sampling, mean, variance, standard deviation, normalization, standardization, etc.

– Digression2: Data Visualization

• how to read a histogram, scatter plot, etc.

DATA, CONCEPTS, INSTANCES, ATTRIBUTES, FEATURES

Raw Data and Data Representation

What is data?

• Data is a collection of facts, such as numbers, words, measurements, observations or evenjust descriptions of things.

• Data can be qualitative or quantitative.

– Qualitative data is descriptive information (it describes something)

– Quantitative data is numeric information (numbers).

Singular or Plural?

• The singular form of data is "datum”. – Ex: "that datum is very high”

• The plural form of ”datum” is ”data”.• ”data” is plural when it indicates many individual datum

– Ex: "the data are available”

• But ”data” can also refer to collection of facts. In this case it is uncountable and takes the singular verb– Ex: "the data is available”

http://www.theguardian.com/news/datablog/2010/jul/16/data-plural-singular

Qualitative Data

• Categorial values

– Nominal (ex: eye colour)

– Ordinal (ex: street numbers)

Quantitative Data• Quantitative data can also be discrete or

continous.• Discrete data is counted, Continuous data is

measured– Discrete data can only take certain values (like

whole numbers)– Continuous data can take any value (within a

range)

Lecture 2: Preliminaries

Concepts, Instances, and Attributes

Components of the input:

Concepts: kinds of things that can be learned

Instances: the individual, independent examples of

a concept

Attributes: measuring aspects of an instance

The importance of feature selectionand representation

Binary data is a special type of categorical data. Binary data takes only two values.

GETTING TO KNOW YOUR DATA

Missing Data/Values

Types: unknown, unrecorded, irrelevant, etc.

Reasons:

collation of different datasets

measurement not possible

Missing data may have significance in itself (e.g.

missing test in a medical examination)

Most ML schemes assume that missing data have no

special significance. So… be careful and make your

own decisions.

Inaccurate values

Typographical errors in nominal attributes values need

to be checked for consistency

Typographical and measurement errors in numeric

attributes outliers need to be identified

• Noise is any unwanted anomaly in the data.

• In ML the presence of noise may cause difficulties in learning the classes and produce unreliable classifiers.

• Noise can be caused by:

– imprecisions in recording input attributes

– errors in labelling

– etc.

Getting to know the data

Simple visualization tools are very useful

Nominal attributes: histograms

Numeric attributes: graphs

Too much data to inspect? Take a sample!

ARFF FORMATWeka (Waikato Environment for Knowledge Analysis)

Weka Software Packagehttp://www.cs.waikato.ac.nz/ml/weka/

Weka (Waikato Environment for Knowledge Analysis) is developed at University of Waikato in New Zealand.

A collection of state-of-the-art machine learning algorithms and data preprocessing tools.

It is open source. It is written in Java.

Contains implementations of learning algorithms that you can apply to your datasets.

Weka input data formats• General formats:

• Weka: – ARFFAttribute-Relation File format. – It is an ASCII file that describes a list of instances

sharing a set of attributes.Lecture 2: Preliminaries 18

The ARFF format

Sparse data In some applications most attribute values in a

dataset are zero E.g.: word counts in a text categorization problem

ARFF supports sparse data

This also works for nominal attributes (where the first value corresponds to “zero”)

0, 26, 0, 0, 0 ,0, 63, 0, 0, 0, “class A”

0, 0, 0, 42, 0, 0, 0, 0, 0, 0, “class B”

{1 26, 6 63, 10 “class A”}

{3 42, 10 “class B”}

SAMPLINGNORMAL DISTRIBUTIONMEASURES OF CENTRAL TENDENCY

Digression: Pills of Statistics

Population and Sample• Population and Sample

– Population: The whole group of ”things” we want to study• Ex: All students born between 1980 and 2000

– Sample: A selection taken from a larger group (the "population") so that youcan examine it to find out something about the larger group.

• Ex: 100 randomly chosen students students born between 1980 and 2000

In other words: the ’population' is the entire pool from which a statistical sample is drawn. The information obtained from the sample allows statisticians to develophypotheses about the larger population. Researchers gather information from a sample because of the difficulty ofstudying the entire population.

Sampling

• Sampling is a science in itself and there are different methods to sample a population

– Ex: random sampling, stratified sampling, multi-stage sampling, quota sampling, etc.

• The main concern: the sample should be representative of the population.

Distributions

Normal Distribution

• A normal distribution is an arrangement of a data set in which most values cluster in the middle of the range and the rest taper off symmetrically toward either extreme.

Skewness

• When data is "skewed", it shows long tail on one side or the other:

Outliers

• An outlier is an observation point that is distant from other observations.

Measures of Central Tendency

• In a normal distribution, the mean, mode and median are all the same.

Right Skewed Distribution

Negative Skewed Distribution

• The mean is the average of the numbers: a calculated"central" value of a set of numbers. To calculate: Just add up all the numbers, then divide by how manynumbers there are.

Ex: what is the mean of 2, 7 and 9? • Add the numbers: 2 + 7 + 9 = 18 • Divide by how many numbers (i.e. we added 3

numbers): 18 ÷ 3 = 6• The Mean is 6

Median

• The Median is the middle number (in a sortedlist of numbers). To find the Median, placethe numbers you are given in value order and find the middle number. (If there are twomiddle numbers, you average them.)

• Find the Median of {13, 23, 11, 16, 15, 10, 26}.

• Put them in order: {10, 11, 13, 15, 16, 23, 26}

• The middle number is 15, so the median is 15.

• The Mode is the number which appears mostoften in a set of numbers.

• In {6, 3, 9, 6, 6, 5, 9, 3} the Mode is 6 (it occurs most often).

Frequency Table

• Ex of a frequency table:

The mean of a frequency table

• In a frequency table, the mean is calculated by:

– multiply the score and the frequency, add up all the numbers and divide by sum of the frequencies

Mean: Formula

• The x with the bar on top means ”mean of x”

• Σ (sigma) means ”sum up”

• Σ fx means ”sum up all the frequencies times the matching scores”

• Σ f means ”sum up all the frequencies”

Quiz: The mean of a frequency table

• Calculate the mean of the following frequency table usingthe mean formula:

Answers (only one is correct)• 2.05• 5.2• 3.7

MEASURES OF DISPERSION

Measures of Dispersion

• Dispersion is a general term for different statistics that describe how values are distributed around the centre

Measures of Dispersion

• range

• quartiles

• interquartile range

• percentiles

• mean deviation

• variance

• standard deviation

• etc.

• The range is the difference between the lowest and highest values.

– Example: In {4, 6, 9, 3, 7} the lowest value is 3, and the highest is 9. So the range is 9-3 = 6.

Quartiles

• Quartiles are the values that divide a list of numbers intoquarters.– First put the list of numbers in order– Then cut the list into four equal parts– The Quartiles are at the "cuts”

• Example: 1, 3, 3, 4, 5, 6, 6, 7, 8, 8 (The numbers must be in order)

• Cut the list into quarters. The result is:• Quartile 1 (Q1) = 4• Quartile 2 (Q2), which is also the Median = 5• Quartile 3 (Q3) = 8

Interquartile Range• The "Interquartile Range" is from Q1 to Q3.

• To calculate it just subtract Quartile 1 from Quartile 3:

Percentiles

• Percentile is the value below which a percentage of data falls (The data needs to be in order)

• Example: You are the 4th tallest person in a group of 20; 80% of people are shorter than you: That means you are at the 80th percentile.

• That is, if your height is 1.85m then "1.85m" is the 80th percentile height in that group.

Mean Deviation

• It is the mean of the distances of each value from their mean.

• Three steps:

– 1. Find the mean of all values

– 2. Find the distance of each value from that mean (subtract the mean from each value, ignore minus signs, and take the absolute value)

– 3. Then find the mean of those distances

Variance: σ2

• The Variance is the average of the squareddifferences from the mean.

• To calculate the variance follow these steps:– Work out the mean.

– Then for each number: subtract the Mean and square the result (the squared difference).

– Then work out the average of those squareddifferences.

Example: Compute the Variance

For the following dataset find the variance: {600, 470, 170, 430, 300}.

Mean = 600+470+170+430+300/5 = 394

For each number subtract the mean:

600-394=206; 470-394=76, 170-394=224, 430-394=36; 300-394=-94

Take each difference, square it, and then avarage the results. The variance is 21,704.

Standard Deviation: σ

• The Standard Deviation is one of the most reliable measure of how spread out numbers are.

• The formula is easy: it is the square root of the variance.

Standard Deviation Formula (population)

• μ = the mean• xi = the individual value of a

dataset• (xi - μ)

2 = for each value subtract the mean and square the result

• N = the total number of values in the dataset

• i=1 = start at this value (here the first number of the dataset)

• Σ = add up all the values• 1/N = divide by total number of

values in the dataset• √ = take the square root of all the

calculation

Standard Deviation Formula (sample)

Standard Deviation is the most reliable measure of dispersion

• Depending of the situation, not all measures of dispersion are equally reliable.

• For ex, the range can sometimes be misleading when there are extremely high or low values.– Example: In {8, 11, 5, 9, 7, 6, 3616}: the lowest value is 5,

and the highest is 3616. So the range is 3616-5 = 3611.

• However: The single value of 3616 makes the range large, but most values are around 10.

• So we may be better off using other measures such as Standard Deviation = 1262.65

Normal Distribution and Standard Deviation

Standard Deviation vs Variance

• A useful property of the standard deviation is that, unlikethe variance, it is expressed in the same units as the data.

• In other words: the StandDev is expressed in the same unitsas the mean is, whereas the variance is expressed in squareunits. So standard deviation is more intuitive…

• Note that a normal distribution with mean=10 and standDev = 3 is exactly the same thing as a normal distribution with mean=10 and variance = 9.

• Watch out and be clear of what you are using!

Quiz: Standard Deviation

68% of the frequency values of the word “and” in a corpus of email (assume emails have equal length) are between 51 and 64. Assuming this data is normally distributed, what are the mean and standard deviation?

1. Mean = 57; S.D. = 6.5

2. Mean = 57.5 ; S.D. = 6.5

3. Mean = 57.5; S.D. = 13

These notions will be resumed later...

• … when dealing with statistical inference and other statistical methods.

• Standard Deviation Calculator: http://www.mathsisfun.com/data/standard-deviation-calculator.html

NORMALIZATION AND STANDARDISATION

Normalization

• To normalize data means to fit the data within unity, so all the data will take on a value between 0 and 1. Many formulas are available:

• Ex:

Standardization

• Standardization coverts all variables to a common scale and reflects how many standard deviations from the mean that the data point falls

• The number of standard deviations from the mean is also called the "Standard Score", "sigma" or "z-score".

How to standardize

• z is the "z-score" (Standard Score)

• x is the value to be standardised

• μ is the mean

• σ is the standard deviation

Why Standardize?

• It can help us make decisions about our data.

CHARTS AND GRAPHSData Visualization

Weka: Data Visualization

Outline

• Bar chart

• Histogram

• Pie chart

• Line chart

• Scatter plot

• Dot plot

• Box plot

Axes and Coordinates

• The left-right (horizontal) direction is commonly called X or abscissaThe up-down (vertical) direction is commonly called Y or ordinate

• The coordinates are always written in a certain order: the horizontal distance first, then the vertical distance.

Repetition: Read careful this web page: https://www.mathsisfun.com/data/cartesian-coordinates.html

Bar Chart• A Bar Chart (also called Bar Graph) is a

graphical display of data using bars of different heights.

• Bar charts are used to graph categorical data. Example:

Histogram• With continuous data, histograms are used.

• Histograms are similar to bar charts, but a histogram

groups numbers into ranges.

Pie Chart• It is a special chart that uses "slices" to show

relative sizes of data.

• Pie charts have been criticized.

Line Chart

• Line chart is a graph that shows information that is connected in some way (such as change over time).

Scatter plot

• A scatter plot has points that show the relationship between two sets of data.

• Example: each dot shows one person's weight versus their height.

Line of best fit

• Draw a "Line of Best Fit" (also called a "Trend Line") on the scatter plot to predict values that might not on the plot

Correlations

• Scatter plots are useful to detect correlations between the sets of data. – Correlation is Positive when the values increase together

– Correlation is Negative when one value decreases as the other increases

More on scatter plots: https://www.mathsisfun.com/data/scatter-xy-plots.html

Quiz: Scatter Plot

• The correlation seen in the graph at the right would be best described as:

1. high positive correlation

2. low positive correlation

3. high negative correlation

4. low negative correlation

Dot Plot• A dot plot is a graphical display of data using dots.

• It is an alternative to the bar chart, in which dots are used to depict the quantitative values (e.g. counts) associated with categorical variables.

Box Plot

• Box plots are useful to highlight outliers, median and the interquartile range.

• aka box-and-whisker plots

The End

introduction to machine...

Documents

friday preliminaries

ch1 preliminaries

export preliminaries

cpa1415 00 preliminaries

v. megalooikonomou preliminaries

6 preliminaries

chapter 7: c language preliminaries 1 chapter 7 c language...

ruskin preliminaries

statistical preliminaries

preliminaries - weebly

section 1 bill no 1 preliminaries - sanbi | … ·...

1 preliminaries - information technology...

preliminaries - wiley

chapter 1 (preliminaries)

mathematical preliminaries strings and languages...

dissertation preliminaries

01 preliminaries

0 preliminaries

final preliminaries

preliminaries & language