mis 111: computers and the inter-networked society class 11: data mining july 25th, 2011
Post on 19-Dec-2015
218 views
TRANSCRIPT
![Page 1: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/1.jpg)
MIS 111: Computers and the Inter-networked Society
Class 11: Data Mining
July 25th, 2011
![Page 2: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/2.jpg)
![Page 3: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/3.jpg)
This Week Data Mining (Today):
humans and machines generate knowledge from data
Decision Support Systems (Tuesday) combining models and data in an attempt to
solve semistructured and some unstructured problems with extensive user involvement
Expert Systems (Machine’s that make decisions) Computer systems that attempt to mimic human
experts by applying expertise in a specific domain.
![Page 4: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/4.jpg)
Learning Objectives List a few current events in information
systems news Recap of quiz 2 learning objectives Use Google analytics to perform data mining
and make business decisions List 3 practical applications of data mining Explain the difference between Descriptive
and Predictive data mining Compare and contrast classification,
association rule, deviation detection data mining
List a tool that can help you perform data mining
![Page 5: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/5.jpg)
Administrative Trivia Quiz
Some people didn’t put their name on their quiz
If you have a zero, come talk with me We’ll go over it together today
Assignment 3 due Wednesday morning before class
![Page 6: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/6.jpg)
Quiz 2 Recap http://eller.qualtrics.com/SE/?SID=SV_eaGLk1J
Gq9kczBy
![Page 7: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/7.jpg)
Data Mining
![Page 8: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/8.jpg)
What exactly IS Data Mining?
Roughly speaking, Data Mining is the process by which humans and machines generate knowledge from data. Data Warehouse
Data Processing
Data Analytics
![Page 9: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/9.jpg)
What is Data Mining?
Many Definitions Non-trivial extraction of implicit, previously
unknown and potentially useful information from data
Exploration & analysis, by automatic or semi-automatic means, of large quantities of data in order to discover meaningful patterns
![Page 10: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/10.jpg)
![Page 11: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/11.jpg)
Knowledge Discovery in Databases
Data Mining is only a small part of the knowledge discovery process. Which part of the process do you think is most
critical? Which part of the process do you think takes
longest?
![Page 12: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/12.jpg)
What is (not) Data Mining?
What is Data Mining?
– Certain names are more prevalent in certain US locations (O’Brien, O’Rurke, O’Reilly… in Boston area)
– Group together similar documents returned by search engine according to their context (e.g. Amazon rainforest, Amazon.com,)
What is not Data Mining?
– Look up phone number in phone directory
– Query a Web search engine for information about “Amazon”
![Page 13: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/13.jpg)
Lots of data is being collected and warehoused Web data, e-commerce purchases at department/
grocery stores Bank/Credit Card
transactions
Computers have become cheaper and more powerful Competitive Pressure is Strong
Provide better, customized services for an edge (e.g. in Customer Relationship Management)
Why Mine Data? Commercial Viewpoint
![Page 14: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/14.jpg)
Why Mine Data? Scientific Viewpoint
Data collected and stored at enormous speeds (GB/hour) remote sensors on a satellite telescopes scanning the skies microarrays generating gene
expression data scientific simulations
generating terabytes of data Traditional techniques infeasible for raw data Data mining may help scientists
in classifying and segmenting data in Hypothesis Formation
![Page 15: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/15.jpg)
Evolution of Data AnalysisEvolutionary Step
Data Collection(1960s)
Data Access(1980s)
DataWarehousing &Decision Support(1990s)
Data Mining(2000s)
Business Question
"What was my total
revenue in the last
five years?"
"What were unitsales in NewEngland last
March?"
"What were unitsales in NewEngland last
March? Drill downto Boston."
"What’s likely tohappen to Boston
unit sales nextmonth? Why?"
EnablingTechnologies
Computers, tapes,disks
Relationaldatabases(RDBMS)
On-line analyticprocessing(OLAP),
multidimensionaldatabases, data
warehouses
Advancedalgorithms,
multiprocessorcomputers, massive
databasesinformation
delivery
![Page 16: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/16.jpg)
In what disciplines do people use data mining?
![Page 17: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/17.jpg)
Google Analytics
![Page 18: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/18.jpg)
AnimalLingo.com Google Analytics What in here is data mining:
Map Overlay Time series analysis New visits Bounce rate Time on site …
What changes should I make to my Web site (this is getting into the role of decision support systems)?
![Page 19: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/19.jpg)
E-commerce
![Page 20: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/20.jpg)
http://www.amazon.com/
![Page 21: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/21.jpg)
Finance
![Page 22: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/22.jpg)
Finance Basic: Finance.yahoo.com Can be much, much more complex (I should
have a finance PhD student come in) IS data mining and finance are a great mix!
Some examples ahead (you don’t have to know these unless you want to)
![Page 23: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/23.jpg)
Finance: Portfolio Management (FYI)1. Collect 30- 40 historical fundamental and technical factors for stock
S1, say for 10-20 years.2. Build a neural network NN1 for predicting the return values for stock
S1.3. Repeat steps 1 and 2 for every stock Si,that is monitored by the
investor. Say 3000 stocks are monitored and 3000 networks, NNi are generated.
4. Forecast stock return Si (t + k)for each stock i and k days ahead (say a week, seven days) by computing NNi(Si(t))=S(t+k).
5. Select n highest Si (t + k)values of predicted stock return.6. Compute a total forecasted return of selected stocks, T and compute
Si(t+k)/T. Invest to each stock proportionally to Si(t+k)/T.DATA MINING FOR FINANCIAL APPLICATIONS 13
7. Recompute NNi model for each stock i every k days adding new arrived data to the training set. Repeat all steps for the next portfolio adjustment.
http://www.math.nsc.ru/AP/ScientificDiscovery/PDF/data_mining_for_financial_applications.pdf
![Page 25: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/25.jpg)
Interpretable Trading Rules (FYI) Categorical rules predict a categorical
attribute, such as increase/decrease, buy/sell.
![Page 26: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/26.jpg)
Discovering Fraud (FYI) http://www.picalo.org/
![Page 27: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/27.jpg)
Sports
![Page 28: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/28.jpg)
Sports and Data Mining Go for it on forth down!
http://www.advancednflstats.com/2009/09/4th-down-study-part-1.html
![Page 29: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/29.jpg)
Moneyball: The Art of Winning an Unfair Game
![Page 30: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/30.jpg)
The Main Message of Moneyball
By analyzing baseball statistics you could see through a lot of baseball nonsense.
For instance, when baseball managers talked about scoring runs, they tended to focus on team batting average, but if you ran the analysis you could see that the number of runs a team scored bore little relation to that team's batting average. It correlated much more exactly with a team's on-base and slugging percentages.
![Page 31: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/31.jpg)
Other applications
![Page 32: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/32.jpg)
Just some examples… Linguistics Economics Farming Government Defense / homeland ssecurity Education Production forecasting Sales forecasting Fast food Just about ANY DISCIPLINE can benefit from
data mining
![Page 33: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/33.jpg)
Data Mining Tasks
![Page 34: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/34.jpg)
Data Mining Tasks Prediction Methods
Use some variables to predict unknown or future values of other variables.
Description Methods Find human-interpretable patterns that describe
the data.
From [Fayyad, et.al.] Advances in Knowledge Discovery and Data Mining, 1996
![Page 35: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/35.jpg)
Data Mining Tasks... Classification [Predictive]
Clustering [Descriptive]
Association Rule Discovery [Descriptive]
Sequential Pattern Discovery [Descriptive]
Regression [Predictive]
Deviation Detection [Predictive]
![Page 36: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/36.jpg)
Lots of tools to help: Weka R SPSS SAS Google Picalo Google Correlate / Graphs
![Page 37: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/37.jpg)
Classification
![Page 38: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/38.jpg)
Classification: Definition Given a collection of records (training set )
Each record contains a set of attributes, one of the attributes is the class.
Find a model for class attribute as a function of the values of other attributes.
Goal: previously unseen records should be assigned a class as accurately as possible. A test set is used to determine the accuracy
of the model. Usually, the given data set is divided into training and test sets, with training set used to build the model and test set used to validate it.
![Page 39: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/39.jpg)
Classification Example
Tid Refund MaritalStatus
TaxableIncome Cheat
1 Yes Single 125K No
2 No Married 100K No
3 No Single 70K No
4 Yes Married 120K No
5 No Divorced 95K Yes
6 No Married 60K No
7 Yes Divorced 220K No
8 No Single 85K Yes
9 No Married 75K No
10 No Single 90K Yes10
cate
gorica
l
cate
gorica
l
contin
uous
class
Refund MaritalStatus
TaxableIncome Cheat
No Single 75K ?
Yes Married 50K ?
No Married 150K ?
Yes Divorced 90K ?
No Single 40K ?
No Married 80K ?10
TestSet
Training Set
ModelLearn
Classifier
![Page 40: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/40.jpg)
FYI: LOTS of different Classification Algorithms
Neural network Mimics the way
that humans Learn with NEURONS
Decision trees K-means
clustering
![Page 41: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/41.jpg)
Classification: The Iris Flower Data Set Which factors help us determine which Iris
type a flower will be?
Petal Length Petal Width Sepal Length Sepal Width
We can make the machine “learn” which attributes = which iris types.
![Page 43: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/43.jpg)
Classification: A Business Application Direct Marketing
Goal: Reduce cost of mailing by targeting a set of consumers likely to buy a new cell-phone product.
Approach: Use the data for a similar product introduced before. We know which customers decided to buy and which
decided otherwise. This {buy, don’t buy} decision forms the class attribute.
Collect various demographic, lifestyle, and company-interaction related information about all such customers.
Type of business, where they stay, how much they earn, etc.
Use this information as input attributes to learn a classifier model.
From [Berry & Linoff] Data Mining Techniques, 1997
![Page 44: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/44.jpg)
Association Rule
![Page 45: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/45.jpg)
Association Rule Discovery: Definition Given a set of records each of which contain some
number of items from a given collection; Produce dependency rules which will predict
occurrence of an item based on occurrences of other items.
TID Items
1 Bread, Coke, Milk
2 Beer, Bread
3 Beer, Coke, Diaper, Milk
4 Beer, Bread, Diaper, Milk
5 Coke, Diaper, Milk
Rules Discovered: {Milk} --> {Coke} {Diaper, Milk} --> {Beer}
Rules Discovered: {Milk} --> {Coke} {Diaper, Milk} --> {Beer}
![Page 46: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/46.jpg)
Association Rule Discovery: A Business Application
Supermarket shelf management. Goal: To identify items that are bought
together by sufficiently many customers.
Approach: Process the point-of-sale data collected with barcode scanners to find dependencies among items.
A classic rule -- If a customer buys diaper and milk,
then he is very likely to buy beer.So, don’t be surprised if you find six-
packs stacked next to diapers!
![Page 47: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/47.jpg)
Application of Association Rule Discovery: Shelf-Management
![Page 48: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/48.jpg)
Correlations and time http://correlate.googlelabs.com/tutorial Tree IPad and Apple stock
![Page 49: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/49.jpg)
Any problems with the association rule? Correlation does not cause causation
![Page 50: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/50.jpg)
Deviation/Anomaly Detection
![Page 51: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/51.jpg)
Deviation/Anomaly Detection
Detect significant deviations from normal behavior Applications:
Credit Card Fraud Detection
Network Intrusion Detection
Typical network traffic at University level may reach over 100 million connections per day
![Page 52: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/52.jpg)
Deviation/Anomaly Detection What kinds of deviations from normal
behavior might you want to mine for?
For Credit Card Fraud Detection Picalo!
For Network Intrusion Detection
![Page 53: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/53.jpg)
Applications and Issues
![Page 54: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/54.jpg)
Recommender Systems
Use Data Mining techniques to recommend products/services based on user behavior
Use Data Mining techniques to recommend products/services based on user specification
![Page 55: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/55.jpg)
Data Mining: Some Concerns
Is your interpretation valid?
Do you have enough data to process for “good” results?
Finally… should the government be able to data mine?
![Page 56: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/56.jpg)
Data Mining is NOT MAGIC• Data Mining will not do any of the following:
• Automatically find answers to questions you do not ask
• Constantly monitor your database for new and interesting relationships
• Eliminate the need to understand your business and your data
• Remove the need for good data analysis skills
![Page 57: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/57.jpg)
“The Great Giveaway” 1. In what way did Amazon, Ebay, and Google “expose
themselves?”
2. Why did they make the decision to open their doors?
3. How might this move “change business in ways they don’t understand?”
4. How are “hacker entrepreneurs” taking advantage of this move?
5. Do you think that this move was a smart one for Amazon, Google, and Ebay? Why or why not?
6. Pretend that you’re a “hacker entrepreneur. What kind of application are you going to develop with all of that data?
![Page 58: MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d2c5503460f94a02972/html5/thumbnails/58.jpg)
Tomorrow’s class Read:
http://en.wikipedia.org/wiki/Decision_support_system
Bring your computer and excel (meet here and we’ll move to a room with computers halfway through class)