kdd for science data analysis issues and examples
Post on 22-Dec-2015
225 views
TRANSCRIPT
Contents
Introduction Data Considerations Brief Case Studies
Sky Survey Cataloging Finding Volcanoes on Venus Biosequence Databases Earth Geophysics Atmospheric Science
Issues and Challenges Conclusion
Data Considerations
Image Data Time-series and sequence data Numerical Vs Categorical values Structured and sparse data Reliability of Data
Brief Case Studies
Sky Survey Cataloging Finding Volcanoes on Venus Earth Geophysics Atmospheric Science Biosequence Databases
Sky Survey Cataloging
The survey consists of 3 terabytes of image data containing an estimated 2 billion sky objects
The basic problem is to generate a survey catalog which records the attributes of each object along with its class: star or galaxy
To achieve this scientists developed the SKICAT system
Reasons why SKICAT was successful
The astronomers solved the feature extraction problem Data mining methods contributed to solving difficult
classification problems Manual approaches were simply not feasible.
Astronomers needed an automated classifier to make the most out of the data
Decision tree methods proved to be an effective tool for finding the important dimensions for this problem
Finding Volcanoes on Venus
Data collected by Magellan spacecraft The first pass of Venus using the left looking radar
resulted in 30,000 1000 x 1000 pixel images To help geologists analyze this data set, the JPL
Adaptive Recognition Tool (JARtool) was developed
Motivation for using Data mining methods
Scientists did not know much about image processing or about the SAR properties. Hence they could easily label images but not design recognizers
There was little variation in illumination and orientation of objects of interest. Hence mapping from pixel space to feature space can be performed automatically
Geologists did not have any other easy means for finding the small volcanoes, hence they were motivated to cooperate by providing training data and other help
Earth Geophysics
Two images taken before and after an earthquake and by repeatedly registering different local regions of the two images, it is possible to infer the direction and magnitude of ground motion due to the earthquake.
Example of a geoscientific data mining system is Quakefinder which automatically detects and measures tectonic activity in the earths crust by examination of Satellite data
Atmospheric Science
Data mining tool used is called CONQUEST Parallel testbeds were employed by Conquest to
enable rapid extraction of spatio-temporal features for content based access.
Some of the goals of the this tool is the development of “learning” algorithms which look for novel patterns, event clusters etc.
Biosequence Databases
The largest DNA database is GENBANK with a database of about 400 million letters of DNA from a variety of organisms
The pressing data mining tasks for biosequence are
Find genes in the DNA sequences of various organisms.
Some of the gene finding programs such as GRAIL, GeneID, GeneParser, Genie use neural nets and other AI or statistical methods
Issues and Challenges
Feature Extraction Minority Classes High degree of Confidence Data mining task Relevant domain Knowledge Scalable machines and Algorithms