sailing on the ocean of 1s and 0s
TRANSCRIPT
Sailing on the Ocean of 1's and 0's
Chris Woodruff
Chris [email protected] – http://chriswoodruff.comTechnical Architect -- Perficient
• Coordinator, Grand Rapids DevDay• INETA Director• Co-host of Deep Fried Bytes Tech Podcast – http://deepfriedbytes.com
Where are we sailing today?• Lets look at Data• Go on to making Data
valuable• Look at ways to share
Data• Finally lets talk about
making Data look good
Science Paradigms
• 1000’s Years Ago– Science was empirical– Describing
• 100’s Years Ago– Theoretical– Using Models
• Last Few Decades– Computational– Simulations
• Today (eScience)– Data Exploration– Unified Theory– Data Generated by
Instruments or Simulations
– Scientists Analyzes data after curated
from The Fourth Paradigm: Data-Intensive Scientific Discovery
Before we get into the water lets talk about the Digital Ocean
The Internet
Why the Internet Won
• Simple architecture - HTML, URI, HTTP• Networked - value grows with data, services, users• Extensible - from Web of documents to ...• Tolerant - even w/ imperfect mark-up, data, links,
software• Universal - independent of systems and people• Free / cheap - browsers, information, services• Simple / powerful / productive for users - text, graphics,
links• Open standards
What is Data?
The term data refers to qualitative or quantitative attributes of a variable or set of variables. Data (plural of "datum") are typically the results of measurements and can be the basis of graphs, images, or observations of a set of variables. Data are often viewed as the lowest level of abstraction from which information and then knowledge are derived. Raw data, i.e. unprocessed data, refers to a collection of numbers, characters, images or other outputs from devices that collect information to convert physical quantities into symbols.
What really is Data?
Information that has no meaning or
understanding.
What is Data Really?
Where is Data produced?
HOW MUCH DATA IS GENERATED ON INTERNET EVERY
YEAR/MONTH/DAY?
HOW MUCH DATA IS MOVED ON INTERNET EVERY MONTH/DAY?
• 21 exabytes per a month
• Around 675 petabytes per a day
The amount of data produced each year would fill 37,000 libraries the size of the Library of Congress. (2003)
Exabyte == a quintillion (or a million trillion) bytes or units of computer data. One exabyte is equivalent to 50,000 years’ worth of DVD-quality data.
TWITTER USERS ARE AVERAGING 27.3 MILLION TWEETS PER DAY WITH AN ANNUAL RUN RATE OF 10 BILLION TWEETS
According to data from Pingdom
HOW MUCH DATA DOES TWITTER PRODUCE?
How much Data is Facebook generating?
More than 30 billion pieces of content (web links, news stories, blog posts, notes, photo albums, etc.) shared each month.
Average user creates 90 pieces of content each month
INTERNET USERS ARE GENERATING PETABYTES OF DATA EVERY DAY
How much Data does your organization
produce?
Curating Data
Definition
“Data curation is the selection, preservation, maintenance, collection and archiving of digital assets.”
What is involved in D Curation?
• Collecting verifiable digital assets• Providing digital asset search and
retrieval• Certification of the trustworthiness and
integrity of the collection content• Semantic and ontological continuity and
comparability of the collection content
Challenges of D Curation
• Storage format evolution and obsolescence
• Rate of creation of new data and data sets
• Broad access and searching flexibility and variety
• Comparability of semantic and ontological definitions of data sets
Setting up a Curation Process
• Identify what data you need to curate• Identify who will curate the data• Define the curation workflow• Identity the most appropriate data-in and
data-out formats• Identify the artifacts, tools, and processes
needed to support the curation process
Tools to Curate Data
Physical• SQL Databases• Wiki’s• SharePoint• Data Warehouses
Collaborative• DBPedia• Azure Datamarket
Semantics!!
“Open” Data
Semantic Web• XML provides an elemental syntax for
content structure within documents, yet associates no semantics with the meaning of the content contained within.
• XML Schema is a language for providing and restricting the structure and content of elements contained within XML documents.
• RDF is a simple language for expressing data models, which refer to objects ("resources") and their relationships.
• RDF Schema extends RDF and is a vocabulary for describing properties and classes of RDF-based resources, with semantics for generalized-hierarchies of such properties and classes.
• OWL adds more vocabulary for describing properties and classes: among others, relations between classes (e.g. disjointness), cardinality (e.g. "exactly one"), equality, richer typing of properties, characteristics of properties (e.g. symmetry), and enumerated classes.
• SPARQL is a protocol and query language for semantic web data sources.
Open Data Protocol (OData) The Open Data Protocol (OData) enables the
creation of HTTP-based data services, which allow resources identified using Uniform Resource Identifiers (URIs) and defined in an abstract data model, to be published and edited by Web clients using simple HTTP messages.
The Key to “Open Data”?
• Shared Agreed upon Protocols• Metadata• Shared Vocabularies
Visualization of Data
Think about your Data
Produce Great Graphical Information
Minard's Diagram of Napoleon's March on Moscow
Have Integrity in your Graphical Information
Edward Tufte’sThe Lie Factor
Have Context with your Graphical Information
Use less “Ink”
Get Rid of the Junk
Thanks Dave Giard!!!
Examples of Great Visual Data
Data Experience (DX)
Wrap Up
• Think about your data• Learn more about how your users work with
the data you curate• Learn about better ways to share your data• Visualize and show the information your data
best for your users• Be a Data Experience Expert
Required Reading
The Fourth Paradigm: Data-Intensive Scientific Discovery
Required Reading
The Visual Display of Quantitative Information
Required Reading
Beautiful Visualization: Looking at Data through the Eyes of Experts
Discussions