1
AgendaAgenda
Data and System TrendsData and System Trends
ChallengesChallenges
ObservationsObservations
ConclusionConclusion
2
19731973Winchester Winchester
DriveDrive
The Big TrendsThe Big Trends
1986198650 50
MegabyteMegabytes s
198219823 ½ Floppy Disk3 ½ Floppy Disk
19831983CD ROMsCD ROMs
19671967First Floppy DiskFirst Floppy Disk
197219725 ¼ Floppy5 ¼ Floppy
StorageStorage
1984198410 MB HD10 MB HD
2000’s2000’sTerabytesTerabytes
1990’s1990’sGigabytesGigabytes
20052005
4
Increased Data Increased Data Under ManagementUnder Management
MasterMasterFilesFiles
SalesSalesDataData
IWIWFilesFiles PicturesPictures
MusicMusic
Product Product Design Design
DataData
ConversationsConversations
MasterMasterTapesTapes
1960’s1960’s 1970’s1970’s 1980’s1980’s 1990’s1990’s 2000’s2000’s
ElectronicElectronicTradesTrades
GISGIS
LocationsLocations
5
TransactionTransactionProcessingProcessing
OLTPOLTP
InternetInternet
XMLXML
ReportingReporting
DataDataProcessingProcessing
1960’s1960’s 1970’s1970’s 1980’s1980’s 1990’s1990’s 2000’s2000’s
CommunicationsCommunications
Reengineering IT: Reengineering IT: EfficiencyEfficiencyWhere have the gains come from?Where have the gains come from?
B2B ServicesB2B Services
Knowledge RetentionKnowledge Retentionand Retrievaland Retrieval
Reduce LatencyReduce LatencyIn Human ProcessesIn Human Processes
6
Data Moving UpstreamData Moving Upstream
Post Post TransactionalTransactional
SummarySummaryAfter the After the factfact
TransactionalTransactional
Record of factRecord of factEnables Enables AnalyticsAnalytics
Pre-Pre-TransactionalTransactional
Record of Record of fact and fact and intentintentEnables Enables deeper deeper insightinsight
7
The Data MonolithThe Data Monolith
Previous 30 years Previous 30 years trended towards trended towards data consolidationdata consolidation
SOA, connected SOA, connected systems, common systems, common schema, pulling schema, pulling data apartdata apart
Data ProcessingData Processing
OLTPOLTP
EAEA
InternetInternet
B2B ServicesB2B Services
DevicesDevices
SOASOA
Dat
a C
onve
rgen
ce
Dat
a C
onve
rgen
ce
Data D
istribution
Data D
istribution
8
Trends Recap…Trends Recap…Huge Gains in PSIHuge Gains in PSI
ProcessingProcessingStorageStorageI/OI/O
Exponential Improvement in all of Exponential Improvement in all of the abovethe aboveData now online and available with Data now online and available with schemaschema
Need to derive value from itNeed to derive value from it
Human costs now dominate Human costs now dominate solutions!solutions!The data monolith is getting broken The data monolith is getting broken apartapart
9
ChallengesChallenges
Data Challenges:Data Challenges:ProgrammingProgramming
DistributingDistributing
Getting ValueGetting Value
Relating and OrganizingRelating and Organizing
SharingSharing
10
Challenge: Programming Challenge: Programming DataDataDifferent EnvironmentsDifferent Environments
Data query environment distinct from Data query environment distinct from rest of programrest of program
No Intellisense, late bound, verbose…No Intellisense, late bound, verbose…
T-SQL versus .NET LanguagesT-SQL versus .NET Languages
““Impedance Mismatch” between Impedance Mismatch” between code and databasecode and database
TransactionsTransactions
Nulls - 3-value logicNulls - 3-value logic
Normalized DataNormalized Data
Declarative QueriesDeclarative Queries
TransparencyTransparency
Exception HandlingException Handling
““Different” nullsDifferent” nulls
ObjectsObjects
Imperative operationsImperative operations
EncapsulationEncapsulation
Database WorldDatabase World Programming WorldProgramming World
The LINQ ProjectThe LINQ Project
StandardStandardQueryQuery
OperatorsOperators
ObjectsObjects
DLinqDLinq(ADO.NET)(ADO.NET)
XLinqXLinq(System.Xml)(System.Xml)
<book> <title/> <author/> <year/> <price/></book>
XMLXML
.NET Language Integrated Query.NET Language Integrated Query
C#C# VBVB Others…Others…
SQLSQL WinFWinFSS
TLN306 – Wed, 1:45PMTLN306 – Wed, 1:45PMDAT323 – Thu, 2:15PMDAT323 – Thu, 2:15PMDAT324 – Fri, 10:30AMDAT324 – Fri, 10:30AM
12
SqlConnection c = new SqlConnection(…);SqlConnection c = new SqlConnection(…); c.Open(); c.Open(); SqlCommand cmd = new SqlCommand(SqlCommand cmd = new SqlCommand( @"SELECT c.Name, c.Phone@"SELECT c.Name, c.Phone
FROM Customers cFROM Customers c WHERE c.City = @p0"WHERE c.City = @p0"
);); cmd.Parameters.AddWithValue("@po", cmd.Parameters.AddWithValue("@po", ""LondonLondon""); ); DataReader dr = c.Execute(cmd); DataReader dr = c.Execute(cmd); while (dr.Read()) {while (dr.Read()) { string name = dr.GetString(0);string name = dr.GetString(0); string phone = dr.GetString(1);string phone = dr.GetString(1); DateTime date = dr.GetDateTime(2);DateTime date = dr.GetDateTime(2); }} dr.Close();dr.Close();
Data Access Data Access in APIs Todayin APIs Today
Queries in Queries in quotesquotes
Arguments Arguments loosely loosely boundbound
Results Results loosely typedloosely typed
Compiler Compiler cannot help cannot help
catch catch mistakesmistakes
13
Data Access Data Access in DLinqin DLinq
public class Customerpublic class Customer{{ public int Id;public int Id; public string Name;public string Name; public string Phone;public string Phone; … …}}
Table<Customer> customers = db.Customers;Table<Customer> customers = db.Customers;
var contacts =var contacts = from c in customersfrom c in customers where c.City == "London"where c.City == "London" select new { c.Name, c.Phone };select new { c.Name, c.Phone };
Classes Classes describe describe
datadataTables are Tables are collectionscollections
Query is Query is natural part natural part
of the of the languagelanguage
The compiler The compiler helps you helps you
outout
14
ArchitectureArchitecture
LINQ Query Objects SubmitChanges()
SQL Query Rows SQL or Stored Procs
DLinq(ADO.NET)
SQLServer
from c in db.Customersfrom c in db.Customerswhere c.City == "London"where c.City == "London"selectselect new { c.Name, c.Phone } new { c.Name, c.Phone }
select Name, Phoneselect Name, Phonefrom customersfrom customerswhere city = 'London'where city = 'London'
Application
Services:Services:- Change tracking- Change tracking- Concurrency control- Concurrency control- Object identity- Object identity
15
Hello World “DLinq” StyleHello World “DLinq” Style
Dinesh KulkarniDinesh KulkarniProgram ManagerProgram ManagerC#C#
16
ADO.NET vNextADO.NET vNext
Pablo CastroPablo CastroProgram ManagerProgram ManagerDataworksDataworks
17
Challenge: Distributing Challenge: Distributing DataData
Need data everywhereNeed data everywherePDA → Desktop → App Server → EnterprisePDA → Desktop → App Server → Enterprise
Need synchronization/replicationNeed synchronization/replication
Need reliable messaging between Need reliable messaging between storesstores
Need reference data everywhereNeed reference data everywhereContactsContacts
CatalogsCatalogs
Price listsPrice lists
18
Challenge: Value From Challenge: Value From DataData
Search versus QuerySearch versus Query
Information from existing dataInformation from existing dataAnalyticsAnalytics
ReportingReporting
Managing the data lifecycleManaging the data lifecycle
19
Search Versus QuerySearch Versus QueryShow pictures of Show pictures of
people who attended people who attended Eric’sEric’s
2004 Birthday Party2004 Birthday Party
QueryQuery
SearchSearch
Eric Eric andand Birthday Birthday
20
PortalPortal(Office Server)(Office Server)
Integrated OfferingIntegrated OfferingOfficeOffice
SQLSQL
Data WarehouseData Warehouse(SQL RDBMS)(SQL RDBMS)
IntegrateIntegrate(SSIS)(SSIS)
AnalyzeAnalyze(SSAS)(SSAS)
ReportReport(SSRS)(SSRS)
Business ScorecardingBusiness Scorecarding(BSM)(BSM)
End-user AnalysisEnd-user Analysis(Excel + Excel (Excel + Excel
Server)Server)
BIBIPlatformPlatform
AnalyticAnalyticApplicationsApplications
End-userEnd-userToolsTools
DAT315 – Thu, 10:00AMDAT315 – Thu, 10:00AMOFF 323 – Fri, 10:30AMOFF 323 – Fri, 10:30AM
21
Challenge: Exploiting Challenge: Exploiting Rich DataRich Data
How to get more value from rich dataHow to get more value from rich data
AsAsOffice documents content more Office documents content more accessibleaccessible
New XML formats for Office “12”New XML formats for Office “12”
More intrinsic metadata becomes More intrinsic metadata becomes availableavailable
We enable data sharingWe enable data sharing
22
Challenges: Exploiting Rich Challenges: Exploiting Rich DataData
UnifyUnifyIt’s not just about filesIt’s not just about files
OrganizeOrganizeIt’s not just about folders (or tables)It’s not just about folders (or tables)
ExploreExploreIt’s not just about searchIt’s not just about search
InnovateInnovateIt’s about a platformIt’s about a platform
23
DAT 209 – Wed, 1:45PMDAT 209 – Wed, 1:45PMDAT 209 – Thu, 5:15PMDAT 209 – Thu, 5:15PMDAT 310 – Wed, 3:15PMDAT 310 – Wed, 3:15PMDAT 312 – Wed, 5:00PMDAT 312 – Wed, 5:00PMADO.NETADO.NET
WinFSWinFS
Item Data Model
Query
Services
File Services Metadata Handlers,…
OperationsBackup, AV,…
Item Association Extension
Schemas
Tasks Media
Docs …
SyncAdapters, …
People Mail
File Services Metadata Handlers,…
OperationsBackup, AV,…
SyncAdapters, …
Update
Logic
“DLinq”
Unified APIUnified APIADO.NET support – query, update, app ADO.NET support – query, update, app logic, language integrated query, …logic, language integrated query, …
It’s Still a File SystemIt’s Still a File SystemFS Semantics, Security, Manageability, FS Semantics, Security, Manageability, Backwards Compatible with Win32 Backwards Compatible with Win32
OperationsOperationsBackup/Restore, AV, …Backup/Restore, AV, …
SyncSyncMulti-master SyncMulti-master Sync
New Windows TypesNew Windows Types “ “Everyday info” schemas + ISV Everyday info” schemas + ISV extensionsextensions
Structured “Item” Store Structured “Item” Store Files & Folders Files & Folders Items & Associations Items & Associations
Integrated storage componentsIntegrated storage componentsLeverage investments in DB Leverage investments in DB technologies, CLR, NTFS supporttechnologies, CLR, NTFS support
Unified APIUnified APIADO.NET support – query, update, app ADO.NET support – query, update, app logic, language integrated query, …logic, language integrated query, …
It’s Still a File SystemIt’s Still a File SystemFS Semantics, Security, Manageability, FS Semantics, Security, Manageability, Backwards Compatible with Win32 Backwards Compatible with Win32
OperationsOperationsBackup/Restore, AV, …Backup/Restore, AV, …
SyncSyncMulti-master SyncMulti-master Sync
New Windows TypesNew Windows Types “ “Everyday info” schemas + ISV Everyday info” schemas + ISV extensionsextensions
Structured “Item” Store Structured “Item” Store Files & Folders Files & Folders Items & Associations Items & Associations
Integrated storage componentsIntegrated storage componentsLeverage investments in DB Leverage investments in DB technologies, CLR, NTFS supporttechnologies, CLR, NTFS support
What is WinFS?What is WinFS?
File Services Metadata Handlers,…
OperationsBackup, AV,…
SyncAdapters, …
Database Engine
NTFS CLR
25
ConclusionConclusion
Data Management value moving -Data Management value moving -From: base services – store, query, From: base services – store, query, retainretain
To: semantic services – mine, analyze, To: semantic services – mine, analyze, repurposerepurpose
We can make it We can make it muchmuch easier to easier to program against dataprogram against data
We will make it much easier for you We will make it much easier for you to:to:
Interact with dataInteract with data
Program against dataProgram against data
Derive value from dataDerive value from data