Parallel Data Warehouse : The Data Warehouse Consolidation Appliance Lionel Pénuchot

Download Parallel  Data  Warehouse : The Data  Warehouse  Consolidation Appliance  Lionel Pénuchot

Post on 12-Feb-2016

44 views

Category:

Documents

0 download

Embed Size (px)

DESCRIPTION

Parallel Data Warehouse : The Data Warehouse Consolidation Appliance Lionel Pnuchot PDW Center of Excellence. Agenda. Aperu de SQL Server PDW SSAS + PDW: la dream team Consolidation Les principales volutions de lappliance Une dmo peut-tre ?. Configuration HP. - PowerPoint PPT Presentation

TRANSCRIPT

<p>PowerPoint Presentation</p> <p>Les journesSQL Server 2013 Un vnement organis par GUSS#JSS2013Parallel Data Warehouse:The Data Warehouse Consolidation Appliance Lionel PnuchotPDW Center of ExcellenceLes journesSQL Server 2013 Un vnement organis par GUSS#JSS2013Aperu de SQL Server PDWSSAS + PDW: la dream teamConsolidationLes principales volutions de lapplianceUne dmo peut-tre ?Agenda#JSS20133/4 Rack45.3TB (Raw)</p> <p>Reserved Space (9U)Data Integration Platform serverPassive Unit (adds Failover Node)Future expansionReserved Space (8U)Data Integration Platform serverPassive Unit (adds Failover Node)Future expansionInfinibandInfinibandEthernetEthernetControl NodeFailover NodeJBOD 1Compute Node 1Compute Node 2JBOD 2Compute Node 3Compute Node 4JBOD 3Compute Node 5Compute Node 6JBOD 4Compute Node 7Compute Node 8ReservedSpacePDW Backplane (6U):Redundant InfinibandRedundant EthernetManagement and control (Active)Rack Failover Node (Passive)Base Unit (7U):2 HP 1U servers (16 cores/ea. Total: 32)JBOD 5U1 TB drivesUser data capacity: 75 TBScale Unit (7U):2 HP 1U servers (16 cores/ea. Total: 32)JBOD 5U1 TB drivesUser data capacity: 75 TB Rack15.1TB (Raw)</p> <p>1/2 Rack30.2TB (Raw)</p> <p>Scale Unit (7U):2 HP 1U servers (16 cores/ea. Total: 32)JBOD 5U1 TB drivesUser data capacity: 75 TBScale Unit (7U):2 HP 1U servers (16 cores/ea. Total: 32)JBOD 5U1 TB drivesUser data capacity: 75 TBFull Rack60TB (Raw)</p> <p>InfinibandInfinibandEthernetEthernetFailover NodeJBOD 5Compute Node 9Compute Node 10JBOD 6Compute Node 11Compute Node 12JBOD 7Compute Node 13Compute Node 14JBOD 8Compute Node 15Compute Node 16ReservedSpaceExtension Base Unit (5U):Redundant InfinibandRedundant EthernetRack Failover Node (Passive)Extension Base Unit (7U):2 HP 1U servers (16 cores/ea. Total: 32)JBOD 5U1 TB drivesUser data capacity: 75 TBScale Unit (7U):2 HP 1U servers (16 cores/ea. Total: 32)JBOD 5U1 TB drivesUser data capacity: 75TB1 Rack75.5TB (Raw)</p> <p>Scale Unit (7U):2 HP 1U servers (16 cores/ea. Total: 32)JBOD 5U1 TB drivesUser data capacity: 75 TBScale Unit (7U):2 HP 1U servers (16 cores/ea. Total: 32)JBOD 5U1 TB drivesUser data capacity: 75 TB3 Rack181.2TB (Raw)</p> <p>1 1/2 Rack90.6TB (Raw)</p> <p>2 Rack120.8TB (Raw)</p> <p>InfinibandInfinibandEthernetEthernetFailover NodeJBOD 9Compute Node 17Compute Node 18JBOD 10Compute Node 19Compute Node 20JBOD 11Compute Node 21Compute Node 22JBOD 12Compute Node 23Compute Node 24ReservedSpaceExtension Base Unit (5U):Redundant InfinibandRedundant EthernetRack Failover Node (Passive)Extension Base Unit (7U):2 HP 1U servers (16 cores/ea. Total: 32)JBOD 5U1 TB drivesUser data capacity: 75 TBScale Unit (7U):2 HP 1U servers (16 cores/Ea. Total: 32)JBOD 5U1 TB drivesUser data capacity: 75 TBReserved Space (9U)Data Integration Platform serverPassive Unit (adds Failover Node)Future expansionScale Unit (7U):2 HP 1U servers (16 cores/Ea. Total: 32)JBOD 5U1 TB drivesUser data capacity: 75 TBScale Unit (7U):2 HP 1U servers (16 cores/ea. Total: 32)JBOD 5U1 TB drivesUser data capacity: 75 TBConfiguration HP#JSS20131/3 Rack22.6TB (Raw)</p> <p>2/3 Rack45.3TB (Raw)</p> <p>Full Rack67.9TB (Raw)</p> <p>Base Unit (6U):Redundant InfinibandRedundant EthernetManagement and Control (Active)Rack Failover Node (Passive)Base Unit (10U):3 servers in 2U enclosure (16 cores/ea. Total: 48)2 JBOD 4U ea.1 TB drivesUser data capacity: 79 TBScale Unit (10U):3 servers in 2U enclosure (16 cores/ea. Total: 48)2 JBOD 4U ea.1 TB drivesUser data capacity: 79 TBReserved Space (6U)Passive Unit (adds Failover Node)Future expansionInfinibandInfinibandEthernetEthernetControl NodeFailover NodeJBOD 2Compute Node 2Compute Node 3JBOD 1Compute Node 1JBOD 5Compute Node 8Compute Node 9JBOD 6Compute Node 7Scale Unit (10U):3 servers in 2U enclosure (16 cores/ea. Total: 48)2 JBOD 4U ea.1 TB drivesUser data capacity: 79 TBReservedUseJBOD 3Compute Node 5Compute Node 6JBOD 4Compute Node 4JBOD 1Compute Node 2Compute Node 3JBOD 2Compute Node 1 Dell configuration#JSS2013Architecture logicielDtails gnrauxTous les serveurs sont sous Windows Server 2012 StandardToutes les VMs sont sous Windows Server 2012 StandardLes tches de type fabric et workload sont dans des VMsFabric VM, MAD01, and CTL partagent un serveurCela permet de rduire le surcot spcialement pour les petites configurationsLagent PDW tourne sur tous les hosts et toutes les VMsDWConfig and Admin Console existent toujoursLa technologie Windows Storage Spaces gre le mirroring et les secours, ainsi cela permet de rduire les cots en utilisant du DAS (JBODs) plutt que du SAN</p> <p>Dtails du moteur PDWSQL Server 2012 Enterprise Edition (PDW build) sur le control node et sur les compute nodes </p> <p>Dtails sur le stockageSimilaire la V1Double de datafiles par filegroupPlus de disques physiques en parallleDtails logicielsHST02HST01HSA01HSA02JBODIB andEthernetDirect attached SASBase unitCTLMADFADVMMCompute 2Compute 1Window Server 2012 Standard PDW engineDMS ManagerSQL Server 2012 Enterprise Edition (PDW build)Shell databasesWindow Server 2012 Standard DMS Core SQL Server 2012 Enterprise Edition (PDW build)#JSS2013Extensions possiblesDELLBaseActiveComputeCapacity inc.SpareTotalRaw disk: 1TBRaw disk: 3TBCapacityOne third-rack103N/A1522.6567.9579-340 TBTwo thirds116100%1845.3135.9159-680 TBFull rack12950%11167.95203.85238-1019 TBOne &amp; third221233%21590.6271.8317-1359 TBOne &amp; two thirds231525%218113.25339.75396-1699 TBTwo racks241820%221135.9407.7476-2039 TBTwo &amp; a third342117%325158.55475.65555-2378 TBTwo &amp; two thirds352414%328181.2543.6634-2718 TBThree racks362713%331203.85611.55713-3058 TBFour racks483633%441271.8815.4951-4077 TBFive racks5104525%551339.751019.251189-5096 TBSix racks6125420%661407.71223.11427-6116 TBHPBaseActiveCompute Incr.SpareTotalRaw disk: 1TBRaw disk: 3TBCapacityQuarter-rack102N/A1415.145.353-227 TBHalf114100%1630.290.6106-453 TBThree-quarters12650%1845.3135.9159-680 TBFull rack13833%11060.4181.2211-906 TBOne-&amp;-quarter231025%21375.5226.5264-1133 TBOne-&amp;-half241220%21590.6271.8317-1359 TBTwo racks261633%219120.8362.4423-1812 TBTwo and a half372025%324151453529-2265 TBThree racks392420%328181.2543.6634-2718 TBFour racks4123233%437241.6724.8846-3624 TBFive racks5154025%5463029061057-4530 TBSix racks6184820%655362.41087.21268-5436 TBSeven racks7215617%764422.81268.41480-6342 TB256 nodes15 TB1.3 PB raw</p> <p>Up to 6 PB user data</p> <p>23 node increments for small topologies#JSS2013PDW + MOLAPPDW + ROLAPPDW + TABULAR Direct QuerySSAS + PDW: la dream team#JSS2013EnableRolapDistinctCountOnDataSource=1ROLAPDimensionProcessingEffort&gt;=300000Connexion via le client natif SQL Server SNAC 11.Les prrequis SSAS#JSS2013Inventaire des cubes et de leurs Data WarehouseMigration vers PDWRpliquer les tables de dimensionsDistribuer les tables de faitsDistribuer les tables de dimensions changeantesPas dindexCCI (Clustered Column Store Index)Prrequis PDW#JSS2013Gain processingLes temps de processing peuvent tre rduits 10% du temps initial.Le processing tire antirement partie du rseau Infiniband de lapplianceCubes et dimensions inchangsGain stockagePas dindexStockage In-memory columnstore .PDW + MOLAP#JSS2013Gain processingProcessing immdiat</p> <p>Gain stockagePas de donnes stocke par SSASPas dindexStockage In-memory columnstore .</p> <p>PDW + ROLAP#JSS2013Gain processingProcessing immdiatGain stockagePas de donnes stocke par SSASPas dindexClustered column store index. Plusieurs modesDirectQueryInMemoryWithDirectQueryDirectQueryWithinMemory</p> <p>PDW + TABULAR Direct Query#JSS2013Exemple de consolidation 1IB#JSS2013Example de consolidation 2IB#JSS2013Example de consolidation 3IB#JSS2013Appliance UpdateFrquence : tous les 6 mois</p> <p>Les principales volutions de lappliance#JSS2013Rgion ddie HadoopAppliance prassemble et prconfigureMassively Parallel Processing (MPP) jusqu 6 PBsIn-memory columnstore : jusqu 100x plus rapideRgion ddie HadoopRequtes SQL sur donnes relationnelles et HadoopDisponible auprs de HP et Dell</p> <p>SQL Server Parallel Data Warehouse#JSS2013Extension du stockage en petabytesMassively Parallel Processing (MPP) parallelise les requtesMultiple noeuds avec CPU, mmoire et stockage ddisAjout incrmental de HW pour une augmentation quasi linaire en volume et performanceAjout de capacit par incrment dans la mme appliance</p> <p>Scale OUTDes Terabytes aux Multi-Petabytes#JSS201319Box Software</p> <p>CloudAppliancesDiffrentes options de dploiementet solutions hybridesSQL ServerHortonworks Data PlatformParallel Data WarehouseSQL Server for data warehousing in Windows Azure VMsHDInsight for Windows AzureSQL Server Fast Track#JSS2013Explorer les donnes non-relationnellesGre les donnes non-relationnelles100% ApacheSimplicit of WindowsHadoop sous toutes les formes : software, appliance, cloudBig Data avec simplicitNon-relational</p> <p>Windows AzureParallel Data Warehouse</p> <p>Hortonworks Data Platform </p> <p>Hadoop cluster in HDP for Windows and HDInsight#JSS2013Key goal of slide: Describe the three ways a customer can deploy Big Data from Microsoft.</p> <p>Slide talk track:To give customers the Microsoft stack of solutions from software (HDP for Windows, appliance (PDW v2 AU1), cloud (Windows Azure HDInsight). Windows Azure HDInsight Microsofts cloud based Hadoop distribution. It is 100% Apache compatible distribution that is available on Windows Azure.SQL Server Parallel Data Warehouse Parallel Data Warehouse will have a Hadoop region where it will have HDInsight on Windows Server integrated in the appliance.Hortonworks Data Platform For Windows Hortonworks created HDP for Windows to have their Hadoop distribution available as a software offering available on Windows.</p> <p>These 3 Big Data offerings on software, appliance, cloud brings the simplicity of Windows to manage your non-relational data with simplicity.</p> <p>21Intgration relationel et non-relationelRequte relationelle et Hadoop en parallleRequte uniquePa besoin dETL dHadoop vers DWRequte Hadoop laide de T-SQLRequte relationelle + non relationelle SQLRsultatDonnes relationellesPolyBaseRequte integrant PolyBase in SQL PDW#JSS2013Key goal of slide: Describe the three ways a customer can deploy Big Data from Microsoft.</p> <p>Slide talk track:Pioneered in the Jim Gray Systems Labs by David DeWitt, PolyBase is an integrated query processor in SQL Server 2012 Parallel Data Warehouse which represents a breakthrough innovation from traditional query processing to join structured and unstructured data from Hadoop together. Without manual intervention, PolyBase Query Processor can accept a standard SQL query and combine tables from a relational source with tables from a Hadoop source directly through external tables. As well, PolyBase Query Processor parallelizes the ability to import/export data to and from Hadoop giving PDW speed, simplicity, and responsiveness in addressing these new types of queries.</p> <p>Ability to issue standard T-SQL that joins relational data with unstructured data in Hadoop PolyBase rapidly imports/exports data between Hadoop and PDW in parallelPolyBase can query data in Hadoop directly without movement (with external tables)Created in Gray Systems Labs by David DeWitt</p> <p>22Dmo#JSS2013</p> <p>Questions ?#JSS2013Merci nos sponsors</p> <p>#JSS2013http://www.microsoft.com/en-us/sqlserver/solutions-technologies/data-warehousing/pdw.aspxhttp://download.microsoft.com/download/4/2/6/42616D71-3488-46E2-89F0-E516C10F6576/SQL_Server_to_SQL_Server_PDW_Migration_Guide.pdfhttp://blogs.technet.com/b/sql/archive/2013/11/12/sql-server-chez-les-clients-parallel-data-warehouse.aspx</p> <p>Rfrences#JSS2013</p> <p>#JSS2013#JSS2013</p>

Recommended

View more >