scale-out with ssis 2017 - · pdf filesql pass austria sqlsaturday vienna organizer ... ssis...
TRANSCRIPT
![Page 2: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/2.jpg)
THANK YOU TO OUR SPONSORSPLATINUM
GOLD SILVER
![Page 3: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/3.jpg)
PARTICIPATING COMMUNITIESCLOUD PRO PT
![Page 4: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/4.jpg)
About me – Wolfgang Strasser
Consultant
Software, Business Intelligence and DWH
SQL Server, SSIS, SSAS, SSRS, Power BI, Azure, …
SQL Pass Austria
SQLSaturday Vienna Organizer
@wstrasser
wolfgang.strasser_AT_gmx.at
workingondata.wordpress.com
![Page 5: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/5.jpg)
DTS SSIS
SSIS Catalog
Project Deployment Model
Custom Logging Levels
SSISDB
https://static.pexels.com/photos/320265/pexels-photo-320265.jpeg
Incremental Package Deployment
Control Flow Parts
AutoAdjustBufferSize
The Story of SSIS
![Page 6: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/6.jpg)
htt
ps:
//d
ocs
.mic
roso
ft.c
om
/en-
us/
sql/
inte
grat
ion
-ser
vice
s/w
hat
-s-n
ew-i
n-i
nte
grat
ion
-ser
vice
s-in
-sq
l-se
rver
-20
17
![Page 7: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/7.jpg)
https://blogs.msdn.microsoft.com/ssis/2017/05/17/ssis-helsinki-is-available-in-sql-server-vnext-ctp2-1/
![Page 8: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/8.jpg)
dtexec
package mode
No SSIS catalog support
No third-party components
![Page 9: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/9.jpg)
htt
ps:
//d
ocs
.mic
roso
ft.c
om
/en-
us/
sql/
inte
grat
ion
-ser
vice
s/w
hat
-s-n
ew-i
n-i
nte
grat
ion
-ser
vice
s-in
-sq
l-se
rver
-20
17
![Page 10: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/10.jpg)
Scalability
![Page 11: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/11.jpg)
Scale-What?
Scalability is the capability of a system, network, or process to handle a growing amount of work,
or its potential to be enlarged to accommodate that growth.(Bondi, André B. (2000). Characteristics of scalability and their impact on performance. Proceedings of the second international workshop on Software and performance – WOSP '00. p. 195. )
![Page 12: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/12.jpg)
Scalability
Scalability is the capability of a system, network, or process to handle a growing amount of work,
or its potential to be enlarged to accommodate that growth.(Bondi, André B. (2000). Characteristics of scalability and their impact on performance. Proceedings of the second international workshop on Software and performance – WOSP '00. p. 195. )
![Page 13: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/13.jpg)
Types of Scalability
Functional
Geographic
Administrative
Load
Generation
ScalabilityMore users? No problem!
New functionality with minimal effort
No problems with going global
Add new generations of components
React to changing loadsHigher number of requests
https://en.wikipedia.org/wiki/Scalability
![Page 14: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/14.jpg)
Vertical versus Horizontal Scaling
Scale Up/Down„Vertical Scalability“
More CPU, Memory
Virtualization
Scale Out/In„Horizontal Scalability“
More Nodes
„Cluster“
![Page 15: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/15.jpg)
Scale Out - Personas
Master Workers
Gets instructions from client
Knows the list of active workers
Job overview
Current state of work
Claim tasks from Master
Work on tasks
Return results and work report
Know their profession
![Page 16: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/16.jpg)
Scale-Out in SSIS 2017
![Page 17: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/17.jpg)
Scale Out in SSIS 2017
Master
SSISDB
SSIS
Mas
ter
Serv
ice
Worker
Wo
rke
r Se
rvic
e
….
Worker
Wo
rke
r Se
rvic
e
Worker
Wo
rke
r Se
rvic
e
![Page 18: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/18.jpg)
Installation & Configuration
![Page 19: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/19.jpg)
Scale Out Master - Installation
PrerequisitesDB-Engine: Mixed Authentication
![Page 20: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/20.jpg)
SSIS Scale Out Master - Configuration
Port NumberDefault: 8391
Do not forget Firewall
SSL Certificate! „self“ Self-signed certificates
![Page 21: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/21.jpg)
Scale Out Master – SSISDB
![Page 22: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/22.jpg)
Scale Out Worker - Installation
![Page 23: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/23.jpg)
SSS Scale Out - Certificates
Every machine in SSIS Scale Out topology needs own certificate
Master needs workers-certificates
Workers need master-certificate
During (master and worker) setup, certificates are created
But you can use your own certificates!
M w1 w2 wx
![Page 24: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/24.jpg)
SSIS Scale Out – Certificates Workflow
Install SSIS Scale Out Master service
Copy worker certificate from worker machine
Install worker certificate (storage location)
Copy Master certificate to worker
Install master certificate (storage location)
Install Scale Out Worker service
Restart Worker service
M w1 w2 wx
C:\Program Files\Microsoft SQL Server\140\DTS\Binn
Master Machine Worker Machine
![Page 25: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/25.jpg)
Improve the Worker Setup experience
setup.exe/ACTION=Install/FEATURES=IS_Worker/ISWORKERSVCACCOUNT="NT SERVICE\SSISScaleOutWorker140"/ISWORKERSVCSTARTUPTYPE=Automatic/ISWORKERSVCMASTER=https://ssismaster01:8391/ISWORKERSVCCERT="\\pathTo\SSISScaleOutMaster.cer" /QS/IACCEPTSQLSERVERLICENSETERMS
Certificates, certificates, certificates!!
![Page 26: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/26.jpg)
Scale Out – Check the Environment (SSISDB)
![Page 27: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/27.jpg)
SSIS Scale Out Workers
![Page 28: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/28.jpg)
Execute in Scale Out
![Page 29: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/29.jpg)
Scale Out Execution (T-SQL)
Create execution
Set parameters
Define workers
Start execution
![Page 30: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/30.jpg)
Execution Monitoring
![Page 31: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/31.jpg)
Execution Monitoring
![Page 32: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/32.jpg)
Execution Monitoring (T-SQL) SSISDB
![Page 33: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/33.jpg)
Execution Monitoring (behind the scenes)
Worker – Execution LoggingC:\Users\SSISScaleOutWorker140\AppData\Local\SSIS\ScaleOut\AgentC:\Users\SSISScaleOutWorker140\AppData\Local\SSIS\ScaleOut\Tasks
Master – LoggingC:\Users\SSISScaleOutMaster140\AppData\Local\SSIS\ScaleOut\Master
Worker
![Page 34: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/34.jpg)
Execution Monitoring (behind the scenes)
How are logging messages transferred to the server?##MS_SSISLogDBWorkerAgentUser##
Works fine if network is available
Documentations states, that logging is cached locally..
Master Worker
![Page 35: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/35.jpg)
Configuration Options
![Page 36: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/36.jpg)
Scale Out – Configuration Options
C:\Program Files\Microsoft SQL Server\140\DTS\Binn
WorkerSettings.configMasterSettings.config
Restart services!
![Page 37: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/37.jpg)
Scale Out in SSIS 2017
Master
SSISDB
SSIS
Mas
ter
Serv
ice
certificates
M w1 w2 wx
Worker
Wo
rke
r Se
rvic
e
Package Executions
certificates
w1Mconfiglogs
config
logs
….
Worker
Wo
rke
r Se
rvic
e
Package Executions
certificates
w2Mconfiglogs
Worker
Wo
rke
r Se
rvic
e
Package Executions
certificates
wxMconfiglogs
![Page 38: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/38.jpg)
SSIS Scale Out – Santa, can you hear me?
https://www.pexels.com/photo/santa-clause-figurine-41963/
License / Edition ?
Cross Platform?
Parallel Foreach Container
![Page 39: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/39.jpg)
SSIS Scale Out – Santa, can you hear me?
https://www.pexels.com/photo/santa-clause-figurine-41963/
Initial configurationcertificates workflow
Centralized dashboardWorker stateRunning, pending, .. tasksCurrent performance situationCentralized loggingLogging in error state (network)
Manageability„grouping“ of Workers(i.e. third-party components)~Tags in VSTS Builds
![Page 40: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/40.jpg)
Scale-Out with SSIS 2017
@wstrasser workingondata.wordpress.com
Wolfgang Straßer
![Page 41: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/41.jpg)
PLEASE FILL IN EVALUATION FORMSFRIDAY, MAY 19th SATURDAY, MAY 20th
https://survs.com/survey/cprwce7pi8 https://survs.com/survey/l9kksmlzd8
YOUR OPINION IS IMPORTANT!
![Page 42: Scale-Out with SSIS 2017 - · PDF fileSQL Pass Austria SQLSaturday Vienna Organizer ... SSIS SSIS Catalog Project Deployment Model Custom Logging Levels SSISDB https: ... Scale-Out](https://reader031.vdocuments.mx/reader031/viewer/2022020100/5aa7a2b17f8b9ad31c8c2ec1/html5/thumbnails/42.jpg)
THANK YOU TO OUR SPONSORSPLATINUM
GOLD SILVER