ssis monitoring deep dive
TRANSCRIPT
![Page 2: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/2.jpg)
Davide Mauri• Microsoft SQL Server MVP• Works with SQL Server from 6.5, on BI from 2003• Specialized in Data Solution Architecture, Database Design,
Performance Tuning, High-Performance Data Warehousing, BI, Big Data• President of UGISS (Italian SQL Server UG)• Regular Speaker @ SQL Server events• Consulting & Training, Mentor @ SolidQ• E-mail: [email protected]• Twitter: @mauridb • Blog: http://sqlblog.com/blogs/davide_mauri/default.aspx
![Page 3: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/3.jpg)
Agenda• Deep Dive into SSISDB Log-Related Catalog• Logging Levels• Building a Dashboard• Native Solutions• 3rd Community Party Solutions• My Solutions:
• HTML 5• Power BI
![Page 4: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/4.jpg)
Notice to SSIS Men• Everything said here works for SQL Server 2012 and
over and applies to the Project Deployment Model *only*
• If you’re sill using the Package Deployment model…
• …change it! It’s OLD!• And even O.L.D.: Omitted Log Data
![Page 5: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/5.jpg)
SSISDB – SSIS Catalog• Contains *everything* needed to run, manage and
monitor SSIS Packages• (only when using the “Project Deployment Model”)
• “internal” schema := private API/Objects, for MS use only
• “catalog” schema := public, supported, API/Objects
![Page 6: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/6.jpg)
SSIS Server Package Execution• Each time a Package is executed:• Create Package Execution “context”• Set Execution Parameters Values• Execute the Package
![Page 7: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/7.jpg)
SSIS Server Package Execution Logging• Logs and Execution Statistics are written to several
tables• catalog.executable• catalog.executable_statistic• catalog.event_messages
• catalog.operations, catalog.operation_messages• The execution_id binds everything together
Named “operation_id” sometimes because event_messages is an extension of operation_messages
![Page 8: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/8.jpg)
SSIS Server Package Logging Levels• Logging is automatically performed by the server• None: Logging is turned off. Only the package execution status is logged.• Basic: (Default) All events are logged, except custom and diagnostic
events.• Performance: Only performance statistics, and OnError and OnWarning
events, are logged.• Verbose: All events are logged, including custom and diagnostic events,
including the DiagnosticEx event.
• Of course, logging level can be changed at execution-time, no special design patterns needed
![Page 9: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/9.jpg)
Reference Solution Used for Tests• Master, Child, Child case• Master and Child have objects with same name• Master and Child have loops• Master and Child have parallel executions• Package and Projects connection managers have the same
name• Package and Project Parameters• Scripts (Dts.Events.FireInformation) logging via Script Task• DataFlows
![Page 10: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/10.jpg)
DemoThe reference solution
![Page 11: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/11.jpg)
Before Execution• Where data is stored in the SSISDB?
• Especially interesting the parameter and connection managers configured and default values• They will be useful to have the complete execution context
when doing log analysis
![Page 12: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/12.jpg)
Demo
![Page 13: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/13.jpg)
Package Execution
Created (1) Pending(5)
Running(2)
Stopping(8)
Canceled (3)
Success(7)
Completed(9)
Failed(4)
Unexpected Termination / Crash (6)
catalog.start_execution
catalog.create_execution catalog.stop_operation
ISServerExec Unexpected Crash
![Page 14: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/14.jpg)
Logging “None”• Really means “Minimal”
• No logged events, but still some data is logged in some “basic” log tables :• catalog.executions• catalog.execution_property_override_values• catalog.execution_parameter_values• catalog.executables• catalog.executable_statistics
• Log-Like table
![Page 15: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/15.jpg)
Logging “Basic”• Default Option
• Data Available in “basic” log tables and• catalog.event_messages
• Logged Events:• OnError, OnInformation, OnPostExecute, OnPostValidate, OnPreExecute,
OnPreValidate, OnWarning
• No specific information on related to DataFlow
![Page 16: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/16.jpg)
Logging “Performance”• Data Available in “basic” log tables and
• catalog.event_messages
• Logged Events:• OnError, OnWarning
• Additional specific information on component “phases”• catalog.execution_component_phases
• No specific information on related to DataFlow
![Page 17: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/17.jpg)
Logging “Verbose”• Like Basic and Perfomance together PLUS DataFlow
details• DataFlow is logged in detail to
catalog.execution_data_statistics
• Logged Events• All!• Even Diagnostic and Custom log messages
(Dts.Events.FireCustom)
• Logging in event_messages is something like 100X more verbose• Hence the name
![Page 18: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/18.jpg)
Logging “Custom”• Available only in SSIS 2016
• You can decide your own logging level…like• Performance +• OnInformation +• Data Flow
![Page 19: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/19.jpg)
Demo
![Page 20: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/20.jpg)
Something to keep in mind• Dataflow logs some interesting data as a text message
only• Lookup Duplicate Rows warning• Memory Warning
• Memory Allocation• Low Virtual Memory
• Rows sent to each destination
• Verbose is just to verbose, but we still want some DataFlow logging
![Page 21: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/21.jpg)
Demo
![Page 22: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/22.jpg)
Building a Dashboard - Native
• Natively Available• Basic User
Experience• “It just works”• Not really a
dashboard
![Page 23: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/23.jpg)
3Rd Party Community Project• SSIS Reporting Pack (by Jamie Thomson)• No more maintained (AFAIK)• Provides more than reporting
• usp_ssiscatalog
http://ssisreportingpack.codeplex.com/
![Page 24: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/24.jpg)
HTML5• Based on • HTML 5 / CSS 3• Bootstrap• jQuery, Morris.Js, MetisMenu• Python + Flask
• Code on GitHub• https://github.com/yorek/ssis-queries • https://github.com/yorek/ssis-dashboard
![Page 25: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/25.jpg)
HTML5
![Page 26: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/26.jpg)
HTML5• An even more powerful version has been created by @niphlod here
• https://github.com/niphlod/ssis_dash
• Still based on Python• But moved from Flask To Web2Py• Supports security, authentication and authorization• Can monitor more than one SSISDB instance• …much more
• Very cool!
![Page 27: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/27.jpg)
PowerBI• Using PowerBI Desktop it is really easy to create a nice
SSIS Dashboard
• If you don’t have problems sending data to the Azure cloud, you can also move the Dashboard to the PowerBI Service• And use SSIS or Azure Data Factory or PowerBI Data Gateway
to keep data in Azure updated
![Page 28: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/28.jpg)
One last note• Due to how security is applied in SSIS, if you’re not sysadmin or
ssis_admin queries can be *really* *really* slow• Each row is under custom row security, using certificates• *A lot of certificates* needs to be open and used: One for each Execution!!!!• SQL Server 2012 SP2 fix this by allowing you to do it on project-basis
instead of Execition-basis• https://support.microsoft.com/en-us/kb/2972285
• A *unsupported* workaround is to fix the [internal].[current_user_object_permissions] procedure, avoiding the usage of the scalar function [get_principal_id_by_sid], use a JOIN instead
![Page 29: SSIS Monitoring Deep Dive](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5881a23f1a28ab1a398b5891/html5/thumbnails/29.jpg)