ServiceNow Discovery Overview

March 18, 2020Andrew McCabeInfrastructure Technology Services

ServiceNow Discovery OverviewAgenda

• Motivation

• What is ServiceNow Discovery?

• How Does Discovery Work?

• Simple Enough … Maybe Not

• Architecture Review

• Non-Technical Metadata

• Discovery Challenges

Motivation

• Find and Identify “All” Devices on the Partners Network

• Record the Identified Devices in the ServiceNow CMDB

What is ServiceNow Discovery?

• Provides a Census of All Devices Residing on the Network

• Agentless

– No Installed Software

– Uses Standard Protocols to Answer Questions About Device Configuration

• ServiceNow® Discovery keeps the ServiceNow Configuration Management Database (CMDB) current by using a secure, agentless architecture to probe computers and other IP-enabled devices in an organization’s network for configuration details.

– ServiceNow Discovery Data Sheet

What is ServiceNow Discovery?Discoverable Assets

•Computers, Clusters, and Virtual Machines:•AIX Computers•ESX Servers•HPUX Computers•Hyper-V•Linux Computers•Linux Kernel-based Virtual Machines (KVM)•Mac Computers•Netware Computers•Solaris Computers•Solaris Zones•Virtual Machines•VMware vCenter•Windows Computers

Hardware Devices:•Dell DRAC•Load Balancers and Load Balancer Proxy Software•Network Printers•Routers•Storage Devices•Switches•Uninterruptible Power Supplies (UPS)

•Web and Database Servers:•Apache Web Servers•General Software Packages•HBase on UNIX•JBoss Servers•Microsoft IIS Servers•Microsoft SQL Servers•MongoDB Instances•MySQL Servers•NGINX Web Servers•Oracle Databases•PostgreSQL Instances•Tomcat Servers•WebLogic Application Servers•WebSphere Servers

Other Technologies:•Amazon Web Services (AWS) Cloud•Connections•IP Networks•Puppet Automation Software•Relationships•Services/Daemons

Discovery collects the following general data in a network.And More …

Source: ServiceNow Wiki

http://wiki.servicenow.com/index.php?title=Data_Collected_by_Discovery#gsc.tab=0

How Does Discovery Work?Overview

Image Source: ServiceNow Wiki

http://wiki.servicenow.com/index.php?title=Discovery_Agentless_Architecture#gsc.tab=0

How Does Discovery Work?Communications Protocols

Image Source: ServiceNow Wiki

http://wiki.servicenow.com/index.php?title=Discovery_Agentless_Architecture#gsc.tab=0

How Does Discovery Work?Probes and Sensors and Patterns

Image Source: ServiceNow Wiki

http://wiki.servicenow.com/index.php?title=Discovery_Agentless_Architecture#gsc.tab=0

• Probes and Servers Will Not Be Supported in Orlando Version

How Does Discovery Work?Probes and Sensors and Patterns

• Probes and Servers Will Not Be Supported in Orlando Version

• In Order to Implement Pattern Based Discovery, We Need To

– Implement the Identification and Reconciliation Engine

How Does Discovery Work?Windows Server Example

ServiceNow Initiates Discovery by IP Address

• Port Scan Phase Initiated (Can I Talk to You?)

– Shazzam Probe (Port Scan) Request Queued

– MID Server Polls Queue (Periodically)

– MID Server Launches Shazzam Probe

– MID Server Gathers Probe Results and Sends to ServiceNow

– ServiceNow Sensor Interprets Results

– ServiceNow Enqueues Classifier Probe Requests

• Classification Phase Initiated (What are You?)

– MID Server Polls Queue (Periodically)

– MID Server Launches Classifier Probes

» Requires Login Credentials for Each Protocol

– MID Server Gathers Probe Results and Sends to ServiceNow

– ServiceNow Sensor Interprets Results

– ServiceNow Enqueues Identification Probe Requests

How Does Discovery Work?Windows Server Example (Continued)

• Identification Phase Initiated (Have I Seen You Before?)

– MID Server Polls Queue (Periodically)

– MID Server Launches Identification Probes

» Gather Identifying Information: Hostname, IP, Serial #s, etc.

» Requires Login Credentials for Each Protocol

– MID Server Gathers Probe Results and Sends to ServiceNow

– ServiceNow Sensor Interprets Results

» Is this Device Already in the CMDB?

– ServiceNow Enqueues Exploration Probe Requests

• Exploration Phase Initiated (Tell Me About Yourself?)

– MID Server Polls Queue (Periodically)

– MID Server Launches Exploration Probes

» Gather Detailed Information from Device

» Need Login Credentials for Each Protocol

– MID Server Gathers Probe Results and Sends to ServiceNow

– ServiceNow Sensor Interprets Results and Updates CMDB

– ServiceNow Enqueues Additional Exploration Probe Requests as Needed Based on Findings

Simple Enough … Maybe Not

• Discovery Time Depends on Device Complexity

– Windows Server Running an IIS Web Server

» CMDB CI Created for Each Web Site

– Windows Server Running a SQL Server

» CMDB CI Created for Each SQL Server Instance and Database

– Discovering One Device Can Take from 2 to 10+ Minutes

• The Partners Network Consists of Approximately:

Device Type Current Growth Count

Probes

per Device

Probe

Weigth

Total

Probes

Printers 17,500 10% 19,250 5 1 96,250

Network Gear 3,500 10% 3,850 5 1 19,250

Servers 10,000 10% 11,000 12 1 132,000

vCenter 18 10% 20 14 5 1,386

Totals 31,018 34,120 248,886

Simple Enough … Maybe Not

• Lots of Devices … O(34,000)

• Lots of Time …

– Low End Estimate: 68,000 Minutes (1,133 Hours, 47 Days)

– High End Estimate: 340,000 Minutes (5,667 Hours, 236 Days)

• Solution … Employ Parallel Computing Techniques

– Discovery is Embarrassingly Parallel

– ServiceNow Supports Parallel Processing Out-Of-The-Box

Simple Enough … Maybe Not

• We Have Six ServiceNow Instances

Image Source: MeTV Web Site

https://metvnetwork.s3.amazonaws.com/9yCaN-1443477522-blog-13.jpg

– Production

– QA

– Development

– Test (Sandbox)

– SUP

– Training

Architecture SolutionDistributed Processing

• Discovery Workload Will Be Distributed Across Multiple MID Server Instances

• MID Servers Will Be Located in Each Data Center Reducing Network Latency

• Distributed, Multiple Data Center Deployment Supports Disaster Recovery

Non-Technical Metadata

• Discovery is Very Good at Gathering Technical Data

– CPU, Memory, and Disk Data

– Network Interface Data and Connection to Switches

– Running Processes

– TCP/IP Connections

– And Much More

• It Does Not Gather Non-Technical Information

– Critical to Operations

– Must Be Set Manually or Via Automation

• Related Application, Business or Infrastructure Service

• Owning Entity

• Owner

• Business Owner

• Technical Owner

• Criticality

• Support Group

• Maintenance Group

• Description and Keyword Terms

• Location

• PHS Build Type

Discovery ChallengesSecurity

• Credentials

– Discovery is Agentless:

» User Name and Password Credentials are Required for Each Protocol Used

» Servers, Network Devices, etc. Must Have Credentials Set Up Consistently

– Password Changes as Required by IS Security

» Future: Implement ServiceNow CyberArk Integration

• Some Devices are Invisible to MID Servers

– Network Firewalls Hide Devices

– Server Based Firewalls Hide Ports Needed to Communicate

» Can Punch Holes in Firewalls or Deploy MID Servers Behind Firewalls

Discovery ChallengesTools and Technologies

• Discovery is Based on IP Address

– ping –a

– nmap (https://nmap.org)

• Discovery Uses XML for Requests and Responses

– XML Notepad (Available from Microsoft)

– XML Spy

• SNMP Discovery

– Unexpected SNMP Responses

» Intermec Bar Code Printers versus Linux Servers

– snmpwalk (http://www.net-snmp.org/)

• Maintaining Multiple Servers with Same Image

Discovery ChallengesCustomization

• Port Definitions Can Be Customized

• Classifiers Can Be Customized

• Identifiers Can Be Customized

– Especially Important When CMDB is Partially Populated

– Technologies: JavaScript, ServiceNow Java Libraries

• Exploration Probes and Sensors Can Be Customized

• Out-Of-The-Box is Best …

• However, Be Prepared for Customizations

Additional Resources

• http://wiki.servicenow.com/index.php?title=Getting_Started_with_Agentless_Discovery#gsc.tab=0

• http://wiki.servicenow.com/index.php?title=Discovery_Made_Easy#gsc.tab=0

• http://wiki.servicenow.com/index.php?title=Discovery_Agentless_Architecture#gsc.tab=0 With Diagrams

• http://wiki.servicenow.com/index.php?title=MID_Server_Plugin#gsc.tab=0

• http://wiki.servicenow.com/index.php?title=MID_Server#gsc.tab=0

• https://docs.servicenow.com/bundle/helsinki-it-operations-management/page/product/mid-server/concept/c_MIDServerInstallation.html

ServiceNow Discovery Overview

Thank You for Your Time and Attention!