addmi 13-discovery overview

© 2009 BMC Educational Services

Discovery Overview

Getting Data from the Estate


Outline

The Basic Philosophy First Scan Second Scan Optimization Bringing It All Together

Completed State Charts Summary


The Basic Philosophy


Basic Discovery Sequence (1)

We start a scan of a collection of IPs

We try to contact each one – we record success/failure

We try to establish what sort of device is on the IP from a combination of heuristics and direct access and record that

Discovery

Run

Discovery

Access

Device

Info

Discovery

Run

Discovery

Run

Discovery

Access



If the device is a Host and we were able to log on we try and get more information about it and it’s list of interfaces

If we get the Host Information and Interface List then we have enough to try to infer a host, and we carry on with our standard set of discovery

Discovery

Run

Discovery

Access

Device

Info

Host

Info

Interface

List

Discovery

Run

Discovery

Access

Device

Info

Host

Info

Interface

List

Host

Processes



Once we have finished standard discovery we can infer other hardware items like Network Interfaces and link the Host to global nodes like subnet

At this stage in the sequence we will start inferring software products, correlating virtual hosts with their physical hosts etc

Discovery

Run

Discovery

Access

Device

Info

Host

Info

Interface

List

Process

List

Host

Discovered

Network

Interfaces

NIC

Subnet



Although our standard discovery has finished, we still have the ability to request more information if we need it

This could be the pattern to create software looking for further data or a platform related pattern looking for additional information to do with the Host

Additional discovery will be done via commands, files, registry queries, WMI queries, SNMP queries, SQL queries…

Discovery

Run

Discovery

Access

Device

Information

Host

Information

Interface

List

Process

List

Host

Oracle

Discovered

File


First Scan Details


We will build up the following Discovery Access state chart:

The DiscoveryAccess node contains 3 key summary attributes to record what happened during session establishment:

ShouldAccess/ SHOULDACCESS

Excluded

/ DONTACCESS

DeviceIdentified/ DEVICE

NoResponse

/ NODEVICE

HostFound/ HOST

UnsupportedDevice

/ NOTHOST

HostInferred/ GOTACCESS

NoAccess

/ NOACCESS

Opt1stScan

/ DONTACCESS

GoodAccess/ SHOULDACCESS

OptNotBestIP

/ DONTACCESS

OptAlreadyProcessing

/ DONTACCESS

Skipped

NoAccess

Success

NoResponse

Error

/ TRACEBACK

Error

OptRemote

Result attribute contents

/ DONTACCESS

Discovery Walk Through

result [ Success | NoAccess | Skipped | NoResponse ]

end_state [ From state diagram ]

reason [ Free text summary reason for lack of success ]


Is Access Allowed?

Check to see if we are allowed to access the endpoint

result = Skipped

end_state = Excluded

Excluded

/ DONTACCESS


Ping Response?

Pings the endpoint to see if anything responds


Check For Open Ports

If something responds then we see if any of the ports we can use are open


Ports

Ports we can use UNIX

22 SSH 23 Telnet 513 rlogin

Windows 135 RPC

SNMP 161 (UDP) SNMP

External OS Detection Only 4 (closed port for IP fingerprint) 80 HTTP


Dark Space

If we have not got a response at this point we regard this endpoint as Dark Space

end_state = NoResponse result = NoResponse


NoResponse

/ NODEVICE


Credential Vault

We look in the vault for a credential that matches: The IP of the endpoint The service ports seen open on

the device (SSH, Telnet..)


Credential Selection

We try to establish a session with each one of the credentials that match

Used in the order they are defined in the vault (as seen on the credential page)


UNIX Access

We try the UNIX access methods first

Only tried if we found a UNIX port open

If we get a response we ask the device what it is


Windows Access

If the UNIX methods don’t get a result then we try the Windows methods

Only tried if we found a Windows port open

Slaves are used in the order defined



SNMP Access

If the UNIX and Windows methods don’t get a result then we try the SNMP methods

Only tried if we found the SNMP port open



Other Attempts

If no access methods have worked we try to determine what the device might be from external evidence SNMP SysDescr IP Stack Fingerprinting Telnet Banner HTTP HEAD


Host Classification

All the results so far go through the OS/Device classifier

If the device is a “Host” we will continue

Otherwise we skip this endpoint

end_state = UnsupportedDevice

result = skipped



HostFound/ HOST

UnsupportedDevice

/ NOTHOST


Once We Determine a Host…

Now we know it is a Host we return to the session and ask for getHostInfo getInterfaceList

Both these are critical for running the Host Identification algorithm


Access Failure

If these methods fail to complete then discovery stops here

end_state = NoAccess result = NoAccess



HostFound/ HOST

NoAccess

/ NOACCESS


Host Identity Algorithm

Host Algorithm uses strong identity properties to compute a Host.key OS, Kernel, MAC, IP, Serial,…

A new Host is created

end_state = GoodAccess result = Success



HostFound/ HOST




Credential Caching

We cache the successfully used credential and slave for use next time


Further Discovery

Standard discovery continues collecting Processes Packages Etc

TPL based discovery starts after Standard discovery

No further change to end_state


Second Scan Details


Is Access Allowed?

Check to see if we allowed to access the endpoint

result = Skipped end_state = Excluded, end_state =

OptNotBestIP or end_state = OptRemote

Excluded

/ DONTACCESS

OptNotBestIP

/ DONTACCESS

OptRemote

/ DONTACCESS


Cached Credential

We check to see if there are cached results from the previous access to this endpoint


Use Last Slave

We try the previous credential/slave to see if we make contact

This shortcuts establishing a session


Does This Cached Attempt Succeed?

If we do not re-establish a session we have to go back to the full analysis

If we do succeed we start at the OS / Device classifier


Back to the Standard Tasks…

OS/Device classifier

HostInfo/InterfaceList

Host Algorithm


Optimization Details


Optimize – Skipped Endpoints

In order to maximise throughput and reduce load on the targets there are a series of optimisations

These can result in skipped endpoints


Duplicate IP in Progress

To prevent collisions if an IP is already in progress duplicates are dropped

result = Skipped end_state = OptAlreadyProcessing


/ DONTACCESS


Best IP

Many Hosts have more than one active interface

Many endpoints in a range that relate to the same Host

No point scanning the same Host 12 times in one range


Best IP - Aims

Scan over a single endpoint Try and keep the single

endpoint chosen stable over time

Minimize network access


Best IP – 1st Scan Optimization

Kicks in when we try to update an existing Host

If the Host has already been updated by an endpoint reaching the GoodAccess end_state discovery stops

result = Skipped end_state = Opt1stScan



HostFound/ HOST


Opt1stScan

/ DONTACCESS


Multiple IP – 2nd Scan Optimization

Kicks in when we scan the same endpoint again

We check to see if it was optimised last time, if it was we assume it will be again

Discovery stops here

result = Skipped end_state = OptNotBestIP

OptNotBestIP

/ DONTACCESS


Multiple IP – 2nd Scan Optimization

We don’t assume the optimisation is correct forever

Every so often we will contact the Host again to confirm If it turns out the IP is new

Host full discovery will occur Otherwise you will get 1st Scan

Optimization again



HostFound/ HOST


Opt1stScan

/ DONTACCESS


Bringing It All Together


Errors

If an error occurs the result will be set Error

The end_state will be set to the last state reached, or Error if it occurs before any other state

Error

/ TRACEBACK


Complete State Chart

Bringing together all the states allows us to draw a complete state chart for standard discovery


Excluded

/ DONTACCESS


NoResponse

/ NODEVICE

HostFound/ HOST

UnsupportedDevice

/ NOTHOST


NoAccess

/ NOACCESS

Opt1stScan

/ DONTACCESS


OptNotBestIP

/ DONTACCESS


/ DONTACCESS

-end_state [from states above]-result [Success|NoAccess|Skipped|NoResponse]-reason [Full text or details]

DiscoveryAccess::.

Skipped

NoAccess

Success

NoResponse

Error

/ TRACEBACK

Error

OptRemote


If a traceback occurs before any end_state is set then end_state will be set to ‘Error’, otherwise end_state is left to record progress.

Result will always be set to ‘Error’

/ DONTACCESS


Complete State Chart – Additional Discovery

Discovery doesn’t stop once a session is established Patterns will cause further discovery as required


Excluded

/ DONTACCESS


NoResponse

/ NODEVICE

HostFound/ HOST

UnsupportedDevice

/ NOTHOST


NoAccess

/ NOACCESS

Opt1stScan

/ DONTACCESS


OptNotBestIP

/ DONTACCESS


/ DONTACCESS


DiscoveryAccess::.

Skipped

NoAccess

Success

NoResponse

Error

/ TRACEBACK

Error

OptRemote




Standard Discovery Additional Discovery

/ DONTACCESS


Complete State Chart – Sweep Scan

If the scan level is restricted to Sweep Scan then the DeviceIndentified state is regarded as a success

result = Success end_state = DeviceIdentified


Excluded

/ DONTACCESS


NoResponse

/ NODEVICE

HostFound/ HOST

UnsupportedDevice

/ NOTHOST


NoAccess

/ NOACCESS

Opt1stScan

/ DONTACCESS


OptNotBestIP

/ DONTACCESS


/ DONTACCESS

SCAN LEVEL = SWEEP SCAN

SCAN LEVEL = FULL DISCOVERY


DiscoveryAccess::.

Skipped

NoAccess

Success

NoResponse

Error

/ TRACEBACK

Error

OptRemote




Standard Discovery Additional Discovery

/ DONTACCESS


Summary

Atrium Discovery can be restricted from scanning sensitive/high risk endpoints

Atrium Discovery needs to be able to see network ports on the target to pick the right access methods and credentials

Atrium Discovery does full discovery of Host devices and basic discovery of other devices

Atrium Discovery caches successful access methods and credentials for faster future session establishment

Atrium Discovery optimizes it’s access to ensure that Hosts with multiple IPs are not repeatedly scanned

Atrium Discovery needs to succeed with getDeviceInfo, getHostInfo and getInterfaceList in order to create/update a Host node


Online Documentation: http://www.tideway.com/confluence/display/81/The+Discovery+Process

Tideway Foundation

Version 7.2

Documentation

Title

Further Resources


Discovery Overview State Charts

addmi 13-discovery overview

Technology

discovery standard discovery

host additional discovery

discovery walk

discovery overview

basic discovery sequence

snmp access

windows access

new host