troubleshooting - cisco · cisco stealthwatch learning network license ucs e-series server...

Troubleshooting

The following describes the most common troubleshooting scenarios.

• Time Synchronization, page 1

• Initial Anomaly Display Issues, page 1

• Maximum Managed Agents, page 2

• Disabled Functionality, page 2

• Controller Administrator Password Reset, page 2

• Performance Issues, page 3

• Certificate Fingerprint Retrieval, page 3

• Connectivity Issues, page 5

• Agent Status Messages, page 5

Time SynchronizationYour controller, agents, and ISRs should all reference the same NTP servers for proper time synchronization,and to report anomalies correctly.

If you deploy your agents to a UCS E-Series blade server, you must configure NTP on each agent.

If you do not configure NTP servers on a agent deployed as a virtual service, configure them on your ISRs,as the agents pull time from the host router.

Initial Anomaly Display IssuesIf you have installed the Learning Network License system and you do not see any reported anomalies, waitfor seven days. The system requires an initial learning phase to create a baseline model of your network trafficand identify anomalies. Note that during this initial learning phase, the system may start reporting anomalies.Because the baseline is not yet complete, these anomalies may not be of interest or relevant to you.

Cisco Stealthwatch Learning Network License UCS E-Series Server Installation Guide, Version 1.1 1

Maximum Managed AgentsIf you have not registered your controller with Smart Licensing, you are in Evaluation Mode, and limited tomanaging 10 agents with that controller. Register the controller with Smart Licensing before the 90-dayEvaluation Mode expires to remove the limit.

Disabled FunctionalityIf the system no longer detects or reports new anomalies, or you can no longer create mitigations, or modifyexisting mitigations, system registration is expired. If the 90-day Evaluation Mode elapsed, make sure youhave the proper license entitlements, and register your controller with the Licensing Authority. Otherwise, ifyour controller has not communicated with the Licensing Authority in more than 90 days, manually renewyour registration with the Licensing Authority.

Controller Administrator Password ResetIf you forget the admin user account's password for the controller web UI, you can reset it from the controllerCLI. When you reset the password, the system prints a randomly generated password to the console. Thisnew password is valid for 3 days, by default. When you next login to the controller web UI as admin, thesystem prompts you to change the password.

You must have access to the ~/SCA/sca.sh script to reset the password.

Resetting the Controller Administrator Password

Before You Begin

• Log into the controller VM as a user with access to the ~/SCA/sca.sh script.

SUMMARY STEPS

1. cd ~/SCA

2. sudo service ciscosln-sca stop, then enter your password when prompted3. ./sca.sh reset-admin-password

4. sudo service ciscosln-sca start

DETAILED STEPS

PurposeCommand or Action

Change directories to ~/SCA.cd ~/SCA

Example:

Step 1

user@host:~$ cd ~/SCA

Cisco Stealthwatch Learning Network License UCS E-Series Server Installation Guide, Version 1.12

TroubleshootingMaximum Managed Agents


Stop the controller processes.sudo service ciscosln-sca stop, then enter your password whenprompted

Step 2

Example:user@host:~/SCA$ sudo service ciscosln-sca stop

Reset the admin user account's password../sca.sh reset-admin-password

Example:user@host:~/SCA$ ./sca.sh reset-admin-passworduser@host:~/SCA$ Resetting the admin password in sln

Step 3

user@host:~/SCA$ New password is 'AbCd1234'user@host:~/SCA$ Admin password reset done.

Start the controller processes.sudo service ciscosln-sca start

Example:

Step 4

user@host:~/SCA$ sudo service ciscosln-sca start

What to Do Next

• Log into the controller web UI as admin, then update the password.

Performance IssuesIf you are having performance issues, remember that there are several factors that affect your virtual appliances.See System Performance for a list of factors that may affect your performance. To monitor ESXi hostperformance, you can use your vSphere Client and the information found under the Performance tab.

Certificate Fingerprint RetrievalTo help troubleshoot public key certificate issues, you can retrieve stored certificate fingerprints from thecontroller VM console, controller web UI, and agent VM console.

Viewing a Controller Client Certificate Fingerprint from the Agent

Before You Begin

• Log into the agent VM console.


TroubleshootingPerformance Issues

SLN_UCS_Installation_Guide_chapter_00.pdf#unique_5

DETAILED STEPS


View the stored controller client certificate SHA256fingerprint in the console.

cat DLA/certificates/authorized_cert

Example:user@host:~$ cat DLA/certificates/authorized_cert

Step 1

Viewing a Controller Client Certificate Fingerprint from the Controller

Before You Begin

• Log into the controller VM console.

DETAILED STEPS


View the stored controller client certificateSHA256 fingerprint in the console.

keytool -v -list -storepass <password> -keystore

SCA/keystore.jks | egrep "Alias|SHA256"

Example:user@host:~$ keytool -v -list -storepass sln123 -keystoreSCA/keystore.jks | egrep "Alias|SHA256"

Step 1

Viewing an Agent Server Certificate Fingerprint from the Agent

Before You Begin


DETAILED STEPS


View the stored agent server certificate SHA256fingerprint in the console.

openssl x509 -in DLA/certificates/server.pem -noout

-fingerprint -sha256

Example:user@host:~$ openssl x509 -in DLA/certificates/server.pem-noout -fingerprint -sha256

Step 1


TroubleshootingViewing a Controller Client Certificate Fingerprint from the Controller

Viewing an Agent Server Certificate Fingerprint from the Controller Web UI

Before You Begin

• Log into the controller web UI.

Step 1 Click AGENTS.Step 2 Click Certificate next to an agent.

Connectivity IssuesYou can view and confirm connectivity for management and sensing interfaces using VMware vSphere Client.

If a firewall or other security appliance sits between the controller and agents, or between the user and thecontroller, ensure that certain communication ports are open. See Communication Ports for more information.

Confirming Interface Connectivity

Step 1 Right-click the name of the virtual appliance in vSphere Client and select Edit Settings.Step 2 Select Network adapter 1 in the Hardware list and make sure the Connect at power on check box is selected.Step 3 Repeat step 2 for each remaining network adapter.

Agent Status MessagesThe following lists the various agent status codes and messages that the system logs during agent configurationin the controller web UI, as well as recommended steps to resolve the error. You can also view the agent logfile at LOG/DLC.log to determine which error occurred, and resolve the issue.

Status Code: 2000• Status Message - Agent Not Responding


TroubleshootingViewing an Agent Server Certificate Fingerprint from the Controller Web UI

Learning_Network_License_UCS_E_Server_Installation_Guide_v_1_1_chapter_010001.pdf#unique_30

• Description - The controller tried to establish a connection with the agent, and did not receive a response,possibly because the agent is down or unreachable.

• RecommendedResolution - From the controller VM console, ping the agent by IP address and hostnameto verify the controller can reach the agent. If you do not receive a response, check your networkdeployment settings.

Before You Begin

• Log into the controller VM console

SUMMARY STEPS

1. ping <agent-IP-address> -c 5

2. ping <agent-hostname> -c 5

DETAILED STEPS


Send five packets to the agent's IP address and receive aresponse for each packet.

ping <agent-IP-address> -c 5

Example:user@host:~$ ping <agent-IP-address> -c 5

Step 1

Send five packets to the agent's host name and receive aresponse for each packet.

ping <agent-hostname> -c 5

Example:user@host:~$ ping <agent-hostname> -c 5

Step 2

Status Code: 2001• Status Message - Agent Certificate Rejected

• Description - The controller rejected the agent certificate, possibly for one of the following reasons:

◦The agent certificate does not match the certificate fingerprint pinned in the controller web UI.

◦The agent certificate is self-signed, and the system is not configured to support self-signedcertificates.

◦The agent certificate is not self-signed, and a CA or root certificate in the chain of trust is missingfrom the controller's truststore.

◦The certificate is expired.

• Recommended Resolution - Take the following actions:


TroubleshootingStatus Code: 2001

◦If you recently upgraded the agent, generate an agent certificate fingerprint, and upload it to thecontroller web UI. See Uploading an Agent Certificate Fingerprint, on page 7 for moreinformation.

◦If your certificate is self-signed, enable support for self-signed certificates. See Enabling Supportfor Self-Signed Certificates, on page 9 for more information.

◦If your certificate is not self-signed, verify the truststore contains the necessary root and CAcertificates.

◦If your certificate is expired, renew your certificate.

Uploading an Agent Certificate Fingerprint

SUMMARY STEPS

1. Log into the agent VM console.2. cd DLA

3. ./dla_admin

4. 4) Certificate and trust management

5. 6) Export DLA certificate

6. 1) Export to remote system, then hostname, then username, then ~/SCA/filename, then password7. 11) Exit

8. Log into the controller VM console.9. cd ~/SCA

10. open ssl x509 -in <dla-filename>.pem -noout -fingerprint -sha256

11. Copy the fingerprint into a text editor.12. Log into the controller web UI.13. Click AGENTS.14. Click Certificate next to an agent.15. Delete the Hash value and enter your new certificate fingerprint hash.16. Check theCheck to overwrite the active certificate checkbox to overwrite the existing pinned certificate

fingerprint.17. Click Pin certificate.

DETAILED STEPS


Log into the agent VM console.Step 1

Change directories.cd DLA

Example:

Step 2

user@host:~$ cd DLA




Run the agent administrator script../dla_admin

Example:

Step 3

user@host:~/DLA$ ./dla_admin

Access the CertificateManagement menu options.4) Certificate and trust management

Example:

Step 4

Enter a number: 4

Export the certificate associated with the agent.6) Export DLA certificate

Example:

Step 5

Enter a number:

Export the certificate to the controller. Give eachseparate certificate you export a different name,such as the agent's hostname.

1) Export to remote system, then hostname, thenusername, then ~/SCA/filename, then password

Example:Enter a number: 1Name or address of remote host []? remotehost

Step 6

Destination username []? adminThe destination filename path can absolute, orrelativeto home dir.Destination filename [server.pem]:~/SCA/<dla-filename>.pemadmin@remotehost's password: <password>

Quit the admin script and return to the commandprompt.

11) Exit

Example:

Step 7

Enter a number: 11

Log into the controller VM console.Step 8

Change to the ~/SCA directory.cd ~/SCA

Example:

Step 9


Generate a SHA256 fingerprint for the agentcertificate.

open ssl x509 -in <dla-filename>.pem -noout

-fingerprint -sha256

Example:user@host:~$ open ssl x509 -in <dla-filename>.pem-noout -fingerprint -sha256

Step 10

Store the fingerprint in a text editor file.Copy the fingerprint into a text editor.Step 11

Log into the controller web UI.Log into the controller web UI.Step 12

The agents management window appears.Click AGENTS.Step 13

The certificate management window appears.Click Certificate next to an agent.Step 14




The displayed certificate fingerprint is updated.Delete the Hash value and enter your new certificatefingerprint hash.

Step 15

The pinned certificate fingerprint is overwritten.Check the Check to overwrite the active certificatecheckbox to overwrite the existing pinned certificatefingerprint.

Step 16

The system pins the certificate fingerprint.Click Pin certificate.Step 17

Enabling Support for Self-Signed CertificatesThe sca.conf configuration file contains several layers of nested brackets. When you update the file to addor update the dla node, make sure that you nest it within the sln bracket. See the following for an example.sln {dla {security {allowSelfSignedCert = true

}}

}

Before You Begin


SUMMARY STEPS

1. cd ~/SCA

2. sudo vi sca.conf, then input your password when prompted3. Update the configuration file to include or modify the configuration.4. Press Esc, then enter :wq! and press Enter.5. sudo service ciscosln-sca restart

DETAILED STEPS


Change to the /SCA directory.cd ~/SCA

Example:

Step 1


Edit the sca.conf configuration file.sudo vi sca.conf, then input your password when prompted

Example:

Step 2

user@host:~/SCA$ sudo vi sca.conf




Update the configuration file to includeallowSelfSignedCert = true.

Update the configuration file to include or modify theconfiguration.

Step 3

Save your changes and exit the editor.Press Esc, then enter :wq! and press Enter.Step 4

Restart the controller processes.sudo service ciscosln-sca restart

Example:

Step 5

user@host:~/SCA$ sudo service ciscosln-sca restart

Status Code: 2002• Status Message - Connection Refused or Closed

• Description - The agent refused to accept or closed the connection with the controller, possibly for oneof the following reasons:

◦The controller certificate does not match the certificate fingerprint pinned on the agent.

◦The controller certificate is self-signed, and the agent is not configured to support self-signedcertificates.

◦The controller certificate is not self-signed, and a CA or root certificate in the chain of trust ismissing from the agent's truststore.

◦The controller certificate is expired.


◦If the fingerprint pinned on the agent does not match the certificate, you enabled TOFU, and youdo not want to upload the new controller certificate fingerprint to the agent, clear the pinnedcontroller certificate from the agent, and manage your agent with the controller. See Clearing aPinned Controller Certificate from an Agent, on page 11 for more information.

If TOFU is enabled, and you clear the pinned controller certificate fingerprint, the agentis vulnerable to any entity that connects to it over TLS with a trustable certificate.Manage the agent from the controller as soon as possible after you clear the fingerprint.

Note

◦If the fingerprint pinned on the agent does not match the controller certificate, and you did notenable TOFU, generate a controller certificate fingerprint, and pin it on the agent, as described inUploading a Controller Certificate Fingerprint, on page 12.

◦If your certificate is self-signed, enable support for self-signed certificates. See Enabling Trust onFirst Use, on page 14 for more information.

◦If your certificate is not self-signed, verify the trusted CA certificates on the agent hold the issuingCA certificate.



◦If your certificate is expired, renew your certificate.

Clearing a Pinned Controller Certificate from an AgentIf you enabled TOFU, and you clear the pinned controller certificate fingerprint, make sure you connect theagent to the controller as soon as possible, or pin the new controller certificate fingerprint.

Before You Begin


SUMMARY STEPS

1. cd DLA

2. ./dla_admin


4. 1) Manage Certificate Pinning

5. 6) Clear Trusted SCA certificate fingerprint

6. y to confirm

DETAILED STEPS



Example:

Step 1

user@host:~$ cd DLA


Example:

Step 2


Access the certificate and trust management options.4) Certificate and trust management

Example:

Step 3

Enter a number: 4

Access the certificate pinning options.1) Manage Certificate Pinning

Example:

Step 4

Enter a number: 1

Choose to clear the pinned controller certificatefingerprint.

6) Clear Trusted SCA certificate fingerprint

Example:

Step 5

Enter a number: 6




Clear the pinned controller certificate fingerprint.y to confirm

Example:

Step 6

Confirm removal of existing SCA certificate

fingerprint [confirm] y

What to Do Next

• If you enabled TOFU, log into the controller web UI and manage the agent with the controller. SeeAdding an Agent to the Controller for more information.

• If you did not enable TOFU, pin the controller certificate fingerprint to the agent. See the next sectionfor more information.

Uploading a Controller Certificate Fingerprint

SUMMARY STEPS

1. Log into the controller VM console on the ESXi hypervisor.2. cd ~/SCA

3. open ssl x509 -in sca_cert.pem -noout -fingerprint -sha256

4. Copy the fingerprint into a text editor.5. Log into the agent VM console.6. cd DLA

7. ./dla_admin



10. 5) Set Trusted SCA certificate fingerprint

11. SHA25612. sca-fingerprint

DETAILED STEPS


Log into the controller VM console on the ESXi hypervisor.Step 1

Change directories.cd ~/SCA

Example:

Step 2




SLN_UCS_Installation_Guide_chapter_01101.pdf#unique_108


Generate a SHA256 certificatefingerprint.

open ssl x509 -in sca_cert.pem -noout -fingerprint -sha256

Example:

Step 3

user@host:~/SCA$ open ssl x509 -in sca_cert.pem -noout -fingerprint

-sha256

Store the fingerprint in a text editorfile.

Copy the fingerprint into a text editor.

Example:SHA256Fingerprint=37:9A:DD:72:B6:91:8F:3E:D7:26:63:86:96:42:83:C3:39:AE:86:96:8F:3C:B8:CA:63:66:65:37:90:0C:51:DC

Step 4

Log into the agent VM console.Step 5


Example:

Step 6

user@host:~$ cd DLA


Example:

Step 7


Access the certificate and trustmanagement options.

4) Certificate and trust management

Example:

Step 8

Enter a number: 4

Access the certificate pinningoptions.

1) Manage Certificate Pinning

Example:

Step 9

Enter a number: 1

Pin the controller certificatefingerprint.

5) Set Trusted SCA certificate fingerprint

Example:

Step 10

Enter a number: 1

Enter the SHA256 hash algorithm.SHA256

Example:

Step 11

Please enter hash algorithm name: SHA256

Enter the sca-fingerprint.sca-fingerprint

Example:Please enter hash value as XX:XX:XX:XX...:37:9A:DD:72:B6:91:8F:3E:D7:26:63:86:96:42:83:C3:39:AE:86:96:8F:3C:B8:CA:63:66:65:37:90:0C:51:DC

Step 12



Enabling Trust on First Use

Before You Begin

• Log into the agent VM console as sln.

SUMMARY STEPS

1. cd ~/DLA

2. ./dla_admin



5. 1) Enable Trust SCA Certificate on First Use

DETAILED STEPS


Change directories.cd ~/DLA

Example:

Step 1

user@host:~$ cd ~/DLA

Run the administrator script../dla_admin

Example:

Step 2


Enter the Certificate and trust management menu.4) Certificate and trust management

Example:

Step 3

Enter a number: 4

Enter the Certificate Pinning menu.1) Manage Certificate Pinning

Example:

Step 4

Enter a number: 1

Enable TOFU, to trust the controller certificate thefirst time it is detected.

1) Enable Trust SCA Certificate on First Use

Example:

Step 5

Enter a number: 1

Status Code: 2003• Status Message - Message Decode Error



• Description - The controller cannot decode a message from the agent.

• Recommended Resolution - Ensure that the controller and agent are on the same version. Upgrade theout-of-band component. See theCisco Stealthwatch Learning Network License Virtual Service InstallationGuide, theCisco Stealthwatch Learning Network License UCS E-Series Blade Server Installation Guide,and the Cisco Stealthwatch Learning Network License Release Notes for more information.

Status Code: 2004• Status Message - Message ACK Timeout

• Description - The agent did not send an ACK in time, cause the controller to close the agent connectionand reconnect to the agent.

• Recommended Resolution - Make sure that your agent is turned on, and ping it from the controller.

Before You Begin

• Log into the controller VM console

SUMMARY STEPS

1. ping <agent-IP-address> -c 5

2. ping <agent-hostname> -c 5

DETAILED STEPS


Send five packets to the agent's IP address and receive aresponse for each packet.

ping <agent-IP-address> -c 5

Example:user@host:~$ ping <agent-IP-address> -c 5

Step 1

Send five packets to the agent's host name and receive aresponse for each packet.

ping <agent-hostname> -c 5

Example:user@host:~$ ping <agent-hostname> -c 5

Step 2

Status Code: 2005• Status Message - Message Too Big

• Description - The controller received a message from the agent that exceeded the maximum supportedmessage size.

• Recommended Resolution - Contact Cisco Support for more information.



Status Code: 2006• Status Message - Secure connection misconfigured

• Description - The controller cannot create an SSL context to validate the certificate.

• Recommended Resolution - View the keystore and truststore contents, and provide the store passwordto check their integrity.

Before You Begin


SUMMARY STEPS

1. cd ~/SCA

2. keytool -list -keystore keystore.jks, then provide your password when prompted3. keytool -list -keystore truststore.jks, then provide your password when prompted

DETAILED STEPS


Change to the /SCA directory.cd ~/SCA

Example:

Step 1


View the keystore contents, and provide thepassword to check the keystore's integrity

keytool -list -keystore keystore.jks, then provide yourpassword when prompted

Example:user@host:~/SCA$ keytool -list -keystore keystore.jks

Step 2

View the truststore's contents, and provide thepassword to check the keystore's integrity

keytool -list -keystore truststore.jks, then provide yourpassword when prompted

Example:user@host:~/SCA$ keytool -list -keystoretruststore.jks

Step 3

Status Code: 2010• Status Message - Unknown Connection error

• Description - The connection with the agent closed for an unexpected reason.

• Recommended Resolution - If this issue persists, contact Cisco Support for more information.



Status Code: ALLOCFAIL• Status Message - Failed to allocate memory

• Description - The agent failed to allocate memory.


Status Code: DNSQEVENTSPERBINLIMIT• Status Message - Too many events have been observed for too many 1-minute bins in the

recent past, for DNS queries.

• Description - The agent reached the maximum on observed unique DNS queries and stopped trackingsome DNS queries.

• Recommended Resolution - Check the maximum detected flows and DNS query capacity and scalingrecommendation in the Cisco Stealthwatch Learning Network License Data Sheet, and verify that yourenvironment falls within the recommendation.

Status Code: DNSQKEYSPERBINLIMIT• Status Message - Too many different keys have been observed for too many 1-minute bins

in the recent past, for DNS queries.

• Description - The agent groups DNS queries using unique keys. It reached the maximum on observedDNS query groups and stopped tracking DNS queries that do not have a key, and thus do not belong toa tracked group.

• Recommended Resolution - Check the maximum detected flows and DNS query capacity and scalingrecommendations in the Cisco Stealthwatch Learning Network License Data Sheet, and verify that yourenvironment falls within the recommendations.

Status Code: DNSREVENTSPERBINLIMIT• Status Message - Too many events have been observed for too many 1-minute bins in the

recent past, for DNS replies.

• Description - The agent reached the maximum on observed unique DNS query replies and stoppedtracking some DNS query replies.



TroubleshootingStatus Code: ALLOCFAIL

Status Code: DNSRKEYSPERBINLIMIT• Status Message - Too many different keys have been observed for too many 1-minute bins

in the recent past, for DNS replies.

• Description - The agent groups DNS query replies using unique keys. It reached the maximum onobserved DNS query reply groups and stopped tracking DNS query replies that do not have a key, andthus do not belong to a tracked group.


Status Code: HOSTLIMITEXT• Status Message - Limit of tracked external hosts has been reached for too long in the

recent past

• Description - The agent reached the maximum number of tracked external hosts and stopped trackingsome hosts.

• Recommended Resolution - Check the maximum external host capacity and scaling recommendationin the Cisco Stealthwatch Learning Network License Data Sheet, and verify that your environment fallswithin the recommendation.

Status Code: HOSTLIMITINT• Status Message - Limit of tracked internal hosts has been reached for too long in the

recent past

• Description - The agent reached the maximum number of tracked internal hosts and stopped trackingsome hosts.

• Recommended Resolution - Check the maximum internal host capacity and scaling recommendationin the Cisco Stealthwatch Learning Network License Data Sheet, and verify that your environment fallswithin the recommendation.

Status Code: HOSTSDROPPEDEXT• Status Message - Too many external hosts have been observed in the recent past

• Description - The agent reached the maximum number of tracked unique external hosts and stoppedtracking some hosts.

• Recommended Resolution - Check the maximum external host capacity and scaling recommendationin the Cisco Stealthwatch Learning Network License Data Sheet, and verify that your environment fallswithin the recommendation.


TroubleshootingStatus Code: DNSRKEYSPERBINLIMIT

Status Code: HOSTSDROPPEDINT• Status Message - Too many internal hosts have been observed in the recent past

• Description - The agent reached the maximum number of tracked unique internal hosts and stoppedtracking some hosts.

• Recommended Resolution - Check the maximum internal host capacity and scaling recommendationin the Cisco Stealthwatch Learning Network License Data Sheet, and verify that your environment fallswithin the recommendation.

Status Code: IPLOCCHANGED• Status Message - Too many recently seen hosts have a changed IP locality

• Description - The agent identified hosts as internal or external, and the classification of those hosts laterchanged, possibly due to router configuration updates.

• Recommended Resolution - Take the following steps:

◦From the Network Element, verify your interface configuration, especially if you reconfigured aninterface's direction from internal to external or vice versa.

◦From the controller web UI, verify theDirection configuration of all Network Element interfaces,including recently reconfigured interfaces.

◦If the updated Network Element interface configuration changed a subnet's label from internal toexternal or external to internal, the traffic models must be updated. Shut down the agent and restartit.

Step 1 From the Network Element, verify your interface configuration by running the following commands:enableshow interfacesexit

Step 2 From the controller web UI, verify the Direction configuration of the Network Element interfaces.

• Click Configure next to the agent.

• For each interface, choose Internal if the interface faces the branch, External if the interface faces the core, orUnconfigured if the interface should be ignored.

• Click Submit.

• Click Submit.

Step 3 If a subnet's label changed from internal to external or external to internal, shut down and restart the agent.


TroubleshootingStatus Code: HOSTSDROPPEDINT

Status Code: IPLOCINVAL• Status Message - Too many recently seen hosts have an invalid IP locality

• Description - The agent detects hosts behind interfaces that are not labeled as internal or external,possibly due to newly enabled or reconfigured interfaces on the Network Element.

• Recommended Resolution - Take the following steps:

◦From the Network Element, verify your interface configuration, especially if you enabled orreconfigured an interface.

◦From the controller web UI, verify theDirection configuration of all Network Element interfaces,including recently enabled or reconfigured interfaces.

Step 1 From the Network Element, verify your interface configuration by running the following commands:enableshow interfacesexit

Step 2 From the controller web UI, verify the Direction configuration of the Network Element interfaces.



• Click Submit.

• Click Submit.

Status Code: NECONNFAIL• Status Message - Network Element connection failure

• Description - The agent cannot establish an SSH connection with the Network Element, due to one ormore of the following causes:

◦The configured Network Element IP address is incorrect.

◦The Network Element is not configured for SSHv2 access.

◦An access control list is preventing SSH access from the agent.

◦The Network Element configuration does not allocate sufficient Virtual Teletype (VTY) resourcesfor SSH.



TroubleshootingStatus Code: IPLOCINVAL

◦Examine the Network Element's log to determine a specific error.

◦Ensure the Network Element has SSHv2 configured.

◦Ensure the Network Element does not have an access control list preventing SSH access from theagent.

◦Ensure the Network Element has sufficient VTY resources.

◦From the controller web UI, configure the Network Element IP address for the agent.

Step 1 Review the Network Element's logged error messages to for SSH connection failure.Step 2 From the Network Element command line, run the following commands to verify that SSHv2 is configured:

enableshow sshexit

Step 3 From the Network Element command line, run show access-lists and verify that none of the access control lists blocksthe agent's IP address.

Step 4 From the Network Element command line, run the following commands to verify there are sufficient VTY resources:enableshow usersexit

Step 5 From the controller web UI, take the following steps to configure the Network Element IP address:

• Select AGENTS.

• Click Configure next to an agent.

• Enter the IPv4 address in the Network Element IP field.

• Click Submit.

• Click Submit.

Status Code: NENOAUTH• Status Message - Unable to authenticate to Network Element

• Description - The agent cannot authenticate the SSH connection with the Network Element becausethe credentials are incorrect or not configured.

• RecommendedResolution - From the agent, use the administrator menu to correct the Network Elementcredentials for SSH login.

Step 1 Access the administrator menu. You have the following options:


TroubleshootingStatus Code: NENOAUTH

• If your agent is deployed as a virtual service, log into the agent VM.

• If you agent is installed on a UCS-E server, log into the agent VM, then run the following commands:cd ~/DLA./dla_admin

Step 2 Select the following options in the administrator menu. Enter the Network Element username and password whenprompted.5) Password management1) Change router credentials

Status Code: NENOIP• Status Message - Network Element IP not configured

• Description - The agent configuration does not have a Network Element IP address.

• Recommended Resolution - From the controller web UI, configure the Network Element IP addressfor the agent.

Step 1 From the controller web UI, select AGENTS.Step 2 Click Configure next to an agent.Step 3 Enter the IPv4 address in the Network Element IP field.Step 4 Click Submit.Step 5 Click Submit.

Status Code: NFDRPFLD• Status Message - Dropping NetFlow: required fields missing

• Description - The NetFlow flow record is missing required fields.

• Recommended Resolution - Ensure that the SLN-NF-RECORD flow record configuration is correct. Afteryou verify the flow record configuration, save the Network Element running configuration as a startupconfiguration.

Step 1 From the Network Element command line, run the following commands and verify that the flow record is properlyconfigured:enableconfigure terminal


TroubleshootingStatus Code: NENOIP

show running-config flow record SLN-NF-RECORDexit

Step 2 If the flow record is improperly configured, from the Network Element command line, run the following commands toconfigure the flow record:enableconfigure terminalflow record SLN-NF-RECORD

match ipv4 protocolmatch ipv4 source addressmatch ipv4 destination addressmatch transport source-portmatch transport destination-portcollect datalink mac source address inputcollect datalink mac destination address outputcollect transport tcp flagscollect interface inputcollect interface outputcollect flow directioncollect counter bytescollect counter packetscollect timestamp [absolute | sys-uptime] firstcollect timestamp [absolute | sys-uptime] lastcollect application namecollect routing forwarding-statusend

Step 3 From the Network Element command line, run the following commands to copy the current running configuration tothe startup configuration:enablecopy running-config startup-configend

Status Code: NFDRPNOINTF• Status Message - Dropping NetFlow: internal intfs not configured

• Description - The Network Element interface Direction configuration has not been performed, or hasnot been pushed to the agent.

• Recommended Resolution - From the controller web UI, check the agent Configured status. If it isWaiting, wait for the controller to push the configuration to the agent. If it is Incomplete or Error,verify and correct the interface configuration.

Step 1 From the controller web UI, select AGENTS.Step 2 If the Configured status for an agent isWaiting, wait for the controller to push the configuration to the agent.Step 3 If the Configured status is Incomplete or Error, take the following steps:


TroubleshootingStatus Code: NFDRPNOINTF


• Enter the IPv4 address in the Network Element IP field.


• Click Submit.

• Click Submit.

Status Code: NFDRPSYNT• Status Message - Dropping NetFlow: config file syntax error

• Description - The agent cannot parse the internal_ranges.csv internal IP address file due to a syntaxerror.


◦If you have not intentionally added the file, remove the file from the agent, then restart the agent.

◦If you have intentionally added the file, verify the file format and syntax, then log into the agentVM console, and use the dla_admin script to reimport the file.

Step 1 If you have not intentionally added internal_ranges.csv, from the agent command line, run rm {internal_ranges.csv}

to remove the file, then power down and start the agent VM.Step 2 If you intentionally added internal_ranges.csv, verify that the file is well-formed. Copy the well-formed file to the

agent VM and overwrite the file at /CONF/internal_ranges.csv.Step 3 You have the following options:

• If your agent is deployed as a virtual service, log into the agent VM.

• If your agent is installed on a UCS-E server, log into the agent VM, then run the following commands:cd ~/DLA./dla_admin

Step 4 From the administrator menu, select the following options to copy the file. Provide an IP address, username, filepath forinternal_ranges.csv, and password when prompted.1) File access3) Configuration files3) Get config file from remote system2) internal_ranges.csv


TroubleshootingStatus Code: NFDRPSYNT

Status Code: NFDRPVER• Status Message - Dropping NetFlow: unsupported version

• Description - The configured NetFlow version does not match the version the system expects.

• Recommended Resolution - Ensure NetFlow v9 is configured on the Network Element.

From the Network Element command line, run the following commands to configure NetFlow version 9 on an interface,and repeat for all interfaces:enableconfigure terminalip flow-export version 9interface interface-type interface-numberip flow {ingress | egress}exitend

Status Code: NFEVENTSPERBINLIMIT• Status Message - Too many events have been observed for too many 1-minute bins in the

recent past, for NetFlow events.

• Description - The agent reached the maximum on observed unique NetFlow flows and stopped trackingsome NetFlow flows.

• Recommended Resolution - Check the maximum detected NetFlow flows capacity and scalingrecommendation in the Cisco Stealthwatch Learning Network License Data Sheet, and verify that yourenvironment falls within the recommendation.

Status Code: NFKEYSPERBINLIMIT• Status Message - Too many different keys have been observed for too many 1-minute bins

in the recent past, for NetFlow events.

• Description - The agent groups NetFlow flows using unique keys. It reached the maximum on observedNetFlow flow groups and stopped tracking NetFlow flows that do not have a key, and thus do not belongto a tracked group.

• Recommended Resolution - Check the maximum detected NetFlow flows capacity and scalingrecommendation in the Cisco Stealthwatch Learning Network License Data Sheet, and verify that yourenvironment falls within the recommendation.


TroubleshootingStatus Code: NFDRPVER

Status Code: NFNORCV• Status Message - Not receiving NetFlow

• Description - The agent has not received NetFlow packets from the Network Element in over 10minutes,possibly due to NetFlow misconfiguration.

• Recommended Resolution - From the Network Element, ensure that the Network Element is running,that NetFlow v9 is configured, and that the Learning Network License flow exporter is properlyconfigured.

Step 1 Ensure that the Network Element is running.Step 2 From the Network Element command line, run the following commands and verify that NetFlow version 9 is configured.

enableshow mls ndeexit

Step 3 From the Network Element command line, run the following commands and verify that the flow exporter is properlyconfigured and exporting to the agent IP address on port 6666.enableconfigure terminalshow running-config flow exporter SLN-NF-EXPORTERexit

Step 4 If the flow exporter is incorrectly configured, from the Network Element command line, run the following commandsto configure the flow exporter, replacing <dla-ip-address> with your agent's IP address.configure terminalflow exporter SLN-NF-EXPORTER

destination <dla-ip-address>transport udp 6666template data timeout 300exit

end

Status Code: SOLTCOLLECTIONSLIMIT1• Status Message - The maximum number of level 1 model collections has been reached,

therefore no more model may be created.

• Description - The agent reached the maximum on observable application groups, and cannot createadditional traffic models based on the excess application groups.

• Recommended Resolution - Check the maximum detected application group capacity and scalingrecommendation in the Cisco Stealthwatch Learning Network License Data Sheet, and verify that yourenvironment falls within the recommendation.


TroubleshootingStatus Code: NFNORCV

Status Code: SOLTCOLLECTIONSLIMIT2• Status Message - The maximum number of level 2 model collections has been reached,

therefore no more model may be created.

• Description - The agent reached the maximum on observable source cluster and application group pairs,and cannot create additional traffic models based on the excess source cluster and application grouppairs.

• Recommended Resolution - Check the maximum detected cluster and application group capacity andscaling recommendations in the Cisco Stealthwatch Learning Network License Data Sheet, and verifythat your environment falls within the recommendations.

Status Code: TOPOFAIL• Status Message - Failed to read required topology file

• Description - A topology file, used to process network traffic information, is missing or corrupt.

• Recommended Resolution - From the agent, check the log at LOG/DLC.log to determine the specificerror.

◦If the custom clusters file is missing or corrupted and the agent is deployed as a virtual service,reinstall the agent.

◦If the custom clusters file is missing or corrupted and the agent is installed on a UCS-E server,copy the file from another UCS-E-based agent.

◦If the internal_hosts file is in the error message, use the controller web UI to verify thatconfig.json does not reference internal_hosts_filename file. Contact Cisco Support for moreinformation on whether you should be using the internal_ranges.csv file.

Step 1 If the clusters file is missing or corrupted, and your agent is deployed as a virtual service, reinstall the agent. See theCisco Stealthwatch Learning Network License Virtual Service Installation Guide and the Cisco Stealthwatch LearningNetwork License Release Notes for more information.

Step 2 If the clusters file is missing or corrupted, and your agent is deployed on a UCS-E server, copy the file from anotheragent deployed on a UCS-E server.

Step 3 If the internal_hosts file is in the error message, verify that config.json does not reference internal_hosts_filename.

• From the controller web UI, select AGENTS.

• Next to the affected agent, click Configure.

• Click Edit raw JSON configuration.


TroubleshootingStatus Code: SOLTCOLLECTIONSLIMIT2

Status Code: VERSCOMPONENT• Status Message - Incompatible DLA component versions

• Description - The agent has component executables at different versions from each other.

• Recommended Resolution - Download an upgrade file and upgrade your agent to that version.

Do NOT manually copy a component executable file from one agent to another.Note

Download an upgrade file and upgrade your agent to that version. See the Cisco Stealthwatch Learning Network LicenseVirtual Service Installation Guide, the Cisco Stealthwatch Learning Network License UCS E-Series Blade ServerInstallation Guide, and the Cisco Stealthwatch Learning Network License Release Notes for more information.

Status Code: WARMBADFILE• Status Message - Failed to load warmstart model file

• Description - The agent failed to load a warmstart file.


Status Code: WARMNOFILE• Status Message - Required warmstart file is missing

• Description - The agent is configured with the force_load setting enabled, and a warmstart file ismissing.

• Recommended Resolution - From the controller web UI, remove the force_load setting fromconfig.json.

Step 1 From the controller web UI, select AGENTS.Step 2 Next to the affected agent, click Configure.Step 3 Click Edit raw JSON configuration.Step 4 Remove the force_load setting and save your changes.


TroubleshootingStatus Code: VERSCOMPONENT

Status Code: WARMSTATEVAL• Status Message - Invalid model state before saving warmstart file

• Description - The agent could not save the internal traffic model state, because it was invalid orinconsistent.



TroubleshootingStatus Code: WARMSTATEVAL


TroubleshootingStatus Code: WARMSTATEVAL

troubleshooting - cisco · cisco stealthwatch learning network license ucs e-series server...

Documents