TelScale SNMP and JMX Management

Mobicents 2011 Summit

Agenda

• MSS Achievements• Community• Introducing TelScale• Roadmaps

JMX Management

Mobicents 2011 Summit

JMX Console Hands on

confidential

• MSS Core Beans• Connector Beans• Application Beans• JBoss Cache Beans

Sip Servlets Management Console Hands on

confidential

• Manage DAR configuration and the DAR extensions• Manage the basic controls• Deploy examples• Not secure by default• Only for development environment• Only manages the local instance• It is not aware of the clustering and other instances• http://localhost:8080/sip-servlets-management/

Jopr/JON Management Console Hands on

confidential

• Manage and Monitor • Deploy/Undeploy• Can be secured• For development and production environment• manages the local instance• Can be aware of the clustering and other instances• http://localhost:8080/admin-console

Twiddle

confidential

• For command line management and shell scripting• Access JMX information from command line

./twiddle.sh -s jnp://localhost:1099 invoke "jboss.web:host=localhost,path=/click2call,type=SipManager" listSessionIds

SNMP Management

Mobicents 2011 Summit

SNMP Description

confidential

• Simple Network Management Protocol (SNMP) is the most popular network management protocol in the TCP/IP protocol suite, using a simple request/response protocol that communicates management information about the configuration and status of nodes between two types of SNMP software entities: SNMP applications,

also called SNMP managers and SNMP agents. • SNMP agents expose management data on the managed systems as variables. The variables accessible via

SNMP are organized in hierarchies (tree-based). SNMP uses an extensible design, where the available information and other metadata (such as type and description of the variable) is described by Management Information Bases (MIBs). MIBs describe the structure of the management data of a device subsystem

• Objects in the MIB are defined using a subset of Abstract Syntax Notation One (ASN.1) called "Structure of Management Information Version 2 (SMIv2)"

• Object identifiers (OID) uniquely identifies managed objects in the MIB hierarchy

SNMP Protocol Data Units

confidential

• SNMPv1 specified five core protocol data units (PDUs). Two other PDUs, GetBulkRequest and InformRequest were added in SNMPv2 and carried over to SNMPv3.

• The seven SNMP protocol data units (PDUs) are as follows:• Response Returns variable bindings and acknowledgement from agent to manager for below GetRequest, SetRequest, GetNextRequest, GetBulkRequest

and InformRequest. Error reporting is provided by error-status and error-index fields. Although it was used as a response to both gets and sets, this PDU was called GetResponse in SNMPv1.

• GetRequest A manager-to-agent request to retrieve the value of a variable or list of variables. • SetRequest A manager-to-agent request to change the value of a variable or list of variables. • The 3 above PDUs are covered by Accessing JMX Beans Attributes (SNMP GET and SET)• GetNextRequest A manager-to-agent request to discover available variables and their values. Returns a Response with variable binding for the

lexicographically next variable in the MIB. The entire MIB of an agent can be walked by iterative application of GetNextRequest starting at OID 0. Rows of a table can be read by specifying column OIDs in the variable bindings of the request.

• GetBulkRequest Optimized version of GetNextRequest. A manager-to-agent request for multiple iterations of GetNextRequest. Returns a Response with multiple variable bindings walked from the variable binding or bindings in the request. PDU specific non-repeaters and max-repetitions fields are used to control response behavior.

• The 2 above PDUs are covered by Accessing JMX wildcards or Beans Attributes as Tables (SNMP getnext, SNMP getbulk)• Trap Asynchronous notification from agent to manager. Includes current sysUpTime value, an OID identifying the type of trap and optional variable bindings. • InformRequest Acknowledged asynchronous notification from manager to manager. This PDU uses the same format as the SNMPv2 version of Trap.

Manager-to-manager notifications were already possible in SNMPv1 (using a Trap), but as SNMP commonly runs over UDP where delivery is not assured and dropped packets are not reported, delivery of a Trap was not guaranteed. InformRequest fixes this by sending back an acknowledgement on receipt. Receiver replies with Response parroting all information in the InformRequest.

SNMP Adaptor

confidential

• Mobicents now features a revamped (from the old one present in JBoss AS) SNMP Adaptor, leveraging snmp4j as a Java implementation of SNMP, that wraps the JMX MBean and Notifications to allow monitoring and configuration of the Application Server and applications deployed on it through the SNMP protocol

• The Adaptor now supports all versions of the protocol, v1, v2c and v3 including security for v2c and v3. It also has supports for tables and both traps and informs.

• The Adaptor is broke down in 2 pieces :o The SNMP Adaptor JBoss Serviceo The SNMP Adaptor JBoss Deployer

SNMP Adaptor JBoss Service

confidential

• The SNMP Adaptor JBoss Service is located in $JBOSS_HOME/server/<profile>/deploy/snmp-adaptor.sar

• Contains the general configuration of the SNMP service and the JMX to SNMP mapping for Mobicents Sip Servlets, JBoss AS and its subcomponents

• The SNMP Adaptor Service Agent listen on a given UDP port (default 1161). The manager may send requests from any available source port to port 1161 in the Agent. The Agent responses will be sent back to the source port of the manager sent requests.

• The SNMP Adaptor Service also provides a manager that can receive notifications (Traps and Inform Requests) on port 1162.

• The SNMP Adaptor Service Agent may generate notifications from any available port on behalf of the JBoss Application Server or its components and applications deployed within it.

SNMP Adaptor JBoss ServiceConfiguration File

confidential

• The SNMP Adaptor JBoss Service configuration file is located in $JBOSS_HOME/server/<profile>/deploy/snmp-adaptor.sar/META-INF/jboss-service.xml

• Contains 3 MBeans :o The SNMP Adaptor Manager org.jboss.jmx.adaptor.snmp.trapd.TrapdServiceo The SNMP Adaptor Agent org.jboss.jmx.adaptor.snmp.agent.SnmpAgentService : It defines a set of

properties to override the default behavior of the Agent and the location of the other configuration files used by the agent. It also defines the list of Subscriptions to JMX notifications from the JBoss Application Server, its components and applications deployed within it that the agent filters and will map to SNMP Notifications. This subscriptionList attribute warrants a specific section as it is used in conjunction with the Notifications configuration file. It defines a list of JMX Notifications emitted by MBeans (the name can use wilcard to match multiple MBeans at the same time) to intercept that will be converted by the SNMP Adaptor as an SNMP Notification through the Notifications configuration file. The type of the notification will be used to find in the Notifications file how to map it, which OID to give it and which information will be sent.

o System Information as specified in RFC 1213 org.jboss.jmx.adaptor.snmp.system.MIB2SystemGroupService

SNMP Adaptor JBoss ServiceManagers File

confidential

• The SNMP Adaptor JBoss Service configuration file is located in $JBOSS_HOME/server/<profile>/deploy/snmp-adaptor.sar/managers.xml

• configure the SNMP managers to which the generated SNMP traps should be forwarded.

• Each manager entry must specify:o address : the manager's address (IP address or host name)o port : the listening porto local-address : the local binding address (optional)o local-port : the local binding port (optional)o version : the SNMP version to use (1 or 2 or 3)o community-string : If version is 1 or 2, the community string (public or private)

SNMP Adaptor JBoss ServiceUsers File

confidential

• The SNMP Adaptor JBoss Service configuration file is located in $JBOSS_HOME/server/<profile>/deploy/snmp-adaptor.sar/users.xml

• Configure the SNMP users when using version 3 of the protocol.• Each user entry must specify :

o securityName : the security name of the user (typically the user name).o authenticationProtocol : the authentication protcol to be associated with this user (MD5 or SHA). If not

set, this user only supports unauthenticated messages.o authenticationPassphrase : If present, the authentication Protocol must also be present. RFC3414

§11.2 requires passphrases to have a minimum length of 8 bytes. If the length is less than 8 bytes an IllegalArgumentException will be thrown.

o privacyProtocol : the privacy protcol ID to be associated with this user (DES, 3DES, AES128, AES192, AES256). If not set, this user only supports unencrypted messages.

o privacyPassphrase : If present, the privacy Protocol must also be set. RFC3414 §11.2 requires passphrases to have a minimum length of 8 bytes. If the length is less than 8 bytes an IllegalArgumentException will be thrown.

SNMP Adaptor JBoss ServiceAttributes Configuration File

confidential

• The SNMP Adaptor JBoss Service configuration file is located in $JBOSS_HOME/server/<profile>/deploy/snmp-adaptor.sar/mss-attributes.xml

• Controls the mapping of JMX mbean attributes to SNMP OIDs accessible through the SNMP Adaptor. It defines a list of MBean and their attributes to map to SNMP OID.

• Each mbean entry must specify :o name : The MBean name for which its attributes need to be exposed. The MBean name can be a

wildcard, in which case the returned set will be exposed as an SNMP Table o oid-prefix (optional) : if set it will be prepended to the attributes oid.o a list of attribute that specifies :

name it needs to match the one that has to be exposed from the MBean oid identifier of the attribute. It can either be the full oid or only the last part of it if the oid-prefix was

set above. All attributes are Scalar objects by default which means that to access them you will need to add a .0 to the attribute oid to access it

table if the attribute has to be exposed as a Tabular object ie a SNMP table (currently if set to true only array, List and Map are supported and within those only basic types (String, int, long, boolean, otherwise Object.toString will be called)

mode if set to *rw*, it means that the attribute is writable (can be set) otherwise by default it is read-only

SNMP Adaptor JBoss ServiceAttributes Scalar/Tabular

confidential

• Scalar object A Scalar Object defines a single object instance. A scalar object by definition has only one instance as opposed to a table. Therefore, an instance index of zero is always specified when retrieving the value of a scalar object. An instance index higher than 0 can only be used in the case of tabular objects, which can have multiple instances.

• Tabular object A Tabular Object defines multiple related object instances that are grouped in MIB tables. The JBoss SNMP Adaptor uses two types of ways to access values in a table :

• a separate index column for tabular objects that represents arrays and java.util.List since they are indexed, but to query the Java index that starts at 0 in an array or List, it will start at index .1 for SNMP as .0 is reserved for scalar objects

• For tabular objects whose type is a java.util.Map or a wildcard used in a MBean, the index column will be named as per the key of the data in the row.

SNMP Adaptor JBoss ServiceNotifications Configuration File

confidential

• The SNMP Adaptor JBoss Service configuration file is located in $JBOSS_HOME/server/<profile>/deploy/snmp-adaptor.sar/mss-notifications.xml

• Controls the mapping of JMX notifications to SNMP Traps. The key for the mapping is considered to be the notification type as defined in the subscriptions list of the SNMP Adaptor JBoss Service configuration file

• Each mapping entry must specify :o notification-type : The type of JMX Notifications to wrap as a SNMP Notification. Please note that the

notification has to be caught first by the subscription list filter defined in SNMP Adaptor JBoss Service configuration file

o inform : if set to "true", this trap will be send as an inform (acknowledged trap)o security-name : the user to associate with the sending of the trap (will be used for version 3 of the

protocol only)o var-bind-list whose defines a wrapper-class that allows to to extract info from arbitrary notification

subclasses. The generated trap may contain an arbitrary payload of variable bindings (var-bind) with explicitly specified oid and values extracted from the emitted notification (tag).

SNMP Adaptor JBoss ServiceNotifications Configuration File

confidential

Tag Description

n:message javax.management.Notification.getMessage()

n:sequenceNumber javax.management.Notification.getSequenceNumber()

n:timeStamp javax.management.Notification.getTimestamp()

n:type javax.management.Notification.getType()

n:all javax.management.Notification.toString()

n:class javax.management.Notification.getClass().getName()

a:startTime snmp agent/adaptor start time

a:uptime snmp agent/adaptor up time

a:trapCount snmp agent/adaptor trap count

u:<user-defined-tag> looked-up in n.getUserData()

SNMP Adaptor Deployer

confidential

• The SNMP Adaptor JBoss Service is located in $JBOSS_HOME/server/<profile>/deployers/snmp.deployer

• The SNMP Adaptor Deployer dynamically looks for snmp-attributes.xml and snmp-notifications.xml files in any applications deployed within JBoss Application Server and register the notifications, attributes and their JMX mapping with the SNMP Adaptor JBoss Service.

SNMP Adaptor MIB Generator

confidential

• A way for a user to generate an SNMP MIB file based on already defined attributes.xml and notifications.xml. This MIB file allows an SNMP manager to display metrics in a much more human readable way than would otherwise be possible, and the generator allows the user to skip the arduous task of manually creating an MIB file.

SNMP Adaptor WalkThroughAccessing JMX Scalar Attrs

confidential

• MSS SNMP Adaptor is exposing out of the box a certain number of MBean and their attributes in the mss-attributes.xml configuration file

• Exposing a JMX Attribute as Scalar Object and reading its value through SNMP GetRequest :

snmpget -v 3 -u TEST -a MD5 -A "maplesyrup" -x DES -X "maplesyrup" -l authPriv 127.0.0.1:1161 .1.3.6.1.2.1.1.1.0

• Exposing a JMX Attribute as Scalar Object and setting its value through SNMP SetRequest

snmpset -v 3 -u TEST -a MD5 -A "maplesyrup" -x DES -X "maplesyrup" -l authPriv 127.0.0.1:1161 .1.3.6.1.2.1.1.1.0 s "Mobicents Sip Servlets rocks"

• Discovering the next OID/JMX Attribute to query through SNMP GetNextRequestsnmpgetnext -v 3 -u TEST -a MD5 -A "maplesyrup" -x DES -X "maplesyrup" -l authPriv 127.0.0.1:1161 .1.3.6.1.4.1.2312.100.1.1.20.1.1.0

Application showcasingSNMP/JMX Mapping

confidential

• Deploy snmp-servlet-5.1.0.GA.war to default profile.• showcase a basic web application that provides a web page which displays a counter, each time the page is

refreshed the counter is incremented. Alternatively, a message can be set through JMX that will display on the web page as well. The application also register a MBean providing the following set of monitoring information :

• counter : the current value of the counter• message : allows to retrieve and set the value of the message that will be displayed on the page along with the counter• messageHistory : all the values that have been set since the application is running returned as a java.util.List<String>• messageHistoryAsArray : all the values that have been set since the application is running returned as a String array.• countHistory : all the values that have been displayed since the application is running returned as a primitive int array.• messageCountHistory : a Map which have as a key the message and as a value the corresponding counter value that was

present when the message was set.• The application also send will make use of the SNMP Agent MBean Service to send a SNMP trap on when the application is

deployed.• Set a message then refresh the pages 3 times, then set another message and refresh again 3 times.

SNMP Adaptor WalkThroughAccessing JMX Tabular Attrs(Index Based)

confidential

• MessageHistory Attribute is of type java.util.List<String> so is a Tabular Object index based that will be used for the following sub sections as an example.

• The query OID should be the concatenation of the oid prefix (.1.3.6.1.4.1.2312.100.10.11) + the MessageHistory OID (.3.1) + the value column (.2) + the index to retrieve the value from (.1) (remember that to access the index 0, .1 should be used because .0 are reserved for scalar objects

• Exposing a JMX Attribute as Tabular Object (Index Based) and reading its value through SNMP GetRequest

snmpget -v 3 -u TEST -a MD5 -A "maplesyrup" -x DES -X "maplesyrup" -l authPriv 127.0.0.1:1161 .1.3.6.1.4.1.2312.100.10.11.3.1.2.1

• To get the second messagesnmpget -v 3 -u TEST -a MD5 -A "maplesyrup" -x DES -X "maplesyrup" -l authPriv 127.0.0.1:1161 .1.3.6.1.4.1.2312.100.10.11.3.1.2.2

• Exposing a JMX Attribute as Tabular Object (Index Based) and setting its value through SNMP SetRequest

snmpset -v 3 -u TEST -a MD5 -A "maplesyrup" -x DES -X "maplesyrup" -l authPriv 127.0.0.1:1161 .1.3.6.1.4.1.2312.100.10.11.3.1.2.2 s "overriding second message"

SNMP Adaptor WalkThroughIterating JMX Tabular Attrs(Index Based)

confidential

• Iterating through a Tabular Object (Index Based) through SNMP GetNextRequestsnmpgetnext -v 3 -u TEST -a MD5 -A "maplesyrup" -x DES -X "maplesyrup" -l authPriv 127.0.0.1:1161 .1.3.6.1.4.1.2312.100.10.11.3.1snmpgetnext -v 3 -u TEST -a MD5 -A "maplesyrup" -x DES -X "maplesyrup" -l authPriv 127.0.0.1:1161 .1.3.6.1.4.1.2312.100.10.11.3.2snmpgetnext -v 3 -u TEST -a MD5 -A "maplesyrup" -x DES -X "maplesyrup" -l authPriv 127.0.0.1:1161 .1.3.6.1.4.1.2312.100.10.11.4.1.2.1

• Iterating through a Tabular Object (Index Based) through SNMP GetBulkRequestsnmpbulkget -v 3 -Cr2 -u TEST -a MD5 -A "maplesyrup" -x DES -X "maplesyrup" -l authPriv 127.0.0.1:1161 .1.3.6.1.4.1.2312.100.10.11.3.1

SNMP Adaptor WalkThroughAccessing JMX Tabular Attrs(Key Based)

confidential

• MessageCountHistory Attribute is of type java.util.Map<String, Integer> so is a Tabular Object key based that will be used for the following sub sections as an example.

• As opposed to an index based Tabular Object, there is no index to iterate through for the rows of a table, so there is no way to issue a GetRequest PDU directly except if the table contents is already known to the querier. So as for index based tabular objects by specifying the table OID ie for MessageCountHistory table attribute the OID will be .1.3.6.1.4.1.2312.100.10.11.6.1, it is actually possible to iterate through the full table by issuing a set of GetNextRequest SNMP PDU.

• Iterating through a Tabular Object (Key Based) through SNMP GetNextRequestsnmpgetnext -v 3 -u TEST -a MD5 -A "maplesyrup" -x DES -X "maplesyrup" -l authPriv 127.0.0.1:1161 .1.3.6.1.4.1.2312.100.10.11.6.1which gives iso.3.6.1.4.1.2312.100.10.11.6.1.2.109.101.115.115.97.103.101.32.49 = INTEGER: 3The OID is a concatenation of the table entry OID (.1.3.6.1.4.1.2312.100.10.11.6.1) and of the ASN.1 representation of the first entry key of the java.util.Map MessageCountHistory attribute (ie "message 1"), to get the next key of the map, the OID of the response has to be used just like for regular index based tabular objects so to get the next entry OID and value the following command should be issuesnmpgetnext -v 3 -u TEST -a MD5 -A "maplesyrup" -x DES -X "maplesyrup" -l authPriv 127.0.0.1:1161 .1.3.6.1.4.1.2312.100.10.11.6.1.2.109.101.115.115.97.103.101.32.49which gives iso.3.6.1.4.1.2312.100.10.11.6.1.2.109.101.115.115.97.103.101.32.50 = INTEGER: 6

SNMP Adaptor WalkThroughAccessing JMX Tabular Attrs(Key Based)

confidential

• The GetBulkRequest was introduced in SNMP version 2 because iterating through a table with GetNext proved to be tedious and inefficient. So GetBulkRequest was introduced to allow fetching a table completely with one request instead of multiple GetNext requests. So to iterate through the MessageHistory table the following command should be issued

• Iterating through a Tabular Object (Key Based) through SNMP GetBulkRequestsnmpbulkget -v 3 -u TEST -a MD5 -A "maplesyrup" -x DES -X "maplesyrup" -l authPriv 127.0.0.1:1161 .1.3.6.1.4.1.2312.100.10.11.6.1 which gives iso.3.6.1.4.1.2312.100.10.11.6.1.2.109.101.115.115.97.103.101.32.49 = INTEGER: 3iso.3.6.1.4.1.2312.100.10.11.6.1.2.109.101.115.115.97.103.101.32.50 = INTEGER: 6

• Exposing a JMX Attribute as Tabular Object (Key Based) and reading its value through SNMP GetRequest

Query the OID that should be the concatenation of the oid prefix (.1.3.6.1.4.1.2312.100.10.11) + the MessageCountHistory OID (6.1) + the value column (.2) + the ASN.1 representation of the key of the table whose value wants to be retrieved (message 1)snmpget -v 3 -u TEST -a MD5 -A "maplesyrup" -x DES -X "maplesyrup" -l authPriv 127.0.0.1:1161 iso.3.6.1.4.1.2312.100.10.11.6.1.2.109.101.115.115.97.103.101.32.49

• Exposing a JMX Attribute as Tabular Object (Key Based) and setting its value through SNMP SetRequest

snmpset -v 3 -u TEST -a MD5 -A "maplesyrup" -x DES -X "maplesyrup" -l authPriv 127.0.0.1:1161 iso.3.6.1.4.1.2312.100.10.11.6.1.2.109.101.115.115.97.103.101.32.49 i 69

SNMP Adaptor WalkThroughAccessing JMX Wilcard asa Table

confidential

• The SNMP Adaptor allows for the mss-attributes.xml configuration file to specify in the MBean name a mbean query by using wildcards to get back a list of MBean which can be particularly useful for Sip Applications :<mbean name="jboss.web:host=localhost,path=/*,type=SipManager" oid-prefix=".1.3.6.1.4.1.2312.100.1.1.13.1" definition-name="sipApplications" table-name="sipMetrics" description="Table containing standard metrics for any webapp deployed on JBossAS" status="current">

• Iterating through a Tabular Object (Key Based) through SNMP GetBulkRequestsnmpgetnext -v 3 -u TEST -a MD5 -A "maplesyrup" -x DES -X "maplesyrup" -l authPriv 127.0.0.1:1161 .1.3.6.1.4.1.2312.100.1.1.13.1 which gives iso.3.6.1.4.1.2312.100.1.1.13.1.2.106.98.111.115.115.46.119.101.98.58.116.121.112.101.61.83.105.112.77.97.110.97.103.101.114.44.112.97.116.104.61.47.99.108.105.99.107.50.99.97.108.108.44.104.111.115.116.61.108.111.99.97.108.104.111.115.116 = INTEGER: 0The OID is the concatenation of the OID prefix (.1.3.6.1.4.1.2312.100.1.1.13.1), first attribute (distributable, .2) and the ASN.1 representation of the following web application ObjectName "jboss.web:type=SipManager,path=click2call/,host=localhost"Important Note: SNMP don't iterate through a full row before moving to the next one, it rather do it by column so keeping on the same attribute until all rows have been iterated before moving to the next attribute, it means that querying with GetNextRequest PDU will give you the same attribute (distributable) but for the next application

• Iterating through a Tabular Object (Key Based) through SNMP walksnmpwalk -v 3 -u TEST -a MD5 -A "maplesyrup" -x DES -X "maplesyrup" -l authPriv 127.0.0.1:1161 .1.3.6.1.4.1.2312.100.1.1.13.1

Sending SNMP Traps

confidential

• When started the SNMP Adaptor sends a JMX Notification that is converted into an SNMP Trap

• When a JMX Notification is sent through JBoss Application Server, the SNMP Adaptor converts it with the SNMP Adaptor Notifications configuration File then the SNMP Trap (or INFORM if the inform attribute is set to "true") will be sent to each Manager defined in SNMP Adaptor Managers Configuration File

confidential

Thank you !

http://telestax.com/