monitoring guide - tibco software

TIBCO Mashery® LocalMonitoring GuideVersion 4.x

November 2020

Copyright © 2006-2020. TIBCO Software Inc. All Rights Reserved.

Contents

TIBCO Documentation and Support Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Monitoring Mashery Local 4.x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4

Node Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Component Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6

Traffic Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

MySQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Memcache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15

Memcache Loader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Status of Cloud Sync Services, Developer, Tokens and Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Legal and Third-Party Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

2

TIBCO Mashery® Local Monitoring Guide

TIBCO Documentation and Support Services

How to Access TIBCO Documentation

Documentation for TIBCO products is available on the TIBCO Product Documentation website, mainly inHTML and PDF formats.

The TIBCO Product Documentation website is updated frequently and is more current than any otherdocumentation included with the product. To access the latest documentation, visit https://docs.tibco.com.

Product-Specific Documentation

Documentation for TIBCO products is not bundled with the software. Instead, it is available on the TIBCODocumentation site at TIBCO Mashery® Local documentation page.

The following documents for this product can be found on the TIBCO Documentation website:

● TIBCO Mashery® Local Release Notes

● TIBCO Mashery® Local Installation and Configuration Guide

● TIBCO Mashery® Local Migration and Upgrade Guide

● TIBCO Mashery® Local Reporting Services

For information on TIBCO Cloud Integration with Mashery, refer to Integrating with TIBCO Mashery.

How to Contact TIBCO Support

You can contact TIBCO Support in the following ways:

● For an overview of TIBCO Support, visit http://www.tibco.com/services/support.

● For accessing the Support Knowledge Base and getting personalized content about products you areinterested in, visit the TIBCO Support portal at https://support.tibco.com.

● For creating a Support case, you must have a valid maintenance or support contract with TIBCO. Youalso need a user name and password to log in to https://support.tibco.com. If you do not have a username, you can request one by clicking Register on the website.

How to Join TIBCO Community

TIBCO Community is the official channel for TIBCO customers, partners, and employee subject matterexperts to share and access their collective experience. TIBCO Community offers access to Q&A forums,product wikis, and best practices. It also offers access to extensions, adapters, solution accelerators, andtools that extend and enable customers to gain full value from TIBCO products. In addition, users cansubmit and vote on feature requests from within the TIBCO Ideas Portal. For a free registration, go to https://community.tibco.com.

3


https://docs.tibco.com

https://docs.tibco.com/products/tibco-mashery-local

https://integration.cloud.tibco.com/docs/using/using-apps/managing-app-endpoints/integrating-with-mashery.html

http://www.tibco.com/services/support

https://support.tibco.com

https://support.tibco.com

https://ideas.tibco.com/

https://community.tibco.com

Monitoring Mashery Local 4.x

Each node of a Mashery Local cluster has the following main components - Traffic Manager, MySQL andMemCache.

To ensure reliability of the Mashery Local cluster, it is important to monitor these components and set upalerts when resources go low either at the node level or at the process level, for each application. At thenode level, resources such as CPU, Memory, Disk and Network I/O can be monitored. At the process level,each process-specific resource can be monitored. For example, monitoring the percentage of availablememory in MemCache is important to ensure reliability and optimal performance of MemCache.

Data Collection

Statistics are collected in the following ways:

● by making HTTP calls from your monitoring system.

● by forwarding log files to your monitoring system.

Both ways are needed as some statistics can be collected only by making HTTP calls and other statistics byforwarding logs from the host machine where the logs are generated.

Monitoring System

A monitoring system is a log processing software that may provide log collection agents like Splunkforwarder for Splunk or FileBeat for elastic search. Please consult your devops team on which monitoringsolution to use.

4


Node Metrics

The health of the node of a Mashery Local cluster can be monitored by tracking the use of resources likeCPU, Memory, Disk and Network I/O. Alerts can be set if the resources are going low below certain levels.For example, alerts can be set if:

● Available CPU resources are below 50%,

● Swap file is being used which means that RAM is fully utilized, or

● Available disk is space is below 20%

The above numbers are just examples. Contact your Mashery Customer Success team for your specificthreshold values.There are many third-party tools available to collect these metrics, for example, ELK userscan use MetricBeat.

Diagnostic Recipe / Alerts

Metrics Notes

CPU resources are low Consider adding more nodes to distribute traffic

Swap file usage has increased and availablememory is low

Consider increasing the RAM size for the node

Disk space running low Check log file sizes and consider purging logs.

Check expired token count in MySQL andconsider deleting more aggressively if the count ishigh.

Consider increasing the disk space.

5


Component Monitoring

● Traffic Manager

● MySQL

● Memcache

● Memcache Loader

● Status of Cloud Sync - Services, Developer, Tokens and Logs

Traffic Manager

Jetty Pool Connection Statistics

Traffic Manager maintains a thread pool to process incoming requests. Monitoring the statistics of this poolcan help in determining the health of the traffic manager.

The stats can be collected by adding a cron job to every node of Mashery Local cluster. The cron job makes acURL call and appends the results in a log file which can be forwarded to the log processing system. This isavailable for Mashery Local 4.4 and above for both Docker and OVA versions. If using the Docker version,log into the container ml-tm before running the following command.curl -s --compressed -H "Accept: application/json" -X GEThttp://127.0.0.1:8082/jmx/com.mashery.proxy.management:name=JettyConnectionPool,type=ConectionPool

Connection Pool Statistics{ "results": [{ "connectionData": [{ "Connections": 0, "ConnectorUptimeSec": 2195, "MaxConnections": 1, "MaxConnectionDuration": 0.056, "ConnectorUptime": " 0 hours 36 minutes 35 seconds", "Port": 80, "MaxRequestsPerConnection": 1, "TotalRequests": 36, "ConnectionDurationStdDev": 0.0075163440961300175, "RequestsPerConnectionStdDev": 0.0, "Keystore": "Non SSL", "TotalConnectionDuration": " 0 hours 0 minutes 0 seconds", "AvgConnectionDuration": 0.008638888888888889, "TotalConnections": 36, "AvgRequestsPerConnection": 1.0, "TotalConnectionDurationSec": 0 }], "threadPoolIdleThreads": 48, "threadPoolMaxThreads": 500, "threadPoolMinThreads": 50, "threadPoolThreads": 50 }]}

6


Parameters

Name Field Notes Details

Max requests threads threadPoolMaxThreads Denotes themaximumconcurrent requestspossible on thisTraffic Managerinstance

Inital number of threads threadPoolMinThreads Threads to becreated on startup

Current created threads threadPoolThreads Number of threadscreated in the pool

Idle Threads threadPoolIdleThreads Number of threadscreated but are idle(at the time of takingthe snapshot)

A high value of idlethreads denotes lowutilization of thisinstance. Check ifother TrafficManager instancesare over-loaded.

Total connections sincestart

Connections Number ofconnection requeststo this instance

Avg time for requests AvgConnectionDuration Average time takenby the this instanceto server traffic

A low value denoteshigh throughput


Metric Field Name / Computation Notes

Efficiency threadPoolThreads/

threadPoolMaxThreads

This ratio provides the utilization of aTraffic Manager. Though it may dependupon traffic patterns, a desired range is75%-80%.

Low water mark - 80%

High water mark - 90%

A continuous breach of high water markindicates that this Traffic Manager instanceis over loaded.

7



Under-utilization

threadPoolIdleThreads/

threadPoolMaxThreads

A very high ratio denotes that the TrafficManager is handling fewer calls and mightindicate deficiency in load balancing rules.However, planning for peak usage is veryimportant and during other times, it ispossible that nodes show under-utlization.

Max connections connectionData.MaxConnections This is the maximum connections open atthe same time, thereby giving the peakusage value. An alert can notify if this valueis within 80%. Upon receiving this alert,consider increasing the number of nodes.Increasing the threadPoolMaxThreads is anoption but it depends if there are enoughCPU and Memory resources to handle ahigher number of concurrent requests in thesame node. A higher number of concurrentrequests will require more resources byGarbage Collection Processes.

Old Garbage Collection Statistics

The stats can be collected by adding a cron job to every node of the Mashery Local cluster. The cron job callsa URL and appends the results in a log file that can be forwarded for further processing. This is available inMashery Local 4.3 and above for the OVA version only.

For example:curl -s --compressed -H "Accept:application/json" -X GEThttp://127.0.0.1:8082/jmx/java.lang:name=PS%20MarkSweep,type=GarbageCollector

The default configuration to run the Traffic Manager does not specify any specific Garbage Collector andJVM chooses "PS MarkSweep" as the old generation garbage collector. For more information about GarbageCollection Tuning, see the HotSpot Virtual Machine Tuning Guide for your version of Java Standard Editionin Oracles's online documentation.{ "results": [{ "LastGcInfo": { "gcInfo": { "compositeType": { "description": "CompositeType for GC info for PS MarkSweep", "array": false, "typeName": "sun.management.PS MarkSweep.GcInfoCompositeType", "className": "javax.management.openmbean.CompositeData" }, "startTime": 544366, "endTime": 544426, "duration": 60, "memoryUsageBeforeGc": { "PS Old Gen": { "max": 716177408, "committed": 44040192, "used": 21829792, "init": 41943040 } }, "memoryUsageAfterGc": { "PS Old Gen": { "max": 716177408, "committed": 61865984,

8


HTTPS://DOCS.ORACLE.COM/EN/

"used": 11530416, "init": 41943040 } }, "id": 2 }, "compositeType": { "description": "CompositeType for GC info for PS MarkSweep", "array": false, "typeName": "sun.management.PS MarkSweep.GcInfoCompositeType", "className": "javax.management.openmbean.CompositeData" } }, "CollectionCount": 2, "CollectionTime": 100, "Valid": true, "MemoryPoolNames": ["PS Eden Space", "PS Survivor Space", "PS Old Gen"], "Name": "PS MarkSweep", "ObjectName": { "domain": "java.lang", "pattern": false, "domainPattern": false, "propertyPattern": false, "propertyListPattern": false, "propertyValuePattern": false, "keyPropertyList": { "name": "PS MarkSweep", "type": "GarbageCollector" }, "keyPropertyListString": "type=GarbageCollector,name=PS MarkSweep", "canonicalKeyPropertyListString": "name=PS MarkSweep,type=GarbageCollector", "canonicalName": "java.lang:name=PS MarkSweep,type=GarbageCollector" } }]}


See the Oracle Garbage Collection Tuning guide for more information.

New Garbage Collection Statistics

The stats can be collected by adding a cron job to every node of Mashery Local cluster. The cron job calls aURL and appends the results in a log file that can be forwarded for further processing. This is available forMashery Local 4.3 and above for the OVA versions only. curl -s--compressed -H "Accept: application/json" -X GEThttp://127.0.0.1:8082/jmx/java.lang:type=GarbageCollector,name=PS%20Scavenge

The default configuration to run the Traffic Manager does not specify any specific Garbage Collector andJVM chooses "PS Scavenge" as the default New generation garbage collector. For more information aboutGarbage Collection Tuning, see the HotSpot Virtual Machine Tuning Guide for your version of JavaStandard Edition in Oracles's online documentation.{ "results": [{ "LastGcInfo": { "gcInfo": { "compositeType": { "description": "CompositeType for GC info for PS Scavenge", "array": false, "typeName": "sun.management.PS Scavenge.GcInfoCompositeType", "className": "javax.management.openmbean.CompositeData" }, "startTime": 2045258, "endTime": 2045263, "duration": 5, "memoryUsageBeforeGc": { "PS Survivor Space": {

9


https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/index.html

HTTPS://DOCS.ORACLE.COM/EN/

"max": 3670016, "committed": 3670016, "used": 1148864, "init": 2621440 }, "Metaspace": { "max": -1, "committed": 40239104, "used": 36625520, "init": 0 }, "PS Eden Space": { "max": 350224384, "committed": 155713536, "used": 155713536, "init": 15728640 } }, "memoryUsageAfterGc": { "PS Survivor Space": { "max": 3670016, "committed": 3670016, "used": 2022880, "init": 2621440 }, "Metaspace": { "max": -1, "committed": 40239104, "used": 36625520, "init": 0 }, "PS Eden Space": { "max": 350224384, "committed": 157286400, "used": 0, "init": 15728640 } }, "id": 62 }, "compositeType": { "description": "CompositeType for GC info for PS Scavenge", "array": false, "typeName": "sun.management.PS Scavenge.GcInfoCompositeType", "className": "javax.management.openmbean.CompositeData" } }, "CollectionCount": 62, "CollectionTime": 372, "Valid": true, "MemoryPoolNames": ["PS Eden Space", "PS Survivor Space"], "Name": "PS Scavenge", "ObjectName": { "domain": "java.lang", "pattern": false, "domainPattern": false, "propertyPattern": false, "propertyListPattern": false, "propertyValuePattern": false, "keyPropertyList": { "name": "PS Scavenge", "type": "GarbageCollector" }, "keyPropertyListString": "type=GarbageCollector,name=PS Scavenge", "canonicalKeyPropertyListString": "name=PS Scavenge,type=GarbageCollector", "canonicalName": "java.lang:name=PS Scavenge,type=GarbageCollector" } }]}


10


For more information about Garbage Collection Tuning, see Oracle's Garbage Collection Tuning guide.

MySQL

Get Status

The status can be obtained periodically by calling the URL:https://<IP address of the node>:5480/service/mashery/cgi/mysql_status_cgi.py

The response is a JSON that has the status of the mysql server.

MySQL Statistics{ "Connection_errors_accept": 0, "Connection_errors_internal": 0, "Connection_errors_max_connections": 0, "Connection_errors_peer_address": 0, "Connection_errors_select": 0, "Connection_errors_tcpwrap": 0, "Connections": 234, "Delayed_insert_threads": 0, "Innodb_buffer_pool_dump_status": "Dumping", "Innodb_buffer_pool_load_status": "Buffer", "Innodb_buffer_pool_resize_status": "", "Innodb_buffer_pool_pages_data": 1802, "Innodb_buffer_pool_bytes_data": 29523968, "Innodb_buffer_pool_pages_dirty": 0, "Innodb_buffer_pool_bytes_dirty": 0, "Innodb_buffer_pool_pages_flushed": 2834, "Innodb_buffer_pool_pages_free": 6302, "Innodb_buffer_pool_pages_misc": 87, "Innodb_buffer_pool_pages_total": 8191, "Innodb_buffer_pool_read_ahead_rnd": 0, "Innodb_buffer_pool_read_ahead": 0, "Innodb_buffer_pool_read_ahead_evicted": 0, "Innodb_buffer_pool_read_requests": 3247529, "Innodb_buffer_pool_reads": 1603, "Innodb_buffer_pool_wait_free": 0, "Innodb_buffer_pool_write_requests": 279100, "Innodb_data_fsyncs": 11193, "Innodb_data_pending_fsyncs": 0, "Innodb_data_pending_reads": 0, "Innodb_data_pending_writes": 0, "Innodb_data_read": 29266432, "Innodb_data_reads": 2006, "Innodb_data_writes": 13727, "Innodb_data_written": 126338560, "Innodb_dblwr_pages_written": 2799, "Innodb_dblwr_writes": 148, "Innodb_log_waits": 0, "Innodb_log_write_requests": 36527, "Innodb_log_writes": 10540, "Innodb_os_log_fsyncs": 10722, "Innodb_os_log_pending_fsyncs": 0, "Innodb_os_log_pending_writes": 0, "Innodb_os_log_written": 33955840, "Innodb_page_size": 16384, "Innodb_pages_created": 134, "Innodb_pages_read": 1668, "Innodb_pages_written": 2834, "Innodb_row_lock_current_waits": 0, "Innodb_row_lock_time": 0, "Innodb_row_lock_time_avg": 0, "Innodb_row_lock_time_max": 0, "Innodb_row_lock_waits": 0, "Innodb_rows_deleted": 11712, "Innodb_rows_inserted": 11924, "Innodb_rows_read": 4709484, "Innodb_rows_updated": 90,

11


https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/index.html

"Innodb_num_open_files": 76, "Innodb_truncated_status_writes": 0, "Innodb_available_undo_logs": 128, "Max_used_connections": 6, "Max_used_connections_time": "2019-03-25", "Performance_schema_accounts_lost": 0, "Performance_schema_cond_classes_lost": 0, "Performance_schema_cond_instances_lost": 0, "Performance_schema_digest_lost": 0, "Performance_schema_file_classes_lost": 0, "Performance_schema_file_handles_lost": 0, "Performance_schema_file_instances_lost": 0, "Performance_schema_hosts_lost": 0, "Performance_schema_index_stat_lost": 0, "Performance_schema_locker_lost": 0, "Performance_schema_memory_classes_lost": 0, "Performance_schema_metadata_lock_lost": 0, "Performance_schema_mutex_classes_lost": 0, "Performance_schema_mutex_instances_lost": 0, "Performance_schema_nested_statement_lost": 0, "Performance_schema_prepared_statements_lost": 0, "Performance_schema_program_lost": 0, "Performance_schema_rwlock_classes_lost": 0, "Performance_schema_rwlock_instances_lost": 0, "Performance_schema_session_connect_attrs_lost": 0, "Performance_schema_socket_classes_lost": 0, "Performance_schema_socket_instances_lost": 0, "Performance_schema_stage_classes_lost": 0, "Performance_schema_statement_classes_lost": 0, "Performance_schema_table_handles_lost": 0, "Performance_schema_table_instances_lost": 0, "Performance_schema_table_lock_stat_lost": 0, "Performance_schema_thread_classes_lost": 0, "Performance_schema_thread_instances_lost": 0, "Performance_schema_users_lost": 0, "Qcache_queries_in_cache": 0, "Queries": 415232, "Slow_launch_threads": 0, "Slow_queries": 0, "Threads_cached": 2, "Threads_connected": 4, "Threads_created": 6, "Threads_running": 1, "Uptime": 19807, "Uptime_since_flush_status": 19807, "ingestion_time": "2019-03-25T15:07:38+00:00"}

Parameters

Name Data Field UnitDataType Details

Connection_errors_accept

Connection_errors_accep

t

N/A Number

The number of errors that occurredduring calls to accept() on thelistening port.

Connection_errors_max_connections

Connection_errors_max_c

onnectionsN/A Numbe

rThe number of connections refusedbecause the server max_connectionslimit was reached.

Connection_errors_peer_address

Connection_errors_peer_

address

N/A Number

The number of errors that occurredwhile searching for connecting clientIP addresses.

12



Innodb_pages_read Innodb_pages_read N/A Number

The number of pages read from theInnoDB buffer pool by operations onInnoDB tables.

Innodb_rows_deleted Innodb_rows_deleted N/A Number

The number of rows deleted fromInnoDB tables.

Innodb_rows_inserted Innodb_rows_inserted N/A Number

The number of rows inserted intoInnoDB tables.

Innodb_rows_read Innodb_rows_read N/A Number

The number of rows read fromInnoDB tables.

Innodb_rows_updated

Innodb_rows_updated N/A Number

The number of rows updated inInnoDB tables.

Max_used_connections

Max_used_connections N/A Number

The maximum number of connectionsthat have been in use simultaneouslysince the server started.

Max_used_connections_time

Max_used_connections_ti

me

N/A Time The time at whichMax_used_connections reached itscurrent value.

Queries Queries N/A Number

The number of statements executedby the server. This variable includesstatements executed within storedprograms, unlike the Questionsvariable. It does not countCOM_PING or COM_STATISTICScommands. The discussion at thebeginning of this section indicateshow to relate this statement-countingstatus variable to other such variables.

Slow_queries Slow_queries N/A Number

The number of queries that havetaken more than long_query_timeseconds. This counter incrementsregardless of whether the slow querylog is enabled. For information aboutthat log, see Section 5.4.5, "The SlowQuery Log" .

Threads_cached Threads_cached N/A Number

The number of threads in the threadcache.

Threads_connected Threads_connected N/A Number

The number of currently openconnections.

13


https://dev.mysql.com/doc/refman/5.5/en/slow-query-log.html

https://dev.mysql.com/doc/refman/5.5/en/slow-query-log.html


Threads_created Threads_created N/A Number

The number of threads created tohandle connections. IfThreads_created is big, you may wantto increase the thread_cache_sizevalue. The cache miss rate can becalculated as Threads_created/Connections.

Threads_running Threads_running N/A Number

The number of threads that are notsleeping.

Uptime Uptime Seconds

Number

The number of seconds that theserver has been up.

Uptime_since_flush_status

Uptime_since_flush_stat

us

Seconds

Number

The number of seconds since the mostrecent FLUSH STATUS statement.

Runbooks

Metrics Computation Notes

Bufferutilization

Innodb_buffer_pool_bytes_data/(Innodb_page_size*Innodb_buffer_pool_pages_total)

If buffer utilization is very high (for example, exceeds98%) run the MySQLTuner script to determine the newvalue for innodb_buffer_pool_size and then increasethe value of the parameter innodb_buffer_pool_sizein the /etc/my.cnf file.

Rejectedconnections

Max_used_connections /Max_Connections

Max Connections can be seen from /etc/my.cnf file. If theratio is high (for example, above 90%), considerincreasing the number of max connections in the /etc/my.cnf file. On the contrary, if the ratio is very low (forexample, less than 10%), consider decreasing the numberof max connections in the /etc/my.cnf file)

AvailableCache

Qcache_free_blocks /Qcache_total_blocks

If the free space is less (for example, 20%), considerincreasing the value of the parameter query_cache_sizein /etc/my.cnf

Replication Status

Replication status of a "follower" node can be retrieved by periodically calling the URL:https://<IP Address of the node>:5480/service/mashery/cgi/replication_status.py

The response is in JSON format.{ "replication_status": { "restore": { "error": false, "errors": "", "log": "", "complete": false }, "replication": { "last_error": "",

14


https://github.com/major/MySQLTuner-perl

"seconds_behind_master": "", "follower_io_running": "", "follower_sql_running": "" } }, "error": null}


Metrics Computation Notes

seconds_behind_master

None An alert can be generated if the value of seconds_behind_masterexceeds more than 120 (2 min). Seriously consider increasing theinnodb_buffer_pool_size. However, before increasing the value, makesure that there is enough RAM available. To calculate the used RAM, addthe memory allocated to the traffic manager, memcache and the database.

To view the memory used by traffic manager, run the command ps -ef |grep java. Look at the parameter -xMx.

To view the memory used by memcache, add the CACHESIZE of all theconfiguration files for various memcache pools. The names of the configfiles are available in the memcache section.

To view the memory used by MySQL, run the top command see the valueof RES for the process mysql. Another way to compute memoryrequirements by MySQL is to use MySQL Calculator.

MemcacheThe memcache status can be periodically collected using the URL:https://<IP address of the node>:5480/service/mashery/cgi/memcached_status.py

The memcahe stats consists of the following pools:

● counter (master node only)● service● mapi● packager● content● httpsclientsecurityThe statistics for each pool are shown below.

Memcache Statistics{ "counter": { "lru_maintainer_juggles": 9441668.0, "lru_crawler_starts": 675240.0, "auth_cmds": 0.0, "crawler_items_checked": 0.0, "reclaimed": 0.0, "rejected_connections": 0.0, "get_expired": 0.0, "curr_items": 0.0, "pid": 26638.0, "malloc_fails": 0.0, "time_in_listen_disabled_us": 0.0, "slab_reassign_evictions_nomem": 0.0, "expired_unfetched": 0.0, "moves_to_cold": 0.0,

15


https://www.mysqlcalculator.com/

"lru_crawler_running": 0.0, "slab_global_page_pool": 0.0, "hash_is_expanding": 0.0, "port": 11212, "lru_bumps_dropped": 0.0, "max_connections": 10240.0, "cas_hits": 0.0, "uptime": 9442506.0, "moves_to_warm": 0.0, "touch_hits": 0.0, "delete_misses": 0.0, "listen_disabled_num": 0.0, "slab_reassign_rescues": 0.0, "cas_misses": 0.0, "decr_hits": 0.0, "direct_reclaims": 0.0, "cmd_touch": 0.0, "incr_hits": 0.0, "version": "1.5.4", "active_slabs": 0.0, "limit_maxbytes": 268435456.0, "total_items": 0.0, "bytes_written": 3773743.0, "incr_misses": 0.0, "accepting_conns": 1.0, "rusage_system": 55.626542999999998, "slab_reassign_running": 0.0, "slabs": { }, "log_watcher_sent": 0.0, "moves_within_lru": 0.0, "get_flushed": 0.0, "slab_reassign_busy_items": 0.0, "slabs_moved": 0.0, "cmd_get": 0.0, "total_malloced": 0.0, "curr_connections": 10.0, "log_worker_written": 0.0, "log_watcher_skipped": 0.0, "touch_misses": 0.0, "host": "localhost", "threads": 4.0, "get_hits": 0.0, "cmd_set": 0.0, "libevent": "2.0.22-stable", "conn_yields": 0.0, "get_misses": 0.0, "evicted_active": 0.0, "reserved_fds": 20.0, "bytes_read": 3762137.0, "hash_bytes": 524288.0, "slab_reassign_chunk_rescues": 0.0, "evicted_unfetched": 0.0, "cas_badval": 0.0, "cmd_flush": 0.0, "lrutail_reflocked": 0.0, "evictions": 0.0, "bytes": 0.0, "slab_reassign_busy_deletes": 0.0, "crawler_reclaimed": 0.0, "slab_reassign_inline_reclaim": 0.0, "connection_structures": 11.0, "hash_power_level": 16.0, "log_worker_dropped": 0.0, "auth_errors": 0.0, "rusage_user": 429.89164599999998, "time": 1559600959.0, "delete_hits": 0.0, "pointer_size": 64.0, "decr_misses": 0.0, "total_connections": 156766.0

16


}, "service": { "lru_maintainer_juggles": 95891483.0, "lru_crawler_starts": 675240.0, "auth_cmds": 0.0, "crawler_items_checked": 1232651.0, "reclaimed": 3.0, "rejected_connections": 0.0, "get_expired": 0.0, "curr_items": 467.0, "pid": 26677.0, "malloc_fails": 0.0, "time_in_listen_disabled_us": 0.0, "slab_reassign_evictions_nomem": 0.0, "expired_unfetched": 3.0, "moves_to_cold": 1338990.0, "lru_crawler_running": 0.0, "slab_global_page_pool": 0.0, "hash_is_expanding": 0.0, "port": 11214, "lru_bumps_dropped": 0.0, "max_connections": 10240.0, "cas_hits": 0.0, "uptime": 9442506.0, "moves_to_warm": 0.0, "touch_hits": 0.0, "delete_misses": 0.0, "listen_disabled_num": 0.0, "slab_reassign_rescues": 0.0, "cas_misses": 0.0, "decr_hits": 0.0, "direct_reclaims": 0.0, "cmd_touch": 0.0, "incr_hits": 0.0, "version": "1.5.4", "active_slabs": 10.0, "limit_maxbytes": 134217728.0, "total_items": 1407989.0, "bytes_written": 38287775.0, "incr_misses": 0.0, "accepting_conns": 1.0, "rusage_system": 256.59099200000003, "slab_reassign_running": 0.0, "slabs": { "1": { "cas_hits": 0.0, "used_chunks": 1.0, "touch_hits": 0.0, "cmd_set": 31474.0, "cas_badval": 0.0, "free_chunks": 10921.0, "slab_no": "1", "total_chunks": 10922.0, "total_pages": 1.0, "decr_hits": 0.0, "incr_hits": 0.0, "get_hits": 0.0, "chunk_size": 96.0, "chunks_per_page": 10922.0, "delete_hits": 0.0, "free_chunks_end": 0.0, "mem_requested": 88.0 }, "20": { "cas_hits": 0.0, "used_chunks": 20.0, "touch_hits": 0.0, "cmd_set": 60358.0, "cas_badval": 0.0, "free_chunks": 127.0, "slab_no": "20", "total_chunks": 147.0,

17


"total_pages": 1.0, "decr_hits": 0.0, "incr_hits": 0.0, "get_hits": 0.0, "chunk_size": 7104.0, "chunks_per_page": 147.0, "delete_hits": 0.0, "free_chunks_end": 0.0, "mem_requested": 117174.0 } }, "log_watcher_sent": 0.0, "moves_within_lru": 0.0, "get_flushed": 0.0, "slab_reassign_busy_items": 0.0, "slabs_moved": 0.0, "cmd_get": 1601164.0, "total_malloced": 12582912.0, "curr_connections": 6.0, "log_worker_written": 0.0, "log_watcher_skipped": 0.0, "touch_misses": 0.0, "host": "localhost", "threads": 4.0, "get_hits": 2628.0, "cmd_set": 1407989.0, "libevent": "2.0.22-stable", "conn_yields": 0.0, "get_misses": 1598536.0, "evicted_active": 0.0, "reserved_fds": 20.0, "bytes_read": 6558918069.0, "hash_bytes": 524288.0, "slab_reassign_chunk_rescues": 0.0, "evicted_unfetched": 0.0, "cas_badval": 0.0, "cmd_flush": 0.0, "lrutail_reflocked": 0.0, "evictions": 0.0, "bytes": 2368985.0, "slab_reassign_busy_deletes": 0.0, "crawler_reclaimed": 0.0, "slab_reassign_inline_reclaim": 0.0, "connection_structures": 8.0, "hash_power_level": 16.0, "log_worker_dropped": 0.0, "auth_errors": 0.0, "rusage_user": 971.62729000000002, "time": 1559600959.0, "delete_hits": 0.0, "pointer_size": 64.0, "decr_misses": 0.0, "total_connections": 188302.0 }, "mapi": { "lru_maintainer_juggles": 37369967.0, "lru_crawler_starts": 675240.0, "auth_cmds": 0.0, "crawler_items_checked": 120333.0, "reclaimed": 14.0, "rejected_connections": 0.0, "get_expired": 96.0, "curr_items": 48.0, "pid": 26611.0, "malloc_fails": 0.0, "time_in_listen_disabled_us": 0.0, "slab_reassign_evictions_nomem": 0.0, "expired_unfetched": 1.0, "moves_to_cold": 81041.0, "lru_crawler_running": 0.0, "slab_global_page_pool": 0.0, "hash_is_expanding": 0.0,

18


"port": 11211, "lru_bumps_dropped": 0.0, "max_connections": 10240.0, "cas_hits": 0.0, "uptime": 9442506.0, "moves_to_warm": 13088.0, "touch_hits": 0.0, "delete_misses": 0.0, "listen_disabled_num": 0.0, "slab_reassign_rescues": 0.0, "cas_misses": 0.0, "decr_hits": 0.0, "direct_reclaims": 0.0, "cmd_touch": 0.0, "incr_hits": 0.0, "version": "1.5.4", "active_slabs": 3.0, "limit_maxbytes": 402653184.0, "total_items": 68009.0, "bytes_written": 6330166.0, "incr_misses": 0.0, "accepting_conns": 1.0, "rusage_system": 98.352047999999996, "slab_reassign_running": 0.0, "slabs": { "1": { "cas_hits": 0.0, "used_chunks": 3.0, "touch_hits": 0.0, "cmd_set": 63058.0, "cas_badval": 0.0, "free_chunks": 10919.0, "slab_no": "1", "total_chunks": 10922.0, "total_pages": 1.0, "decr_hits": 0.0, "incr_hits": 0.0, "get_hits": 31364.0, "chunk_size": 96.0, "chunks_per_page": 10922.0, "delete_hits": 0.0, "free_chunks_end": 0.0, "mem_requested": 271.0 }, "13": { "cas_hits": 0.0, "used_chunks": 0.0, "touch_hits": 0.0, "cmd_set": 1.0, "cas_badval": 0.0, "free_chunks": 708.0, "slab_no": "13", "total_chunks": 708.0, "total_pages": 1.0, "decr_hits": 0.0, "incr_hits": 0.0, "get_hits": 0.0, "chunk_size": 1480.0, "chunks_per_page": 708.0, "delete_hits": 0.0, "free_chunks_end": 0.0, "mem_requested": 0.0 } }, "log_watcher_sent": 0.0, "moves_within_lru": 18166.0, "get_flushed": 0.0, "slab_reassign_busy_items": 0.0, "slabs_moved": 0.0, "cmd_get": 31500.0, "total_malloced": 3145728.0, "curr_connections": 11.0,

19


"log_worker_written": 0.0, "log_watcher_skipped": 0.0, "touch_misses": 0.0, "host": "localhost", "threads": 4.0, "get_hits": 31364.0, "cmd_set": 68009.0, "libevent": "2.0.22-stable", "conn_yields": 0.0, "get_misses": 136.0, "evicted_active": 0.0, "reserved_fds": 20.0, "bytes_read": 11960883.0, "hash_bytes": 524288.0, "slab_reassign_chunk_rescues": 0.0, "evicted_unfetched": 0.0, "cas_badval": 0.0, "cmd_flush": 0.0, "lrutail_reflocked": 5316.0, "evictions": 0.0, "bytes": 39421.0, "slab_reassign_busy_deletes": 0.0, "crawler_reclaimed": 0.0, "slab_reassign_inline_reclaim": 0.0, "connection_structures": 13.0, "hash_power_level": 16.0, "log_worker_dropped": 0.0, "auth_errors": 0.0, "rusage_user": 607.44365400000004, "time": 1559600959.0, "delete_hits": 0.0, "pointer_size": 64.0, "decr_misses": 0.0, "total_connections": 188282.0 }, "content": { "lru_maintainer_juggles": 9441673.0, "lru_crawler_starts": 675240.0, "auth_cmds": 0.0, "crawler_items_checked": 0.0, "reclaimed": 0.0, "rejected_connections": 0.0, "get_expired": 0.0, "curr_items": 0.0, "pid": 26616.0, "malloc_fails": 0.0, "time_in_listen_disabled_us": 0.0, "slab_reassign_evictions_nomem": 0.0, "expired_unfetched": 0.0, "moves_to_cold": 0.0, "lru_crawler_running": 0.0, "slab_global_page_pool": 0.0, "hash_is_expanding": 0.0, "port": 11213, "lru_bumps_dropped": 0.0, "max_connections": 10240.0, "cas_hits": 0.0, "uptime": 9442506.0, "moves_to_warm": 0.0, "touch_hits": 0.0, "delete_misses": 0.0, "listen_disabled_num": 0.0, "slab_reassign_rescues": 0.0, "cas_misses": 0.0, "decr_hits": 0.0, "direct_reclaims": 0.0, "cmd_touch": 0.0, "incr_hits": 0.0, "version": "1.5.4", "active_slabs": 0.0, "limit_maxbytes": 402653184.0, "total_items": 0.0,

20


"bytes_written": 3773731.0, "incr_misses": 0.0, "accepting_conns": 1.0, "rusage_system": 51.921106000000002, "slab_reassign_running": 0.0, "slabs": { }, "log_watcher_sent": 0.0, "moves_within_lru": 0.0, "get_flushed": 0.0, "slab_reassign_busy_items": 0.0, "slabs_moved": 0.0, "cmd_get": 0.0, "total_malloced": 0.0, "curr_connections": 5.0, "log_worker_written": 0.0, "log_watcher_skipped": 0.0, "touch_misses": 0.0, "host": "localhost", "threads": 4.0, "get_hits": 0.0, "cmd_set": 0.0, "libevent": "2.0.22-stable", "conn_yields": 0.0, "get_misses": 0.0, "evicted_active": 0.0, "reserved_fds": 20.0, "bytes_read": 3762137.0, "hash_bytes": 524288.0, "slab_reassign_chunk_rescues": 0.0, "evicted_unfetched": 0.0, "cas_badval": 0.0, "cmd_flush": 0.0, "lrutail_reflocked": 0.0, "evictions": 0.0, "bytes": 0.0, "slab_reassign_busy_deletes": 0.0, "crawler_reclaimed": 0.0, "slab_reassign_inline_reclaim": 0.0, "connection_structures": 6.0, "hash_power_level": 16.0, "log_worker_dropped": 0.0, "auth_errors": 0.0, "rusage_user": 433.868042, "time": 1559600959.0, "delete_hits": 0.0, "pointer_size": 64.0, "decr_misses": 0.0, "total_connections": 156761.0 }, "httpsclientsecurity": { "lru_maintainer_juggles": 18941435.0, "lru_crawler_starts": 675240.0, "auth_cmds": 0.0, "crawler_items_checked": 0.0, "reclaimed": 0.0, "rejected_connections": 0.0, "get_expired": 0.0, "curr_items": 1.0, "pid": 26651.0, "malloc_fails": 0.0, "time_in_listen_disabled_us": 0.0, "slab_reassign_evictions_nomem": 0.0, "expired_unfetched": 0.0, "moves_to_cold": 31474.0, "lru_crawler_running": 0.0, "slab_global_page_pool": 0.0, "hash_is_expanding": 0.0, "port": 11216, "lru_bumps_dropped": 0.0, "max_connections": 10240.0,

21


"cas_hits": 0.0, "uptime": 9442506.0, "moves_to_warm": 0.0, "touch_hits": 0.0, "delete_misses": 0.0, "listen_disabled_num": 0.0, "slab_reassign_rescues": 0.0, "cas_misses": 0.0, "decr_hits": 0.0, "direct_reclaims": 0.0, "cmd_touch": 0.0, "incr_hits": 0.0, "version": "1.5.4", "active_slabs": 1.0, "limit_maxbytes": 134217728.0, "total_items": 31474.0, "bytes_written": 737904.0, "incr_misses": 0.0, "accepting_conns": 1.0, "rusage_system": 58.092168000000001, "slab_reassign_running": 0.0, "slabs": { "2": { "cas_hits": 0.0, "used_chunks": 1.0, "touch_hits": 0.0, "cmd_set": 31474.0, "cas_badval": 0.0, "free_chunks": 8737.0, "slab_no": "2", "total_chunks": 8738.0, "total_pages": 1.0, "decr_hits": 0.0, "incr_hits": 0.0, "get_hits": 0.0, "chunk_size": 120.0, "chunks_per_page": 8738.0, "delete_hits": 0.0, "free_chunks_end": 0.0, "mem_requested": 105.0 } }, "log_watcher_sent": 0.0, "moves_within_lru": 0.0, "get_flushed": 0.0, "slab_reassign_busy_items": 0.0, "slabs_moved": 0.0, "cmd_get": 0.0, "total_malloced": 1048576.0, "curr_connections": 6.0, "log_worker_written": 0.0, "log_watcher_skipped": 0.0, "touch_misses": 0.0, "host": "localhost", "threads": 4.0, "get_hits": 0.0, "cmd_set": 31474.0, "libevent": "2.0.22-stable", "conn_yields": 0.0, "get_misses": 0.0, "evicted_active": 0.0, "reserved_fds": 20.0, "bytes_read": 2140375.0, "hash_bytes": 524288.0, "slab_reassign_chunk_rescues": 0.0, "evicted_unfetched": 0.0, "cas_badval": 0.0, "cmd_flush": 0.0, "lrutail_reflocked": 0.0, "evictions": 0.0, "bytes": 105.0, "slab_reassign_busy_deletes": 0.0,

22


"crawler_reclaimed": 0.0, "slab_reassign_inline_reclaim": 0.0, "connection_structures": 7.0, "hash_power_level": 16.0, "log_worker_dropped": 0.0, "auth_errors": 0.0, "rusage_user": 482.107708, "time": 1559600959.0, "delete_hits": 0.0, "pointer_size": 64.0, "decr_misses": 0.0, "total_connections": 31486.0 }, "packager": { "lru_maintainer_juggles": 18941431.0, "lru_crawler_starts": 675240.0, "auth_cmds": 0.0, "crawler_items_checked": 0.0, "reclaimed": 0.0, "rejected_connections": 0.0, "get_expired": 0.0, "curr_items": 1.0, "pid": 26664.0, "malloc_fails": 0.0, "time_in_listen_disabled_us": 0.0, "slab_reassign_evictions_nomem": 0.0, "expired_unfetched": 0.0, "moves_to_cold": 31474.0, "lru_crawler_running": 0.0, "slab_global_page_pool": 0.0, "hash_is_expanding": 0.0, "port": 11215, "lru_bumps_dropped": 0.0, "max_connections": 10240.0, "cas_hits": 0.0, "uptime": 9442506.0, "moves_to_warm": 0.0, "touch_hits": 0.0, "delete_misses": 0.0, "listen_disabled_num": 0.0, "slab_reassign_rescues": 0.0, "cas_misses": 0.0, "decr_hits": 0.0, "direct_reclaims": 0.0, "cmd_touch": 0.0, "incr_hits": 0.0, "version": "1.5.4", "active_slabs": 1.0, "limit_maxbytes": 134217728.0, "total_items": 31474.0, "bytes_written": 737888.0, "incr_misses": 0.0, "accepting_conns": 1.0, "rusage_system": 56.798364999999997, "slab_reassign_running": 0.0, "slabs": { "1": { "cas_hits": 0.0, "used_chunks": 1.0, "touch_hits": 0.0, "cmd_set": 31474.0, "cas_badval": 0.0, "free_chunks": 10921.0, "slab_no": "1", "total_chunks": 10922.0, "total_pages": 1.0, "decr_hits": 0.0, "incr_hits": 0.0, "get_hits": 0.0, "chunk_size": 96.0, "chunks_per_page": 10922.0, "delete_hits": 0.0,

23


"free_chunks_end": 0.0, "mem_requested": 94.0 } }, "log_watcher_sent": 0.0, "moves_within_lru": 0.0, "get_flushed": 0.0, "slab_reassign_busy_items": 0.0, "slabs_moved": 0.0, "cmd_get": 0.0, "total_malloced": 1048576.0, "curr_connections": 6.0, "log_worker_written": 0.0, "log_watcher_skipped": 0.0, "touch_misses": 0.0, "host": "localhost", "threads": 4.0, "get_hits": 0.0, "cmd_set": 31474.0, "libevent": "2.0.22-stable", "conn_yields": 0.0, "get_misses": 0.0, "evicted_active": 0.0, "reserved_fds": 20.0, "bytes_read": 1794152.0, "hash_bytes": 524288.0, "slab_reassign_chunk_rescues": 0.0, "evicted_unfetched": 0.0, "cas_badval": 0.0, "cmd_flush": 0.0, "lrutail_reflocked": 0.0, "evictions": 0.0, "bytes": 94.0, "slab_reassign_busy_deletes": 0.0, "crawler_reclaimed": 0.0, "slab_reassign_inline_reclaim": 0.0, "connection_structures": 7.0, "hash_power_level": 16.0, "log_worker_dropped": 0.0, "auth_errors": 0.0, "rusage_user": 485.45619900000003, "time": 1559600959.0, "delete_hits": 0.0, "pointer_size": 64.0, "decr_misses": 0.0, "total_connections": 31485.0 }}

Parameters

Name Data Field Unit Data Type Notes Details

Pool Name pool_name N/A String Possible valuescounter,services, mapis,httpclientsecurity, packager,content

The pool name maps toa Pool

24



pool pool N/A Number The portnumber onwhich the poolis listening forMemcachedfunction callsPossible values11211, 11212,11213, 11214,11215,11216

11211 = mapis, 11212 =counter, 11213 = content,11214 = services, 11215 =packages, 11216 =httpsclientsecurity

Particular slab slab_class N/A Number For eachMemcache,there could bemultiple slabs,slab_classdenotes whichslab it is.

Process id pid N/A Number Process id ofmemcachedservices- therewill be one PIDper memcachedpool

Up time uptime N/A Number Time since startof process -denotes howlong theprocess hasbeen active.

Maximumconnections

max_connectio

ns

N/A Number Maxconnectionsallowed for thispool

Total connections total_connect

ions

N/A Number Total number ofsuccessfulconnectionssince start

Current clientconnections

curr_connecti

ons

N/A Number Number ofMemcachedclientsconnected atthis point

25



Total GETs cmd_get N/A Number Number oftimes a Getrequest wasmade- includesboth successfulandunsuccessful

Total SETs cmd_set N/A Number Number oftimes a Setrequest wasmade - includesboth successfulandunsuccessful

Total Get hits get_hits N/A Number Number ofsuccessful GETrequets

Total GET misses get_misses N/A Number Number offailed "get"requestsbecausenothing wascached forrequested key.

get hits get_hits N/A Number Number of hitsfor the slab -this should be >sn_cmd_set

chunk size chunk_size Bytes Number Size of itemchunks.

Number of chunks ina page

chunks_per_pa

ge

Bytes Number Total number ofchunks in aPage - This willcompute aschunk_size*total_chunks = 1MBapproximately.


You can keep track of the health of the Memcached pools.

26



Fill ratio/percent bytes/limit_maxbytes Low water mark 85%; High water mark 90%.

If the Fill ratio is very low (say 10%) considerreducing the CACHESIZE in the configurationfile.

Eviction Rate evictions It is vital that for the Services pool, the numberof evictions be 0. If evictions > 0, then increasethe CACHESIZE of the services memcache poolin file /etc/memcached-services.conf. Withother pools, it is NOT crucial to have evictionsbe 0, but is always preferred.

Connection LimitReached

listen_disabled_num This is a poorly named counter whichdescribes how many times you've reachedmaxconns. Each time memcached hitsmaxconns it will delay new connections, whichmeans you'll possibly get timeouts. Iflisten_disabled_num > 0, increase MAXCONNin the memcache configuration files.

How to change connection limit and cache size

Memcache parameters are generally present in a configuration file present in the /etc folder. The number ofmax connections from traffic manager to a memcache pool are determined by the parameter MAXCONN. Themax memory allocated to the memcache pool is determined by the parameter CACHESIZE.

Locate the configuration file for the memcache pool for which you want to change the value. Theconfiguration files along with the pool name is as follows.

Pool Name Port Conf File MAXCONN CACHESIZE

MAPIs 11211 /etc/memcached.conf 10240 384M

Counter 11212 /etc/memcached-count.conf 10240 256M

Content 11213 /etc/memcached-

content.conf

10240 384M

Services 11214 /etc/memcached-

services.conf

10240 128M

Packages 11215 /etc/memcached-

packager.conf

10240 128M

HTTPsClientSecurity

11216 /etc/memcached-

httpsclientsecurity.conf

10240 128M

To get the settings into effect, restart the Mashery Local nodes. That will trigger restart of the memcacheand traffic manager will use the newly-warmed up cache.

27


Memcache LoaderMemcache loader loads data from the MySQL database to Memcache so that Traffic Manager can get fasteraccess to configuration data and process incoming traffic calls. Memcache loader is invoked via a cron jobspecified in the script /etc/cron.d/cache-loader. Every time memcache loader runs, it records the count of thefollowing data - Services, Packages, Plans, Methods, MethodOverrides, ResponseFilters, Mapis, DevClasses,HttpsClientProfile, Trust and Identity in the log files.03/Jun/2019:18:53:14 +0000 log_level=INFO, tid=c7df27c2-92ee-4155-95f7-0af5604acf61, stack=java, loader_class=Mapi, primary_param=updatedmapi=10, area_param=, action=end countMapisLoaded=003/Jun/2019:18:53:14 +0000 log_level=INFO, tid=c7df27c2-92ee-4155-95f7-0af5604acf61, stack=java, loader_class=DevClass, primary_param=updateddevclass=10, area_param=, action=end countDevClassesLoaded=003/Jun/2019:18:53:15 +0000 log_level=INFO, tid=c7df27c2-92ee-4155-95f7-0af5604acf61, stack=java, loader_class=Packager, primary_param=updatedpackager=30, area_param=, action=processing areaid=258; done loading Plans; countPlansLoaded=003/Jun/2019:18:53:15 +0000 log_level=INFO, tid=c7df27c2-92ee-4155-95f7-0af5604acf61, stack=java, loader_class=Packager, primary_param=updatedpackager=30, area_param=, action=processing areaid=258; done loading Method Overrides; countMethodOverridesLoaded=003/Jun/2019:18:53:15 +0000 log_level=INFO, tid=c7df27c2-92ee-4155-95f7-0af5604acf61, stack=java, loader_class=Packager, primary_param=updatedpackager=30, area_param=, action=processing areaid=258; done loading Methods; countMethodsLoaded=003/Jun/2019:18:53:15 +0000 log_level=INFO, tid=c7df27c2-92ee-4155-95f7-0af5604acf61, stack=java, loader_class=Packager, primary_param=updatedpackager=30, area_param=, action=processing areaid=258areaid=258; done loading Response Filters; countResponseFiltersLoaded=003/Jun/2019:18:53:15 +0000 log_level=INFO, tid=c7df27c2-92ee-4155-95f7-0af5604acf61, stack=java, loader_class=Services, primary_param=, area_param=, action=end countServicesLoaded=003/Jun/2019:18:53:15 +0000 log_level=INFO, tid=c7df27c2-92ee-4155-95f7-0af5604acf61, stack=java, loader_class=HttpsClientSecurity, primary_param=updatedhttpsclientsecurity=30, area_param=, action=end countHttpsClientProfileLoaded=903/Jun/2019:18:53:15 +0000 log_level=INFO, tid=c7df27c2-92ee-4155-95f7-0af5604acf61, stack=java, loader_class=HttpsClientSecurity, primary_param=updatedhttpsclientsecurity=30, area_param=, action=end countTrustLoaded=003/Jun/2019:18:53:15 +0000 log_level=INFO, tid=c7df27c2-92ee-4155-95f7-0af5604acf61, stack=java, loader_class=HttpsClientSecurity, primary_param=updatedhttpsclientsecurity=30, area_param=, action=end countIdentityLoaded=0


The logs are written in the file /var/log/mashery/memcacheload.log. At every run the file is overwritten(you can change this by modifying the file /etc/cron.d/cache-loader). This log file can be forwarded toyour log processing software.

When to receive alerts:

● If the logs are not being generated● The last log statement generated was more than 30 minutes ago (The memcache loader runs every 10

minutes.)● If the count of each type of data consistently stays at 0 even though configuration data has beenmodified in the API Control Center

In case the memcache loader is not running, consider stopping the javaproxy (service javaproxy stop),restarting memcache (service memcached restart) and starting javaproxy (service javaproxy start)

Status of Cloud Sync Services, Developer, Tokens and Logs

Cloud Sync downloads configuration data that is updated in API Control Center. The configuration data ismade up of Services, (Service) Keys, Packages, PackageKeys, Apps, Classes, HttpsClientSecurity. The cloudsync also uploads logs and synchronizes tokens.

28


The status for "Cloud Sync" can be periodically collected using the URL:https://<IP address of the node>:5480/service/mashery/cgi/cloud_sync_status_cgi.py

The response is a JSON that shows the time stamp of last successful sync for each type of sync. Thefollowing is a sample response.

Cloud Sync Response{ "Logs": { "start": "2019-06-12 15:08:01", "is_success": 1, "end": "2019-06-12 15:08:02", "details": null, "record": 0 }, "PackageKeys": { "start": "2019-06-12 15:15:02", "is_success": 1, "end": "2019-06-12 15:15:02", "details": null, "record": 0 }, "Keys": { "start": "2019-06-12 15:15:02", "is_success": 1, "end": "2019-06-12 15:15:02", "details": null, "record": 0 }, "Apps": { "start": "2019-06-12 15:15:02", "is_success": 1, "end": "2019-06-12 15:15:02", "details": null, "record": 0 }, "Tokens": { "start": "2019-06-12 15:22:02", "is_success": 1, "end": "2019-06-12 15:22:02", "details": "0", "record": 0 }, "Classes": { "start": "2019-06-12 15:15:02", "is_success": 1, "end": "2019-06-12 15:15:02", "details": null, "record": 0 }, "HttpsClientSecurity": { "start": "2019-06-12 15:15:02", "is_success": 0, "end": "2019-06-12 15:15:02", "details": "MOM responded with 596 596\ncom.mashery.proxy.onpremloader.OperationException: MOM responded with 596 596\n\tat com.mashery.proxy.onpremloader.OnpremLoader.doSyncHttpsClientSecurity(OnpremLoader.java:2817)\n\tat com.mashery.proxy.onpremloader.OnpremLoader.syncHttpsClientSecurity(OnpremLoader.java:2747)\n\tat com.mashery.proxy.internal.onpremloader.OnpremLoaderManager$LoaderTask.runInterruptibly(OnpremLoaderManager.java:318)\n\tat com.mashery.proxy.internal.onpremloader.OnpremLoaderManager$LoaderTask.run(OnpremLoaderManager.java:257)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n", "record": 0 }, "Services": { "start": "2019-06-12 15:15:02",

29


"is_success": 1, "end": "2019-06-12 15:15:02", "details": null, "record": 0 }, "Packages": { "start": "2019-06-12 15:15:02", "is_success": 1, "end": "2019-06-12 15:15:02", "details": null, "record": 0 }}

Parameters


start start N/A datetime

Start time

end end N/A datetime

End time

is_success is_success N/A boolean Indicates if the sync operation wassuccessful

record record N/A integer Number of records downloaded

details details N/A text Message from the sync operation


You can monitor the sync of each type of configuration data (Services and their Keys, Packages and theirKeys, Apps, HttpsClientSecurity, Tokens, and Logs). An alert can be set if a successful sync has nothappened three times in a row. That is, if the last successful sync happened more than three times of thecloud sync interval in the past, an alert can be generated.

Symptom Action

The parameter "details" contains "MOMresponded with 596".

Call TIBCO Support immediately, as it indicatesthat TIBCO Cloud may be unhealthy.

The parameter "details" indicates that MasheryOnPrem Manager (MOM) cannot be reached.

Check network connectivity of the Mashery Localnode to the internet.

No Last successful CloudSync in last three tries. See the log files for possible clues on failures.

30


Legal and Third-Party Notices

SOME TIBCO SOFTWARE EMBEDS OR BUNDLES OTHER TIBCO SOFTWARE. USE OF SUCHEMBEDDED OR BUNDLED TIBCO SOFTWARE IS SOLELY TO ENABLE THE FUNCTIONALITY (ORPROVIDE LIMITED ADD-ON FUNCTIONALITY) OF THE LICENSED TIBCO SOFTWARE. THEEMBEDDED OR BUNDLED SOFTWARE IS NOT LICENSED TO BE USED OR ACCESSED BY ANYOTHER TIBCO SOFTWARE OR FOR ANY OTHER PURPOSE.

USE OF TIBCO SOFTWARE AND THIS DOCUMENT IS SUBJECT TO THE TERMS AND CONDITIONSOF A LICENSE AGREEMENT FOUND IN EITHER A SEPARATELY EXECUTED SOFTWARE LICENSEAGREEMENT, OR, IF THERE IS NO SUCH SEPARATE AGREEMENT, THE CLICKWRAP END USERLICENSE AGREEMENT WHICH IS DISPLAYED DURING DOWNLOAD OR INSTALLATION OF THESOFTWARE (AND WHICH IS DUPLICATED IN THE LICENSE FILE) OR IF THERE IS NO SUCHSOFTWARE LICENSE AGREEMENT OR CLICKWRAP END USER LICENSE AGREEMENT, THELICENSE(S) LOCATED IN THE “LICENSE” FILE(S) OF THE SOFTWARE. USE OF THIS DOCUMENT ISSUBJECT TO THOSE TERMS AND CONDITIONS, AND YOUR USE HEREOF SHALL CONSTITUTEACCEPTANCE OF AND AN AGREEMENT TO BE BOUND BY THE SAME.

ANY SOFTWARE ITEM IDENTIFIED AS THIRD PARTY LIBRARY IS AVAILABLE UNDER SEPARATESOFTWARE LICENSE TERMS AND IS NOT PART OF A TIBCO PRODUCT. AS SUCH, THESESOFTWARE ITEMS ARE NOT COVERED BY THE TERMS OF YOUR AGREEMENT WITH TIBCO,INCLUDING ANY TERMS CONCERNING SUPPORT, MAINTENANCE, WARRANTIES, ANDINDEMNITIES. DOWNLOAD AND USE OF THESE ITEMS IS SOLELY AT YOUR OWN DISCRETIONAND SUBJECT TO THE LICENSE TERMS APPLICABLE TO THEM. BY PROCEEDING TO DOWNLOAD,INSTALL OR USE ANY OF THESE ITEMS, YOU ACKNOWLEDGE THE FOREGOING DISTINCTIONSBETWEEN THESE ITEMS AND TIBCO PRODUCTS.

This document is subject to U.S. and international copyright laws and treaties. No part of this documentmay be reproduced in any form without the written authorization of TIBCO Software Inc.

TIBCO, the TIBCO logo, the TIBCO O logo, and TIBCO Mashery are either registered trademarks ortrademarks of TIBCO Software Inc. in the United States and/or other countries.

All other product and company names and marks mentioned in this document are the property of theirrespective owners and are mentioned for identification purposes only.

This software may be available on multiple operating systems. However, not all operating system platformsfor a specific software version are released at the same time. Please see the readme.txt file for the availabilityof this software version on a specific operating system platform.

THIS DOCUMENT IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS ORIMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY,FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT.

THIS DOCUMENT COULD INCLUDE TECHNICAL INACCURACIES OR TYPOGRAPHICAL ERRORS.CHANGES ARE PERIODICALLY ADDED TO THE INFORMATION HEREIN; THESE CHANGES WILLBE INCORPORATED IN NEW EDITIONS OF THIS DOCUMENT. TIBCO SOFTWARE INC. MAY MAKEIMPROVEMENTS AND/OR CHANGES IN THE PRODUCT(S) AND/OR THE PROGRAM(S) DESCRIBEDIN THIS DOCUMENT AT ANY TIME.

THE CONTENTS OF THIS DOCUMENT MAY BE MODIFIED AND/OR QUALIFIED, DIRECTLY ORINDIRECTLY, BY OTHER DOCUMENTATION WHICH ACCOMPANIES THIS SOFTWARE, INCLUDINGBUT NOT LIMITED TO ANY RELEASE NOTES AND "READ ME" FILES.

This and other products of TIBCO Software Inc. may be covered by registered patents. Please refer toTIBCO's Virtual Patent Marking document (https://www.tibco.com/patents) for details.

Copyright © 2006-2020. TIBCO Software Inc. All Rights Reserved.

31


https://www.tibco.com/patents

monitoring guide - tibco software

Documents