troubleshooting cisco unified contact center...

Troubleshooting Cisco Unified Contact Center Enterprise BRKCCT-3005

Gino D’Addario

Technical Solutions Manager

Customer Solutions Success

2

‟Have you tried turning it off and on again?” • “The IT Crowd”

• Graham Linehan

‟We cannot solve our problems with the same thinking we used when we created them.”

• Albert Einstein

Troubleshooting

© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public

Agenda

Serviceability

– Performance Monitors

– SNMP

– Logging Improvements

– OPC Capture

– The Truth About Minidumps

– Network Optimization

– Updated Platform Support

– Diagnostic Framework

4


Agenda

Troubleshooting Flagship Features in UCCE 9.0

– CCE Web Administration – Tomcat 101

– Deployment Types

– Congestion Control

– Precision Routing

– Virtualization

Q&A

– Also available for Meet the Engineer

BONUS! Multimedia Integration

BONUS! Solution Approach to Troubleshooting

5

Serviceability


Performance Monitors

Performance counters are measurements of system state or activity. – Microsoft TechNet

Reactive Usage: UCCE Serviceability Guide section 8.3 recommends which PerfMon counters to enable when diagnosing issues.

Proactive Benefits: Counters can also be enabled for trending and predicting upcoming capacity/performance requirements.

Counters added in UCCE 8.0 provide richer visibility into system activity and health.

7



Platform Diagnostic Counters still (7.0) collected automatically by Node Manager. High-level system resources.

– 1 Minute Intervals / 1MB Per File / 45 Days Max

– Saved to \icm\log\Perf_MACHINENAME_YYYYMMDDHHMMSS.CSV

8

Component Counter Name Units Threshold

Processor %Processor Time (_Total) Percentage <50% | 50% - 60% | >60%

(sustained)

Process Handle Count (_Total) # Handles <=2 x [# CPUs] (continuous)

Memory Page Faults / sec # Faults <10 | >=10 | >10 (sustained)

Memory Committed Bytes # Bytes Relative to paging, demand for

virtual memory

Memory Pages / sec # Pages / sec <10 | >10 (continuous)

System Threads # Threads Reference



Component-specific Counters for Further Diagnosis

* = Useful for Capacity Planning. Available Prior to UCCE 8.0.

9

Cisco Router Object Counter Default State

Agents Logged On* On

Calls In Progress* On

Calls/sec* On

Calls In Router On

Router State Size Off

Messages Processed/sec Off

Bytes Processed/sec Off

Avg Process Time/Message (ms) Off



Component-specific Counters for Further Diagnosis

* = Useful for Capacity Planning. Available Prior to UCCE 8.0.

10

Cisco Router Object Counter Default State

Agents Logged On* On

Calls In Progress* On

Calls/sec* On

Calls In Router On

Router State Size Off

Messages Processed/sec Off

Bytes Processed/sec Off

Avg Process Time/Message (ms) Off



Other new objects available for

– PIM – EA & VRU

– CTI Server, CTI OS

– OO Campaign Mgr, Import Process

Reference the UCCE Serviceability Guide for more information http://www.cisco.com/en/US/products/sw/custcosw/ps1844/products_installation_and_configuration_guides_list.html

11

Administration & Data Server New Counters

Default State

CallType DB Write Records Processed On

Route Queue Depth On

Route DB Write Average Time On

Route DB Write Records Processed On

Service Queue Depth On

Service DB Write Average Time On

Agent Queue Depth On

…and many more!

OPC New Counters

Default State

Call Count On

Agent Count On

Skill Group Count On

Services Count On

Logged-In Agent Count On

Ready Agent Count On

Not-Ready Agent Count On

…and many more!

Logger New Counters

Default State

Number of DB Write Records On

DB Write Average Time On

DB Write Records Processed On

Outbound Option Dialer New Counters

Default State

Number of DB Write Records On

DB Write Average Time On

DB Write Records Processed On

…and many more!

http://www.cisco.com/en/US/products/sw/custcosw/ps1844/products_installation_and_configuration_guides_list.html



Simple Network Management Protocol

SNMP is a family of specifications that provide a means for collecting network management data from the devices residing in a network. – Microsoft TechNet

SNMP Agent installed on “managed devices” can report problems via “traps” to the Network Management System – and you(r team)!

Subagent Management Information Bases (MIBs) Installed:

– CISCO-CONTACT-CENTER-APPS-MIB

– CISCO-CVP-MIB

Reference the UCCE Serviceability Guide for Install, Config, Notifications: http://www.cisco.com/en/US/products/sw/custcosw/ps1844/products_installation_and_configuration_guides_list.html

12







UCCE SNMP Event Flow – High Level

13

SNMP Network Management System

SNMP

syslog

EMS

EMS

EMS

Router

ADS

PG

Logger

Syslog Collector




14

SNMP Network Management System

SNMP

syslog

EMS

EMS

EMS

Router

ADS

PG

Logger

Syslog Collector

Cisco Prime Collaboration


Cisco Prime Collaboration



»For more information about Cisco Prime Collaboration: http://www.cisco.com/en/US/products/ps12363/index.html

15

SNMP

syslog

EMS

EMS

EMS

Router

ADS

PG

Logger

http://www.cisco.com/en/US/products/ps12363/index.html




t_Event


UCCE SNMP Event Flow – Digging Deeper

16

ProTip: Cisco SNMP Service starts SNMPDM, which starts

CCCAAgent Logger

CSFS

CCCA

Agent

SNMP

DM

MDS

Router

Syslog Collector

SNMP NMS

EMS EMS

CW2K

Feed

Other Agents (Generic traps,

platform-specific subagents)

EMS

EMS

SNMP

syslog



Logging Differences for SNMP vs Syslog Events

– Certain events hard-coded as “Objects”

– SNMP looks for this flag and will only process “Objects”

Example: Syslog-only “Simple” Event (no Object)

CW2KFeed picks the event from the pipe and pushes it out to Syslog collector

17

Logger

CSFS

CCCA

Agent

CW2K

Feed

03:37:06:096 la-csfs Trace: ProcName:mds SysName:85PG2B StatusCode:The remote process has exited. Severity:Error Category:Message Delivery

03:37:06:096 la-csfs Trace: ID: e1048013 Msg: Unexpected disconnect from client pim1.

03:37:06:096 la-csfs Trace: EMS ID [e1048013] in Primary Positive Table

03:37:06:096 la-csfs Trace: Using [e104801385PG2Bpim1] as String for Suppression Event

03:37:06:096 la-cw2kfeed Trace: CSysMsg::PutQMsg at Tail -- Q size = 1.

03:37:06:096 la-cw2kfeed Trace: CSyslog::GetCSFSMsg Reading data from CSFS pipe feed.

03:37:06:096 la-cw2kfeed Trace: CSysMsg::GetQMsg -- event for data in Q has been signalled.

03:37:06:096 la-cw2kfeed Trace: CSysMsg::GetQMsg -- Retrieving Q item # 1.

03:37:06:096 la-cw2kfeed Trace: <131>101: 85pg2b: Mar 8 2012 00:37:03.679 +0000: ICM_PG_MessageDelivery-3-1048013: [comp=PG2-

B][pname=mds][iid=ucce][mid=1048013][sev=error]: Unexpected disconnect from client pim1.

03:37:06:096 la-cw2kfeed Trace: CSysMsg::GetQMsg -- Q is empty. Waiting for data.






Example: SNMP Event (Object!)

CW2KFeed still picks the event from the pipe and sends to Syslog collector

18

Logger

CSFS

CCCA

Agent

CW2K

Feed

03:37:06:096 la-cw2kfeed Trace: <132>102: 85pg2b: Mar 8 2012 00:37:03.695 +0000: ICM_PG_NodeManager-4-102C113: [comp=PG2-

B][pname=nm][iid=ucce][mid=102C113][sev=warning]: ICM\ucce\PG2B node process pim1 exited from Control-C or window close.

03:37:06:096 la-csfs Trace: ProcName:nm SysName:85PG2B StatusCode: Severity:Warning Category:Node Manager

03:37:06:096 la-csfs Trace: ID: a102c113 Msg: ICM\ucce\PG2B node process pim1 exited from Control-C or window close.

03:37:06:096 la-csfs Trace: Using [3_2_85PG2B_ICM\ucce\PG2B_pim1] as Unique Object String

03:37:06:096 la-csfs Trace: Created New Base Object '3_2_85PG2B_ICM\ucce\PG2B_pim1'

03:37:06:096 la-csfs Trace: Using [a102c113_3_2_85PG2B_ICM\ucce\PG2B_pim1] as String for Object Suppression Event






Example: SNMP Event (Object!)

Events for Host\Instance\Process use same Object created since last CSFS restart

19

Logger

CSFS

CCCA

Agent

CW2K

Feed


03:37:06:096 la-csfs Trace: ID: a102c113 Msg: ICM\ucce\PG2B node process pim1 exited from Control-C or window close.


03:37:06:096 la-csfs Trace: Created New Base Object '3_2_85PG2B_ICM\ucce\PG2B_pim1'

03:37:06:096 la-csfs Trace: Using [a102c113_3_2_85PG2B_ICM\ucce\PG2B_pim1] as String for Object Suppression Event`


03:37:15:596 la-csfs Trace: ID: a102c110 Msg: ICM\ucce\PG2B node process pim1 successfully reinitialized after restart.


03:37:15:596 la-csfs Trace: Event for Existing Base Object '3_2_85PG2B_ICM\ucce\PG2B_pim1'

03:37:15:596 la-csfs Trace: Using [a102c110_3_2_85PG2B_ICM\ucce\PG2B_pim1] as String for Object Suppression Event


Logging Improvements

Call Router RTRTrace

Check boxes for:

– Config Changes

– Route Request

– Network VRU

– Translation Route

– Call Queueing

– CallType RealTime

– INRC (RC Dialogue)

20

Peripheral Gateway CTI OS

– Registry EMSTraceMask 0x60A0F

Peripheral Gateway EA PIM

Enable in Procmon:

– trace tp* /on

– trace precall /on

– trace *event /on

– trace csta* /on

– trace periph* /on

– trace opc* /on

Peripheral Gateway OPC:

‒ OPCTEST debug /agent /routing

/closedcalls /cstacer /ctimsg

/inrcmsg /calls

‒ Registry EMSTraceMask 0x40

Peripheral Gateway CTI Gateway

‒ Registry EMSTraceMask 0xF8

Peripheral Gateway JGW Enable in Procmon:

‒ trace JT_TPREQUESTS /on

‒ trace JT_JTAPI_EVENT_USED* /on

‒ trace JT_PIM_EVENT /on

‒ trace JT_ROUTE_MESSAGE /on

‒ trace *CONF* /on

Peripheral Gateway VRU PIM

‒ Registry EMSUserData 7F F7 C0 Peripheral Gateway MR PIM

‒ Registry EMSTraceMask 0xF8

‒ Registry EMSUserData DF

Logger Configlogger

‒ Registry EMSTraceMask 0xFFFF



Simplified Tracing Introduced in UCCE 8.0(1)

– Level 0 – Default / Error :: Zero to minimal performance impact

– Level 1 – Warning :: Small performance impact, Cumulative tracing

– Level 2 – Informational :: Medium performance impact, Cumulative tracing

– Level 3 – Debug :: High performance impact, Cumulative tracing

How do they translate?

21



Simplified Tracing Introduced in UCCE 8.0(1)

– Level 0 – Default / Error

– Level 1 – Warning

– Level 2 – Informational

– Level 3 – Debug

How do they translate?

Goal: Only ask for logs

ONCE

Reference the UCCE Serviceability Guide for more information! http://www.cisco.com/en/US/products/sw/custcosw/ps1844/products_installation_and_co

nfiguration_guides_list.html 22

PG Process Level 0

(Default/Error) Level 1

(Warning) Level 2

(Informational) Level 3 (Debug)

JTAPIGW

JT_JTAPI_EVENT_ USED

JT_TPREQUESTS JT_PIM_EVENT

JT_ROUTE_ MESSAGE

JT_CONNECTION *CONF*

JT_JTAPI*

JT_HEX

JT_ROUTE*

JT_TERM*

JT_LOW*

JT*

OPC default, cstacer

0x40

agent, inrcmsg, closedcalls,

tpmsg, routing 0x40

calls, NCT, simplified

0x40

Missingdata, halfhour

0x40

PGAGENT 0x00 0x03 0x0F 0xFF

CTI SVR 0xF0 0xF6 0xFE 0xFF

CTI OS 0x60A0F 0x240A2F 0x260A2F 0x2E0A2F







EMS Log Compression = Increased Retention (x10!) for:

– CTI OS Server

– OPC-CCE

OPC Capture Binary Retention also increased x4

~10% increase in CPU and Disk I/O during 150% load testing.

Introduced with 7.5(10), 8.0(3), 8.5(1)

Corresponding Dumplog enhancements made on Peripheral Gateways

– Uses \icm\bin\gzip.exe to unzip compressed EMS before dumping

– User must manually decompress files if using dumplog on non-PG machine.

23

‒ OPC-TDM

‒ CTISVR

‒ EAGTPIM

‒ JGW

‒ VRUPIM


OPC Capture

How is OPC Capture different from traditional component logs?

Also: OPCconfig, Pgreg, OPCXferData

Not a Quick Process! 24

Debug Level 1 opc_120301_170156.ems(.gz)

10101000100101100010 Dumplog

OPCdata_opc_0301_170156.cap

111111111111111111111111 Merge Debug

Missing lots of trace messages!

Free Disk

Space > 5GB


OPC Capture

Where are the OPC Capture files?

25

PG Service (Re)Start, OPC State Transfer

Triggers dated folder creation containing:

• Capture data

• OPC Config Export

• Registry Export

• State Transfer (optional)

Timestamps must match!

Non-matching OPCdata = Capture File Wrap

Root Cause Unlikely!

M

K

D

I

R


OPC Capture Pop Quiz!

If OPC asserted at 6:39PM on February 15th, which folder will have the capture files that include the assertion?

A.

B. C.

D.

Dated Folder is created when OPC starts, so 18:39:44 will have the capture data AFTER the assertion. The folder from the PREVIOUS start is correct.

But what if the Capture Files have wrapped & timestamps don’t match?

26


OPC Capture

Hard-Coded 2GB Capture Limit Per Dated Folder

– OPCdata_opc_xxxx_yyyyyy.cap files limit ~10MB each. Wrapping = Failure.

How can we baseline the Capture Files?

7.2(5) & 7.5(1) through 8.5(1): ‘exit_opc’

– Intrusive / downtime required

– Could clear / delay condition recurrence

8.5(2) & Later: ‘newcap’

– Does not kill OPC, but WILL delay requests from agent desktop (i.e. consult calls) for 1-2 seconds! These requests typically have a 60 second timer, so 1-2 seconds will not impact

– Perform during low call volume / agent activity

27


OPC Capture

Where are the OPC Capture files?

28

PG Service (Re)Start, OPC State Transfer

Triggers dated folder creation containing:

• Capture data

• OPC Config Export

• Registry Export

• State Transfer (optional)

Timestamps must match!

Non-matching OPCdata = Capture File Wrap

Root Cause Unlikely!

M

K

D

I

R

NewCap in

8.5(2) or later

Updated!


The Truth About Minidumps

Minidumps: Quick, limited information. Root cause HIGHLY unlikely!

Full crash dump: Complete process snapshot

– Larger file size: 40MB vs 40KB

– Takes few seconds longer to write…but worth it

RCA Rule #1 – If you see a mdmp, enable the full dump! – Minidump does NOT guarantee root cause

– Risk producing multiple minidumps VS single full crash dump

Always Collect The Following – .mdmp file

– <process>.exe file – Needed to confirm exact build info to match binaries when debugging

– <process>.pdb file – Symbol file containing debug info

29


The Truth About Minidumps

30

Registry Value: EMSGenerateSmallMemoryDump

– Default: 0x20000001 Minidump

– Recommended: 0x20000002 Full crash dump


Network Optimization

NIC Teaming Not Supported

– Load balancing can negatively affect critical HW I/O continuity

Binding Order

– Public NIC should be listed first followed by Private NIC

Speed And Duplex Settings

– Always full duplex (exceptions with some peripherals, e.g. Avaya)

– Statically set and must match on both the NIC and Switch Port, respectively

– Use Auto/Auto ONLY when running Gigabit NIC/Switch

Best Practices tech tip

– http://www.cisco.com/en/US/products/sw/custcosw/ps1001/products_tech_note09186a00808160f4.shtml

You’ve Heard It A Thousand Times…+1

31

http://www.cisco.com/en/US/products/sw/custcosw/ps1001/products_tech_note09186a00808160f4.shtml






TCP/IP Offloading, Potential Loss of Server Network Connection

– Offloads some TCP/IP processing to network adapters

– Causes serious issues with UCCE & CVP components

– Enabled by default with Windows 2003 SP2, 2008

– http://www.cisco.com/en/US/ts/fn/632/fn63215.html

– http://docwiki.cisco.com/wiki/Contact_Center_Networking:_Offload,_Receive_Side_Scaling_and_Chimney Always disable in all possible places (NIC properties, NIC drivers, Windows registry)

Microsoft TCP/IP Stack Logic Affects MDS TCP Characteristics (RTT)

– http://www.cisco.com/en/US/ts/fn/620/fn62508.html

Important! Urgent! Please Read! Nota Bene!

32

http://www.cisco.com/en/US/ts/fn/632/fn63215.html


http://docwiki.cisco.com/wiki/Contact_Center_Networking:_Offload,_Receive_Side_Scaling_and_Chimney







Intermittent Silent Monitoring Disconnections, Failures – 10:15:17.517 552 SilentMonitorService Thd(1292) CUDPPort(008AFB90)::ReceiveData, recvfrom,

Blocking call interrupted by WSACancelBlockingCall. (WSAEINTR), Error Number(10004).

Lost Heartbeats, Test-Other-Side (ToS) Seen in MDS – ra-mds Trace: EMT 10: Missing heartbeat, countdown is 5, last received seqno 269795

– pg1b-mds Trace: Sending TOS request: sequence = 1. pg1b-mds Trace: Received TOS response: sequence=1 status=ENABLED.

Logger Sync issues during business day, then clearing up at night

– Recovery keys different between A-side and B-side


33

Validate All Windows 2003 Servers: netsh int ip show offload

Validate All Windows 2008 Servers: netsh in tcp show global

+ NIC Drivers!

+ NIC Properties!



Updated Platform Support

8.5(2) adds support for Windows 2008 R2 SP1 (64-Bit Only)

– Enhanced Native Security = no need for UCCE Installer OS Hardening

– Unsupported PGs on 2008: Aspect, Spectrum, Symposium (Ref: Cisco ICM ACD Matrix)

9.0(1) requires Windows 2008 R2 SP1, SQL Server 2008 R2 SP1

OS architecture change = UCCE process windows no longer appear

Title bar indicators gone with the wind(ows).

34

http://www.cisco.com/en/US/docs/voice_ip_comm/cust_contact/contact_center/ipcc_enterprise/compatibility_matrix/icmacdmx.pdf










35

EMSmon

Usage Syntax:

emsmon <cust_instance> <node_name> <process> {system}

Note: ‘system’ argument is optional and only

needed if connecting to remote server

Example (local):

EMSMON ucce RouterA rtr

Example (remote):

EMSMON ucce PG1A pim1 UCCEPG1A

*Max of 1 local and 5 remote connections





36

EMSmon

Usage Syntax:

emsmon <cust_instance> <node_name> <process> {system}

Note: ‘system’ argument is optional and only

needed if connecting to remote server

Example (local):

EMSMON ucce RouterA rtr

Example (remote):

EMSMON ucce PG1A pim1 UCCEPG1A

*Max of 1 local and 5 remote connections

? ? ?


Enter…The Diagnostic Framework Portico

37

Services Installed Locally

Not Bold = No Processes Running

Direct Link

Process Title Bar

Blue: Parentheses status

change within 10 min

Uptime

Red < 10 minutes

10 < Yellow < 30

Smart refresh

Remembers

current location

Bookmark

Your Servers


Diagnostic Framework Portico

Single-system “Support Tool” – More than just Process Status!

Set tracing for single process

38



Single-system “Support Tool” – More than just Process Status!

Set tracing for single process

39

… and verify



What if a trace is set using “old” tools?

For example, RTRTrace:

40



Level 99 = Custom (Does Not Match Defined Trace Levels)

Set a trace level 0-3 to overwrite using SetTraceLevel

41



Dump logs for a single process

42


Diagnostic Framework API

How does it work?

Other devices also have their own Web Services for similar functions

Isn’t there a tool to bring this all together?

43

RoggerA RoggerB

UCM PUB UCM SUB

IP IVR 1 IP IVR 2

HTTPS

HTTPS

PG1A PG1B

ADS 1 ADS 2

HTTPS

RTMT

RTMT


Analysis Manager

Tool within the CUCM Real-Time Monitoring Tool (RTMT)

Node, Trace template configuration steps required

Set and collect traces in real-time or scheduled

Collect system environment info

Supported Products:

– UCM – IOS Gateways

– UCCE – Unity Connection

– UCCX – Unified Presence

– CVP

Solution Tool Option #1

44


Analysis Manager

How does it work?

45

RoggerA RoggerB

UCM PUB UCM SUB

IP IVR 1 IP IVR 2

HTTPS

HTTPS

PG1A PG1B

ADS 1 ADS 2

HTTPS

RTMT

RTMT


Analysis Manager

How does it work?

46

RoggerA RoggerB

UCM PUB UCM SUB

IP IVR 1 IP IVR 2

PG1A PG1B

ADS 1 ADS 2

ICM

UCM

UCCX

Node Type Nodes

Node Group

Trace Template RTMT


Analysis Manager

Add Nodes

47

Node Type

Node

Node Group


Analysis Manager

Create Trace Template

48

Select

Processes


Analysis Manager

Create Trace Template

49

Set Trace

Levels Level 0 – Default / Error

Level 1 – Warning

Level 2 – Informational

Level 3 – Debug


Analysis Manager

Schedule Periodic Collection

Trace level applied at Start Time

Set back to original at End Time

50

Select Node or

Node Group

Set Duration

Set Interval

Select Trace Template


Analysis Manager

So this supports all of the core “Cisco Unified” contact center products, right?

– Analysis Manager does NOT support CUIC, Finesse, MediaSense, or SocialMiner

What about the other products that use RTMT? Is this version compatible?

What if I have multiple ICM instances?

– Analysis Manager does not support multiple ICM instances.

Let’s check out Option #2!!

51


System CLI

Solution-wide tool

– System Mode – run commands on multiple devices

– Easily Add/Remove devices.

Syntax similar to Cisco IOS gateway CLI

– Without Tab auto-complete

Capture, debug, trace from:

– All CCE servers – IOS Gateways – UCM

– CVP / IP IVR – Presence – CUIC

Low Priority Tasks Throttled – Reduced Priority, Low CPU threshold

– Will only use the IDLE CPU on the system. Max of 20 concurrent requests

– No timeouts on the client; controlled by the server.

– Load risk still applies to IOS devices.

Solution Tool Option #2

52

Multi-Instance Support


System CLI

How does it work?

53

RoggerA RoggerB

UCM PUB UCM SUB

IP IVR 1 IP IVR 2

HTTPS

HTTPS

PG1A PG1B

ADS 1 ADS 2

HTTPS

RTMT

RTMT


System CLI

How does it work?

54

RoggerA RoggerB

UCM PUB UCM SUB

PG1A PG1B

ADS 1 ADS 2

PG1A PG1B ADS

IP IVR 1 IP IVR 2 CUIC PUB CUIC SUB


System CLI

How does it work?

55

RoggerA RoggerB

UCM PUB UCM SUB

IP IVR 1 IP IVR 2

CUIC PUB CUIC SUB

PG1A PG1B ADS

ICM

UCM

UCCX

CUIC

Device Type Devices

CLI

Device List

Location differs

with/without

CVP in

deployment


System CLI Deployments Without CVP

How does it work?

56

RoggerA RoggerB

UCM PUB UCM SUB

IP IVR 1 IP IVR 2

CUIC PUB CUIC SUB

PG1A PG1B ADS

ICM

UCM

UCCX

CUIC

Device Type Devices

CLI

Device List

devices.csv



Devices.csv must be present for CLI to use System Mode (without CVP)

– Otherwise, the CLI will run as a single server tool

Create from Devices-sample.csv (included with every UCCE installation)

57



58

<server or IP>, <description>, <device type>, <group>, <username>, <password>

10.10.10.30, RoggerA, ICM, ucce-sideA, VMLOAD\Administrator, cvpcisco

10.10.10.31, PG1A, ICM, ucce-sideA, VMLOAD\Administrator, cvpcisco

Required

Recommended

ProTip: If System CLI will be run only on ADS, then devices.csv is only required on ADS.



How does it work?

59

RoggerA RoggerB

UCM PUB UCM SUB

IP IVR 1 IP IVR 2

CUIC PUB CUIC SUB

PG1A PG1B ADS

ICM

UCM

UCCX

CUIC

Device Type Devices

CLI

Device List

devices.csv

Group

“All ICM”

Device Type

Trace Template


System CLI

How does it work?

60

RoggerA RoggerB

UCM PUB UCM SUB

CUIC PUB CUIC SUB

PG1A PG1B ADS

ICM

UCM

CUIC

Devices

CLI

Device List

devices.csv

Group

“All ICM”

Device Type

Device Type


System CLI Deployments With CVP

How does it work?

61

RoggerA RoggerB

UCM PUB UCM SUB

CVP 1 CVP 2

CUIC PUB CUIC SUB

PG1A PG1B ADS

ICM

UCM

CVP

CUIC

Devices

Device List

Group

“All ICM”

CVP

OAMP

CVP Remote

Operations

Manager CLI Device Type

Device Type


System CLI Deployments With CVP

Devices Managed from CVP OAMP

62

Encrypted!


System CLI

Initial Login is to single machine only (CVP OAMP or a UCCE server)

Enter System Mode to connect to solution devices (defined in OAMP or local CSV) admin:system

Initializing system mode …

Retrieving device list. This process may take a few minutes to complete.

. . .

Unable to connect to 10.10.10.130:7890(icm)

. . . . .

Once in System Mode, commands will be run against all devices. admin(system):show version

Retrieving [version] data from device [uccergra30] ProductType [icm] …

ICM 8.5(2) BuildNumber=37931 ES=0 PatchInstallerVer=465

Retrieving [version] data from device [uccepg1a31] ProductType [icm] …

ICM 8.5(2) BuildNumber=37931 ES=0 PatchInstallerVer=465

Useful Commands

63

Success

Fail

Success


System CLI

Commands can still be limited to a single device… admin(system):show debug server 10.10.10.30 Retrieving [debug] data from device [10.10.10.30] ProductType [icm] …

Component: icm:Router A, subcomponent: icm:rtr Trace level = 0

…Device Type… admin(system):show devices devicetype icm Device host name: uccergra30 Port: 7890 Product type: ICM Group: ucce-sidea Device host name: uccepg1a31 Port: 7890 Product type: ICM Group: ucce-sidea

64

…Group… admin(system):debug level 1 group ucce-sidea

Component: icm:Logger A, subcomponent: icm:hlgr

Debug level is set successfully

…repeats for all Logger A processes

Component: cvp:CallServer, subcomponent: cvp:MSGBUS


…repeats for all CVP processes

Component: icm:CTI Server 1A, subcomponent: icm:ctisvr


…repeats for all CTI Server 1A processes

Component: icm:PG1A, subcomponent: icm:jgw1


…repeats for all PG1A processes


System CLI

…or on-the-fly combination of devices admin(system):show config server 10.10.10.30|10.10.10.31 redirect dir c:\temp Retrieving [configuration] data from device [10.10.10.30] ProductType [icm] … Fetching configuration resource for component DumpCfg Fetching configuration resource for component ExportICMCfg Fetching configuration resource for component ConfigExport Fetching configuration resource for component Registry Retrieving [configuration] data from device [10.10.10.31] ProductType [icm] … Fetching configuration resource for component Registry Output is saved to “c:\temp\clioutput0.zip”

65

File Download

Specify Save

Location

Rogger

DumpCfg,

DB Export

Peripheral Gateway

Registry Export Only


System CLI

Best Of All, The Game-Changer

admin(system):show tech-support Retrieving [Every Single Piece Of] data from

device [Your Entire Deployment]

ProductType [ICM, UCM, CVP, IOS, UCCX, CUIC, ETC!] …

Output is saved to

“C:\Cisco\CVP\wsm\CLI\download\clioutput1.zip”

66

Instant 24-hour Snapshot

Produces zip of ALL defined server component

logs (including OPC Capture), ICM config,

dumpcfg, versions, registries,

OS info, PerfMon, debug levels, etc.

For >/<24 hours, add:

“absdatetime = MM-DD-YYYY:hh:mm”

Organized By Device

EVERYTHING TAC/DE would need

to start investigation.


System CLI Quick-Reference Guide

http://docwiki.cisco.com/wiki/Unified_System_CLI_Quick-Reference_Guide

+ Command Examples!

+ Troubleshooting! 67





System CLI Limitations

CSCub56442 – CVP 9.0 does not allow Remote Operations on client Windows

– Use CVP 8.5 installer for Remote Operations in interim. Fixed in 10.0.

No Device Types exist for Finesse (CSCue98453) or MediaSense (CSCue98487)

– CLI does not speak the same language.

– Can get basic info by using device type CUIC (common platform), but not too useful: Device is marked offline due to unreachable device or invalid user name or password [10.10.10.38:8443(cuic)]

Please run "system init" command to retry.

Product Version List:

Active Server Version: 9.0.1.10000-2

Cisco Finesse

9.0.1.10000-2

Finesse has basic website for logs:

– http://<Primary>/finesse/logs/webservices/

MediaSense has a more advanced Serviceability portal for tracing, services, etc:

– http://<Primary>/oraservice/

SocialMiner not integrated, but enhancement filed too: CSCuf85862 68


System CLI Scripting With Batch Files

Samples in C:\icm\serviceability\wsccli\sample

Bat file can contain commonly-used commands

– Set tracing on CVP

– Collect logs from all ICM devices

– Endless possibilities

Bat file can be run manually or automatically with Windows Scheduler

Bring quick resolution to an ongoing issue!

69


Diagnostic Framework Match-up

70

Ability to schedule periodic collections

GUI

No multi-instance support

Requires UCM

Thick-Client

No Configuration Needed

Browser Accessible

Quick Single-Process Transactions

No multi-instance support

Limited Scope (Single Device)

Positives Positives

Negatives

Negatives

Positives

Multi-instance support

System Mode

ANY command can be scheduled

No support for OEMs, Finesse & MediaSense

…No GUI?

Negatives

Flagship Features in UCCE 9.0


CCE Web Administration

CCE 9.0 adds Web Administration (Apache Tomcat 7 webapp)

“Deployment Type Aware”: Used for Precision Routing in UCCE, more in PCCE

72



Not to be confused with Web Setup (added in 8.0)!

73



Limited management gadgets for UCCE

UCCE Deployment Types

No change to

Configuration

Manager

74



Plenty of management gadgets

PCCE Deployment Types

75

Consolidated

Configuration

Manager



Plenty of management gadgets


76

Consolidated

Configuration

Manager

Packaged CCE is the SAME code as Unified CCE!

Same base installer, same maintenance releases

Same components, same serviceability tools

Only difference = Deployment Type!

More Info: http://www.cisco.com/en/US/products/ps12586/index.html





Important Detail:

If a PCCE customer decides they have outgrown the PCCE capacity, they can change the deployment type to UCCE.

BUT, they must know it is a one-way path.

– Moving from UCCE to PCCE is not supported!

77



Deployment Type validates system before unlocking simplified PCCE Gadgets

78



Packaged CCE limits configuration items – finally!


79



Limited error descriptions with

standard tools.

Example: Administrator must

trace references in order to

delete a Route.

New tools provide

better detail why a

config change was

not allowed.

Applies to PQs, ECC

variable sizing, etc. MUCH easier for Administrators!

80


CCE Web Administration Problem Scenario

Problem Statement: Login attempts to CCE WebAdmin intermittently hang or fail with “server not responding” error

Business Impact: Administrators and Supervisors unable to access PCCE Web Administration console for agent management. Queues are not properly staffed to handle incoming call volume and agents must be adjusted.

Version:

– CCE 9.0(2)

Troubleshooting Performed:

– Restarting the Apache Tomcat service temporarily resolves the issue

Login Timeouts

81



Check the IE9 Developer Tools Console (F12) for script errors

Timeouts from the API are causing UI issues!

WebAdmin = Tomcat WebApp. Increase tracing to see API timeouts.

– Edit /icm/tomcat/conf/logging.properties: com.cisco.ccbu.cce.security.ccead.CCEADConfig.level = FINE

– Stop/Start Tomcat service

82

, check Tomcat logs , reproduce login issue

Verify

Task

Manager



tomcat7-stderr.date.log shows failed login attempt. Feb 15, 2013 4:55:57 PM com.cisco.ccbu.cce.security.ccead.CCEADConfig checkConfigDomainUser

FINE: Could not find admin123@ad in any domain.

Feb 15, 2013 4:55:57 PM com.cisco.ccbu.cce.security.ccead.CCEADConfig getAdministratorPrincipal

FINE: Authentication as administrator failed for admin123@ad

AD authentication is failing!

How does Tomcat authenticate?

Next Step: Enable sADlib debugging:

– Set HKLM\SOFTWARE\Cisco Systems, Inc.\ICM\SystemSettings\Debug_Sadlib to “Yes”

– Restart Tomcat. sADlib output will write to C:\temp\sadlib.log.

– Negligible production impact, but log will grow without purging. Watch disk space, set registry key back to “No” when finished

Time to learn how sADlib works! 83

Tomcat sADlib.dll Windows

API Active

Directory



sADlib recognizes two domain controllers: AD.cisco.com & ADB.cisco.com

1. Search for UPN [email protected] in ADB (arbitrary) 11:05:09 Find External Trusted Domains

11:05:09 Searching for (&(objectCategory=person)(objectClass=user) ([email protected])) in DC=ADB,DC=CISCO,DC=COM

11:05:09 Execute search -- Initialize the search

11:05:09 Get each row from the result table

No hits mentioned. Does the search ever even finish?

– Nope! No hit in ADB, but no failover to AD!

AD delay issues cause sADlib search to hang

– Defensive fix CSCue95184 adds a 5 second timeout to handle this AD condition Available in 9.0(4), 9.0(3) ES3 (link in slide notes)

Fix applied, onward!

84

AD ADB



2. Search for UPN [email protected] in AD with defensive fix 11:05:15 Execute search -- Initialize the search


11:05:20 Search Over for User

3. Search without suffix: admin123 11:05:20 Execute search -- Initialize the search


11:05:20 SadStatus::addError User or Group not found DC=AD,DC=CISCO,DC=COM

11:05:20 Search Over for User

4. Search for sAMAccountName admin123 (pre-Windows 2000 name) 11:05:20 The search for user info is completed

11:05:20 0 "SadMan::findUserInfo" "SadMan::searchB (using dn) Found dnDomain:

OU=USERS, DC=AD,DC=CISCO,DC=COM userName: admin123“

85

AD ADB

No results!

5 second delay = timeout fix!

No results!

No timeout! AD better now?

Found!



Check if user is a member of the CCE Security Groups (config or setup) 11:05:20 0 "SadGroup::getAdAttrib" "threadId: 4583 sAMAccountName"

11:05:20 SadIcm::isIcmSGMember dnUser: CN=admin123, OU=USERS, DC=ad,DC=cisco,DC=com

The user is in the Config group, so login will be permitted. 11:05:21 SadIcmInstance::IsUserMemberOfConfigSgEx user is a member of Config. usserId: CN=admin123,OU=USERS,DC=ad,DC=cisco,DC=com

With the timeout fix in place, the user finally logged in – but only after 12 seconds!

– 11:05:09 11:05:21

– This stresses the importance of a rock-solid AD infrastructure

– Staging Guide references MS tools to ensure AD health and connectivity (dcdiag, netdiag) http://www.cisco.com/en/US/products/sw/custcosw/ps1844/prod_installation_guides_list.html

86

AD ADB

http://www.cisco.com/en/US/products/sw/custcosw/ps1844/prod_installation_guides_list.html

http://www.cisco.com/en/US/products/sw/custcosw/ps1844/prod_installation_guides_list.html



Also...Configuring Deployment Type unlocks Congestion Control feature

What purpose does Congestion Control serve? 87


Congestion Control

Prior to 9.0

88


What is Congestion Control?

Central Controller Guardian

– Keeps the system running close to rated capacity under extreme overload conditions

– Deployment Type sets system capacity (in CPS at Router)

Satisfactory service to a limited percentage of calls rather than degraded service to ALL calls

– Call never reaches routing script! “Handled” at Routing Client.

– What are the “handling” options?

89


Congestion Control Treatment Options

Treat Call with Dialed Number Default Label [1]

Treat call with Routing Client Default Label [2]

Treat call with System Default Label [3]

Terminate call with a Dialog Fail or RouteEnd [4]

Treat call with Release Message to Routing Client [5]

90


Congestion Control Configuration

1. Set Deployment Type & Enable in Web Administration

2. Select Treatment Mode

– “Use System Congestion Control” = Global mode set in Web Administration gadget

OR

– Set on a per-routing client basis

**All PGs must be at 9.0 for Congestion Control to work properly**

Reference 9.0 SRND for details and upgrade options 91


Congestion Levels and Thresholds

System capacity is adjusted based on real time conditions. Assume system configured for 100 CPS (avg)

Onset/Abate allow short spikes when hovering near configured capacity

Prevents high rate of level oscillation & alarming when approaching limits

NOT adjustable. Defined by system Deployment Type

Level 1

Level1Onset

110% of Capacity

L1Reduction

10% Handled

Level1Abate

90% of Capacity

Level 2

Level2Onset

130% of Capacity

Level2Reduction

30% Handled

Level2Abate

100% of Capacity

Level 3

Level3Onset

150% of Capacity

Level3Reduction

30 to 100% Handled

Level3Abatement

100% of Capacity

92


Congestion Control

Protects the system from a sudden overload

Existing “committed” calls protected

Restricts capacity by “handling” excess calls

Options:

– Redirect using Label

– Reject/Release

93

Level 1

• Triggered at 110% volume

• 10% reduction in new calls

Level 2


• 30% reduction in new calls

Level 3


• 30-100% reduction in new calls


Congestion Control

PG Startup shows current Congestion Control settings

PG1A-opc Trace: ADDED 1 Peripheral.

PG1A-opc Trace: Routing Client congestion control data: Treatmentmode = 0 label = 12345 rcid = 500

Logs also show Congestion level transition

pg1a-opc trace: congestion trigger received: congestion level=3 reduction per=100

Notifications via Event Viewer & SNMP

94


Congestion Control

“System Capacity Real Time Data” CUIC report, multiple views:

– Congestion Information, Reduction Percentage

– Key Performance Indicators

95


Congestion Control Recap

Assess the system

Deployment Type

Set the Deployment

Type

Select the Congestion Treatment

Mode

- System Level

Select the Congestion Treatment

Mode

- Routing Client

Optionally

Override the system

capacity settings if required.

Setup call treatment

system

Enable the Congestion

Control

Steps to Deployment

System Protected

against overload

System Unprotected

from overload

96


Precision Routing

Traditional Skill Group Routing

97

Sales

Support

Returns


Precision Routing

Attribute Routing

– Simpler scripting

– Multidimensional

– Multi-step

98

Attribute Value

English 9

Sales 8

Returns 5

Boston True

Attribute Value

English 7

Spanish 8

Support 10

Baltimore True

Attribute Value

English 9

Sales 5

Returns 8

Boston True


Precision Routing

Attribute Routing

– Simpler scripting

– Multidimensional

– Multi-step

99

Attribute Value

English 9

Sales 8

Returns 6

Boston True

Attribute Value

English 7

Sales 2

Returns 10

Baltimore True

Attribute Value

English 9

Sales 5

Returns 8

Boston True


Precision Routing Problem Scenario

Problem Statement: Unable to create new Precision Queues

Business Impact: Unable to complete implementation of new design, causing delays to rollout of Precision Routing.

Versions:

– CCE 9.0(3)


– Checked documentation and on-line help for limit: Total Skill Groups and Precision Queues per agent: 15

15 = Packaged CCE size limit, but same concept applies to UCCE (higher limit)

Agents & Attributes

100



What is seen in the Web Administration when trying to add new PQ?

– Improved error messages within UI eliminate need for checking logs

– Well, *almost* eliminate. Logs show offending Agent ID

One agent is preventing us from creating a new Precision Queue?!

– System is smart! No registry keys/DB hacks/etc to get around this limit!

– How is this agent skilled?

101

15:57:07:982 Dis-uaw Trace: [client: Tomcat7(upcc)] Update of central controller failed, Message: Precision

Queue 5017 (Sales) exceeds the limit of 15 combined Skill Groups/Precision Queues for Agent 5023

(5000.666), (Precision Queues : 16 Skill Groups: 0), LastUpdateKey: 0.



This agent only has 10 attributes!

We should have room still, up to 15, right?

– Not quite. Attempting to add an 11th attribute also fails. Attributes != Precision Queues

102



103

Agent has 4 Attributes

Assume there is a single Precision Queue match for each Attribute

Additional Precision Queues may also trigger those same Attributes

In PCCE, this agent can belong to 7 additional PQs (15 total)

Attribute

English

French

Italian

Spanish

English French Italian Spanish

Europe Canada USA

Attribute

English

French

Italian

Spanish

Attribute

French

Italian

Attribute

English

French

LATAM

Attribute

English

Spanish

8 PQs from

4 Attributes!


Quick Word on APIs

APIs available in CCE 9.0 for:

– Precision Queues – Deployment Type – Bucket Intervals

– Attributes – Congestion Control – AD Domains for Login

Developers Guide: http://www.cisco.com/en/US/products/sw/custcosw/ps1844/products_ programming_reference_guides_list.html

Enables powerful integrations across components within the solution

List, Get, Create, Update, Delete – REST

104

Agent’s Attributes

Fluent in English

Certified to sell Home Insurance

HR Database

API

Now certified to sell Auto Insurance

http://www.cisco.com/en/US/products/sw/custcosw/ps1844/products_programming_reference_guides_list.html






Virtualization

7.5(3) – Client Administrator Workstations, certain Peripheral Gateways (PGs)

8.0(2) – Expanded virtualization support on UCS B & C Series

– Central Controller, CVP, CUIC, Multichannel, etc.

– Launched http://docwiki.cisco.com/wiki/Virtualization_for_Unified_CCE

– OVA Templates introduced with VM resources/reservations

– OS/Application “Direct Install” still very manual

9.0(1) – Automated Cloning procedure released

– Packaged CCE Golden Template Automation Process

– Creates reusable images, decreases time to deploy

– Utilizes VMware PowerCLI, customer-specific data sheet XLS

How is Golden Template different from Direct Install?

Evolution of CCE on VMware

105

http://docwiki.cisco.com/wiki/Virtualization_for_Unified_CCE

http://docwiki.cisco.com/wiki/Virtualization_for_Unified_CCE


Unified Computing System (UCS) Platform

UCS C260M2 Server

– Local Storage

Two for redundancy

VMware ESXi

VMware vCenter

– New or existing

Local Storage No SAN!

VMware ESXi VMware ESXi

106


Installation Options

Direct Install

– Ideal for customers deploying themselves, lab environments, etc.

107



Installation Options

Direct Install

– Ideal for customers deploying themselves, lab environments, etc.

Golden Template Automation

– Ideal for partners repeatedly performing PCCE deployments

108


Partner Lab

Customer 1, 2, 3, …, N

VMware ESXi


Virtualization – Automation Procedure

Create Golden Template

Configure Destination

Servers

(Side A & B)

Customization

Spreadsheet

Start Cloning Scripts

Post Cloning Tasks

109


Virtualization – Golden Template Creation

Once for each component

Created on “Source” host

Will be cloned to customer environment

No configuration!

Download OVA Create VM

from the OVA

Install

Windows

Install Anti-

Virus

Enable Remote

Desktop

Install VMware

tools

Convert VM to

Golden Template

Install

Application

UCOS-based Applications

110


Virtualization – Destination Staging

2. Configure the

Destination servers

Golden Templates

(Source Host)

• RAID

• ESXi

• NIC config

• vCenter

• Etc…

ESXi hosts must be added to vCenter with IP address, not FQDN!

Packaged CCE scripts cannot find ESXi hosts by FQDN

VMware ESXi (A-side)

VMware ESXi (B-side)

111


Virtualization – Customization

3. Complete the Automation

Spreadsheet

VMware ESXi (A-side)

VMware ESXi (B-side)

4. Use vSphere PowerCLI to run

DeployVM.PS1 passing the

Automation Spreadsheet and

destination vCenter as parameters

112


Virtualization – Cloning

Cloning script creates customer VMs based off source Golden Templates

113


Virtualization – Cloning


VMware ESXi

UCOS

CUIC A

Windows

Data

Server A

Windows

Call

Server A

UCOS

Finesse A

Windows

CVP 1A

Windows

CVP 2A

Windows

CVP

OAMP

UCOS

UCM Pub

UCOS

UCM Sub1

UCOS

CUIC B

Windows

Data

Server B

Windows

Call

Server B

UCOS

Finesse B

Windows

CVP 1B

Windows

CVP 2B

Windows

CVP

Reporting

UCOS

UCM Sub1

114


Virtualization – Post-Cloning


VMware ESXi

UCOS

CUIC A

Windows

Data

Server A

Windows

Call

Server A

UCOS

Finesse A

Windows

CVP 1A

Windows

CVP 2A

Windows

CVP

OAMP

UCOS

UCM Pub

UCOS

UCM Sub1

UCOS

CUIC B

Windows

Data

Server B

Windows

Call

Server B

UCOS

Finesse B

Windows

CVP 1B

Windows

CVP 2B

Windows

CVP

Reporting

UCOS

UCM Sub1

5. Complete customer-specific configuration

– UCOS PlatformConfig XML FLP

– CCE Domain Manager, Web Setup, Base Config Import

115


Virtualization Problem Scenario

Problem Statement: Golden Template cloning fails

Business Impact: Unable to deploy new Packaged CCE customer environment using Golden Templates. Schedule will be heavily impacted if Direct Install procedure must be used.

Versions:

– PCCE 9.0(3)

– VMware ESXi 5.0

– VMware PowerCLI 5.0, 32-bit


– Confirmed ESXi hosts are added to vCenter using IP address

– Confirmed spreadsheet entry matches the VM name, but the PowerCLI will not find it

Golden Template Cloning Confusion

116



What error does the PowerCLI give? Quite a few, actually: Invalid Golden Template Name

Template does not exist

Resource validation failed

Errors related to:

– Source/destination host

– Source/dest datastore

– Template

– VM Name

Work backwards – what does the Automation Spreadsheet say?

117



PowerCLI looking for VM template named “CCE-Staged-W2K8-SP1-8G”

This VM template exists on the source vCenter

Cannot “Edit Settings” and view the properties of a template (like with a VM), but can check the .vmtx config file in the datastore browser.

118



Hmmm…something looks different.

– The file names do not match the folder name!

119



Hmmm…something looks different.

– The file names do not match the folder name!

Download the .vmtx config file and open it in your favorite text editor 120



Parts of vmtx config file also do not match the template name shown in vCenter

This template was renamed in vCenter, which does not change the datastore files

So can we just update our Automation Spreadsheet with the original name?

– No. From the PCCE Install/Config Guide:

121

Original: CCE Staged W2K8_SP1-8G

Renamed: CCE-Staged-W2K8-SP1-8G



To properly rename: Convert to VM > Clone > Convert to Template

– http://communities.vmware.com/message/1568820

– http://kb.vmware.com/kb/1029513

Convert original problematic template to VM

Clone original problematic VM to new VM

122

Convert new VM to Template

http://communities.vmware.com/message/1568820

http://kb.vmware.com/kb/1029513



Run DeployVM.ps1 cloning script

123

Original Golden Template

New Golden Template

Cloned CCE VM



Run DeployVM.ps1 cloning script

124

Original Golden Template

New Golden Template

Cloned CCE VM


Virtualization – Useful Links

Cisco Packaged CCE Automation: Setup & Install Summary Video-on-Demand

– http://www.youtube.com/watch?v=3IqtBiIbecc&list=UULUuIagjnt1XDoc9rfRUUkA&index=1

DeployVM.ps1 script is released with Packaged CCE, but concepts can be applied to any VMware deployment.

– BRKCCT-2052 - UCCE Deployments on Cisco UCS Platforms Best Practices Oops! Tuesday, Jun 25, 8:00 AM - 10:00 AM has passed!

Catch the recording on CiscoLive365.com!

Main Unified Communications virtualization wiki:

– http://docwiki.cisco.com/wiki/Unified_Communications_Virtualization

Important sub-section for ESXi networking:

– http://docwiki.cisco.com/wiki/UCS_Network_Configuration_for_UCCE #Network_Requirements_for_UCS_C_Series_Servers

– Answers common questions for mapping Public/Private/Mgmt networks

125

http://www.youtube.com/watch?v=3IqtBiIbecc&list=UULUuIagjnt1XDoc9rfRUUkA&index=1

http://www.youtube.com/watch?v=3IqtBiIbecc&list=UULUuIagjnt1XDoc9rfRUUkA&index=1

http://docwiki.cisco.com/wiki/Unified_Communications_Virtualization



http://docwiki.cisco.com/wiki/UCS_Network_Configuration_for_UCCE






Troubleshooting Unified CCE

Serviceability

– New Tools

– Old Tools with New Tricks

Real Customer Scenarios

– TCP Offload Anomalies

– Web Administration

– Congestion Control

– Precision Routing

– Virtualization

BONUS Multimedia & Solution Approach to Issue Resolution slides in Appendix!

Time for Q&A!

Summary

126

Diagnostic

Framework

Attribute

English

French

Italian

Spanish


Troubleshooting Unified CCE

Keep This Presentation as a Reference!

Homework: Implement CLI in your environment & USE IT!

Have a safe trip home.

Core Message

128

Thank You

Serviceability Tools

Automated Improvements

Troubleshooting

Smarter


Maximize your Cisco Live experience with your

free Cisco Live 365 account. Download session

PDFs, view sessions on-demand and participate in

live activities throughout the year. Click the Enter

Cisco Live 365 button in your Cisco Live portal to

log in.

Complete Your Online Session Evaluation

Give us your feedback and you could win fabulous prizes. Winners announced daily.

Receive 20 Cisco Daily Challenge points for each session evaluation you complete.

Complete your session evaluation online now through either the mobile app or internet kiosk stations.

129


Cisco Collaboration Community and User Group

Discuss strategy, market trends, IT architecture, product features, user adoption, and more with peers

contact centers, social software, telepresence, unified communications, and web conferencing

Interact with Cisco Product Managers, Technical Marketing Engineers and Services Consultants

Hear announcements and see demos of new solutions

Join the Collaboration User Group

– Influence product direction

– Participate in advisory groups & early product releases

– VIP perks & exclusive roadmap sessions at Cisco Live

Connect with Peers and Cisco

130

Visit Collaboration Booth - Cisco Campus:

Explore the Collaboration Community

Join the Collaboration User Group (CUG)

CUG Members: get gift ticket & badge ribbon

www.cisco.com/go/joinconversation

http://www.cisco.com/go/joinconversation

Multimedia Integration


Multimedia Integration Combining Email, Chat, and Voice

133

High-level Flow

MR

PG

Agent

PG

GED-188 (CTI) ‒ Agent

availability

for Task

GED-125 (MR) ‒ New Task

‒ Do This With

Task


Multimedia Integration

Key Differences from Voice:

Email/Chat Media Routing Domains (MRD) A collection of skill groups and services associated with a specific communication channel. – UCCE Scripting and Media Routing Guide

Concurrency How many tasks can an agent handle in an MRD?

Interruptibility Can one task be interrupted by another task? Concept of “Available” vs. “ICM Available”

Combining Email, Chat, and Voice

134

Email MRD

Sales Support Billing

I.E. Agent “Available” for email but not “ICM Available”


Interruptibility & Handle Time

Problem Statement: Agent Productivity confusion.

Business Impact: Agents paid based on activity efficiency. Reported numbers are unclear to supervisors.

Versions:

– UCCE 8.5(3)

– EIM/WIM 4.4(1)


– Verified interruptibility settings for Chat and Email MRDs are correct.

– Verified concurrent task limits defined in EIM/WIM.

– Paid attention to this crash course!

Combining Email, Chat, and Voice

135



136

Media Routing Domain Email Chat

Concurrent Task Limit 3 2



137





138





139





140





141

Interrupted!





142





143

Interrupted!





144

Interrupted!




Interruptibility & Handle Time How does UCCE see all of this?

145

External Agent Assignment Service MR PIM

MR PIM Finished!

MR

PG

Agent

PG

GED-125

18:38:19:693 pg2A-pim2 Trace: Application->PG:

Message = NEW_TASK; Length = 102 bytes

DialogueID = (10) Hex 0000000a

SendSeqNo = (1) Hex 00000001

MRDomainID = (5003) Hex 0000138b

PreviousTask = -1:-1:-1

PreferredAgent = Undefined

Service = (0) Hex 00000000

CiscoReserved = (0) Hex 00000000

ScriptSelector: 8000

ECC Variable Name: user.cim.activity.id

Value: 1372

18:38:19:693 pg2A-pim2 Trace: PG->Application:

Message = DO_THIS_WITH_TASK; Length = 121

bytes

DialogueID = (10) Hex 0000000a

SendSeqNo = (1) Hex 00000001

IcmTaskID = 150083:310: 1

SkillGroup = (5036) Hex 000013ac

Service = Undefined

Agent = (5003) Hex 0000138b

AgentInfo: 1003

Label:


Value: 1372

Router CallKey

EIM/WIM AgentID

Agent STID



146

Listener CTI Server

Pre-Call Indicator, Links The RouterCallKey & DialedNumber

Indicators are generally sent from an integrated application

– EXCEPT DEVICE_TARGET_PRE_CALL_IND which is sent from Router.

Events are calculations that occur within OPC for interval reporting & All Event Clients

Agent

PG

GED-188

18:38:19:693 cg1A-ctisvr Trace: DEVICE_TARGET_PRE_CALL_IND - Instrument= RouterCallKey=150083 310

NetworkTargetID=0

18:38:19:709 cg1A-ctisvr Trace: SvSkTargID=-1(-1) SkGroupSkTarID=5036(531)

18:38:19:709 cg1A-ctisvr Trace: AgSkTargID=5003 ANI= CED=

18:38:19:709 cg1A-ctisvr SESSION 2: MsgType:AGENT_PRE_CALL_EVENT (MonitorID:0 NumNamedVariables:2

NumNamedArrays:0 ServiceNumber:N/A

18:38:19:709 cg1A-ctisvr SESSION 2: ServiceID:N/A SkillGroupNumber:531 SkillGroupID:5036

SkillGroupPriority:0 MRDID:5003

18:38:19:709 cg1A-ctisvr SESSION 2: AgentInstrument:"" RouterCallKeyDay:150083 RouterCallKeyCallID:310

18:38:19:709 cg1A-ctisvr SESSION 2: RouterCallKeySequenceNum:1 DialedNumber:"8000" )



147

Listener CTI Server

Email Offered & Started

Handle Time Starts Counting

Agent

PG

GED-188

18:38:19:724 cg1A-ctisvr SESSION 3: MsgType:OFFER_TASK_IND (InvokeID:0x2c767 MRDID:5003 ICMAgentID:5003

18:38:19:724 cg1A-ctisvr SESSION 3: TaskID:150083/310/1ARMSkillGroupID:5036 ARMServiceID:N/A )

18:38:19:724 cg1A-ctisvr Trace: OFFER_TASK_EVENT: PeripherlID=5000 PeripherlType=4 MRDomainID=5003

18:38:19:724 cg1A-ctisvr Trace: TaskID=150083:310:1 ICMAgentID=5003 SkillGroupID=5036

18:38:19:724 cg1A-ctisvr Trace: ServiceID=5617416 ApplicationRouted

18:38:25:537 cg1A-ctisvr SESSION 3: MsgType:START_TASK_IND (InvokeID:0x2c768 MRDID:5003 ICMAgentID:5003

18:38:25:537 cg1A-ctisvr SESSION 3: TaskID:150083/310/1ARMSkillGroupID:5036 ARMServiceID:N/A )

18:38:25:537 cg1A-ctisvr Trace: START_TASK_EVENT: PeripherlID=5000 PeripherlType=4 MRDomainID=5003





148

Listener CTI Server

Agent “Active” In Multimedia SG

Agent Can Still Receive A Second Task

– NumTasks=1, MaxTaskLimit=2

Agent

PG

GED-188

18:38:25:537 cg1A-ctisvr Trace: AGENT_EVENT: ID=21 Periph=5000 Ext= Inst= Sig=

18:38:25:537 cg1A-ctisvr Trace: SGState=ACTIVE SGDuration=0 OverallState=ACTIVE OverallDuration=0 Reason=-1

18:38:25:537 cg1A-ctisvr Trace: MRDID=5003 NumTasks=1 MaxTaskLimit=2 AgtMode=1 AgtAvailStatus=1

ICMAgentID=5003

18:38:25:537 cg1A-ctisvr Trace: SkTgtID=5003 SkGrpNo=0x213 SkGrpID=5036 NumLines=0 CurLine=-1

ClientStatus=0x0



149

Listener CTI Server

Agent Gets Second Email (Pre_Call, Offer) and Reads It

– Handle Time “pauses” for original email and starts for new email

Second Email Completed, Original Email Resumed

Agent

PG

GED-188

18:40:33:632 cg1A-ctisvr Trace: PAUSE_TASK_EVENT: PeripherlID=5000 PeripherlType=4 MRDomainID=5003

18:40:33:632 cg1A-ctisvr Trace: TaskID=150083:310:1 ICMAgentID=5003 Reason=5000

18:40:33:647 cg1A-ctisvr Trace: START_TASK_EVENT: PeripherlID=5000 PeripherlType=4 MRDomainID=5003



18:40:39:929 cg1A-ctisvr Trace: END_TASK_EVENT: PeripherlID=5000 PeripherlType=4 MRDomainID=5003

18:40:39:929 cg1A-ctisvr Trace: TaskID=150083:311:1 PreviousTaskID=-1:-1:-1

18:40:39:929 cg1A-ctisvr Trace: ICMAgentID=5003

18:40:42:788 cg1A-ctisvr Trace: RESUME_TASK_EVENT: PeripherlID=5000 PeripherlType=4 MRDomainID=5003

18:40:42:788 cg1A-ctisvr Trace: TaskID=150083:310:1 ICMAgentID=5003 Reason=5000



150

Listener CTI Server

Original Email Interrupted By Chat

What are these MRDs? Where can we find their names? USE bxb_awdb select * from t_Media_Routing_Domain

Agent

PG

GED-188

18:40:49:710 cg1A-ctisvr SESSION 3: MsgType:AGENT_INTERRUPT_ADVISORY_EVENT (InvokeID:0x2 MRDID:5001

InterruptingMRDID:5002 ICMAgentID:5003 )

18:40:49:710 cg1A-ctisvr SESSION 3: MsgType:AGENT_INTERRUPT_ADVISORY_EVENT (InvokeID:0x2 MRDID:5003

InterruptingMRDID:5002 ICMAgentID:5003 )



151

Listener CTI Server

Agent Reserved For Chat SG; Interrupted For Email SG

Offer, Start, End Task Indicators/Events and state transitions mimic Email

Note: Handle time for chat increments for all running sessions.


18:40:49:710 cg1A-ctisvr Trace: SGState=RESERVED SGDuration=0 OverallState=RESERVED OverallDuration=0 Reason=-1

18:40:49:710 cg1A-ctisvr Trace: MRDID=5002 NumTasks=1 MaxTaskLimit=1 AgtMode=1 AgtAvailStatus=0 ICMAgentID=5003

18:40:49:710 cg1A-ctisvr Trace: SkTgtID=5003 SkGrpNo=0x21d SkGrpID=5039 NumLines=0 CurLine=-1 ClientStatus=0x0

18:40:49:725 cg1A-ctisvr SESSION 3: MsgType:AGENT_INTERRUPT_ACCEPTED_IND (InvokeID:0x2 MRDID:5001 ICMAgentID:5003 )

18:40:49:725 cg1A-ctisvr SESSION 3: MsgType:AGENT_INTERRUPT_ACCEPTED_IND (InvokeID:0x2 MRDID:5003 ICMAgentID:5003 )


18:40:49:725 cg1A-ctisvr Trace: SGState=INTERRUPTED SGDuration=0 OverallState=INTERRUPTED OverallDuration=0 Reason=0

18:40:49:725 cg1A-ctisvr Trace: MRDID=5003 NumTasks=1 MaxTaskLimit=2 AgtMode=1 AgtAvailStatus=0 ICMAgentID=5003

18:40:49:725 cg1A-ctisvr Trace: SkTgtID=5003 SkGrpNo=0x213 SkGrpID=5036 NumLines=0 CurLine=-1 ClientStatus=0x0

Agent

PG

GED-188

MRD

ID

Media

5001 Outbound

Email

5002 Chat

5003 Inbound

Email



152

How Can a Failure Event Be Mapped to a Task/Activity?

Use DialogueID to Correlate

Agent

PG

18:38:19:693 pg2A-pim2 Trace: Application->PG:

Message = NEW_TASK; Length = 102 bytes

DialogueID = (10) Hex 0000000a SendSeqNo = (1) Hex 00000001

MRDomainID = (5003) Hex 0000138b

PreviousTask = -1:-1:-1

PreferredAgent = Undefined

Service = (0) Hex 00000000

CiscoReserved = (0) Hex 00000000

ScriptSelector: 8000


Value: 1372


Message = DO_THIS_WITH_TASK; Length = 121 bytes


IcmTaskID = 150083:310: 1

SkillGroup = (5036) Hex 000013ac

Service = Undefined

Agent = (5003) Hex 0000138b

AgentInfo: 1003

Label:


Value: 1372


Message = NEW_TASK_FAILURE_EVENT; Length = 12 bytes


ReasonCode = (209) Hex 000000d1

MR

PG

Solution Approach to Troubleshooting


Communicate, Collaborate, Succeed Resolving Problems with a Solution-based Approach

154

Accurately Confirming Critical Items Helps With…

Baseline(s)

Accurate troubleshooting

Improving feedback and collaboration

Reducing cycles, guess work, relying on luck, etc.

“Trust, but verify.” ~Ronald Reagan “What changes have you made?”

“None!”

“What version are you running?” “9.b(7c)”

“Who is affected?” “Everyone!”

“Did you look at the logs?” “Yep.”

“Did you enable tracing?” “Yes.”

“What and when did this happen?” “Don’t ask me.”

“It’s been 10 minutes. When are you going to fix this?!”


Communicate, Collaborate, Succeed

155

Clarify Problem

Description

Determine Possible Causes

Investigate

Test Likely Causes

Problem

Solution

Think

Beyond the

Fix…

15

5



156

Emotions

Politics

Fear

Technical

•Brokering Reasonable Expectations

•Fair and Balanced Approach

Situational Analysis

•Formulate Baseline

•Deliver Logs, Recordings, Screenshots

Technical Analysis

•Clear Expectations

•Formulate Next Steps, Workarounds Action Plan

•Alignment and Utilization Resource

Management

•Collaborate on Big Picture Solution

Approach

15

6



157

•Establishes respect

•Acts as a pressure valve

•Shows you care

Listen

•Ask questions to re-enforce what you were told

•Echo and summarize your understanding

Understand

•Log Validation

•Reproduce

•Test Cause & Effect

•Remember, “Trust, but verify.”

Check And Assess Action Plan

•Avoid transactional dialog

•“Tell me, I will forget. Show me, I may remember. Involve me and I will understand.”

Collaborate


Solution Problem Isolation

Problem Statement: Agents getting stuck in RESERVED state for 30 seconds, not receiving the call being routed to them.

Business Impact: Calls being rerouted introduce delays, exceed SLAs. This is affecting 20% of calls, causing financial penalties for the call center.

Versions:

– UCCE 8.5(3)

– CVP 8.5(1)

– CUCM 8.5(1)

Troubleshooting Performed: Isolated issue to single site.

What’s the call flow?

158


Solution Problem Isolation Narrowing Down The Problem

159

Call comes in to CVP from Voice Gateway

Caller requests to speak with an agent

VVVV

PG PG

Aug 30 2011 15:20:52.360 +0530: %CVP_8_0_SIP-7-CALL: {Thrd=DATAI.11} NEW CALL with guid=1A1935C8100001326975748653ED69C3 legid=8D03395A-D22311E0-9734A76F-421B7E24 dn=61401211 ani=+918946861782 uui=null calldate=Tue Aug 30 15:20:52 IST 2011 video=false

cachecallcontext = false is_postcallsurvey = false RouterCallKey = null RouterCallKeyDay = null RouterCallKeySequenceNumber = null

Aug 30 2011 15:26:12.031 +0530: %CVP_8_0_ICM-7-CALL: {Thrd=pool-1-thread-34-ICM-16843525} CALLGUID =

1A1935C8100001326975748653ED69C3, DLGID = 401702 [IVR_LEG] - Publishing ,, [ICM_RUN_SCRIPT_RESULT], dialogueId=401702,

sendSeqNo=3, invokedId=2, resultCode=true, ced=-1, newTransaction=false, CallContext:, user.microapp.caller_input: 1,

user.microapp.error_code: 0, user.microapp.metadata: N|000|01|00|00|019390|GS,Server,V, user.microapp.FromExtVXML[0]: +918946861782,

user.microapp.FromExtVXML[1]: hin,, LEGID = , DNIS = 10582222217939, ANI = sip:[email protected]:5060



160

Router sends a Connect message for extension 19797

CVP receives the Connect message

VVVV

PG PG

15:26:16:045 ra-rtr Trace: (401693 401702 17939 : 0 0) DeviceTargetPreCall_V11: CID=(149990,405494), PerID=5000, SGSTID=7037, AGSTID=8294,

NetworkCallFlags=0, InvokeID=1830299, GREET=

15:26:16:045 ra-rtr Trace: (401693 401702 17939 : 0 0) Connect: CID=(149990,405494), EventSelect=0, ServiceType=0, RCID=5005, Label(s)=19797


1A1935C8100001326975748653ED69C3, DLGID = 401693 [SIP_LEG] - Publishing ,, [MsgBus:CONNECT], ssId=SYS_SIP2, labelType=NORMAL,

label=19797, correlationId=null, command=null, rckey=405494, rcday=149990, rcseq=0, CallContext:, CallVariable1: 8946861782,

CallVariable2: NA|NA, CallVariable3: 1A1935C8100001326975748653ED69C3, etc...…..






161

Router sends a Connect message for extension 19797

CVP receives the Connect message

VVVV

PG PG





1A1935C8100001326975748653ED69C3, DLGID = 401693 [SIP_LEG] - Publishing ,, [MsgBus:CONNECT], ssId=SYS_SIP2, labelType=NORMAL,

label=19797, correlationId=null, command=null, rckey=405494, rcday=149990, rcseq=0, CallContext:, CallVariable1: 8946861782,

CallVariable2: NA|NA, CallVariable3: 1A1935C8100001326975748653ED69C3, etc...…..



162

Use InvokeID or RCK to match PreCall message in CTI Server logs

Agent was AVAILABLE in call’s destination SG just before this PreCall

VVVV

PG PG

15:26:15:723 cg1B-ctisvr Trace: DEVICE_TARGET_PRE_CALL_IND - Instrument=19797 RouterCallKey=149990 405494 15:26:15:723 cg1B-ctisvr Trace: SvSkTargID=-1(-1) SkGroupSkTarID=7037(617)

15:26:15:723 cg1B-ctisvr Trace: AgSkTargID=8294 ANI=+918946861782 CED=-1

15:26:15:723 cg1B-ctisvr Trace: CallTypeID=5535 PreCallInvokeID=1830299

15:26:15:504 cg1B-ctisvr Trace: AGENT_EVENT: ID=20415 Periph=5000 Ext=19797 Inst=19797 Sig=CTIOSServer

15:26:15:504 cg1B-ctisvr Trace: SkgState=AVAILABLE SkgDuration=0 OverallState=AVAILABLE OverallDuration=0 Reason=0

15:26:15:504 cg1B-ctisvr Trace: MRDID=1 NumTasks=0 MaxTaskLimit=1 AgtMode=1 AGTAvailabilityStatus=1 ICMAgtID=8294

15:26:15:504 cg1B-ctisvr Trace: SkTgtID=8294 SkGrpNo=0x269 SkGrpID=7037 NumLines=1 CurLine=0 ClientStatus=0x1



163

Use InvokeID or RCK to match PreCall message in CTI Server logs

Agent was RESERVED in call’s destination SG concurrent with PreCall

VVVV

PG PG

15:26:15:723 cg1B-ctisvr Trace: DEVICE_TARGET_PRE_CALL_IND - Instrument=19797 RouterCallKey=149990 405494 15:26:15:723 cg1B-ctisvr Trace: SvSkTargID=-1(-1) SkGroupSkTarID=7037(617)

15:26:15:723 cg1B-ctisvr Trace: AgSkTargID=8294 ANI=+918946861782 CED=-1

15:26:15:723 cg1B-ctisvr Trace: CallTypeID=5535 PreCallInvokeID=1830299

15:26:15:723 cg1B-ctisvr Trace: AGENT_EVENT: ID=20415 Periph=5000 Ext=19797 Inst=19797 Sig=CTIOSServer

15:26:15:723 cg1B-ctisvr Trace: SkgState=RESERVED SkgDuration=0 OverallState=RESERVED OverallDuration=0 Reason=0

15:26:15:723 cg1B-ctisvr Trace: MRDID=1 NumTasks=1 MaxTaskLimit=1 AgtMode=1 AGTAvailabilityStatus=0 ICMAgtID=8294

15:26:15:723 cg1B-ctisvr Trace: SkTgtID=8294 SkGrpNo=0x269 SkGrpID=7037 NumLines=1 CurLine=0 ClientStatus=0x1



164

UCCE and CUCM notify CVP that call has been Answered

But the Agent PG is never notified that this call arrived at device 19797.

What does a normal call look like? What do we expect to see?

VVVV

PG PG


1A1935C8100001326975748653ED69C3, DLGID = 401693 [SIP_LEG] - Publishing ,, [ICM_EVENT_REPORT], dialogueId=401693, sendSeqNo=3,

eventId=ANSWER, causeCode=NONE,, LEGID = 8D03395A-D22311E0-9734A76F-421B7E24, DNIS = 61401211, ANI = +918946861782

Aug 30 2011 15:26:16.172 +0530: %_UserCB-6-com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.LlSM.client.UserCB: SIP/2.0 200 OK Via: SIP/2.0/UDP 172.18.150.61:5060;branch=z9hG4bKI5Y12NeaMEV6dImqSWb3TA~~8933838

To: "--CVP_8_0_1_0_0_0_1440" <sip:[email protected]>;tag=A269BB6C-77D

From UCCE

From CUCM



165

For a NORMAL call, we should see after the Reservation:

– CSTA_DELIVERED

– CSTA_ESTABLISHED

– RTP_STARTED / RTP_STOPPED

– CSTA_CONNECTION_CLEARED

– Agent transition to NOT_READY state (or wrap-up etc)

None of these messages are seen for the call in question.

VVVV

PG PG



166

PGs internal timer expires in 30 seconds, reverting agent to AVAILABLE

Agent Reserved, PG waiting, CVP Answered. So what’s left?

VVVV

PG PG

15:26:45:945 PG1A-pim1 Trace: RemovePreRoutedCall: Head Inst: N19797 CreatedByPreCall: T PeriphCID: -1 Queue Count: 0

RouterCallKey=(149990/405494)

15:26:45:945 PG1A-pim1 Trace: ProcessReservedTimeout: No call arrived to match PreCall message. netTargID=100780 Ext=19797

AgentID=20415 RTRCallKey=149990.405494 DtAborts=1

15:26:45:945 PG1A-pim1 Trace: SendDeviceTargetAbortInd: agentSkillTargetID = 8294

15:26:45:945 PG1A-pim1 Trace: SetPrevalentAgentState: (Agent 20415 ext 19797) newAgentState=AS_AVAILABLE(3) reasoncode=0

stateDuration=0 currentLineHandle=-1 lineType=LT_UNKNOWN(3) newLineState=LS_UNKNOWN(5) devString=

15:26:45:945 PG1A-pim1 Trace: SetPrevalentState: newState=AS_AVAILABLE(3) reasoncode=0

15:26:45:945 PG1A-pim1 Trace: CancelAgentTimer: type = agentTimerReserved



167

CUCM logs show agent going off-hook and on-hook repeatedly.

CUCM rejects routed call while off-hook

VVVV

PG PG

15:26:15.135 |StationInit: (0002392) SoftKeyEvent softKeyEvent=2(NewCall) lineInstance=1 callReference=0.|6,100,50,1.4164601^

15:26:16.252 |StationInit: (0002392) SoftKeyEvent softKeyEvent=9(EndCall) lineInstance=1 callReference=110085009.|6,100,50,1.4164606^





15:26:17.740 |LineControl(2351) - 1 calls, 0 CiReq, busyTrigger=1, maxCall=2|6,100,57,1.399447^172.18.150.120^*

15:26:17.740 |//SIP/SIPTcp/wait_SdlSPISignal: Outgoing SIP TCP message to 172.18.150.120 on port 37776 index 13 [1449373,NET]

SIP/2.0 486 Busy here









168



VVVV

PG PG


15:26:17.740 |//SIP/SIPTcp/wait_SdlSPISignal: Outgoing SIP TCP message to 172.18.150.120 on port 37776 index 13 [1449373,NET] SIP/2.0

486 Busy here

Hit the Floor!









169



VVVV

PG PG


15:26:17.740 |//SIP/SIPTcp/wait_SdlSPISignal: Outgoing SIP TCP message to 172.18.150.120 on port 37776 index 13 [1449373,NET] SIP/2.0

486 Busy here

Hit the Floor!



170


– Agents do not have enough time in between calls

– Agent Desk Settings issue, possible staffing issue

ACTION PLAN:

– After-Call Wait Time should be increased to meet agent needs.

– Call volume should be evaluated to align with agent staffing.

VVVV

PG PG

troubleshooting cisco unified contact center...

Documents