case study type in a sub-title agenda here 3 agenda business problem & background solution...
TRANSCRIPT
Case StudyType in a sub-title agenda here
3
Agenda
• Business Problem & Background
• Solution Approach
• Along the Way: Learnings
• Summary
4
Trends in Enterprise Call Centers
• Cost reduction
• Multiple Outsourcers (agents)
• Multiple Geographies (global)
• Multiple Networks (transport)
• Multiple Technologies (on call center premises)
5
Call Distribution Evolution – ’80s
Long DistanceNetwork
Single enterprise & single site ACD
Automatic Call Distributors
Single Enterprise/ Single Site1980’s
Although business strategies have evolved, call center technology has not kept pace. Many concepts developed in the early 80’s still exist in TDM and IP solutions
Enterprise Customer Enterprise Customer
Avaya
NORTEL
ASPECT
6
Call Distribution Evolution – ’90s
Long DistanceNetwork
Single enterprise with multiple call centers
Network Routing & CTI
Single Enterprise/ Multiple Sites
1990’s
In the 1990s, vendors addressed single enterprise and multi-site routing with network routing solutions
Enterprise Customer Enterprise Customer
Genesys
Cisco
7
New headache, needing a new aspirin
Outsourcer Enterprise
Client Enterprise
Outsourcer Enterprise Outsourcer Enterprise
Client EnterpriseClient Enterprise
Life has become a
Mess….
Multiple Enterprises
Multiple Sites
Multiple Networks
Multiple Technologies
Multiple Operations
Call Distribution Evolution – Now
8
Implementation Models
• Black Hole– Call sent to offshore outsourcer directly from the network– No direct Visibility, Control or Quality
• Forced Footprint– Outsourcer implements Enterprise chosen Technology (e.g. ICM or
Genesys)– Provides a level of dynamic control over call routing and call reporting
(e.g. at pre-routing phase, reports of agent handling through routing software)
• Mini-Telco– Bring the call into a domestic ACD– Peel off calls to go to domestic teams or offshore teams– Extensive control over routing and reporting– Can do compliance recording– Difficult to do agent monitoring
9
Vendor Management
• Outsourcer provides Reporting and Service Level Management for Blackhole implementations
– Typically, this means end of day reports with metrics like SL, abandons, max queue depth, etc.
• Enterprise has insight into overall call center performance through historic reports
– Some get more depth in reporting
• These features cost more, and provided by outsourcer based on their technology implementation
– The more features that the outsourcer provides the enterprise, the higher the cost (software licenses for switch vendor’s reporting software)
10
Issues
• Management (Visibility, Control, Quality) of call delivery to captive and outsourced call centers
• Assessment of outsourcer performance
• Contract management– Staffing level– Customer satisfaction on service levels
• Non-capex solution required
11
Solution Requirements
• An outsourcer management gateway• Matches any deployed technology• Zero footprint at Enterprise & Outsourcer• Integrates with existing call center solutions• Delivered as a managed service• Cost effective, pay as you go model
Solution Approach
13
Technology Trends
• Migration of enterprises to converged networks• Emergence of SIP in the VoIP space• Downward cost curves of VoIP equipment• Leverageable building blocks• Standards
– VoiceXML– CCXML
14
The Solution: Version 1
• Match demand (calls) with supply (trunks leading to outsourcers)
• Let call center use its own ACD• Not be responsible for transport (carriers do that)• Get visibility into all calls
– Impossible to do this with TDM– Being in the SIP signaling path throughout the call tells all
• Provide some control over call destinations– Define capacity based trunks to send calls– Perform dynamic load-balancing across trunks (towards agents)– Tackle the set of problems that are unique to Outsourcing
• Provide enterprises with silent monitoring capability
15
Separate Application from Network
Application Servers
CCXML
Application Server executes the logic, and controls the call within the networkthrough CCXML documents
SIP
16
Inside the Network
SIP
Media Gateway
Media Servers
Call Control Gateways
HTTP/CCXML
17
Application Servers and Networks
Application Server
CCXML
Application Server controls calls within multiple service provider networksusing CCXML documents
SIP
SIP
SIP
18
What Happened
• Funded the idea of a Management Gateway– Nobody would think of funding yet another ACD
• Built the team• Built the techology• Went to market• Customers said they needed to be able to deliver calls
from the mid-point based on agent availability• Crucial customer feedback on viability of such a solution
– ACD reports not useful if the full lifetime of the call is not known– Call center metrics get skewed (to excellent!) if the non ACD
queuing time not included
19
The Solution: Version 2
• Added call delivery based on agent presence– State of line not needed
• Found first customers• Incorporated additional market feedback to
include call transfers: blind/consult/conference• Added many additional ACD oriented features
20
Extension to Architecture
Application Servers
CCXML
Added Agent Login and Presence from browser-based desktop applet
SIP
21
Sample Enterprise Implementation
PHX
PHX
OMA
DEN
ATL
MCO
JFK VPOP
Verizon PSTN
3 VZB DS3
VZB MPLS
LAX VPOP
3 VZB DS3
C1
C2
O (CR)
S3
S1 (ES)
S1 (Manila)
T (Manila)
S2 (Dom. Rep)
6Mb
6Mb
9Mb
1.5Mb
6Mb
W
15Mb
MIA
ATLAC
SJCAC
Internet
Sprint MPLS
9Mb
SykesIPLC
6Mb
15Mb
ROC
Manh
Rich
6Mb
6Mb
305555....
4692617[0-3]..
585550....
770555....
4075551[5-9]..
602555....
402555[1-4]...
Along the Way: Learnings
23
Choice of Technologies
• Linux as a server platform– Experience has been par excellence
• Call Control Gateway implementation– SIP B2BUA– Java or C++: C++– Choice of language determined by developers available
• SIP stack– Open source: generally bare metal, required a lot of
maintenance– Commercial stack: we chose Dynamicsoft
• Previous experience with it at Telera (Genesys)
– General experience• Definitely required in-house maintenance• Robustness issues under load, race conditions, etc.
24
Choice of Technologies
• Media Server– Desired to use any available VoiceXML media server– Conferencing features not standards based– Chose Snowshore/Brooktrout/Cantata/Dialogic Linux-
based software media server
• Application server implementation– Java is an excellent choice– Scales and performs very well
• Used Cisco MGs for development
25
Deployment Experiences: Interop
• First implementation in a carrier’s network required interop testing with Lucent Telica
• Making the CCG work well with the Telica (Interop test)
– Straight call worked fine (as expected)• Issues with REINVITES, codec negotiations, header contents
(E.164 number vs label), etc.
– Took longer than most, as interpretation of SIP can cause behavioral issues
• Made changes to CCG adapt faster than Lucent could issue patches
26
Tenant Identification & Trunks
• Every tenant in system had a unique (internal) dial prefix• Number with dial prefix delivered to the CCG which
needed the digit string to identify SIP trunk group and number to be presented
• Developed a re-direct server that performed number translation, and route identification
• Also needed to deploy CCG on separate IP address per tenant
– Carrier billing required this– Linux supports multiple IPs: CCGs instantiated per tenant
27
Post-Deployment Experiences
– Customer encountered one-way audio issues (on some calls)
• Isolated a specific SBC to capture traffic (a difficult task in a carrier network) and proved that RTP stream was intelligible in one direction
• Incidents disappeared after MG software updated
– Session timers• Telica reinvite appearing on our second leg using Cseq 1• Turned out to be a stack bug (and calls got dropped when
enough of these invites got dropped by stack)
28
Learnings: SIP & VoIP Products
• Interop tested with numerous SIP products:– Media Gateways– Phones (hard, soft, G.711, G.729)– Registrars– Hosted IP PBX (Broadsoft, Sylantro)
• Straight call always worked• Reinvites as a B2BUA can be tricky to debug especially in
a carrier network• Voice quality and echo cancellation issues are very thorny
– One fixed through vendor DSP firmware update– A set of such intermittent problems fixed themselves when MG
updated
29
WAN Reliability
• Even the best Internet backbones have packet loss
– Moved to MPLS after achieving a certain volume of customers
• Application tolerance to packet loss– Needed tuning to retry and back off– Balance real time responsiveness with compensation
for network
Summary
31
In Conclusion
• A real Enterprise problem that needed solving
• Could not even think of tackling this problem without SIP
• Interesting problems to solve along the way due to initial maturity of technologies
• A lot has happened in 4 years
32
Summary
• Problem & Background
• Solution Approach
• Along the Way: Learnings
• Summary