isc networking

ISC Networking & Telecommunications

Educause National 2006

Mike Palladino & Deke KassabianUniversity of Pennsylvania

Educause National 2006October 2006

Production Quality Open Source VoIP: Next Generation

Telephony at Penn



Copyright Michael Palladino & Dikran Kassabian [2006]. This work is the intellectual property of the authors. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the authors.



Agenda

• Background, Organization, Business and Motivation Discussion – Mike Palladino

• Project and Technology – Deke Kassabian



Background: Penn

• A private, Ivy League university in Philadelphia, PA founded in 1740

• 22,000 students/4,000 faculty/13,000 staff

• 7500 students live on campus• 9000+ students, staff and faculty live in the surrounding community

• Health System has separate management–IT division outsourced/15,000 users



Background: Information Systems & Computing

• ISC is the central computing group– 300 technical staff/contractors

• CIO (Robin Beck) reports to Provost and EVP for administration– Three main groups

• Administrative Systems• Support & Security• Networking & Telecommunications



Background: Networking & Telecommunications

• 100 technical staff/contractors• 100% direct charge budget • 25,000 phone lines/12,000 voicemail users• 50,000 IP addresses/14,000 email accounts• 9,000 analog video connections• 40+ services from wiring, to digital video, to R&D for emerging technologies

• Internet2 GigaPop (MAGPI)/38 institutional subscribers



Our Vision

• A unified staff delivering excellent, integrated (data, voice and video) systems and services over one converged, highly reliable, robust IP network capable of handling today’s applications and tomorrow’s innovations.



Leadership Role-Convergence Readiness

• Organizational• Technical • Financial



Organization

• Networking, Telecommunications and Video Groups merged in 2000

• Much staff re-education has taken place

• In the past 6 years, 109 staff/contractors changed

• New organization is nearly fully integrated and increasingly well cross-trained



Organizational Convergence• Consciously avoided silos through

integration of data, voice and video staff with similar jobs– Centralized Finance Group– Consolidated asset and contract management

– Centralized help desk – Single Operations & Infrastructure Groups

– Single Video Group (analog, streaming, digital, production, podcasts)

– Single Network Management Group– Emerging services group developing



Technical Convergence

• Implemented through Next Generation PennNet Project– Provide infrastructure foundation for next generation data, voice and video services

– Eliminate single points of failure– Network redundancy to every campus building

– Reduce catastrophic disaster recovery time from 2 weeks to under 2 hours

– Self-healing routing core and services



Campus Map



NAP5WESTERN TIER

LEVY

NAP3SOUTHERN TIER

MOD5

NAP1EASTERN TIER

VAGELOS

NAP4NORTHERN TIERSANSOM EAST

NAP2CENTRAL TIER

HUNTSMANNAP-CH

COLLEGE HALL NODE ROOM

Next Generation PennNet



Financial Convergence

• All budgets consolidated into one• 100% direct charge budget • No capital budget• Replacement cycles built into the charges

• All services financially self-sufficient.

• User participation & full budget transparency through Network Planning Task Force (www.upenn.edu/computing/group/nptf)



Voice Services at Penn Today

• Most users have Centrex service– Local Exchange Carrier -provided business telephone service

– Long list of available features– Switch at Telco Central Office– On-campus copper cabling

• Some departments have PBXs and small key systems



Telecommunications Infrastructure

• Local Exchange Carrier brings copper loop infrastructure to campus Rate Demarcation Points (RDPs)

• Penn maintains in-ground and in-building copper cable infrastructure from RDP to building, building to closet, closet to wallplate

• This infrastructure is very old, and in some cases is failing resulting in outages and requiring expensive repairs

• Replacement of these cables would be $3-5M



Rate Demarcation Points



Business Drivers• Aging phone-cabling infrastructure

– In-ground cabling is 25-75 years old– Cable faults are increasing– Cable repairs are expensive and time consuming

• Operational inefficiencies– Separate cabling infrastructure– Moves, adds and changes are time consuming and costly (3000/yr @ $125= $375k)

• Legacy services costly to support– ACD– Voice mail



Business Drivers (Continued)

• Vendor Management– Service delivery issues– High usage rates – Limited application integration/flexibility

– Prefer vendor independence– Prefer open standards– Prefer open source



Voice Services Goals• To convert 25,000 analog voice customers to Integrated Communications (VoIP, Voicemail, etc.) on a converged IP network with added functionality and lower costs in 5 years or less

• Cost–effective, reliable voice communications

• Vendor independence• Single integrated network infrastructure• Flexibility and new functionality to meet evolving demands



Voice Services Strategy

• Negotiate low-cost phone line and usage contracts

• Develop architecture to enable competition among service providers for connectivity and usage

• Eliminate legacy voice services and position for Integrated Communications (ACDs, Voicemail, etc.)

• Programs with 2 cellular phone service providers– Over $125k of revenue-sharing annually – Over $300k savings annually to Penn community



Leadership Challenges: VoIP

• Penn’s research into integrated communication and VoIP started in late 1990s

• There was a lot of activity with minimal progress

• Focused leadership was required– Integrated Communications was made our top departmental priority

– A program director whose sole responsibility was the VoIP project was hired

– Project teams were immediately set up



Deke KassabianDirector, Engineering & Services

University of Pennsylvania

Project and Technology



History of VoIP at Penn

• Protocol research dates back to late 1990s• Worked first with H.323-based VoIP, and later with Session Initiation Protocol (SIP)

• Worked with a range of applications:– Cisco Call Manager, an enterprise solution (2001-2002)

– BroadSoft’s Broadworks, a carrier-class VoIP software solution, piloted on campus (2001-2003)

– Externally hosted IP Centrex solutions (2003-2004)

– Open-source SIP Express Router with Asterisk Comedian Voice Mail (2003-Present)



Key Open Source Question

• Does IP Telephony lend itself to open source server software?– Can it be reliable enough?– Will we have the feature set we need?– Will we be able to maintain it well?

• Experience with open source server software for Email and Web services encouraged us to take this as a serious option



The Breakthrough

• Held a VoIP Project launch meeting with managers and team leads

• Identified biggest issues• Committed significant local staff resources

• Hired a project manager



Project TeamsSteering

Committee

Technical Process BusinessFacilities

Network Architecture

Application Architecture

Telephony Architecture

InstallationsOperations

IntakeSupport

Billing

Provisioning

Communications

Penn iPhone



Open Source Strategy• Avoid over-customization

– Use existing functionality where possible

– Maintain active relationship with developers

– Try to get any essential changes of general interest built back into mainline code

• Notable Asterisk customization– IMAP access to voice mail message store for true unified messaging from phones and email clients



VoIP Building Blocks

• SIP Clients or User Agents

• SIP Proxy Servers• Media & Feature Servers

• Gateways and ITSP services



SIP User Agents (UAs)

• Hardware SIP phone, in the familiar desk set form factor. Penn currently uses the Cisco 7940 phone.

• Software that acts as a SIP client or user agent and runs on a desktop, laptop or handheld. Shown here is eyeBeam from Counterpath.



SIP Proxy Servers

• Server that accepts “registration” from valid users

• Allows client to signal their call information

• Handles call set up and gets out of the way

• Penn uses SER (iptel.org)and OpenSER (openser.org)



Media & Feature Servers

• Server that handles “media” such as voice mail and music on hold

• Provides for some additional call features

• Co-exists with provisioning services

• Penn currently uses Asterisk (www.digium.com)



Gateway

IP ISDN PRIs

PSTNCampusNetwork

Gateway

• Interconnect between campus network and PSTN

• Penn currently uses a Cisco 36xx router



General VoIP Design



VoIP Topology


Penn iPhone Roadmap

Phase 1.X• IMAP Message Store

• Limited Multi-line Pilots

• PiPS Maintenance

• Security Recommendations

• Handset Recommendations

• 600+ phones deployed

Phase 2.0• Feature Release 2.0

• Softphone Pilots - I

• Centrex-Asterisk Pilot(s)

• ITSP/LNP Production

• New Handset Testing

Phase 3.0• Octel voice mail migration

• New Handset Production

• Softphone Pilots - II

• iPhone Communications

• Multi-line Pilots

• 2000 phones deployed

Phase 1.0• VoIP Campus Pilot Begins

• Penn iPhone web portal

• Penn Directory lookups from iPhone

Phase 3.0Phase 1.X Phase 2.0

1H 2006 2H 2006

GeneralAvailabilityPhase 1.0

1H 2007 2H 2007



Campus-wide Pilot Rollout Deployed Penn iPhones

0

100

200

300

400

500

600

700

800

2005-012005-022005-032005-042005-052005-062005-072005-082005-092005-102005-112005-122006-012006-022006-032006-042006-052006-062006-072006-082006-09

Month

total iPhones



Current State of Service

• Production-grade– Redundant servers, gateways and PRIs– Single-line features, email/voicemail integration

– 911 support equal to legacy system

• Layer 2 QoS, separate VLANs & subnets

• Full service- installation, support & billing



Managing Penn iPhone

• Web “provisioning interface” to allow– Installation staff to configure phones for deployment

– Support staff to provide local support

– End-users to select and maintain configuration options



Penn iPhone Web Services (Phase 1.0)

Self-serviceVoice mail

Change Email

Password

Do Not Disturb

Call Forward

Reject Anon.

Blocked Caller ID

More . . .



Unified Messaging

• Some users prefer getting voice mail in email. Others really need to access voice mail by phone.

• Wanted to access voice mail via email and via the telephone without message replication

• Penn-developed extension to Asterisk for IMAP access to voice mail for true unified messaging from phones and email clients– included in version 1.4 with Fall 2006 release



Emergency Services & Location

• Ability to route 511/911 calls, including support for location information

• Three phases towards improvement– Phase 1 - Fixed location phones– Phase 2 - VoIP phones movable by IT Staff, with web form location reporting

– Phase 3 - Dynamic location updates



Security & Privacy

• Key project elements for S&P– Configuration security - protecting phone configuration and user details

– Gateway security - securing gateways so only authorized users make calls

– Call privacy - minimizing the eavesdrop risks



Current Development Projects

• Multi-line and ring-group support• Using ITSPs (Internet Telephony Service Providers) for off campus calls

• Migrating Centrex users to our next generation voice mail

• Power Over Ethernet (PoE) network designs

• Evaluating additional VoIP handsets



New Roles for Campus IT Staff?

• Local tech support staff troubleshoot basic connectivity, occasionally power cycle phones

• Submit trouble tickets when necessary

• Act as proxy for the user in the web interface, to change feature settings and forwarding numbers



Collaboration within Higher-Ed

• Several Internet2 universities are using a very similar approach

• Sharing plans, code and best practices through monthly collaboration calls

• Open to participation by other groups seriously engaged in similar efforts



A Few Key Challenges• “Recruiting” pilot customers that fit the service profile, are in a suitable location, and can provide detailed feedback

• Identifying supportable, high function VoIP handsets

• Managing very substantial staff workload

• Moving fast enough to avoid costly telephone cable upgrades



Lessons & Recommendations

• Make project a priority across the organization

• Cross-train staff, and allow time for adjustment. Augment staff with consultants in key areas

• Pilot one technology at a time (open-source vs. in house commercial vs IP centrex)

• Create/maintain a voice development environment



Lessons & Recommendations• Roll out pilot and new features in your

own department first, as a pre-pilot• Replace existing phones with VoIP

phones on users desk (don’t simply add)• Structure installations to be “cookie

cutter”• Pilot not only technology, but full

support and communications process. Involve all service delivery and support personnel



Looking To The Future

• With the elimination of legacy services and networks, we anticipate:– Increased reliability– Improved functionality– Higher levels of operational efficiencies

– Telecommunications cost reductions of 10-30%

– Better customer service



Mike Palladino & Deke KassabianUniversity of Pennsylvania

Educause National 2006October 2006

Production Quality Open Source VoIP: Next Generation

Telephony at Penn

isc networking

Documents

video services

legacy voice services

voice services strategy

voice services goals

video staff

video systems

phonecabling infrastructure

video groups