outsourcing resnet at ithaca college resnet ithaca college residence hall computer network
TRANSCRIPT
Outsourcing ResNetat Ithaca College
ResN
et
ResN
et
Ithaca CollegeIthaca CollegeRESIDENCE HALL COMPUTER NETWORKRESIDENCE HALL COMPUTER NETWORK
2
Ithaca College
• Located in upstate New York• Founded in 1892• Private, coed, residential college• Offering undergraduate and
graduate programs in 5 schools & aninterdisciplinary division.
• Over 100 degree programs
Faculty 440
Staff 985
Students 6,200In Residence Halls 4,400
3
Information Technology Services
Director of ITS
Information Systems
and Services
Network and Communication
Services
Technology and Instructional
Support Services
Web, Systems & Departmental
Services
• Networks• Phones•9 people
• Field Service•User Support•Academic•14 people
• Web Dev.• Sys Admin• ITS Business•18 people
•ERP, Ops
•19 people
4
Residence Hall Network
• Phased in over 3 years in early to mid 90’s
• “One port per pillow”
• Today - 53 residential halls with 4,400 ports
5
ITS provided ResNet
• Flat network, 10/100 Mbps to the desktop • “Free” – no additional charge for the students• Shared main college Internet connection
– ResNet bandwidth was allocated as a percentage of our main connection
– Eventually allocated bandwidth by time of day (shifted more bandwidth to ResNet at night, less during day)
– Shaped Internet traffic (allocated very little bandwidth to identified P2P traffic)
6
ResNet Pressures Build & Change
At first… • It was all about support to get student machines configured and connected
By 2000 it was about….
• Support &
• Bandwidth &
• P2P Management
And then…. • File sharing (RIAA) &
• Bandwidth &
• P2P Management &
• Security - Viruses and Worms, and their impact on network performance
7
Bandwidth Allocated to ResNet
1998 3 Mb
1999 6 Mb to 12 Mb
2000 16 Mb
2001 24 Mb
Spring 2004 38 Mb
38 Mb
3 Mb
8
Fall 2003 Approaches
• Viruses, Worms, and other self-propagating nasties– And their significant impact on network performance
• RIAA, MPAA, ESA…• Insatiable Bandwidth Desires• P2P “whack-a-mole”• Internal debate regarding the purpose of ResNet and
use of our resources, and what is appropriate use of ResNet (games, P2P, etc)
• Staff needed for other projects
9
Fall 2003 – It boils over
• August 2003 Blaster and Nachi strike and ResNet could not handle the traffic from infected machines
• On the Friday before opening weekend we turned off the entire ResNet to prevent further infections – It ended up being off for 7 days while we
developed procedures to deal with the situation and worked with students to get their machines protected and cleaned
11
Why was it so bad?
• Flat network– The entire ResNet was one subnet, no way to contain an
infected machine
• Full 10/100Mbps Ethernet speeds– Allowed viruses and worms to spread like wildfire – An unprotected machine would be infected within
30 seconds of connecting to ResNet
• No firewall protection for ResNet machines• Individuals were not used to having to worry about
computer security or patching
12
And once past opening weekend…
• We continued to take incremental steps to address many of the performance, support and network architecture issues
• Had to remain vigilant and constantly turn off ports of infected machines– One or two infected machines would have significant impact on
network performance
• Saw a significant increase in P2P usage• RIAA, MPAA complaints increased significantly• Continued to experience a huge drain on support, network and
other ITS staff – many projects were put on hold• Bottom-line: throughout Fall 2003 ResNet performance was
sporadic and ITS staff stretched thin
13
Students (and Parents) React
• Student created “ResNet Sucks” Website with on-line petition and call-to-action
• Editorials and letters in student newspaper
• Students deliver a petition to ITS with 1,175 signatures demanding a better ResNet
• And of course, angry emails and phone calls to the helpdesk, ITS management, and the college administration
16
Something had to change
• If we were to continue to support ResNet ourselves, we would need to:– Significantly change and upgrade the network
– Install better traffic monitoring and compliance systems
– Provide additional network and support staff
– Find additional space for student computer repairs
• However, we were concerned about the ability to find funding and staff resources to support this for the long term
17
Thoughts turned to outsourcing
• Initial requirements:– Tiered service plan including a base no-cost option
for the students– On-site support– Increased bandwidth– Robust network architecture that could better
withstand viruses, Trojans and worms– A true partner interested in a collaborative
relationship and who understands higher education
18
We solicited proposals
• A major nation-wide telecom
• The local cable-TV / broadband franchise
• A small company specializing in supporting residence hall networks
19
The nation-wide telecom
• Limited experience with this type of network and service
• Limited experience with the academic environment
20
Local cable-TV / broadband provider
• Would provide a significant discount for their broadband service – Based upon a guarantee # of subscribers from us– The college would need to make one payment to
them, and have students pay the college for the service
• Would not provide tiered or free service options
• No direct on-site support
21
Local cable-TV / broadband provider
• Would need to build an entirely new network infrastructure in the residence halls– Current cable-TV infrastructure could not support
broadband service– College would be unable to support this cable-TV
based network should the partnership end
22
Small Residence Hall Provider
• Agreed to meet all of our requirements– Tiered service options (including “free” option)– On-site support– Increased bandwidth / network redesign– Understood residence hall networks and the
academic culture• CEO very effective at talking with the students about
their concerns and issues
– Interested in a partnership-type of relationship
23
Small Residence Hall Provider
• Concerns:– Located in Austin Texas with only two current
customers– We would be their largest deployment (and
farthest away)– Small company, operating “lean”
24
Selected Apogee Telecom
• We felt that these concerns were manageable
• Selected the small residence hall provider, Apogee Telecom
• Felt they were the best overall fit for us
25
What we outsourced…
• Redesign of the Residence Hall network
• Network monitoring, troubleshooting, repair and replacement
• Custom service levels and prices that met our particular needs and budgets
• Management of the account sign up process
• Student billing
• Marketing and PR
26
What we outsourced…
• An on-site technician for troubleshooting student network issues
• A call center open 7 days-a-week • The ISP of record for DMCA and other
complaints– Work collaboratively with the college for
responding to DMCA complaints– College gets copies of all notices and still may
impose judicial sanctions
27
Apogee Telecom – The Network
• Redesigned our residence hall network and “over built it”– Provided 3.5 times the bandwidth (OC-3, 155 Mbps total)– Used our switches, but replaced most other components– Installed two Allot Communications NetEnforcers to
control network traffic and bandwidth allocation– Significantly limited the number of computers on any one
subnet• Students allocated bandwidth based upon their
service level– Could use that bandwidth for whatever they wanted (any
“legal use”)
28
Fall 04 Apogee Service Plans
PlanCost per semester
On Campus Resources Download Upload
Basic Free High speed* 64 Kbps 64 Kbps
Bronze $49.95 High speed 1 Mbps 128 Kbps
Gold $69.95 High speed 3 Mbps 256 Kbps
Internet Access
*High Speed = 10/100 Mbps to all ithaca.edu resources
29
Apogee Service Costs
• The College pays Apogee a base fee per month to cover:• 25 Mbps administrative internet service
• “Free” 64 Kbps basic-level Internet service for all students
• High-speed access to on-campus resources for all students
• Apogee help desk support for ResNet students
• This base fee is roughly equal to the amount we used to pay to our old campus network provider for the T3 service – so in the end, there was no increase in costs to the college
31
TimelineContract signed
Communicated plans to campusTechnical site visits and design workCable trenchingImplementation sub-teams created
Support procedures created
Student antivirus strategy determinedNetwork design finalizedOnsite support staff interviewed and hiredLegal / policy issues agreed upon
Site coordinator startsNetwork infrastructure upgraded
ResNet by Apogee Live
March April May June July August
32
Campus Communication
• Apogee CEO came to campus and met with administrators, student government, and conducted an open town hall meeting
• Announcement letters sent to students and parents detailing changes
• Ads placed in the student paper
• On-going dialogs with student leaders and parents to respond to concerns…
34
A petition to ITS with 1,175 signatures responding to a February 26, 2004 student newspaper article. They request that we work with a vendor such as Apogee to fix ResNet and not impose any additional fees for what they consider a basic and necessary service.
35
Account Sign-ups
• No matter what service level, people need to sign up for an Apogee ResNet account
• Advance sign-up mailings sent over the summer to all students living in residence halls
• Freshman sign-ups at orientation (June/July)
• Could sign up via Web, phone or paper form
• 2,892 people signed up in advance (67%)
36
Fall 04 Apogee Subscriptions
Basic 1,937
Bronze 1,120
Gold 1,214
Total Subscribers 4,271
As of December 28, 2004
Basic46%
Bronze26%
Gold28%
37
Activation Process
• A student plugs their computer into a ResNet port
• Automatically directed to Apogee login page– Includes a link to the account
sign-up page if needed
• Student activates computer using Apogee supplied username and password
• Only need to do this once per semester
38
Cutover and Move In – Here’s Murphy
• Last of the new equipment cutovers occurred shortly before opening weekend
• Major fiber-cut 60 miles away took down service the week before opening weekend
• A few days before opening weekend a network problem developed where students were not getting their subscribed-to level of service– Would result in a MAJOR PR problem if it didn’t work!
– Apogee was able to put a workaround in place late at night on the Friday of opening weekend
39
Opening Weekend
• Our helpdesk and offices were quiet (it was strange!)• Apogee’s call center was busy, on-site staff were very
busy and some students had to wait a few days for on-site help
• Overall things went smoothly– Few if any complaints ever came to us regarding problems
• ITS staff were unsettled by not having the ability to directly monitor ResNet performance or call volume to the Apogee support center
40
Early Fall
• A few scattered equipment problems
• One particular residence hall experienced a few interruptions due to someone repeatedly plugging an improperly configured wireless router into the network
• Overall, things went well
41
Other (minor) issues that came up along the way…
• Different expectations for who would be on-site during opening weekend
• ITS staff and management had some trouble “letting go”….
42
How’s it working?
• ITS has had time to focus on other things– “This has been my most productive fall in a very
long time”
• Students seem to like the service– Apogee conducted a customer survey in late fall.
Out of 4,193 survey’s sent out, received 1,067 responses (25.4%)
43
“Connection is stable and reliable”
Total responses: 1,067
Agree 54%
Disagree 33%
Undecided 13%
0
100
200
300
400
500
600
700
44
“I am satisfied with the speed I signed up for”
Total responses: 1,067
Agree 69%
Disagree 21%
Undecided 11%
0
100
200
300
400
500
600
700
800
45
“The Apogee call-center support staff was knowledgeable in troubleshooting my issue”
Total responses: 425
Agree 80%
Disagree 8%Undecided 12%
0
50
100
150
200
250
300
350
400
49
Spring 2005
• Continuation “as-is”
• Continue to optimize network settings and support procedures
50
Future
• Upgrade a few more network components• Look to add possible other services in the
future:– Provide subscription-based TV-like services over
the network– Wireless in residence halls– Better automatic virus and patching updating and
quarantine services
• Revisit price-points
51
Is Outsourcing ResNet For You?
• Depends on a number of factors such as:– In-house staffing levels and expertise
– Your view of the roll / mission of ResNet
– Ability to continually refine the network and services to meet changing needs
– Cost structure
• It’s really a decision that must fit with the values, expectations, and unique situation of each institution
• For us, so far the answer has been yes
52
Other Resources
• The Chronicle of Higher Education– “Outsourcing the Dorm Network”– December 3, 2004
53
Special thanks to David Weil
Director of Web, Systems and Departmental ServicesInformation Technology Services
for his assistance in preparing this presentation.
Outsourcing ResNetOne Institution’s Experience
ResN
et
ResN
et
Ithaca CollegeIthaca CollegeRESIDENCE HALL COMPUTER NETWORKRESIDENCE HALL COMPUTER NETWORK