managing remote operation teams
TRANSCRIPT
2
Definitions
• Mixed in-house and remote employees
• Fulltime remote employees
• Remote office with remote employees
• Remote contractors
• Dedicated technicians via 3rd party support vendor
• Shared technicians via 3rd party support vendor
• 24/7 Technical Support (L1/L2)
• Infrastructure Support (Network/Cloud/Storage)
• Provisioning
• Development
• Management
Who?
Doing What?
3
Why Consider Remote Operation Teams?
• Take advantage of a larger talent pool• Enables Strategic Placement: • Infrastructure/Colocation (Boots on the ground)• Customer-base or language• Emerging markets• 24/7 Coverage
• Sets up company for scale & growth• Employee Flexibility:• Full or Partial remote work• Select from available locations• Start a new location strategically
• Lower Costs• Built in management layer and accountability (3rd party)
4
The Challenges
• Language & Culture Differences• Time Zones• Creating an Inclusive Environment• Real-time Communication• Monitoring• Response Time & SLAs• Customer Interaction• Rewards & Compensation• Long Term Employee Engagement • Technical Knowledge, Training, and standards• Liability, Tax, and Financial implications
5
Our Environment
Founded: 1996Headquarters: Long Island, NYServices Offered: Public, Private & Hybrid Cloud, Dedicated Servers, Colocation, CDN, Security, DRaaS, and moreCustomers: Enterprise, Healthcare, SMBs, VoIP providers, eCommerce, Information Technology, and more
The Webair Value: Over 18 years providing customers with best-in-class Managed Hosting solutions High-touch Support Full ownership of our customer’s infrastructure stack so they can focus on their core
business. Manages most secure and redundant facility east of Manhattan with trans-Atlantic bypass
(Tier III, HIPPA/SSAE16)
6
Global Presence
Los Angeles, California• Technicians local to Colo• Infrastructure Engineers
Bodrum, Turkey• DevOps
Amsterdam, Netherlands• Technicians local to Colo• Systems Administrator
Boston, Massachusetts• Systems Administrators
Coimbatore Tama Nadu, India• Technical Support
Managers• Systems Migration
Specialists• Systems AdministratorsMontreal, Canada• Data Center Technicians
Long Island, New York (Flagship)• Customer Service
Specialists• Infrastructure Engineers• Project Managers• Account Representatives• PHP / Linux Developers• System Administrators
7
8
Language & Culture
• Make a conscious decisions you’re willing to deal with the nuances
• Understand local holidays• Implications of vacation time• Phone support & accents• Provide ESL training where applicable (even in house..)• Be respectful of local cultures, religions, and acceptable
chat topics• Language Differences:• South Africa: Floppy disks vs
Stiffy Disks • Indian English: “Do One
Thing..”• “I have a doubt about ssh
keys” (question)
9
Team Inclusion
• Do whatever you can to promote the human factor. They’re not just names on a chat screen
• Don’t change people’s names!! Promote personalization.• Engage them like in-house employees (small talk, build a
rapport)• ‘Good’ Video chat for regular in-house meetings, stand ups,
and scrums• Opportunistic meeting (conventions)• Meet face to face at least yearly to maintain relationship
and train• Swag & Business cards• Challenge them• Understand you may not be
approachable• Compensation, Rewards, and performance
reviews
10
Managing & Monitoring Complex Infrastructure Remotely
• How can our remote teams support & troubleshoot complex setups with little to no information on them?
• LBs, App, Web, DB, FW, NM, CDN.. (non Cpanel..)
• Distributed, virtualized, and abstracted infrastructure - Network, Storage, Compute.
• Self-healing & Auto scaling
• Microservices based architecture
• Troubleshooting this == An Art.
11
Communication & Context
Make it part of your culture.
12
Centralized Monitoring
13
Centralized Monitoring & Context
14
Monitoring for Success
• Fast resolution starts with a solid monitoring strategy.• Empower DevOps to create layered alerts.• Create simple interfaces, scripts, APIs to allow for easy
additions of monitoring to standardized systems.• Make it part of development & build process.• Monitor the Big and the Small:• Big - The finished product. Look for expected result via end
user interface to ensure all services are properly functioning.• Small - Every layer! OK/FAIL monitoring for multi-
dimensional services and 3rd party providers.• If setup properly, big and small alerts should trigger
simultaneously allowing instant localization.• Build automated actions triggered on alerts (log monitoring,
splunk..)
15
Remove Roadblocks..
• Don’t relegate remote Level1 teams to only observe. Empower them to participate in the long term success of your customers.
• Balance standard methods & procedures with flexibility.
16
Smart Monitoring & Troubleshooting
17
3rd Party Monitoring (NIH)
”In God we trust. Everyone else we Monitor.”
18
Aggregated Statistics
Cluster Wide
CPU
Network
Memory
Disk
19
Aggregated Statistics
Network by Host
Host 1
Host 2
Host 3
Host 4
20
Aggregated Statistics
Network by VM (Host 1)
VM 1
VM 2
VM 3
VM 4
21
Training & Technical Knowledge
• Documentation software & strategy• Owners for various sections• Automated article expiration• Built in diagram creation• Tie ins to other systems• Part of your culture, comments, and chat
• Knowledge sharing & training strategy• Prescheduled internal training• Mentor program, buddy system• Remove Silos• Build policy which discourages over-escalation
• Don’t forget the soft skills – Troubleshooting, Technical writing, Resourcefulness, Googling, Customer interactions
22
Documentation
“If it wasn’t documented, it didn’t happen.”
23
Security
• References & Background checks• VPN• Ability to quickly cut access • Build trust over time, tiered access levels• Activity based alerts (strange queries)• Historical logging (SSH & other), auditing• Unique IPs & hostnames• Device policies• File change monitoring (Tripwire)• Out of Band access for remote teams
24
Legal and Tax liabilities
• Customer privacy concerns (ie ePHI)• Remote team strategy in line with audits & compliance?
(SSAE16, HIPPA, etc)• Data stored in other countries • Impact on BAA agreements• Sufficient Cybersecurity insurance which covers remote
team members or outsourced?
• Tax deductions when paying out of state employees • Are NDAs enforceable?• Payment methods & Currency• Sanctions??
25
Do it Yourself!
• Expand Opportunistically• Strong remote team starts with trust –
Send someone you trust to start one!• Employees moving ‘back home’ can
start a remote office and recruit locally• Ensure remote management and
accountability• Are they capable of it? Do they want it?
26
Working with 3Rd Party Companies
• Presence in your country is a bonus• Do you receive dedicated support or shared?• What management options are available?• What is their employee retention rate?• Do they rotate employees or do you get to keep them
long term?• Who is accountable for that team, who can you trust?• Do they offer contracts?• Will they sign an NDA, who is liable for security?• Will local laws help or hurt you?• References
