cloud hosting - sasktel€¦ · cloud hosting is a data centre service that provides hardware and...
TRANSCRIPT
CLOUD HOSTING
OPERATIONS MANUAL
Revised: Nov 2020
2
Confidentiality and Proprietary Statement This information is SaskTel’s property and it is strictly confidential. Without SaskTel’s prior written permission, this information must not be copied, disclosed or distributed in whole or in part. By receiving this information, the receiving party is bound by these conditions.
3
TABLE OF CONTENTS
1.0 Introduction ................................................................................................. 4
2.0 Service Description .................................................................................... 4
3.0 Responsibility Matrix ................................................................................... 4
4.0 VRGs and VDCS ........................................................................................ 5
5.0 Virtual Servers ............................................................................................ 6
6.0 Physical Servers ......................................................................................... 6
7.0 Operating Systems ..................................................................................... 6
8.0 SaskTel-supported Operating Systems ...................................................... 7
9.0 Service Level Agreement ........................................................................... 7
10.0 Additional Services (Add-ons ) ................................................................... 9
11.0 Network .................................................................................................... 10
12.0 Change Management ............................................................................... 10
13.0 Request Management Process ................................................................ 11
14.0 Incident Management Process ................................................................. 15
15.0 Disaster Preparedness ............................................................................. 17
16.0 Monitoring and Alarming ........................................................................... 17
17.0 Billing ........................................................................................................ 18
18.0 Certifications and Audits ........................................................................... 19
19.0 Security .................................................................................................... 19
20.0 Environmental Controls ............................................................................ 20
21.0 Maintenance ............................................................................................. 21
4
1.0 INTRODUCTION
Welcome to SaskTel Cloud Hosting. This Operations Manual (OM) provides a description of Cloud Hosting services including technical details, contact information, and additional processes.
2.0 SERVICE DESCRIPTION
Cloud Hosting is a data centre service that provides hardware and support in our Enterprise-class Data Centre. This is a multi-tenant service in which SaskTel provides and owns the infrastructure (space, power and cooling; hardware, virtualization layer, storage, and computing resources) and management of this; while you are responsible for the management and support of your applications. The cost of providing and maintaining the infrastructure is shared across many customers, allowing you to realize significant savings over the cost of building and maintaining your own data centre, while also leveraging our technical expertise.
3.0 RESPONSIBILITY MATRIX
The following tables outlines the function that is understood to be completed by you and by SaskTel throughout the different stages of the service delivery process. On-boarding work (migration to the service) breakdown structure
Task SaskTel Customer
Read
iness
Phase o
ne
Identification of system and business functional requirements Perform
Solution design - application Perform
Solution design – hardware, supporting infrastructure, and SaskTel-supported OS NOTE: Additional charges may apply
Perform
Rack and network design Perform
Contract sent to customer Perform
Contract received back from customer Perform
Arranges In-service Date with customer Perform Assist
All one-time fees billed to customer (if required) Perform
Purchases server and network hardware Perform
Procurement, receiving and verification of equipment Perform
Build
an
d H
an
d-o
ff
Phase 2
Installation of equipment into the SaskTel Data Centre (if required) Perform
Implement SaskTel-supported operating system(s) Perform
Perform validation testing for non-SaskTel-supported operating system(s) – Additional charges apply
Perform
Implement validated operating system(s) – Additional charges apply Perform
Antivirus software is installed (if required) Perform
Sets burn-in period where customer can contact SaskTel Operations directly for any issues related to the Service
Perform
Provide login credentials to the Customer for self-serve portal(s) Perform
Implement the application layer Perform
Firewall/network configuration – SaskTel-managed Perform
Firewall/network configuration – Customer-managed Perform
Configuration of customer-owned network equipment Perform
Data migrations as required Perform
5
End-to-end validation testing Assist Perform
Monitoring and alarming is activated (if required) Perform
Server Patching services are initiated (if required) Perform
Service “Goes live” Perform
Ste
ady
Sta
te P
ha
se
3
Initiates billing for all recurring charges Perform
Issues back billing charges (if required) Perform
Contacts BFST for any incidents Perform
*For SaskTel-managed servers only
Operational work breakdown structure
Task SaskTel Customer
Opera
tio
nal
Fun
ctio
ns
Virtualization layer ongoing management and support Perform
Virtual/Remote - start-up, shutdown and other common server procedures
Perform* Perform
Physical hardware resets Perform
Server alarming, and alarm actioning Perform*
Operating system patching Perform*
Application backup and recovery Perform
Application patching including all Microsoft applications Perform
*Additional charges may apply
Support work breakdown structure
SUPPORT SaskTel Customer
Supp
ort
Hardware support Perform
Supporting infrastructure monitoring and alarming Perform
SaskTel-supported operating systems Perform*
Non-SaskTel-supported operating systems Perform
Application support Perform
*Additional charges may apply
4.0 VRGS AND VDCS
A Virtual Resources Group (VRG) is a set of predefined virtual CPUs, virtual GBs of RAM, and 1 Public IP selected by customers and contracted with SaskTel that forms the base for your virtual Data Centre (vDC). We create your vDC based on the contracted VRG and specified storage (in GBs); whenever customers don’t provide the amount of storage required, SaskTel provisions customers’ vDC with the minimum amount required for successful provisioning– 1 GB; customers can afterward adjust storage via the self-server Portal. VRGs have a minimum and maximum number of computing resources, these values represent the upper and lower boundaries of your vDC. If at any time during your contract term you assess your maximum computing resources available are insufficient, or your minimum computing resources available are excessive, to meet your needs you may request to upgrade/downgrade your VRG.
6
Storage used to provision your vDC has also minimum and maximum GBs available (minimum of 0 GBs and maximum varying based on VRG), if at any time during your contract term you assess your available storage is insufficient you may request to increase the upper limit for this. NOTE: Additional charges apply for any downgrade in virtual resources groups. To upgrade or downgrade your contracted VRG, or to request an increase in your vDC’s storage upper limit, please follow the Request Management process depicted in section 12. After provisioning your virtual Data Centre, we will provide you access to the Cloud Hosting self-serve portal where you can self-manage your contracted resources to your specific needs. You can hire additional managed and un-managed resources in a pay-per-use model to meet any unexpected increase in workloads. Please refer to the Client Portal User Guide for detailed information about features and capabilities of the self-serve portal.
5.0 VIRTUAL SERVERS
Our virtual solution leverages industry-leading vendors to provision highly reliable and scalable virtual environments. We use VMware virtualization technology for the hypervisor layer; Cisco and EMC for hardware infrastructure; and leverage Microsoft and Linux operating systems. For optimal performance of your virtual data centre, we recommend following the below parameters when creating virtual machines:
VIRTUAL MACHINES
Resource Minimum Maximum
vCPU 1 16
Increase in increments of at least 1vCPU
RAM (GB) 512MB* 300GB
*Dependent on OS to be installed
If you require virtual servers with computing resources above the previous maximums, please contact SaskTel prior to attempting deployment of such servers.
6.0 PHYSICAL SERVERS
All physical servers are customized to your specified requirements; if you require a new physical server a contract amendment needs to be signed before the provisioning of this new server occurs; for any changes to your physical servers please follow the Request Management processes outlined in section 12.
7.0 OPERATING SYSTEMS
SaskTel includes several standard operating systems for easy deployment with your virtual solution. Operating systems included in your solution at no additional charges are:
• Windows Server 2019
• Windows Server 2016
• Windows Server 2012 R2
• CentOS 7
7
• CentOS 6 Operating systems available for deployment in solution at extra charges are:
• Red Hat Enterprise Linux 7 SaskTel is the sole provider of patching services for all virtual servers deployed with Red Hat operating systems. When customers deploy a Red Hat virtual server, SaskTel will assess and deploy one or more ‘Red Hat Suite’ services based on the number of CPUs each Red Hat virtual server has, this suite is used to bill for both: the Red Hat operating system(s) and the patching service required for such server(s). This suite will appear in your bill as “Cloud Hosting – RHEL OS Licenses”. When customers remove a Red Hat virtual server(s), SaskTel will remove the ‘Red Hat Suite’ used to bill for such server(s) as well. You can upload your own OS template, via the self-serve portal, for your sole use, provided is compatible with our platform. Current compatible OSs with Cloud Hosting’s platform can be found at: https://www.sasktel.com/cloudhostingsupport
8.0 SASKTEL-SUPPORTED OPERATING SYSTEMS
A SaskTel-supported operating system allows for servers with such OS to be on-boarded unto the tools SaskTel uses to provide its management services. Current standard SaskTel-supported operating systems, and only available in the self-serve portal, are:
• Windows Server 2019
• Windows Server 2016
• Windows Server 2012 R2
• Red Hat Enterprise Linux 7 You can request SaskTel to perform validation testing on any OS uploaded to the platform at any time following the Request Management processes outlined in section 12. Additional charges apply for all validation testing.
9.0 SERVICE LEVEL AGREEMENT
All Cloud Hosting customers receive a 99.9% availability SLA at the VRG level; i.e. SaskTel targets for your computing resources (vCPU, and GB RAM) to be available a minimum of 99.9% of the time. You automatically receive a 99.93% availability SLA (Tier 2 availability) at the virtual server level for each server for which you have hired the next services:
• Tier 1 management
• Server Patching
• Antivirus
• Cloud Backup o Backup for servers, or for servers with file agents, or for servers with app agents o Backup storage with additional copy o Cloud Backup management
You automatically receive a 99.96% availability SLA (Tier 3 availability) at the virtual server level for each server for which you have hired the services required for Tier 2 availability, plus:
• Cloud Disaster Recovery: o Any RTO o Journaling storage o At least one of the 3 tiers of storage
8
o Replication bandwidth A VRG or server is considered unavailable only when 2 events have been validated by SaskTel’s systems:
1. When SaskTel’s monitoring system indicates the VRG or server is unreachable from within the Cloud Hosting network; and,
2. When SaskTel’s internal ticket management system has created an incident ticket related to the unavailability of the VRG or server.
The penalties for failing to meet the availability targets are as follow:
Virtual – Tier 2 availability
Monthly Uptime Service Credit
99.93% 0%
<99.93% & >=99.70% 5%
<99.70% & >=99.0% 10%
<99.0% 30%
SLA Credits won’t apply to unavailability or other performance issues caused by:
• Your software or other technology; or,
• Third party software or other technology used by you; or,
• Factors outside the reasonable control of SaskTel, including without limitation, force majeure events or network access problems beyond the demarcation point of the service; or,
• Your unauthorized action or lack of action when required; or,
• Your failure to follow reasonable security practices; or,
• Your failure to adhere to any policies or procedures for acceptable use of the Service; or,
• That results from any maintenance to the Service. SLA Credits Request Process If you believe that a service failure occurred, please follow the next steps to request your SLA Credits:
1. Log into your ITSM Centre account 2. Under the ‘Service Catalog’ section click on ‘Information Request’:
a. Business Service: Cloud Hosting b. Nature of Inquiry: Technical c. Please describe your question: First line of text should read ‘Request for SLA Credits’
after that please accurately detail the credit request including dates if possible. d. Attach SLA report if available.
3. Click on ‘Submit Request’
Note: Customers should request any SLA Credits within 30 days of the service failure.
We will investigate the request and contact you with the findings once the investigation is complete. If a credit is required, a SaskTel Sales Representative will process the credit to your bill.
VRG
Monthly Uptime Service Credits
99.9% 0%
<99.9% & >=99.5% 5%
<99.5% & >=99.0% 10%
<99.0% 30%
Virtual – Tier 3 availability
Monthly Uptime Service Credits
99.96% 0%
<99.96% & >=99.90% 5%
<99.90% & >=99.50% 10%
<99.50% 30%
9
You are solely responsible for providing accurate and current contact information. SaskTel will have satisfied its obligations under the outage reporting when your designated point of contact is updated.
10.0 ADDITIONAL SERVICES (ADD-ONS )
SaskTel offers a range of services that complement our VRGs on a pay-per-use model. Each of these optional services can be added or removed online at any time using your self-serve portal or through our Service Management Centre: Add-ons are classified based on their billing type as:
a. Daily Add-ons: Refers to services which when deployed, are charged on a daily basis; all the daily charges for an add-on are added up to create a single dollar amount to be billed. Billing for daily Add-ons will vary from one month to the next due to the number of days varying from one month to the next.
b. Monthly Add-ons: Refers to services which when deployed, are charged and billed on a monthly basis; billing for Monthly add-ons will not vary from one month to the next. SaskTel charges one month in advanced for monthly add-ons, this advanced charge is credited back to you upon cancellation of the monthly add-on in full or pro-rated in accordance to when the cancelation occurs.
Service DEFINITION Billing type
Additional vCPU The virtual CPU performs basic arithmetic, logic, controlling, and input/output (I/O) operations specified by the instructions.
Daily
Additional GB RAM
Random access memory, a type of computer memory that can be accessed randomly; that is, any byte of memory can be accessed without touching the preceding bytes.
Daily
Storage Tier 1
Highest enterprise-grade storage, with average sub millisecond response time, highest IOPS (>5K). Ideal for transactional data that requires fast, 100% accurate writes and reads either to support customers or meet the requirements of high-speed applications.
Daily
Storage Tier 2
High enterprise-grade storage, with average 2 millisecond response time, up to 5K IOPS. Ideal for major business applications from email to ERP, majority of active business data, where sub second response is not a requirement, but reasonably fast response still is needed.
Daily
Storage Tier 3
Medium enterprise-grade storage, with average 2 millisecond response time, up to 1K IOPS. Ideal for aging data still used for business activities such as trend analysis and complex decision support.
Daily
Storage Tier 4
Low enterprise-grade storage, with no latency or IOPS guarantees. Ideal for historical data. Data in this tier is generally idle or inactive, and seldom accessed.
Daily
Dedicated Firewall
Increase your network security by replacing the edge gateway included in your Cloud Hosting service with this high-performance network solution.
Daily
Shared Load Balancers
Optimize resource usage, maximize throughput, minimize response time, and avoid overload of any single resource by evenly splitting the workload among several devices. Load balancing virtual appliances are shared among several customers.
Daily
Dedicated Load Balancers
Optimize resource usage, maximize throughput, minimize response time, and avoid overload of any single resource by evenly splitting the workload among several devices. The load balancing virtual appliance is dedicated to a single customer.
10
Windows Remote Desktop Services
Remote Desktop Services allows you to remotely access and control virtual machines.
Daily
SQL Licenses Gain access to a relational database management system so you can easily store, managed, and access your data.
Daily
Office Pro Plus Productivity suite of applications which includes Microsoft: Access, Excel, InfoPath, OneNote, Outlook, PowerPoint, Publisher, Skype for Business, and Word.
Daily
Additional Microsoft Licenses
Fulfill your requirements for other Microsoft licenses and editions, including SharePoint, Dynamics, and more, from SaskTel. Daily
Application Monitoring and Alarming
Monitor key performance data from business-critical applications to ensure their continuous health. Receive alerts whenever a performance threshold has been crossed so you can proactively troubleshoot any issues before they become critical.
Daily
Tier 1 Management
SaskTel will monitor and alarm standard metrics from your servers, undertaking any action required to solve technical issues affecting them; We will also provide 24/7 incident and problem management for your servers.
Monthly
Server Patching SaskTel will perform monthly patching up to and including the operating system on customer’s behalf; mitigating vulnerability risks associated with software versioning
Monthly
Antivirus Protect virtual servers from virus and online threats. Monthly
Public IP Purchase additional public IPs to identify your computers and networks so they can communicate over the network
Monthly
Site-to-site VPN Allow multiple fixed locations to establish secure connections with each other or with a central location over a public network such as the internet.
Monthly
VPN Software Clients
Type of virtual private network that uses the Secure Sockets Layer protocol in standard web browsers to provide secure, remote-access VPN capability for remote users.
Monthly
Monthly Custom Reporting
We can provide with reports that are customized to your business’ specific environment.
Monthly
Monthly Operational Meetings
We will meet with you to review business-critical topics such as issues, processes, and reporting. Monthly
11.0 NETWORK
Internet access
A shared Internet access is included with the service; there is a 1 Gbps link provided and split for all customers in the service. We monitor the access to ensure that the capacity is maintained, however, no bandwidth is dedicated to individual customers.
If your applications require additional security and privacy, faster speeds, dedicated bandwidth, or quality of, services such as VPLS, LANspan IP and Dedicated Internet are available at preferential rates for Cloud Hosting customers.
12.0 CHANGE MANAGEMENT
We follow the Information Technology Infrastructure Library (ITIL) framework of best practices, including our ITIL disciplines of Change Management, Incident Management, Problem Management, Capacity Management, and Service Level Management. We closely adhere to the following ITIL practices and processes:
Change Management: All changes to a customer’s network infrastructure can be controlled through a formal change management process that will be aligned with your change management processes. Our primary change management objective is to implement changes with minimal disruption to IT services. Change management ensures that changes are deployed in a controlled way. Changes are evaluated, prioritized, planned, tested, implemented, and documented.
11
Incident and Problem Management: We focus on the primary objectives of Problem Management, which are to prevent problems from happening, to eliminate recurring problems, and to minimize the impact of problems that cannot be prevented. Activities include diagnosing the root cause of the problem, determining an appropriate resolution, and implementing the resolution following the established change management process.
Configuration and Asset Management: We manage all components of our core network. We provide centralized and controlled change management procedures for device and software configurations and changes to devices, software, or networks as required, based on service scope and service levels. Service assets and configuration information are managed in order to support the other service management processes.
Release Management: We manage all network infrastructure releases as a part of the overriding change management process as defined by the service level agreements.
Business Continuity: Industry standards and best practices in regards to backup and recovery services for managed networks are followed. This will ensure network availability, as defined by service level agreements. We back up device and network configurations, as defined by the service level agreements, to be used for service restoration, data corruption recovery, or implementation of IT service continuity plans.
Service Desk: 24x7x365 technical diagnostics and troubleshooting of the network infrastructure is provided as required, based on service scope and service levels.
13.0 REQUEST MANAGEMENT PROCESS
Customer requests are to be managed through the SaskTel’s Online Request Process depicted in this section. Requests will be responded to as follows:
• Requests made during regular business hours (Monday to Friday, 8:00 a.m.- 4:30 p.m. CST – daylight savings not observed, SaskTel-observed holidays excluded) will be responded to within 8 business hours. NOTE: Responded does not mean actioned and completed.
• Requests made outside of regular business hours will be responded to the next business day. NOTE: Depending on the request initiated, time and charges may apply.
Online Request Process via self-serve portal – Customer process
12
Online Request Process via Service Management Centre – Customer process The following steps outline the process you must follow to submit a change request through SaskTel’s Service Management Centre (SMC). With SM Centre you have the ability to track your change requests, add additional information if required, and be updated directly from SaskTel.
1. Log in to mySaskTel.com profile account. 2. Select Service Management Centre. 3. In the Service Management Centre portal, click on Service Catalogue. 4. In the dropdown menu, select Changes to Service. 5. In the new section, click on the drop-down arrow and select one of the changes applicable to the
service. 6. Complete the applicable fields based on the type of Change Request selected. 7. Click Submit Request.
NOTE: You can view the request at any time to check on the status by clicking on View Requests at the top of the page. You will receive notifications via Service Management Centre and by email when the request is Resolved and Closed.
For more information, please review your SMC User Guide.
SaskTel’s Request Management Process This section depicts SaskTel’s process for customers’ request, whether additional service requests or requests to change customer’s Cloud Hosting service: It depicts such process at a high-level in flowchart diagram and at a low-level in table form with an in-depth step-by-step breakdown of activities and the teams involved in completing such activities. To report troubles, please follow the Incident Management process outlined in Section 12.
13
Request Management Process – Flowchart Diagram
Request Management Process – Table
ID Predecessors Step Description Roles
P.1 - Need to Change Service
Customer needs to change the service or CI (a specific server)
Customer
P.2 P.1 Login to Mysasktel.com
Customer logs in to mysasktel.com to access SMC
Customer
1.0 P.2 Submit Service Request for change in service
Customer to submit service request for change of in-scope service
Customer
1.1 P.2 Submit Service Request for change in service
Business sales to submit service request for change of in-scope service on-behalf of customer
Business sales
14
2.1 1.0/1.1 Review Service Requests
Task- Verify service request and ensure all information is complete. Set variable as chargeable if applicable. Set variable for “Contract Update” to initiate a task for ITSM.
SDM
2.2 2.1 Determine cost of service
Task- If variable is chargeable, determine cost of service and consult marketing as required.
SDM
2.3 2.2 Inform customer and get email approval
Task- Communicate customer the cost for requested service to get approval through email.
SDM
2.3.1 2.3 Receive charges for service
Receive charges for requested service through email
Customer
2.3.2 2.3.1 Approve charges for service
Approve/Reject quoted price through email Customer
2.4 2.3.2 Receive response from customer
Task- Receive response from customer and acknowledge. Set the variable to Accepted or Rejected as applicable.
SDM
2.5 2.4 Engage business sales
If variable is set to not accepted, Task- Informs business sales to work with the customers
ITCIT
2.6 2.5 Engage business sales
Receives notification from ITCIT that customer has declined the price. Involves customer prime from business sales to work with customer.
Business Sales (Sales Prime)
2.6.1 2.6 Work with customer
Works with the customer as required to agree on charge of requested service. Renegotiate as required.
Business Sales (Sales Prime)
2.6.2 2.6.1 Customer acceptance
If customer accepts the cost, then ITCIT will update a variable on their task to initiate next step. If customer declined, then Service request should be closed.
Customer
2.7 2.6 Forward Fulfilment Request
Task- Forward the service request for fulfilment. Assign the next task to appropriate workgroup.
SDM
3.1 2.7 Receive Service Requests to be fulfilled
Task- Receives service requests to be fulfilled
a) Initiate Standard change (as
required) and link with service
request
b) Initiate Normal change (as
required) and link with service
request
ITSM
3.2 3.1 Notify SDM Notify SDM regarding the completion of task
ITSM
15
4.1 2.7 Receive Service Requests to be fulfilment
Task- Receives service requests to be fulfilled
a) Initiate Standard change (as
required) and link with service
request
b) Initiate Normal change (as
required) and link with service
request
BSO/DCT
4.2 4.1 Notify SDMT Notify SDM regarding the completion of task
BSO/DCT
4.3 3.02, 4.02 Update Service Contract
Task- Update service contract if any new CI’s or User need to add on existing contract
ITSM
5.1 3.4,4.4 Verification Task- Receive notification from operation team regarding completion of task. Complete Variables: Customer Name (Reference to the Company Table), Customer Account Number (Reference to CAN table), Billing details. Complete required fields related to billing information and close task. Indicate BRM or CRB in the billing details.
SDM
5.2 5.1 Notify for billing Notify ITCIT for billing SDM
5.3 5.2 Receives requests for completion of billing
Task – ITCIT receives requests for
completion of billing and does the needful
to raise the CRB billing based on
information provided by SDM. If BRM,
ensure recurring billing is setup in the
system under customer account.
ITCIT
5.4(End) 5.3 Complete Request
Once all workflow tasks are marked as complete, the system will complete the request which will notify the originator of the request.
System
14.0 INCIDENT MANAGEMENT PROCESS
The objective of Incident Management is to restore normal service operations as quickly as possible and minimize any adverse impact on business operations, ensuring that the levels of service and availability are maintained. The purpose is to ensure timely resolution of all incidents, and to allow for rapid escalation, when required, with the proper communication to management. NOTE: This is a pay-per-use service unless the Tier 1 Management add-on has been hired for your servers Major Incidents Incident Management for major incidents and the escalation of such are the responsibilities of the SaskTel Business First Support Team (BFST). SaskTel BFST is accountable for all incidents and for Data Centre services until resolution. SaskTel BFST operates on a 24/7 basis.
16
Trouble resolution process a) Contact the SaskTel Business First Support Team (BFST) at 1-844-SASKTEL (727-5835) to
report a trouble. You must call into the BFST before the trouble resolution process can begin. All calls will be answered 24/7 at this contact number.
b) BFST will create a trouble ticket and dispatch to the appropriate responder group. Minor Incidents Incident Management for minor incidents and the escalation of such is initiated via the Service Management Centre. Trouble resolution process
1. Log in to mySaskTel.com profile account. 2. Select Service Management Centre. 3. In the Service Management Centre portal, click Report an Incident under the Support menu. 4. Fill all the necessary information and click submit.
For a more detailed description of this process, see your SMC User Guide.
In reporting incidents, impairments are categorized by priority according to the table below to ensure timely response based on severity. NOTE: Response time is the time period the trouble is reported to SaskTel and a trouble ticket is created in our system. Resolution time is the time period SaskTel has a technician actively investigating the trouble. Time to repair will be dependent on the nature of the problem. For Severity 1 problems, work will be performed continuously until the point of resolution of the problem.
PROBLEM SEVERITY
DEFINITION MAXIMUM RESPONSE TIME
MAXIMUM RESOLUTION TIME
UPDATES TO CUSTOMER
Severity 1
Major functionality loss to the suite of services, as defined in the contract, and has a severe impact upon multiple businesses, or a total loss of service for 1 or more businesses.
Thirty (30) minutes response
Four (4) hour resolution
1 hour
Severity 2
Narrow functional limitations that may make use of the service difficult, although do not severely impair the customer’s business activities on the application. While these can adversely affect business performance, a temporary workaround may be available.
Four (4) hour response
Twelve (12) hour resolution. Except weekends and holidays.
4 hours
Severity 3
Specific functions that do not negatively impact daily operations. These would be things that are infrequent in occurrence, have an easy workaround or can be avoided by the end user.
Twenty-four (24) hour response
Forty-eight (48) hour resolution. Except on weekends and holidays.
Best efforts
Severity 4
Issues that do not impact the customer’s business operations, e.g. how-to’s, informational questions, feature requests, that are the responsibility of the customer administrator. (These are subject to SaskTel Professional Service charges.)
Twenty-four (24) hour response
One-hundred-twenty (120) hour resolution. Except on weekends and holidays.
Best efforts
The table below illustrates the current trouble escalation processes:
TIME ELAPSED
ACTION TAKEN
0 minutes Problem discovered by SaskTel or reported to the SaskTel BFST. If the trouble is reported by a customer, you will enter your assigned four-digit PIN and select the appropriate IVR option.
17
30 minutes Escalation to SaskTel Tier II Technician.
1 hour Status updates every hour given to customer contact.
2 hours Escalation to SaskTel internal Tier III support.
3 hours Escalation to BFST manager and support managers (if required).
4 hours Escalation to Director – Customer Services Network Operations.
5 hours Status updates every hour to Director.
6 hours Escalation to all Managers involved. Mandatory involvement with all support groups and equipment vendors/suppliers.
10 hours Status updates must be communicated to all managers every 3 hours.
18 hours Executive decisions. If other resources needed (Corporate Affairs).
15.0 DISASTER PREPAREDNESS
SaskTel Disaster Preparedness plans are guidelines on how a company will react to emergencies and disasters. These plans are not to be confused with Disaster Recovery Services. SaskTel Cloud Hosting Services do not include any Disaster Recovery services. We highly recommend you acquire additional Disaster Recovery services, such as SaskTel’s Cloud DR, to protect your production environment in the event is brought offline without any real possibility of recovery of such. To learn more about SaskTel Cloud Disaster Recovery services, contact your SaskTel Representative. In the event of a disaster, we are committed to maintaining service continuity to the greatest possible extent, with the objective of protecting property, plant, service capability, and resources. An Emergency Operations Centre is maintained by SaskTel, which can be activated to take control of both internal and external situations. Corporate Administrative Procedures that relate to disaster recovery plans and procedures have been developed. These procedures are backed by our on-call certified vendors and partners, and are designed to facilitate service continuity and operations compatibility. In a disaster, the SaskTel Regina Systems Operation Centre (SOC) staff can occupy our Remote Operations Centre (ROC) located on Albert Street South in Regina. There are designated workstations and network connectivity that allow the SOC and other departments to work remotely and continue to monitor data centre systems. In the event that the buildings housing the primary and/or secondary Regina data centers become unavailable, all calls and monitoring can immediately be diverted to our Saskatoon Data Centre.
16.0 MONITORING AND ALARMING
The goal of SaskTel’s Monitoring and Alarming service is to ensure continuous operation and health of customers’ servers by acting proactively to solve potential issues before they become critical.
Monitoring and alarming for SaskTel Cloud Hosting customers is provided via Monitoring Centre for both virtual and physical servers. SaskTel monitors your servers’ CPU, Memory, File System usage; and services under SaskTel responsibility (e.g. antivirus, Splunk, VMware tools, etc.), against a set of thresholds. Whenever any of these thresholds is reached (event) a ticket is autogenerated for the incident:
• Server Availability – Polls every 5 minutes and alerts on 2 failed polls
• CPU Critical Alarm 90% for 30 Minutes (Incident Generated)
• CPU Minor Alarm 80% for 15 minutes (Viewable alarm, No Incident)
18
• Memory Critical Alarm 90% for 30 Minutes (Incident Generated)
• Memory Minor Alarm 80% for 15 minutes (Viewable alarm, No Incident)
• File System Usage Major for 91% (Incident Generated)
• File System Usage Critical for 95% (Incident Generated)
• Monitored Service alert or first not running detection (Incident Generated)
Our Monitoring Centre is integrated with our Service Management (SM) Centre, any event created in Monitoring Centre is pushed to the SM Centre. Whenever an incident is cleared in Monitoring Centre, it is updated in the SM Centre Incident section and the state is changed to awaiting verification until it is reviewed and resolved.
17.0 BILLING
Your Cloud Hosting services will appear in the ‘Special SaskTel Services’ section of your monthly bill, under the ‘Managed Services’ legend.
a. This is the Customer Account Number (CAN) the service is being billed to, the 5 digits after the
dash represent the Child Account, used to differentiate between different SaskTel Managed Services; for example, if you were also a Cloud DR customer, these [CDR] services would immediately follow the Cloud Hosting services in your bill, with a different Child Account.
b. This represents the charge for your vDC; it includes your contracted VRG, the storage your vDC is consuming, and any additional vCPUs and GBs RAM you have hired above the base resources of your VRG.
c. These are all the additional pay-per-use services you have hired, both daily and monthly services are included in here.
d. This represents one-time charges or credits for circumstances such as billing mistakes, back bills, etc.
There are additional codes that might appear from time to time on your bill for concepts such as:
• Cloud Hosting Professional Services: This represents any one-time charges from requests made by you where manual intervention from SaskTel was required; for example, setting a VPN in your self-serve portal, or deploying a VM on your behalf.
• Cloud Hosting Missed SLA Credit: This represents any SLA Credits provided to you in compensation for not achieving our availability targets.
a b
c
d
19
18.0 CERTIFICATIONS AND AUDITS
CSAE3416 • CSAE3416 SOC 1 Type 2 audits are conducted annually, through a contracted 3rd party
provider, on internal controls in the Cloud Hosting environment. Additional certifications or audit requirements can be requested as needed.
• The results of our CSAE3416 audit are available auditor to auditor a request can be made
to [email protected]. • Annual CICA Section 5970 Period of Time audits (CSAE 3416 Period of Time Audits in
2015) are performed by an independent 3rd party auditor on our data centres. This audit is conducted in accordance with the standards established by the Canadian Institute of Chartered Accountants (CICA) for audits of controls at a service organization.
• The audit covers areas such as organizational, logical, and physical security. Procedures and operations are audited against CICA standards, and tests are applied to specific controls. The auditor’s opinion is provided in the annual audit report on the data centre operating environment. The formal audit report provides information on the design, implementation, and effectiveness of controls.
• SaskTel will review and take action regarding recommendations from the audit in an effort to follow industry best practices and meet our objective of continuous process improvement.
• Past audits have repeatedly demonstrated our commitment to operational service and security controls.
19.0 SECURITY
SaskTel Data Centre The Data Centre hosting SaskTel Cloud Hosting offers a safe and secure operating environment for servers and software applications. The Data Centre is a building within a building; a dedicated, multi-floor, site-hardened and windowless structure with a fire rating of 1.5 hours for walls and doors. Mechanical rooms, electronic control units, and Uninterruptible Power Supply (UPS) units are separate from the main server rooms to enhance the safety of your data. The facility features a two-stage fire suppression system with cross-zoned detection that responds to either heat or smoke. In addition, a separate fire detection mechanism constantly samples the air and acts in tandem with detectors mounted on the ceiling Security and monitoring Security is of prime importance in the establishment of a Data Centre. Access to the Data Centre is governed by our general security policies. Access to the building is controlled 24/7, 365 days per year. Physical entry is electronically secured, physically monitored, and staffed by bonded security personnel on a 24/7 basis. The Data Centre is protected by a number of additional measures including:
a) The outer perimeter and corridors of the facility are monitored with closed-circuit cameras and motion detectors to ensure that no unauthorized individuals gain access to the facility.
b) To control access to the Data Centre itself, biometric scanners are used in conjunction with key-card access.
c) Inside, highly trained and experienced technical staff monitor and maintain all systems for performance and availability on a 24/7 basis.
d) For equipment behind a shared firewall, redundant Ethernet connectivity is provided with high-availability load balanced switches, firewall and intrusion detection systems, and multiple connections to core Internet providers.
e) System vulnerability assessments are conducted on a regular basis to ensure maximum protection of all data centre infrastructure and system.
f) To guarantee performance, the Data Centre resides in a low risk setting. Natural hazards such as floods, earthquakes, and tornadoes are rare, and the risk of human threats such as chemical spills or criminal activity is minimal. In case of emergency, police and fire services are readily available.
20
g) Behind our outside firewall, the customer will be able to have edge servers and backend servers segregated through another internal firewall. This will ensure that backend servers and sensitive data are not at risk should an edge server be compromised.
Corporate security Provides Cloud Hosting with a secure hosting environment including restricted physical access controls with 24/7/365 on-site security guard, building access with security kiosk, CCTV cameras, data centre biometric, and card swipe at the entrance. In addition, Corporate Security can be engaged to provide guidance for compliancy checks, governance, security consulting, and business cases for special projects. Information security
a) We have developed a corporate Information Security Policy and an Acceptable Use of Technology Policy. The Information Security Policy establishes roles and responsibilities for information security and defines standards for data security and classification, asset management, personnel security, administrative safeguards, technical safeguards, physical and environmental safeguards, legal compliance, and acceptable use. Employees are required to certify annually that they have read the Acceptable Use of Technology Policy. Additionally, we have established various committees that are responsible for defining information security standards.
b) We have placed into operation an Incident Response Program that is followed to escalate incidents such as security and disaster recovery issues, among others, to the appropriate personnel. The program includes steps for incident identification, event routing and prioritization, incident escalation, containment, resolution, recovery, follow-up, and communication. Incidents are evaluated for trending, and an overview is communicated to the Information Security Steering Committee, which includes our corporate executive management as well as senior business and IT executives.
c) System vulnerability assessments are conducted on a regular basis to ensure maximum protection of all data centre infrastructure and systems.
d) We record all IT Assets in a configuration management database. Hardware is configured and installed in accordance with documented standards. We maintain and enforce security standards for the operating systems.
20.0 ENVIRONMENTAL CONTROLS
Data Centre environment Temperature control ASHRAE Environmental Guidelines in the data centre are followed. The air-conditioned air within the cold aisles or data centre will be measured by the building monitoring systems providing optimal operating conditions for data equipment:
• Temperature will be maintained between 18 to 27 degrees Celsius with a target temperature of 22°C (72°F) with allowable ranges of 15 to 17.9 and 27.1 to 32 degrees Celsius
Humidity control The target range for humidity within the data centre follows ASHRAE Environmental Guidelines of 20% to 65%.
Data Centre power system The Data Centre is connected to the public power grid and operates with three diesel-powered backup generators that can accept load in one minute and are capable of operating for 5 days before being refueled. We also have a contracted supply arrangement for diesel fuel, so the generators can operate indefinitely.
21
Power distribution within the data centre server area is achieved using two power distribution units. Redundant, parallel, and synchronized UPS systems provide a steady and stable power supply to the data centre infrastructure and servers. The UPS support systems provide protection from power surges before they reach any of the data centre equipment.
21.0 MAINTENANCE
Maintenance window: We reserve a six (6) hour maintenance window from 12:00 a.m. – 06:00 a.m. CST to perform maintenance daily, unless otherwise noted within the contract of another SaskTel service with you where interdependencies between the services exist. Maintenance notification: We will inform you of any planned maintenance scheduled in the maintenance window that has the potential to impact you seven (7) business days prior to the maintenance. NOTE: This will exclude any emergency maintenance. Maintenance outside of maintenance windows: We reserve the sole right to schedule maintenance outside of normal maintenance windows. We will inform you of any planned maintenance, scheduled outside the maintenance window, that has potential to impact your business. If maintenance is scheduled to occur outside regular maintenance hours, seven (7) business days’ notice will be provided. You can object at least four (4) business days prior to the scheduled maintenance to allow us an assessment period for alternative times. If an objection cannot be resolved, we will negotiate with you on a mutually agreed upon time and date. Emergency maintenance: From time to time, emergency maintenance may be required, where there is insufficient time to schedule this activity due to the urgent nature of the situation. There may be circumstances where it is not possible or is impractical to notify customers of this type of activity. We reserve the right to perform this maintenance at its sole discretion to ensure the integrity of our facilities and environment. When emergency maintenance is required due to circumstances outside the reasonable control of SaskTel, there will be no service credits available to you. If emergency maintenance was required due to circumstances which could have been reasonably anticipated, then service credits may be available. Maintenance activities performed by SaskTel include, but are not limited to: UPS maintenance Monthly check:
• Pilot cell voltage with and without temperature
• Compensation string voltage
• Specific gravity on the pilot cells Six-month routines:
• Voltage and specific gravity on all cells in the string
• Midtronic water levels
• Strap and post condition
• Float current test on each string
• AC ripple test of rectifiers and power plant room Monthly cleaning:
All UPS batteries, including a visual inspection for cracks or deformities and water top up if required
Generator maintenance
22
Regular maintenance or repair of gensets will be performed by the genset manufacturer’s local outlet. Fuel systems, pumps, filters, and fuel test sampling will be done by SaskTel Power Techs. Monthly runs:
• A minimum of 2 hours to test the building load, verify engine run reliability Every three months:
• An electrical utility failure simulation.
Fire extinguishers Building Technician will inspect the FE's monthly SPI Safety and a complete annual inspection, and a Hydro inspection every 5 years.
Fire suppression Fire panel and associated controls are inspected on annual basis by an outside fire protection service provider. Sprinkler systems and fire pump will be inspected on an annual basis by an outside fire protection service provider. SaskTel maintenance staff conduct weekly inspections of the buildings pre-action stations as well as conducting monthly fire pump run tests.
A/C maintenance Daily visual inspection. Every three months:
• The belts and general operation inspection including cleaning of the humidification system and filters as required
Two times a year:
• A full inspection for refrigerant levels and any other repairs as required while the redundant cooling system will have no impact on the data hall performance