1 55

CJSC CASPIAN PIPELINE CONSORTIUM

AGREED APPROVE

CPC Security department CPC Telecommunication head

_______________________________________ K.I. Savchenko

"_____"___________ 2016 "_____"_______________ 2016

TECHNOLOGY DEPARTMENTTELECOMMUNICATION GROUP

TERMS OF REFERENCE No 01-16/02

To design multiservice information network to ensure security in CPC oil pipeline system

2016

2 55

CONTENTS

1. ABBREVIATION LIST2. GENERAL3. REGULATIONS AND NORMS 4. DEFINITIONS5. OVERVIEW AND BRIEF WORK DESCRIPTION 6. DWDM WAVE MULTIPLEXING 7. CORE LAYER OF ISPLS NETWORK 8. ISPLS NETWORK DISTRIBUTION LEVEL 9. ISPLS NETWORK ACCESS LEVEL10.SECURITY AND NETWORK EQUIPMENT CONTROL 11.QUALITY OF QOS SERVICE 12.SYSTEM TO SYNCHRONIZE ISPLS NETWORK TIME PROTOCOL USING

NETWORK TIME (NTP) PROTOCOL 13.PROTECTING DTN CONTROL AND MANAGEMENT 14.ROUTING PROTOCOL SECURITY15.REQUIREMENT TO AAA 16.REQUIREMENTS TO NETWORK CONTROL SYSTEM 17.REQUIREMENTS TO HARDWARE AND VIRTUALIZATION SYSTEM 18.REQUIREMENTS TO QUALIFICATION AND WORK MANAGEMENT

3 55

1. ABBREVIATION LIST

AS Autonomous System

BGP Border Gateway Protocol

BPDU STP Protocol Frame

CDP Cisco Discovery Protocol

CE Customer Edge

COPP Control Plane Policing

COS Class of service

DSCP Differentiated Services Code Point

EoMPLS Ethernet over MPLS

Ethernet Data Transmission Packet Technology

HSRP Hot Standby Router Protocol

IGP Interior Gateway Protocol

IP Internet Protocol

L2 Layer 2

L3 Layer 3

LACP Link Aggregation Control Protocol

LAN Local Area Network

LDP Label Discovery Protocol

LFIB Label Forwarding Information Base

LSP Label-switched path

MAC Media Access Control

MD5 Message Digest 5

MEC Multichassis Ether-Channel

MP-BGP Multi-protocol Border Gateway Protocol

MPLS MultiProtocol Label Switching

MT Marine Terminal

MTU Maximum Transmission Unit

NSF Non-stop Forwarding

NTP Network Time Protocol

OSI Open Systems Interconnection

4 55

OSPF Open Shortest Path First

PAGP Port Aggregation Protocol

PE Provider Edge

PFC Policy Feature Card

PVLAN Private VLAN

QoS Quality of Service

RD Route Distinguisher

RED Random Early Detection

REP Resilient Ethernet Protocol

RFC Request for Comments

RSTP Rapid Spanning-Tree Protocol

SDH Synchronous Digital Hierarchy

SF Shore Facility

SNMP Simple Network Management Protocol

SPF Algorithm Shortest Path First

STP Spanning-Tree Protocol

SVI Switched Virtual Interface

TCP Transmission Control Protocol

TF Tank Farm

UDP User Datagram Protocol

USB Universal Serial Bus

VLAN Virtual Local Area Network

VPN Virtual Private Network

VRF Virtual Routing and Forwarding

WAN Wide Area Network

AS Autonomous System

OMB Outband Management Block

FPS Future Pump Station

MCC Main Control Center

CPC Caspian Pipeline Consortium

MLBV Main Line Block Valve

5 55

MT Marine Terminal

PS Pump Station

TF Tank Farm

BCC Back up Control Center

ISPLS Integrated System of Pipeline Security

OS Operating System

CI Computation Infrastructure

VSS Virtualization Subsystem

2. GENERAL

2.1 The Terms of Reference herein set out the requirements to design CPC secure multiservice information network specifying the composition and scope of design documentation.2.2 The telecommunications equipment used in CPC security network is currently out of production. CPC is planning a step by step network upgrading and replacing failed components on the basis of developed project design to provide for continuity of operations in security systems.

3. REGULATIONS AND NORMS

The documents listed are an inherent part of technical requirements herein. Except for the changes introduced, the requirements provided in this document suggest using the latest editions of each regulation with addenda in effect on the date of execution of the contract. Work programs covered by the technical requirements herein should be conducted meeting the requirements set forth in applicable provisions of referred documents published (or referred to equivalent in Russian or Kazakhstan standards).

3.1 Electronic Industries Association.TIA/EIA-TSB67 Technical specifications for the data transfer channels under operational testing cables made of non-shielded twisted-pair wires.TIA/EIA-TSB72 Guiding instructions on developing centralized optic fiber cable networks.TIA/EIA-568-A TIA/EIA-568-B Standard on telecommunication cable lines to be developed inside industrial buildings.TIA/EIA-569-A TIA/EIA-569-B Standard on routs and areas to develop telecommunication lines inside industrial buildings.EIA/TIA570 Standard on telecommunication wire lines to be developed in communal areas and zones having industrial development.TIA/EIA-606 Standard to set up a telecommunication infrastructure and data transfer in industrial buildings.TIA/EIA-607 Requirements to earthing and connection of equipment casing inside industrial buildings.TIA/EIA-TSB75 Methods to lay additional cables horizontally inside working areas having open space design.

3.2 Institute of Electrical and Electronics Engineers.

6 55

IEEE 802 Local and metropolitan area networks - General provisions and architecture.IEEE 802.7 Recommended methods to develop broadband local networks.IEEE 802.8 Engineering consulting group on fiber optics.

3.3 ETL testing laboratories.Package of documents to design cable telecommunication systems to be submitted to ETLISO/I EC 11801 14763-1 14763-2 TIA/EIA568A 569A 607 Standard to lay cables inside buildings.CENELEC EN-50173 European technical requirements to cables. Recommendation IEC 61754. Fiber Optic Connector Interfaces, IEC 61755-1. Fiber optic connector optical interfaces.Recommendation IPC -8497-01 (Optoelectronic Assembly and Packaging Technology) Cleaning Methods and Contamination Assessment for Optical Assembly.

3.4 GOST, RD, CPC policies. The documentation pertinent to the project design work to develop SCS and the work itself is carried out in line with requirements set out in GOST series 34 and GOST series 19.GOST 12.1.030-81 SSBT. Electric safety. Protective earthing, and neutralling.GOST 464-79 Grounding for the stationary installations of wire telecommunications, repeating stations, wire broadcasting centers and community antenna television systems. Resistance norms.GOST R 50571.2-94 Indoor electric installations. Part 3. Major parameters.GOST R 50571.3-94 Indoor electric installations. Part 4. Safety requirements. Electric shock protection systems.GOST R 50571.10-96 Indoor electric installations. Part 5. Selection and assembly electric equipment. Part 54. Grounding electrode systems and grounding conductors.GOST 21.110-95 Rules to develop equipment, item and material specifications.OST 45.88-96 Departmental standardization system. Sequence to develop departmental guiding documents.OSTN-600-93 Departmental development of engineering norms on structure assembly and setting up telecommunication, radio broadcasting and television broadcasting means.VSN-332-93 Guidelines on designing electric installation in plants and structures to be used in electric communications, wire telecommunications, radio broadcasting and television broadcasting.VSN 600-IV-87 Safety rules to follow in assembly of process equipment and electric supply.VSN 332-93 Guidelines on designing electric installations in plants and structures to be used in electric communications, wire telecommunications, radio broadcasting and television broadcasting.Text part of the documentation should be developed in line with requirements set forth in RD 50-34.698.90 Information Technology. Methodological guidelines. Set of standards applied to automation systems. Automated systems. Requirements to document composition. ВRD CPC 107.03.2015 Specifications to structured cable network systems.RD 34.21.122-87 Guidelines on setting up lightning protection in buildings and structuresRD 45.091.195-90 Guidelines on designing electric telecommunication complexes. General requirements and norms on equipment, cable and metal structure earthing.RD CPC Guidelines on operating CPC fiber-optic line.RD 45.190-2001 Departmental guiding document Element section of fiber-optic line to transfer data Standard user acceptance test program.RD 45.155-2000 Departmental guiding document Grounding and potential bonding in fiber-optic line equipment installed at wire telecommunications facilities

7 55

Rules to apply fiber-optic cables, passive optics devices and gear to fuse-weld optics fibers. Instruction of the Ministry of Information and Telecommunications No 46 dated April 19, 2006Standards NIST, FSTEC

8 55

4.0 DEFINITIONS

Customer - CJSC Caspian Pipeline Consortium-R.Contractor - a company selected to undertake project designing.ISPLS network - integrated system of CPC secure pipeline network.

5.0 OVERVIEW AND BRIEF WORK DESCRIPTION

5.1 Project designing and consulting should be done by a company having license and/or other legally established permitting documents to engage into this activity, having work record no shorter than 3 years in this specific area.5.1.1 The Contractor should make an obligatory front end engineering survey involving the visit to the job site of CPC MCC in Novorossiisk through getting permits to access the customs area and preliminary issuing permits, passes and undergoing safety orientations in compliance with norms in effect at the moment of making plans to visit CPC facilities.5.1.2 The scope of front end engineering survey should be sufficient to obtain all required and sufficient information to ensure high quality of design work, to clarify all issues dealing with project implementation. 5.1.3 Based on results of the front end engineering survey a certificate should be executed on making survey of facilities to be designed bearing the signature of a responsible engineer from CPC telecommunications group outlining the information gathered.5.1.4 If it is required to conduct additional research in the course of design work these activities will be carried out by the staff of the project design organization without escalating the contract value.5.2 During front end engineering survey Contractor should become familiar (without

limitation) with the following list of materials and documents: Topological architectural ISPLS network layout; ISPLS network address table; Composition of active network equipment at the network core layer; Composition of active network equipment at the distribution level; Composition of active network equipment at the access level; Composition of server equipment in DVM video surveillance systems; Composition of IQ system to access control server equipment; Composition of EBI server equipment ; ISPLS systems interaction logic; Composition and logic of voice telephone equipment; Composition and logic of ISPLS network monitoring systems and planned expansion; Configuration of all network equipment; Development of existing equipment configurations that are planned to connect to

MCC virtualization storage systems; Results of executing verification commands; MDS for the fiber-optic cable line; Parameters of virtual machines, composition, and available services;

5.3 Requirements to the work scope. 5.3.1 Contractor undertakes the following scope of work on designing and developing process solutions, including (but not limited to) the following:

Front end survey of ISPLS network to be made at the "Customer’s" facilities; Detailed analysis of Marine Terminal network configuration (kp 1504) - PS 8 - PS 7

(kp 1353 – PS Kropotkin (kp 1237) – PS 5 (kp 1138);

9 55

Issuing recommendations on configuration of network segments as a separate document;

Provides the functional specification and serves an explanatory note describing design solutions and underlying analysis;

Provides network architectures of PS to be debottlenecked and line pipe shelters indicating the physical circuit to connect equipment;

Provides functional diagrams indicating the logical network interactions; Develops a migration plan or the program to replace off-spec components with new

equipment specifying milestones and risks; Develops a template configuration of network devices consistent with proposed

design; Carries out standard procedures and methods of pilot tests, factory tests, and

acceptance tests; Makes calculations and provides estimates on power consumption of proposed

equipment; Conducts network analysis accounting for convergence of routing and signaling

protocols; Develops and informs the Customer of the safe rules to handle active equipment

proposed; Compiles the new address plan for the network upgraded; Makes calculations of the budget of the optics telecommunication lines to connect

new equipment; Offers options to replace the equipment on the basis set forth in draft proposal

followed by generating cost estimates on equipment to be installed Develops a program to train three Customer telecommunication engineers in a

training center certified by the manufacturing company Makes the feasibility study of design solutions

5.3.2 Taking the requirements to operate, maintain, repair and store the equipment as set out in the documentation of the manufacturer and in guiding CPC documents as a base, Contractor develops the following (without limitation):

Guidelines for system user; Guidelines for system administrator; Procedures on regular system maintenance; Procedures on emergency system maintenance; Guidelines on subsystem installation; Guidelines for back up copying and data recovery.

10 55

6.0 DWDM WAVE MULTIPLEXING

6.1 Information on existing backbone telecommunication network

The existing backbone optics telecommunication network is developed in section 1504 - 204 km using Siemens A-DF (ZN) 6x6E9/125 0.36F3.5 + 0.21H18 LG cable.The fiber optics cable has a capacity of 36 ОВ.The manufacturing company is Siemens.A schematic diagram of cable cross-section is given in Fig. 1.

Figure No 1

The main parameters and features of design Optics twisting consists of 6 elements having a central cable member (CCM) of

diameter 2 mm. Attenuation index, no greater than :

0.2 dB/km for a wavelength of 1.55 µm, 0.4 dB/km for a wavelength of 1.31 µm.

Tensile strength: 3.5 kN. Temperature range: -40 С…+50 С. Fiber manufactured by company Corning SMF-20e, complying with standard

G.652.D. Weight, no greater than: 137 kg/km.

Outside diameter: 13.3 mm. Number of optics fibers: 36.

The parameters of a single optics fiber are given in tables below:Table No 6.1

Optics guide parameters Single mode cable without dispersion shiftCore diameter 8.3 µmOptical fiber cladding diameter 125.0 2.0 µmNon-roundness of optical fiber cladding section

1.0%

Diameter of color optical fiber 250 µmCladding diameter 245 10 µmConcentricity of core and optical cladding 0.8 µmBeam diameter to channelize mode 9.20 0.50 µm at a wavelength of 1310 nm

11 55

10.50 1.00 µm at a wavelength of 1550 nm

Minimum yield strength 100 000 f-F/sq.inchEvenness of attenuation No more than 0.10 dB at 1310 nm or 1550

nm Maximum dispersion 3.2 ps/nm-km over the range from 1285 to

1330 nm< 18 ps/nm-km at 1550 nm

Cutoff optic fiber wavelength < 1260 nmZero dispersion wavelength (λ) 1301.5 nm < λ < 1321.5 nmZero dispersion gradient < 0.092 ps/nm2-kmPolarized mode dispersion < 0.5 ps/km1/2

Table No 6.2

Parameter Brand OVSMF-28e®

Operational wavelength, nm 1260…1625Attenuation factor, dB/km, no more than:

at a wavelength of 1310 nm 0.34at a wavelength of 1383 nm 0.31at a wavelength of 1550 nm 0.20at a wavelength of 1625 nm 0.22Chromatic dispersion factor, ps/nm•km:within the wavelength band of (1285-

1330) nm≤ 3.5

within the wavelength band of (1530-1565) nm

≤ 18

within the wavelength band of (1565-1625) nm

≤ 22

Zero dispersion point, nm 1302…1322Slope of dispersion curve in zero dispersion wavelength, ps/nm2•km, no greater than:

within the wavelength band of (1285-1330) nm

0.089

Polarized mode dispersion, ps/µm, no more than:individual fiber 0.2line (20 connected fibers) 0.06

Cable cutoff wavelength, nm, no more than 1260Mode field diameter, µm

at a wavelength of 1310 nm ±0.4at a wavelength of 1550 nm 10.4±0.5

Glass geometryinherent fiber curvature, m ≥ 4.0reflecting jacket diameter, µm 125.0 ± 0.7core and jacket eccentricity, µm ≤ 0.5jacket out-of-roundness, % ≤ 0.7

CPC is using 36 fiber single mode cable P/N BC01200721 at section 0-204 km as a backbone fiber optics cable; the cable is manufactured by company Teldor LDD-9-06x06-D-ZP-G-FR-BK (SMF-28e) (6 buffer tubings contain 6 cores, the fiber manufactured by Corning SMF-28e, the cable complies with standard G.652.D).

12 55

Figure No 2

LDD-9-06x06-D-ZP-G FR-BK Component number: F90360644B General design: The cable consists of 36 single mode color coded fibers. The fibers are accommodated inside 6 color coded tubings. The tubings are filled with tixotropic gel and SZ are twisted around the central cable member. The space around the cable core is filled with waterproofing gel. A fiber optics layer is used as an additional strength member. The final element of the cable structure is provided by a light-stabilized flame-retardant polyethylene coating. Application: Laying into cableways using blowing technique. Long distance telephony, cable TV and transferring data on line pipe facilities. Outside cable jacket material: FR-PE Outer diameter: 12.5 mm nom. Mass: 145 kg/km Design and materials Buffer material: Polybutylene terephthalate Piping diameter: (ID/OD): 2.8 mm/3.0 mm Color coding: TIA/EIA 598-C Central core member: FRP Piping twisting: SZ Core members: optic fibers Pipings with fibers: 6 Total pipings: 6 Total fibers: 36 Waterproofing: Between piping gel Jacket outer color: Black Marking: Per request Standards Standards: IEC 60794, ISO/IEC 11801, TIA/EIA-568 Installation: IEC 60794-1-1 Annex AOperational featuresMaximum admissible tensile force (assembly) 4000 N Maximum admissible tensile force (operations): 2400 N Impact strength: 20 N•m Cyclic impact resistance: 30 cycles Maximum admissible crushing force: 400 N/cm1 Minimum bending radius (assembly): 20xD mm Minimum bending radius (operations): 10xD mm Bending resistivity: 500 cycles Twisting (L=125 x d): 10 cycles 2 Maximum admissible operational temperature: +70°C Minimum operational temperature: -40°C

13 55

Maximum admissible cable laying temperature: +50°C Minimum admissible cable laying temperature: -10°C Maximum admissible storage temperature: +70°C Minimum admissible storage temperature: -50°C Light stability: Affirmative Waterproofing: Affirmative

Optic properties Index Standard per ITU-T-G.652D IEC 60793-2

Measurement units

Attenuation: Max/Typical @1310 nm 0.38/0.35 dB/km @ 1550 nm 0.23/0.20 dB/km @ 1625 nm 0.25/0.22 dB/km Dispersion: between 1285 and 1330 nm (range O)

≤2.7 ps/(nm*km)

between 1460 and 1530 nm (range S)

- ps/(nm*km)

between 1530 and 1565 nm (range C)

≤1.5 ps/(nm*km)

between 1565 and 1625 nm (range L)

≤2.2 ps/(nm*km)

Zero dispersion wavelength

1301.5-1321.5 nm

Mode field diameter: @1310 nm 9.3±0.5 μm @1550 nm 10.5±1.0 μm Critical wavelength ≤1260 nm Transmission medium sublayer (separate fiber optic)

≤0.2 ps/km1/2

Jacket diameter 125±0.2 μm Core/jacket eccentricity error

≤0.5 μm

Jacket out-of-roundness ≤1.0 % Jacket diameter (unpainted)

245±5 μm

Check test level 0.7 GN/m2

Forced macro bending: 100 turns around a core Core radius 10 7.5 mm Max. @ 1550 nm 0.5 0.4 dB Max. @ 1625 nm 1.5 0.8 dB

The backbone telecommunication line connects 16 PS and 98 line pipe valve shelters over a distance of 1504 km.

14 55

6.2 Requirements to DWDM system

The designed DWDM system should provide for telecom traffic for all existing CPC communication systems built on the basis of the national DWDM "Volga" developed by Russian company T8.The main list of services provided using DWDM (solely for preliminary information) is the following:

Corporate network in offices ranging from PS Astrakhan, PS Atyrau, PS 7 up to CPC On-shore facilities (1 Gigabit Ethernet channel per site).

OTN-150 SDH -STM1: one channel between PS. OTN-2500 SDH -STM16: two channels between PS. SCADA network 1 Gigabit Ethernet: two channels between PS, in over-lap topology

every other PS. ISPLS network: two channels 10 Gigabit Ethernet between PS. ISPLS network: two channels 1 Gigabit Ethernet between PS in over-lap topology. SAN network: up to 8 channels between sites having developed fail-safe

virtualization system; the throughput capacity is determined by design based on the service requirements and requirements on redundant capacity

The number of channels in different network segments, the types of interfaces, locations in CPC telecommunication line should be set forth during the design work.

6.2.1 Requirements to DWDM equipment: Expandability. Modular design. Flexibility of the system. Ability to provide a separate optical channel for each user

service at a separate wavelength. Reliable control using a dedicated optical channel having elevated sensitivity. 1 + 1 redundancy or system backing up to provide service reliability

DWDM system should allow a gradual build-up in line and network capacity on as needed basis without deflating investments made.DWDM equipment should have high reliability: all transponders, amplifiers, systems of power supply should operate in autonomous regime; when an error occurs on one channel the system should continue operating normally; the traffic in adjacent channels should not interrupt. All equipment in the system should support hot-swapping.DWDM equipment should support all types of digital client interfaces in the range between 100 Mbps and 10 Gbps. It should not require converting the client signal protocol into another protocol and it should be completely transparent to traffic in CPC process networks.DWDM system should be flexibly adjusted to the Customer needs; it should be able to organize I/O point for required number of channels at a minimum cost; at the same time it should be free from complete signal demultiplexing in fiber and it should pass part of channels through I/O point in transit without regeneration.

6.2.2. DWDM equipment should support remote visual monitoring and remote equalization of the channel spectrum.

6.2.3. DWDM equipment should support the ability to remotely upgrade software installed. 6.2.4. As part of equipment DWDM amplifiers should do the following:- supporting different stabilization modes: in particular, support stabilization of the gain mode, stabilization of the power output, stabilization of pumping current - be capable of adjusting the power output - have a mechanism to compensate the uneven spectral gain

- support the ability to change automatically the stabilization mode

15 55

6.2.5 Operational modes and composition of sets of equipment should be designed to provide for consistent operations in telecommunication channels designed in compliance with CPC requirements.

7.0 CORE LAYER IN ISPLS NETWORK

7.1 Requirements to network core layerThe existing security network is an MPLS network providing data transfer using technologies VPN L3, VPN L2 for ISPLS network clients.The following models of core layer routers installed in operator control rooms are used in each PS.

7.2 Network core hardware.Specification of the core layer equipment installed at the following facilities Marine Terminal, PS Kropotkin, PS Komsomolskaya, PS Astrakhan, PS Atyrau and PS Tengiz is shown in tables below.

Table 7-1Physical security network Security Network (IPSS) Number

Pump station (server room)Pump Station (Telecom Room)

Router Cisco 7606 Chassis 6 Cisco 7606 Chassis 6 CISCO7606

Processor module RSP720-3C-GERoute switch Processor RSP720-3C-GE

RSP720-3C-GE

Line card WS-X6748-GE-TX WS-X6548-GE-TX WS-X6548-GE-TX

Switch board Cisco 7600 ES Line Card, 20xGE SFP with DFC 3C

7600 ES20 Line Card, 20xGE SFP with DFC 3C

7600-ES20-GE3C

Power supply 2700 WDC Power Supply for 7606

2700 WDC Power Supply for 7606

PWR-2700W-DC

Specification of the core layer equipment installed at the following facilities: Marine Terminal and PS Kropotkin.

Table 7-2Physical security network Security Network (IPSS) Number

Pump station (server room)Pump Station (Telecom Room)

Router Cisco 7606 Chassis 3 Cisco 7606 Chassis 3 CISCO7603

Processor module WS-SUP720-3BXLRoute switch Processor WS-SUP720-3BXL

WS-SUP720-3BXL

Memory module Cisco C7600 RSP720 Compact Flash Memory

C7600 RSP720 Compact flash memory

MEM-RSP720-CF256M

Line card WS-X6548-GE-TX WS-X6548-GE-TX WS-X6548-GE-TX

2700 WDC Power Supply for 7606 2700 WDC Power Supply PWR-2700W-DC

16 55

for 7606

Specification of the core layer equipment installed at the following facilities: PS-3, PS-4, PS-7, PS-2, PS-5А, PS-4А, PS – А4А, PS-3А, PS-5, PS 4 (kp 390)

Table 7-3Physical security network Security Network (IPSS)

Pump station (server room)Pump Station (Telecom Room)

Router Cisco 7606 Chassis 6 slot RSP720-3C OS included Cisco 7600 Route switch Processor 720 Gbps fabric, PFC3C, GE included 2700 WDC power supply for CISCO 7606

Cisco 7606 Chassis 6 slot RSP720-3C OS included Cisco 7600 Route switch Processor 720 Gbps fabric, PFC3C, GE included 2700 WDC power supply for CISCO 7606

7606S-RSP720C-Pincluded RSP720-3C-GEPWR-2700-DC

Memory module Cisco C7600 RSP720 Compact Flash Memory //MEM-RSP720-CF256M

C7600 RSP720 Compact flash memory

MEM-RSP720-CF256M

2700 WDC Power Supply for 76062700 WDC Power Supply for 7606

PWR-2700W-DC

Software Cisco 7600-RSP720 IOS ADVANCED IP SERVICES //S764AIS-12233SRD

Cisco 7600 RSP 720 IOS Advanced IP Services

S764AIS-12233SRD

Switch board Cisco 7600 ES+ Line Card, 20xGE SFP with DFC 3C //7600-ES+20G3C

7600 ES20 Line Card, 20xGE SFP with DFC 3C

7600-ES+20G3C

7.3. Architecture of existing network core.

A typical physical topology of pump stations is the following; it is shown in scheme No1.

17 55

Diagram No 1

7.4. It should be taken into account when designing a new ISPLS network that the core layer architecture provides for a high-speed third layer medium in OSI model. 7.5. Core architecture of the network should provide for the permanent availability level free from single point of failure (SPOF) including, but not limited to the following:

chassis redundancy; redundancy in power supply for each chassis; support of hot swap for network equipment modules without service interruption; use of reserved direct links between pump stations; use of redundant backup links through a pump station (overlap connections)

throughout DWDM infrastructure; using back-door links for project-defined VPN; the total convergence of network should be less than 3 seconds; network availability should be no less than 99.97% the bandwidth should be used in efficient manner; the fastest possible data transfer; transmission of separate traffic flows.

7.6. The network core should comprise two Gigabit Ethernet connections to the backbone high-speed channel. The network core is using switchable D3 up channels to communicate with the network distribution layer. From the standpoint of MPLS network the core network hardware should perform functions of P and PE routers.

18 55

7.7. The network core layer should comply with the following requirements and should be capable of supporting the following:

each pump station should have two multi-layer routers having a modular architecture installed;

the network core layer should provide for interaction with the distribution layer equipment;

core network layer should provide for traffic balancing. high scalability without chassis modifications; communication channel aggregation technology (protocol 802.1ad, static); access lists (ACL based on layers L3, L2); time synch protocol (NTP); secure management/monitoring protocols (SSH, SNMP v2c, v3); Syslog protocol event logging;

7.8. These functions should be implemented through application of the following core layer protocols and technologies:

dynamic routing protocol OSPF v2; Multiprotocol Label Switching for MPLS labels with label transmission using LDP

protocol; transmission of Ethernet services over MPLS network (EoMPLS); Virtual infrastructure in MPLS networks (MPLS VPN) using MP-BGP protocol;

7.9. OSPF dynamic routing protocolDesigning should provide for use of OSPF dynamic routing protocol accounting for the following features and properties:

control over OSPF Hello and OSPF Dead timers; mechanism of identification of bilateral accessibility BFD; maintaining the same type of OSPF networks on interfaces; use of multiple routs having the same convergence; mechanisms for optimization of transmission of routing information; mechanisms of optimization of SPF algorithm; authentication.

7.10. Topology

The following types of telecommunication channels should be designed at the core layer: direct channels between the core layer network equipment; redundant channels organized by OTN 2500 network (for facilities Marine Terminal

and PS Tengiz);Direct links between the core routers should be used as main channels. Direct channels should be connected using Gigabit Ethernet technology with MTU values and duplex parameters at adjacent interfaces.Two loops of OTN 2500 network are used as a backup; each loop features a distributed hub. Connectivity technology to switch redundant Fast Ethernet channels having supported MTU values up to 1522 bytes accounting for the link level headers and duplex parameters. It is required to provide for additional aggregation level network equipment in case this is impossible to match the rate of the core layer routers connected to OTN sites.It is required to provide for an appropriate channel value to prevent Equal Cost routes in redundant channels. The topology should be connected to ensure that the first core routers in each station are connected to the first OTN ring. The second core router in each station is connected to the second OTN ring.

19 55

Two direct point-to-point or point-to-multi-point channels should be designed between the network equipment at the core layer. OSPF protocol is activated only on interfaces and in networks that connect core layer devices.It is required to use logic loopback interface having number 0 (loopback0) to identify the routers within OSPF protocol (OSPF Router ID). It allows improving overall stability in protocol performance.OSPF protocol should be active only on interfaces and in networks connecting devices of the core layer.

7.11. AddressingDesigning should provide for using telecommunication channels between the core equipment ("point to point") using network with mask /29These channels involve:

direct communication channels between core routers; networks to connect to backup communication channels;

Provide for using addresses having mask /32 to identify router within OSPF protocol.Summing routes on the core layer is unacceptable as it affects accurate performance of LDP and MP-BGP protocols. Therefore, there is no need to split different core layers into various OSPF domains. Core layer should belong completely to OSPF Area 0.

7.12. Fast convergence in OSPFProject should provide for changing the basic characteristics of this protocol. Reduce the convergence time of OSPF protocol in the following ways:

optimize mechanisms to detect line incidents (OSPF Hello and OSPF Dead timers, BFD mechanism, Carrier Delay timer);

optimize performance of SPF algorithm (SPF Throttling and SPF mechanisms); optimize the transmission of information about the telecommunication status (LSA

Throttling, LSA Group Pacing, and LSA Flood Pacing mechanisms).

7.13. Optimizing OSPF timersDuring design stage it is required to optimize the timer to send periodic OSPF Hello and OSPF Dead messages.The default value is equal to seconds respectively. Implying that detecting an incident at the data link level can be as long as 40 seconds. This value is unacceptable and requires a significant reduction.In standard configuration the minimum value of OSPF Hello timer can be equal to 1 second. OSPF Dead timer value should be at least 4 times greater than OSPF Hello timer or 4 seconds. This is the time required to determine an incident and it is also too long and requires reduction.The project should provide for the minimum possible values of OSPF Hello and OSPF Dead timers.

7.14. Carrier Delay Timer OptimizationDesign should provide for optimization of the Carrier Delay timer to accelerate process convergence when an incident is detected at the OSI model physical level. The timer is set to 0 for all direct link interfaces between core routers. Operational mechanisms of events containment (Half Life Period, Reuse Threshold, Suppress Threshold, Max Suppress, Restart Penalty) should be kept at their default values.

7.15. Implementation of BFD mechanism

20 55

Design should provide for detection of failure in bilateral connection of the physical link using BFD mechanisms. The following parameters should be optimized in BFD mechanism performance:

BFD Interval; Minimum RX; Multiplier.

7.16. Optimizing distribution of the connection status informationIt should provide for prevention of generation of a large number of packets sent through optimizing LSA packet dispatch intervals. It is required to optimize the following timer parameters:

Start-Interval — the minimum delay before the onset of generating the first update package;

Hold-Interval — the dynamically changeable delay time used to calculate the time to generate update packages. When the second event occurs during the time sending the packet is delayed by the Hold-Interval time. When subsequent events occur the delay will amount to 2*Hold-Interval, 4*Hold-Interval and so on for each event until the delay value becomes equal to the Max-Interval;

Max-Interval — is the maximum delay before the onset of generating next LSA packages;

OSPF Group Pacing.

7.17. SPF calculationsIt is required to provide for optimization of SPF Throttling mechanism to delay triggering SPF process and, as a result, slow down the procedure of calculation of the shortest route when the network is unstable.By default the delay in triggering OSPF process is disabled in routers. However, it should be used to ensure better stability in network performance.

7.18. The maximum number of routsIt is required to provide for load balancing across multiple routs having the same value to the network segment. It is required to optimize maximum path parameter.

7.19. AuthenticationAuthentication should be used to improve OSPF protocol performance security level. Algorithm MD5 - a Digital Signature Algorithm described in RFC1321 - should be used for authentication.

7.20. Recording OSPF system messagesCore layer routers can record the change in OSPF-neighborhood status in the local system message log and send these messages to the Syslog server. This functionality should be enabled in all core switches to facilitate and speed up the search for potential malfunctions and fixing them.

7.21. Optimizing OSPF (incremental OSPF update)A function of incremental OSPF optimization (ISPF) should be provided as an OSPF protocol optimization option allowing to calculate using SPF algorithm not the whole a graph but only the part showing changes.The use of ISPF will reduce the operational time of SPF algorithm. The application of ISPF at the core layer routing is justified and necessary as it is required to speed up the network convergence process.

21 55

7.22. Label Distribution Protocol (LDP) for Multiprotocol Label Switching (MPLS)

7.22.1. It is required to provide for the label distribution of multiprotocol switching (MPLS) in ISPLS network at the core layer using a dedicated Label Distribution Protocol (LDP).

7.22.2. Functional parameters of LLDP are determined according to the following parameters and properties:

label exchange mode; protection of sessions; authentication;

7.22.3. It is required to analyze the compatibility of solutions with protocols currently used on existing equipment at the stage of designing application of LDP and MPLS.

7.22.4. The results of comparative analysis should include the following details: the list of supported MPLS standards in existing and designed equipment; description of Link-layer protocol support in existing and designed equipment; review of assigning mpls label; comparative analysis of special MPLS labels, BGP VPN Route distribution and

penultimate routers and assess of their impact on overall compatibility of developed design solutions and existing design;

comparative analysis of LSP types used based on deployment of solutions on traffic engineering;

analysis of feasibility of using Constrained Shortest Path First algorithm in LSP computation;

analysis of using Fast Reroute and Link Protection;

7.22.5 Starting LDP Use Unsolicited Downflow Label Distribution Mode; label saving mode corresponds to the Liberal Label Retention (LLR) mode; route control mode corresponds to Independent LSP Control Mode.

7.22.6. Basic LDP configuration To simplify troubleshooting each router should capable of setting range of the locally used labels. Router ranges should be set equal to NN 0000 to NN 9999, where NN is the number of the facility where the router is installed. A complete list of ranges should be presented in a separate table.The auto configuration protocol LLDP should not be used. At the same time activation of LLDP protocol is made manually for each core interface. These interfaces include all direct telecommunication link interfaces between core routers.

7.22.7. Router identifier for LDPThe logical loopback interface should be used to identify core routers for LDP protocol (LDP Router-ID); also loopback0 should be used for identification in OSPF protocol.Determine transport address used by LDP as a Router-ID.

7.22.8. TTL propagationThe Time-To-Live (TTL) should be used as a function to rule out occurrence of routing loops at the core layer at the designing stage.

7.22.9. MPLS MTU

22 55

It is required to provide for interface MTU values facing the core layer when a network is set up using MPLS for such technologies as the following:

technology MPLS L2VPN technology MPLS L3VPN technology EoMPLS

7.22.10. Controlling label propagationProvide for support of control of the label propagation mechanisms:

Controlling the Advertisement of Labels; LDP Inbound Label Binding Filtering.

Both technologies are aimed at reducing the number of label entries in the router memory.It is required to evaluate the possibility of applying LDP IPv4 FEC filtering. Design solutions should include using ADR automatic connection policers and describing the application rules in the context designed QoS model in ISPLS network.

7. 22.11. Protection of MPLS LDP Session Provide for protection of LDP sessions for each core layer router. LDP sessions should not terminate to ensure the fastest possible restoration of correct network performance after incident. The timeout to restore the direct telecommunication link between routers is not specified.

7.22.12. Synchronization of LDP-IGP protocolsProvide for synchronization of OSPF routing protocol with LDP protocol in all core routers.

7.22.13. LDP authentication MD5-authentication should be used to improve security level of LDP protocol performance in ISPLS network.

7.22.14. Recording LDP system messages It should provide for a record of changes in LDP-neighborhood status into the local system log and send these messages to Syslog server. Also routers allow reporting authentication errors in LDP-sessions. This functionality should be enabled in all core switches to facilitate and accelerate troubleshooting.It should provide for the activation of collecting statistics in LDP and describe potential limitations.

7.22.15. Convergence of LDPIt should ensure the maximum possible convergence time in LDP protocol. (Unsolicited Downflow Label Distribution Mode, Liberal Label Retention Mode, Independent LSP Control Mode).Assess configuring LDP timers for hello, HOLD timer, and Keepalive timer messages.Assess the potential of using strict targeted hello messages for LDP.Describe application of filtering Inbound Outbound LDP label binding in the process of migration of network segments.It should provide for the use of BFD for LDP LSPs and describe its operating modes.It is required to use LDP link protection both for unicast and multicast LSPs.

23 55

7.23. Ethernet s over network service transmission technology with multiprotocol label switching (EOMPLS).Provide for use of EoMPLS (Ethernet over MPLS, EoMPLS) technologies for connecting loops in segment circuits for the main shut-off valve skids at the second level of OSI model.

The project should provide for the following:- possible access load balancing (via EoMPLS) using MPLS core;- estimation of the maximum package size (MTU) for all intermediate links between

endpoints and use of TE (RSVP) functions for major core communication channels;- describing methods to place information on VLAN membership in Ethernet frames;- spanning tree and REP (or analog) work for VLAN with Ethernet over MPLS;- limitations of using EoMPLS technology for proposed software version in

communication equipment; - compatibility of equipment used; - optimization of migration procedures or equipment replacement.

7.24. EoMPLS switching Requirements to pseudo-conductor to connect circuit segments in MPE blocks:

the possibility of organizing each pseudoconductor on a separate physical interface; the ability to transmit frames with headlines IEEE 802.1Q; the potential of Port Mode choosing; provide for a completely transparent transmission for all second level frames in OSI

model; provide for a single identifier for each pseudoconductor accounting that the first two

numbers comply with the numbers of the object code to terminate the segment, and the third number complies with the segment number.

7.25. Design the topology of the ring segments for entire ISPLS network using Resilient Ethernet Protocol (REP) or similar for the chains of blocks in main shut-off valves.To analyze compatibility of REP protocol in existing equipment with new design solutions for parallel operations and describe migration steps.

7.26 Transparency of dot1q for EoMPLSSegments of the main chains of shut-off valve blocks in ISPLS network should be designed to be capable of using multiple virtual networks (VLAN). EoMPLS technology should close the segment contours with potential tunneling all required virtual networks. Use this technology due to transparent transmission of Ethernet frames having IEEE 802.1Q headers through pseudoconductor. Port Mode allows sending frames both with IEEE 802.1Q headers and without them.

7.27. Starting MPLS TE technologyProject should provide for using MPLS TE EoMPLS pseudoconductor. Use RSVP as a signaling protocol. EoMPLS should use only the basic core channels.RSVP protocol is used in all core layer routers; it is only allowed in direct communication channels between routers. At the same time the analysis should be conducted for the maximum available bandwidth in MPLS TE tunnels.

7.28. Starting MPLS TE tunnelsTo explicitly specify the traffic rout MPLS TE tunnels should used indicating the network traffic passing parameters. In the future these tunnels should be used for EoMPLS traffic transmission. The amount and direction of the tunnel should be determines by project.

24 55

7.29 MPLS MTUThe project should describe the use of the optimal MTU value for EoMPLS-frame with TE tunnel to ensure the necessary level of service.

7.30 The technology of virtual private networks Multiservice network ISPLS should support L2 and L3 MPLS VPN. The number of L3 MPLS VPN in backbone network, the nature of their interaction should be specified within the feed survey and feed design. Currently, it is required to provide setting up the following VPN:

VPN (CL) to connect work stations, engineering stations, stations to maintain UPS systems; this VPN should support multicast. It is present in each physical facility;

VPN (DVM): it is required to connect DVM Database Servers, DVM CameraServers, Streamers or Network Cameras. It is present at each facility; Existing systems limitation:

DVM server should be located in the same subnet with cameras; the cameras should have the capability of switching (re-registering) with different

DVM servers installed in geographically remote PS facilities.The character of interaction is shown schematically in diagram No 2 to be rectified in design (shown for information only). It should support both the fixed addressing, and automatic assignment of address at connecting to the network.

Scheme No 2. Scheme of interaction between ISPLS systems

25 55

TCP Port 80 (HTTP),TCP Port 135 (DCOM),Configurable DCOM PortsTCP Port 10000 (Event Server)TCP Port 60000 (DVA Configuration)TCP Port 443 (HTTPS – DVM Console Logon) NEW!

UDP Port 10001 (Multi-monitor ping)TCP Port 135 (DCOM) (for multi-monitor clients only)

Multicast Address 226.0.x.x Port 1 (Multicast Live Video)

UDP/TCP Port 10001 (Camera Manager Ping and Health Status), TCP Port 135 (DCOM)Configurable DCOM Ports

TCP Port 135 (DCOM)Configurable DCOM Ports

TCP Port 443 (HTTPS – DVM Console Logon NEW!

EBI

DVM Database Server

Station Internet Explorer

Streamers or Network Cameras(All supported DVM models)

TCP Port 135 (DCOM)Configurable DCOM PortsTCP Port 1433 (SQL Server)TCP Port 10000 (Event Server)

UDP/TCP Port 135 (DCOM)Configurable DCOM Ports

TCP Port 135 (DCOM)Configurable DCOM PortsTCP Port 10001 (Live Video)TCP Port 10001 (Playback)TCP Port 10001 (VMD Tuning)TCP Port 10010 (Authentication)

TCP Port 80 (MJPEG Video)TCP Ports 80 & 554 (MPEG-4 Video) TCP Port 5001 (PTZ – for camera control types other than « Fixed Camera » and « Use Streamer Settings »)

DVM Database Server

TCP Port 10000 (Event Server)TCP Port 1433 (SQL Server)

Configurable DCOM PortsTCP Port 135 (DCOM)

TCP Port 40200 (DVM Service Framework) NEW!TCP Port 12000 (DVM Point Server LinkD) NEW!

TCP Port 12001 (DVM Point Server RPC) NEW!TCP Port 12002 (DVM Point Server Notifications) NEW!

TCP Port 135 (DCOM),UDP Port 10002 (IntegrityService Ping)Configurable DCOM Ports,TCP Port 1433 (SQL Server)TCP Ports 137, 139, 445TCP Port 60000 (DVA)

TCP Port 10001(Video loss alarms)

DVM Camera Server

UDP/TCP Port 135 (DCOM), Configurable DCOM PortsTCP Port 10001 (Camera Control)TCP Port 10010 (Authentication)(Custom application/scripts using DVM Object Model)

- VPN (IQ) is used to ensure operability of systems to control perimeter access security. Controls access to CPC facilities. It is present at all key locations (PS, MT and shelters).The project should provide for both L3 VPN, and L2 VPN for such clients. Potentially the need to support multicast will be required. The nature of the interaction is rectified at design stage including interaction with CL VPN, VPN DVM, VPN SUP, and interaction locations at each PS.

VPN IPG is an isolated L 3 VPN set up to control channel generating equipment (optical amplifiers, DWDM). It is present only at PS and in shelters of future PS.

VPN OTN is an isolated L 3 VPN serving systems to control SDH - OTN network, it is present in Marine Terminal and PS Kropotkin.

26 55

VPN PHN is a transport L 3 VPN to support ip phones; it is present in all facilities. VPN RAD is an isolated L 3 VPN to serve systems to support radio

telecommunication equipment. It is present only at PS and in radio shelters of future PS. VPN NMS is an L 3 VPN to control elements at the network access level. It is present

in all facilities and it interacts with all VPN. VPN SUP is a L 3 VPN to connect contractors serving security systems; it is present

at all sites and it provides access to VPN DVM. VPN CL. VPN IQ Project design should determine necessity to set up VPN for virtualization solutions The project should describe the necessity and implementation of VPN central

services and their locations7.31. The design should account for potential scaling up in the future to develop up to 15 L3 VPN solutions (up to 4 L2 VPN).7.32. The project should describe MPLS Data Forwarding operations, BGP VPN Route distribution to demonstrate compatibility with existing equipment; in case of finding risks of incompatibility with existing equipment and solution it is required to describe mechanisms of migration and mitigation for the risks identified.7.33. The project should analyze the interactions between VPN to describe the architecture, taking into account implementation of various topologies (eg, HUB-and-SPOKE, FULL MESH , etc.); a special attention should be paid to analysis of potential traffic asymmetry between L2 L3 VPN having multiple regional points of interactions.Analyze existing addressing scheme and describe the necessary changes (if required). ISPLS network should use private address space (RFC 1918) ipv4.7.34. The project should describe the interaction of PE-CE sites both using L3 OSPFv2 protocol and static connections similar to existing design.7.35. Solutions on Service Level Agreement (SLA) should be provided both for existing equipment and for equipment designed.

7.36. Design solutions on network routing should include the following analysis and description:

the maximum number of routes announced in VPN network and setting relevant restrictions;

backdoor connections between sites for set up in VPNs design using redundant communication lines (e.g., dedicated FastEthernet communication channels in existing SDH ring in OTN -2500);

mechanisms to protect against cycling in routing protocols when backdoor connections are active;

optimization of convergence in routing protocols; balancing traffic during connection.

7.37 Configuration of MP-BGP Protocol Provide the following parameters and protocol configuration including but not limited to the following:

autonomous system; Address family; Route Distinguisher; Route Target Community; Virtual Routing and Forwarding instance; import/Export policy; definition of neighbor groups ; balancing load; monitoring MP - BGP protocol .

27 55

7.37.1 Autonomous systemISPLS network consists of a single autonomous system comprising all levels of the network core routers. A decimal number 64512 will be used as AS number. This value corresponds to a value in existing network lying within the range of numbers determined by IANA for private use.All routers in the autonomous system establish direct neighbor adjacency.

7.37.2 Address familyIt is required that BGP protocol provides for transmission of routing information of protocol IPv4 in the global routing table. Also, it should provide for transmission of VPNv4 family routes and VPN MDT multicast. This address family should be explicitly specified in BGP protocol configuration, thereby enabling multiprotocol extension in BGP (MP-BGP). It is also required to provide the parameters of route redistribution in BGP protocol for specific VRF under MP-BGP protocol for multicast addresses family.

7.37.3 Route featuresIt should provide Route Distinguisher to generate unique VPNv4 prefixes for each VPN.

7.37.4 Import/Export route policiesProvide fully connected topology for each virtual infrastructure through bilateral, symmetrical exchange of routing information for each VPN.

7.37.5 Neighbor groupsIt is required to provide for use of one group of neighbors to simplify configuration.

7.37.6 Load BalancingIn order to optimize channel loading and improve convergence time the mechanisms of BGP load balancing should be used.

7.37.7 Interaction with the network management systemThe network management system should provide for control of MP-BGP protocol the status using SNMP v2c, v3 protocolThe following SNMP objects that control BGP neighborhood are most critical:

bgpPeerLastError; bgpPeerState; cbgpPeerLastErrorTxt; cbgpPeerPrevState.

7.37.8 BGP optimisation and convergenceImplement BGP performance optimizations:

change BGP timers and transportation mechanisms TCP at accordance with recommendations RFC 4271;

Fast peering deactivation; suppression of swinging routs; Bidirectional Forwarding Detection for BGP; BGP NextHop Tracking.

28 55

7.37.9 It should ensure activation of BGP mechanism to Import VPN based on eventsWhen a change occurs in network mechanism BGP Event-Based VPN Import allows importing information obtained from neighboring routers in VRF much faster than through periodic table scanning.

7.37.10 Provide for BGP securityUse the following BGP security mechanisms:

switch off ibgp (internal bgp) synchronization; includes registration of changes status in bgp-neighbors; use of authentication; harmonization of bgp versions is disabled for acceleration of process start up; include a limit on the maximum number of prefixes; loopback interface should be used for ibgp announcements; turns off the automatic summing prefix announcements; use next hop-self in re-routers; avoid availability of excessively detailed routes;

7.37.11 A plane to transfer the dataProvide for using the following mechanisms:

tables CEF; labels; tables LFIB.

7.37.12 LabelsThe range of labels used to label virtual third level infrastructures should be distributed dynamically (dynamic label allocation). Implement the compatibility mode over the label transmission for different network vendors.

7.38. VPLSIn existing network VPLS technology allows combining geographically dispersed LAN segments into a single bridged domain over a packet MPLS network. According to the model to deploy VPLS the backbone network serves as the logical bridge. The topology and signaling in the backbone network are transparent to the customer LAN segments, which are interconnected as if they were connected to a chain of LAN Ethernet switches. The use of VPLS in existing network design is caused by requirement to ensure reliable operations of security applications and accommodation of camcorders and DVM servers, IQ controllers and EBI servers in the same logical subnet.This restriction is important and requires specific comment in the project documentation.It is required to evaluate design options, reliability and stability in use; it is also required to describe the mechanisms of protection from broadcast storms or loops.The following options should be considered:

full mesh VPLS; H-VPLS for boundary MPLS segments; single-domain VPLS for full mesh topology; multidomain VPLS for full mesh topologies at the level of each domain; multidomain VPLS for H-VPLS configuration of HUB and SPOKE types with HUB

redundancy.It is required to evaluate the risks of traffic asymmetry when these domains have multisite connections to other VPN L3.On consent of telecommunication group other technologies can be also used.

29 55

7.39. Interaction with existing networkThe network designed should be connected to the existing network for a period of services migration to ensure continuity in current process operations. All devices in the old network should be accessible from the new network and vice versa.The design process should take into account the feasibility of stage by stage failed hardware replacement, the project should describe various migration methods and describe the risks of migration procedures and risk mitigation strategy.

8.0 ISPLS NETWORK DISTRIBUTION LEVEL

8.1 Requirements to distribution level8.1.1 The existing network has a collapsed core design.

8.1.2 To overcome these operational design limitations the draft design should describe both the use of the new equipment as part of new design of collapsed core and in design of the two-layer approach (PE-CE).

8.1.3 The distribution level should be implemented in the form of a pair of 3 (L3) level switches for the critical pump stations (Onshore Facilities, Kropotkin PS, PS Atyrau, Tengiz PS, PS Astrakhan, PS 3); for other PS the distribution level should involve one switch.

8.1.4 The distribution level should ensure an efficient gathering of routing information as it is leaving the distribution sites to be sent to the core layer using IGP (OSPF v2) protocol.

8.1.5 The distribution level should support the following protocols: OSPF, VLAN, PVLAN, EoMPLS , MPLS, EtherChannel, HSRP, VRRP, GLBP for backup the default gateway, MP-BGP, MPLS-TE , QoS DiffServ.

8.1.6 The distribution level in ISPLS network should meet the following requirements: the fastest data transfer; load balancing across multiple active routs to the core; connection aggregation; routing between VLAN; use of active gateway redundancy protocols; redundancy on chassis; redundancy on power supply for each chassis; providing quick convergence of the network when topology changes; transfer of isolated traffic flows; connection to the core layer; connection of remote buildings and access level; termination of IP traffic from the end directly connected devices and from L2 remote

buildings.

8.1.7 These functions should be implemented through applying the following protocols and technologies at the distribution level:

OSPF dynamic routing protocol; virtual infrastructures (VRF); Etherchannel technology.

8.1.8 The project is required to provide for the distribution of the interaction level with core, aggregation and access levels. Communication of hardware at the distribution level with other network levels is made using aggregated connection channels using EtherChannel technology. Protocols PAgP and LACP should be used to make a logical interface.

30 55

8.1.9 These functions are realised through using the following protocols and technologies at the distribution level:

Etherchannel technology.

8.1.10 Three mechanisms can be used to form EtherChannel channel:1. PAgP - a proprietary protocol to concur channels in EtherChannel.2. LACP (802.3ad) - a standard multivendor concurrency protocol.3. Static «ON» mode without using concurrency protocols.

8.1.11 Design solutions should describe route summing schemes, schemes of filtering, addressing; the solutions should describe methods to prevent loops both at the L2 protocol level and on L3 protocol level; the description should be given how to set up backdoor connections for project VPN.

8.1.12 Project documentation should include network architectures and functional diagrams.

8.1.13 Design solutions should be justified to use the functional network devices.

9.0 ISPLS NETWORK ACCESS LEVEL

9.1 The architecture of access levelISPLS Network Access Level should provide connection of the end network devices (video servers, video tape drives, video cameras, IQ access controllers, operator workstations, printers, telephones, communication systems management ports, etc.) installed in remote buildings of the pipeline facilities both at PS sites and along the line pipe. The access level is connected to the level of the distribution network using two redundant connections to form a

fully-meshed topology.

9.2. The access level in ISPLS network should meet the following requirements: packet switching; port security; PoE support (optional); connection of network clients using channels and protocols corresponding to Gigabit

Ethernet specifications; design solution should provide for scaling up without replacing the equipment and/or

change in architecture provide for each site the port capacity no less than 48 ports; the network should have a fault-tolerant architecture, eliminating a single point of

failure at any level; the equipment should provide for support in quality of service (QoS) mechanisms in

the framework of the whole network; network segmentation using VLAN technology, including setting up trunk channels of

data transfer (802.1q); network protection at L2 level from appearance of Spanning TreeProtocol (PVST +,

Rapid + PVST, MSTP) loops;

31 55

support of 10Gbps and 1Gbps interfaces to interact with core/distribution level; link aggregation technology (protocol 802.1ad, static); access lists (ACL based on levels L3, L2); time synch protocol (NTP); secure management/monitoring protocols (SSH, SNMP v2c, v3); Syslog protocol event logging;

9.3. Hardware and software components of the access level should provide for the following:

high availability for ISPLS service system users; use of redundant power supplies and interface hot-swappable modules; redundant gateways via dual connections to redundant systems (distribution level

switches) using GLBP , HSRP and VRRP protocols. prioritization of critical network traffic using QoS functions. The access level should

provide for implementation of priority mechanisms and traffic labeling as close to the network as possible.

additional network protection from unauthorized access should use the following tools: 802.1x protocol, port security, Dynamic ARP Inspection, IP source guard.

Rapid use of protocols PVST + (802.1w) or Multiple STP (MST 802.1s) with the following improvements of the Spanning Tree protocol:

PortFast - It allows the port to skip listening and learning phases UplinkFast - provides for convergence during 3..5 sec after connection break BackboneFast - decreases the convergence time to MaxAge value for indirect

disruptions Loop Guard - Excludes the capability of choosing an alternative Root Port when

BDPU (Bridge protocol Data Units) data blocks are absent Root Guard - is preventing capability of using the external switch as a root BPDU Guard - disables port supporting PortFast functions on receiving BPDU data

block BPDU filter - prevents dispatching or receiving BPDU data blocks through ports

supporting PortFast functions

9.4. The project should provide for connection of remote buildings using both L2 and L3 (OSPFv2) with description of routing schemes, summing, filtering, route tagging, setting up backdoor connections for VPN specified in project design.

10.0 SECURITY AND NETWORK EQUIPMENT CONTROL

10.1. ISPLS network is an isolated network without access to Internet, SCADA or the corporate network; however, the network design should provide for security solutions at all levels of the network hierarchy.

10.2 The following technologies and methods should be applied to ensure the necessary level of security to the access switches without limitation:

Port Security; DPSP Snooping; Dynamic ARP Inspection;- IP Source Guard;- Storm Control;- security function in STP protocol.

32 55

10.3 It is required to use Port Security mechanism in order to limit the number of MAC-addresses on interfaces.

10.4 The project should describe the following for all equipment (projected and existing): device and service hardening techniques; main data to ensure security in control - plane and data - plane; security in routing and switching protocols; infrastructure security mechanisms; control over using equipment resources; mechanisms to protect signaling network device plane; port mirroring to transfer traffic.

10.5 The project should provide for setting up Cisco ASA firewalls to ensure secure access to network infrastructure monitoring systems in MCC (Onshore Facilities) and BCC (backup control center) in a failover mode. Monitoring and control system is described in the following sections.

10.6 The existing infrastructure allows organizing out-of band management in communication channels for network devices installed indoor the telecommunication buildings at PS. The project should describe a solution on using RS232 ports to provide for access from MCC via existing SDH OTN-2500 telecommunication channels.

33 55

11.0 QUALITY OF QOS SERVICE

11.1 It is required to carry out analysis of traffic transmitted by all systems to ensure subsequent correct labeling and classification when QOS policy is developed. 11.2 The policy of providing quality of service implies the network ability to provide for predictable loss and delay. 11.3 The design should explicitly specify which devices perform QoS on the software and hardware level followed by making risk analysis to ensure compatibility of existing and designed equipment.11.4 Classify and label the application traffic as close to the source as possible. This implies using the network-wide unified Differential Service principles and incremental maintenance strategies. All connected end devices such as servers, ISPLS workstations, IQ controllers will not be trusted in terms of traffic labeling; however camcorders, video tape drives can support the traffic labeling. The choice of the most optimal strategy to deploy QoS service should be based on analysis of the working documentation provided.11.5 The unwanted traffic should be restricted as close to the source as possible. This is especially significant for traffic in "denial of service" attacks or unwanted traffic generated by "worms."11.6 Design should provide for implementation of queue management policy in all sites featuring potential overload regardless of overload frequency. This is especially important for redundant communication channels, in communication channels between switches having overload prerequisites.11.7 Protect control plane and data plane using Control Plane Policing and using restriction on the network scavenger traffic.11.8 It is necessary to implement the following mechanisms to meet QoS requirements without limitation:

Classification and labeling allow classifying traffic on the basis of several characteristics such as values in Ethernet, IP and MPLS headers to be followed by labeling relevant packets using a specified value usually located in Type-of-Service byte IPv 4 header, in experimental MPLS header fields or in 802.1p User Priority fields in Ethernet - frame 802.1q. When packets are assigned a specific labeling their traffic class can be easily identified by other network sites.

Overload control allows establishing traffic priorities inside network devices when overload appears. If outlet channel is overloaded the packets designated for telecommunication form a queue. Overload control mechanisms allow redistribute the packets in transmission queue that makes it possible to guarantee the throughput capacity and delay time for certain traffic classes.

Preventing overload is aimed at avoiding overload in reasonable manner before it starts dominating communication through proactive packet rejection when queue exceeds a certain length.

Traffic limitation and control allow restricting incoming or outbound traffic at established "profile" rate level. In case of restrictions traffic exceeding permissible rate is rejected. Control is trying to adjust the traffic that can temporarily exceed a profile rate through queuing the traffic; as a result the overall traffic rate gets smoother and better meets the profile rate due to additional delay.

11.9 Classification and labeling

11.9.1 The network traffic entering DiffServ domain should be classified and made concurrent. Traffic should be classified against many different parameters, such as source address, destination address, or traffic type; the traffic can be attributed to a certain traffic

34 55

class. Traffic classifiers should accept for processing any DiffServ labeling in packages obtained.11.9.2 During labeling the switches should analyze incoming packets (frames) and assign an appropriate «QoS label» ("Internal DSCP "). «QoS label» determines further QoS actions to be applied to the packet on the route from the input to the output port. «QoS label» is an internal switch label; it is assigned based on DSCP or CoS packet values and determines the queuing techniques and the packet scheduling actions. In the framework of the project «QoS label» will coincide with corresponding packet DSCP. 11.9.3 In case other labeling mechanisms are used the project should provide for a complete description of desired functionality in hardware designed. 11.9.4 Network equipment should be able to classify traffic against the following criteria:

Using DSCP field in IP - packets. Using IP precedence field. Using 802.1P CoS Ethernet frame field. Using MPLS EXP field. Classification should be made using standard or extended IP access lists.

Classification should be made based on VLAN ID Labeling should be made accounting for IP Explicit Congestion Notification (IP ECN)

field value. The labeling should be made on the basis of application signatures using NBAR

technology (optional for routers only).

11.9.5 The types of ISPLS network equipment and necessary means of traffic labeling at the inlet port are given in Table No 11.9.5.

Inlet labeling potential

Table No 11.9.5.

Hierarchy levelIP

Precedence

IP DSCP

802.1p CoS

MAC ACL IP ACL VLAN ID MPLS

EXP

Access level

X X X X X

X X X X X

X X X X X

Distribution level X X X X X X

Core layer X X X X X X X

The main provisions on labeling traffic in ISPLS network are as follows: all connected resources are not trusted from the standpoint of traffic labeling except

the traffic generated by video cameras and streamers; the traffic is labeled at the network inlet (at the access level and on the distribution

level) using access ip lists with filled ip dscp fields corresponding to table of classes; traffic labeling for facilities connected is made in «port - based» and «vlan - based»

modes. Labeling is made individually for each device in "port - based" regime; labeling is made for groups of devices in «vlan-based» case;

all connections between active network equipment are trusted. Labeling is made for ip - traffic using fields ip dscp, 802.1p, mpls exp;

35 55

traffic re-labeling is made from field ip dscp into field mpls exp and vice versa at the boundary mpls - network core;

for transmission along the core mpls ip dscp traffic labeling remains the same and recovers at the outlet of the network mpls - core («pipe mode»).

11.10 Classes of traffic pattern11.10.1 It is required to design ISPLS network traffic labeling pattern on the basis of 8 classes. This labeling is given in Table No 11-10 for informational purposes; it is subject to rectification and adjustment at designing stage.

Traffic classesTable 11-10

Class

Classification L3 Classification L2

AssignmentDSCP PHB Priority

802.1p CoS,

MPLS EXP

Critical 56 CS7 7 7 Critical traffic

Network 48 CS6 6 6 BGP, OSPF, EIGRP, HSRP

Real-time 46 EF 5 5 ISPLS Real Time Traffic

Interactive 32 CS4 4 4 High priority traffic

Normal 24 CS3 3 3 Normal priority traffic

Transactional 16 CS2 2 2 Low priority traffic

Scavenger 8 CS1 1 1 Scavenger traffic

Unclassified (Best Effort) (Unclassified (without guarantees)) 0 0 0 0 The remaining (non-

classified) traffic

11.10.2 This 8-class model should be implemented at all levels, including the network core where traffic labeling is only possible against 3 MPLS EXP bits.These traffic classes should be processed by queues using different mechanisms; different traffic classes should have different guaranteed bandwidth.

11.11 QOS Domains ISPLS network should implement 2 QoS domains. One domain should contain MPLS-core, backbone and redundant communication channels. The second domain should contain facilities connected to MPLS-core, namely all PS, Onshore facilities and Tank Farm.QoS policy should be applied in all interfaces of uplink, downlink, EtherChannel network and others. An important aspect that should be considered when deploying QoS on EtherChannel policy deals with load distribution. It is necessary to distribute the load based on source and destination IP - addresses as this gives a statistically improved load balancing.Algorithms should be divided using two ways for QoS algorithm configuration on EtherChannel interfaces:The input algorithms such as trust or labeling and/or restriction policies associated with PortChannel interface.Output queuing algorithms are applied directly to the (physical) interfaces that make up the EtherChannel package.

36 55

11.12 Setting up queues and resetIt is required to use a priority queue to serve real-time traffic; the rest of the traffic is served using CBWFQ mechanism. Designing should provide for the following functions to set up queues in different models of network equipment:

queue types at outlet; establishing queue restrictions; configuration of priority queue; configuration of standard queues; configuration of thresholds; configuration of threshold for queue wred - reset based on service class; configuration of threshold resetting the last element when a queue is generated on

the basis of dscp field value;

37 55

12.0 SYSTEM TO SYNCHRONIZE ISPLS NETWORK TIME PROTOCOL USING NETWORK TIME PROTOCOL (NTP)12.1 The project should provide for use of synchronization time protocol NTPv4 ensuring authentication of time sources and supporting network equipment.12.2 The following NTPv4 features should be accounted as a selection criteria:

NTPv 4 is compatible with NTPv 3 in case equipment is used that does not support NTPv 4;

NTPv 4 is an new protocol officially supported by software and network equipment producers;

algorithm MD 5 is applied for server time authentication; NTPv 4 is identified in RFC 5905: http://tools.ietf.org/html/rfc5905 .

12.3 Solution on system time synchronization12.3.1 NTP protocol should be used to synchronize the system time in network equipment and equipment of ISPLS security system. It is necessary to implement a two-tier time synchronization scheme.12.3.2 Two servers should be used as first-level time sources (Stratum 0); they should be located at the following sites:

PS Tengiz, radio shelter; Onshore facilities, radio shelter.

12.3.3 These servers should set the system time with an accuracy of 10 microseconds. Time synchronization at the Stratum 0 level should be carried out using GPS satellite channels. The local server clock built on crystal oscillator having a temperature compensation (Stratum 12 level) are used as backup time sources. It is not expected that external NTP-servers are used for this purpose.The second layer (Stratum 1) in the time synchronization system is implemented at the core layer routers. The rest of the network equipment installed at the pipeline facilities is using above sources as a clock. 12.3.4 The project should describe the time synchronization schemes for different levels of network hierarchy.

38 55

13.0 PROTECTING DTN CONTROL AND MANAGEMENT

13.1 The project should provide for mechanisms to protect management and infrastructure planes include the following technologies:

Mechanism to protect Control Plane. Mechanism to protect Data Plane. Mechanism to protect Management Plane. These mechanisms should be implemented for the following network equipment

level: Assess Distribution Core

13.2 Traffic Control PlaneThe project should traffic policing in control plane and security. Engineering solutions should be implemented through defining traffic classes and restricting rate for each class in traffic processing at the Routing Processor. In accordance with engineering solutions traffic control mechanism is implemented in the Core MPLS Routers and Distribution Routers in locations where the network transfers data. 13.3 The following traffic classes should be adjusted to control the traffic in control plane:

office traffic having dynamic routing protocols; Management traffic (SSH, Telnet, HTTP, TFTP, FTP, SYSLOG, NTP, DNS, WEB,

SNMP); malicious traffic. normal traffic; Default Traffic — the rest traffic by default.

14.0 ROUTING PROTOCOL SECURITY

14.1 BORDER GATEWAY PROTOCOL14.1.1 The following BGP protocol protection functions (without limitation) should be implemented at the stage of project designing:

bgp neighbor impersonation attack and resetting transmission control protocol; transmission control protocol session attack using internet control message protocol; bgp session theft attack; route "rocking"; route degradation; unauthorized route announcement; service rejection due to exhausted resources;

14.1.2 Implement secure BGP model including, but not limited to the following: iBGP should not contain synchronization (Internal BGP); disable «Fast external failover» function to prevent route change due to BGP

neighbor temporary disruptions in sending «keepalives communication" packets; enable registering change in BGP neighbor status; use soft reconfiguration; use authentication; disable matching BGP versions to accelerate start process; block incoming route updates from fraudulent prefixes;

39 55

use restriction on the maximum prefix number to avoid inflating Routing Table; use loopback interface for IBGP announcements; disable automatic summation prefix announcements; allow BGP neighbors establishing connection using only port TCP 179; use prefix lists to restrict announcing only selected prefixes.

14.2 Protocol OSPF v214.2.1 The designing should implement the following functions to protect OSPF protocol including, but not limited to the following:

establish MD 5 authentication in each channel; reject LSA from unknown neighbors; install external route filters in boundary routers for autonomous system (if available); provide installation of the latest versions of software with eliminated OSPF

vulnerability.

15.0 REQUIREMENT TO AAA MODEL15.1 The project should implement the model of Authentication, Authorization, and Accounting to ensure three consistent independent protection levels to be passed before a user gets controlled access to the active network equipment or network service. AAA model should provide the following functions:

authentication (a set of methods identifying remote user). user ID should be implemented at the level of selecting the user’s Login and Password.

Authorization (a set of methods to restrict the user access level to operating system installed in the active network equipment and/or network service).

Accounting (a set of methods to gather and export to remote server statistical information on user activity).

15.2 These methods should be implemented both locally and remotely within the framework of Access Control Server (ACS). The design solution should be based on a mixed method where the main user database and policies will be maintained on access control server acting as a distributed disaster-proof cluster whereas the emergency data base (in case ACS servers are not available) will be maintained locally on each device.15.3 The following event logging categories should be provided in ACS system:

making changes in acs configuration parameters. logging all events when changes are introduced into acs configuration parameters. In case a new element is added or a change is made in existing element a detailed

information is entered on modified attributes and new values of these attributes. If the new attributes were not established as a result of change query the log in the

journal to audit configuration parameters is not made; acs administrator access: registration of all events occurring since the administrator’s

access to the system and to moment the administrator logs out. also registered. The type of administrator log out is logged; it can be an explicit request or termination of session after expiry of the maximum idle time. This logbook should are record abortive events of attempted login because of the fact that account is inactive. The journal registers information on a failed login attempt, as well as the reason of login rejection;

entering changes into acs operations: registering all operations requested by administrators including acs activation from the deployment point as the main system, query about full replication, loading software, back-up copying or recovery out of back-up copies, development and recovery of pac, etc .;

changing password of internal user: logging all events dealing with password changes for internal users in all control interfaces;

40 55

in addition, the reports of the administrative and operational audit are logged with the local storage. Additionally, a configuration can be made to record these messages into remote event logs;

aaa audit should cover events of successful and unsuccessful authentication radius and tacacs +, events of successful and unsuccessful authentication of command access, events of changing passwords and answers on radius queries;

aaa diagnostics that should involve information about authentication, authorization and account for radius and tacacs + diagnostics queries, as well as attribute requests from radius and information about repository of identity data in the authentication flow. Registering these messages is not mandatory;

diagnostics system covering events of the start and completion of the work in system as well as diagnostic messages dealing with event logs:

diagnostic administration events dealing with command line interface and Web interface;

messages dealing with external server; local data base messages; the local service messages; logging these messages is not mandatory; system statistics involving information on the system performance and consumption

of resources. This information includes the resource consumption of CPU and memory as well as the process status and waiting time to process queries;

user messages such as messages on beginning and termination or change of sessions as well as messages dealing with command accounting; in addition it is possible to configure registration of these messages in the local storage.

15.4 Each registered message should contain the following information: message code - a unique message code; category of event registration - determines the category to which the registered

message belongs; severity range - determines the severity range for diagnostic messages. message class - determines groups of messages having similar content, e.g.

messages, dealing with RADIUS, policy or EAP; text messages - a brief illustrative text message in English; description - text in English containing description of reasons to register the

message, information about the way to eliminate an errors (if applicable) and reference on external resources containing more information;

failure cause (optional) - indicates whether the registered message deals cause of failure.

15.5 The project should provide a description of the licensing model on a device and the application of licenses, procedures to make an automatic backup configuration copies of ACS systems, a description of elements of policies, user roles, hierarchy of regions and network devices, a description of command shell for user roles accounting for privileges. Provide the following roles and privileges:

administrators of all system having full scope of rights and having accesses to all equipment;

regional administrators having full scope of rights and access to regional equipment; PS administrators having full rights to access PS equipment and selected sites on

the line pipe; administrators of all system having limited rights to execute commands on checking

the system functionality and accesses to all equipment;

41 55

regional administrators having restricted rights to execute commands on checking the system functionality and accesses to regional equipment;

PS administrators having restricted rights to execute commands on checking the PS hardware functionality and accesses to certain sites of the line pipe;

access to NMS management systems, the auditor rights to execute only strictly specified commands.

15.6 Arranging for secure access to hardware using SSH protocolThe project should implement a secure remote access to active ISPLS network equipment using SSH protocol. The second version should be used. To activate the secure remote access mechanism using SSH the following changes should be introduced into the configuration of the active network equipment:

activate a SSH - server and generate a 2048 bit RSA/RSA key pair; this configuration of SSH - server will feature: prohibition of using the first version of SSH protocol; the number of attempts to authenticate/re-authenticate will be increased to five; SSH negotiation timeout will be reduced to 60 seconds; Virtual terminal lines 0-15 will be accessed only through SSH;- AAA model will be indicated as a method to authenticate in these lines.

It is important that the operating system of the active network equipment is changed to have proper hostname and domain prior to the activation of SSH-server and generation of a pair of RSA keys.

16.0 REQUIREMENTS TO NETWORK CONTROL SYSTEM

16.1 The Project should provide integration of connectivity to existing and planned monitoring network and to ISPLS system control. Network monitoring and control infrastructure tools should comply with recommendations of the International Organization for Standardization - FCAPS-model to provide for the key tasks to be addressed by the monitoring system:

Fault Management; Configuration Management; Accounting Management; Performance Management; Security Management.

16.2 Composition of the monitoring and control means in the network infrastructure includes (without limitation) the following:

taking into account the system and SYSLOG protocol network logs, generation of reports on message statistics;

The performance and network hardware availability monitoring system such as Network Node Manager I (Hp);

network traffic monitoring system based on protocol Netflowsflow/Jflow & Cflow, including QOS Statistics and NBAR analysis;

accounting for AAA events such as Access Control System; Accounting for IP address space (IPAM).

16.3 All subsystems of the network monitoring and control system should have role-based access control functionality and integrate with external directories to store account data such as Active Directory, LDAP, RADIUS. The system should consist of several integrated subsystems.

42 55

Services to manage network system should be implemented on the basis of virtualization systems. The hardware of control and monitoring system should be compatible with existing hardware resources. The list of hardware will be specified at the feed survey stage. 16.4 The system should be developed in a three-tier architecture (information system architecture having three components: a database server, an application server and client applications installed on administrator workstations). The exact composition of the functional components to comprise the system should be determined after the project feed survey.16.5 Failure of one of designed subsystems should not lead to termination of the system operations; in this situation it should be possible to perform basic functions using the remaining subsystems operating in the normal mode.If possible, the software in monitoring system should operate using hardware under Linux operating system family.

16.6 Requirements to SYSLOG notification journaling systemThe project should provide for storage of the system messages generated by the operating system installed in the active network, server and other equipment as a response to changing conditions during life cycle.The concept basis should involve the client-server model where Syslog protocol RFC 5424 will be used as the main export mechanism.It is necessary to activate the mechanism to export system messages in the base configuration of the operating system:

activate the journaling event mechanism; enable the time stamp mode in exported messages; enable the numbering mode in exported messages (several messages can be

generated at once); determine the net address of the main and spare server; determine the export criticality levels.

16.7 SYSLOG storage server should provide for the following: consolidation of all events in a single database allowing for the following:

sorting messages on basis of SYSLOG message fields; Analyzing incoming messages and the ability to provide data for monitoring and

auditing in the form of reports; developing reports across the historical data and in real time; GUI; Rate-Limiting incoming messages; Supporting all operating systems and equipment involved at the project; responding to security events in real time through issuing notifications (console

messages and e - mail) recovery of the whole file system and selected files and folders; restoring access rights for users to objects in the file system; Sending alerts about events occurring in the backup system (such events as follows:

successful task completion, crash of task, detecting fault in hardware). the alerts should be delivered using various means: SNMP, e-mail, pop-up windows on

computer displays, an entry made into the System log; disaster tolerance of the system monitoring (ensuring prevention of Site, PS failure).

16.8 Requirements to accessibility monitoring and efficiency of hardwareand server equipment.

43 55

16.8.1 Monitoring the availability and performance should cover the entire scope of the network and server equipment.16.8.2 Monitoring availability and equipment failures is carried out using regular interrogation of network devices against SNMP and ICMP protocols at certain time intervals and receiving SNMP trap. 16.8.3 Network devices should support SNMPV2S, SNMPV3. The project should describe the configuration of both protocol versions.16.8.4 Monitoring equipment performance is carried out using periodic gathering statistics by SNMP protocol as well as using Netflow protocol (described in the following chapters). Performance monitoring should include the full amount of information that allows unambiguously assessing the status and load of network and server hardware components:

1. Composition of components:a. composition and status of modules:

i. chassis;ii. card.

b. composition and status of power supply;c. composition and status of CPU;d. composition and the status of RAM and non-volatile memory; e. composition and status of network interfaces:

i. L2 interfaces;ii. L3 interfaces;iii. ports.

2. the performance of components:a. utilization of network interfaces:

i. packets; ii. connections; iii. buffers;

b. utilizing CPU; c. utilizing Memory

3. for virtualized systems: the virtual objects, their availability and performance. 4. Failover and High Availability service status.

16.9 The monitoring system should provide for a graphical representation of the availability and performance of the network and server infrastructure, creating events (incidents) when the specified metrics thresholds are reached; the system should have potential of generating a notification upon the occurrence of the incident using E-mail and SMS, and it should be capable of finding root causes of failures. The system should generate customized reports reflecting trends or giving a snapshot.

16.10 Requirements to application of NETFLOW protocol

16.10.1 Operation of traffic statistical analysis protocol NETFLOWThe project design should provide for gathering statistical information Netflow to address the following issues:

Monitoring network traffic on the network core layer; Monitoring network traffic at the network distribution level; Monitoring network traffic at the network access level.

The network and application monitoring means the following: gathering Netflow metrics and data aggregation over specified periods of time; module to analyze and generate reports by equipment and interfaces monitoring

status (volume of traffic, amount of packets amount of flows, network ports (interfaces) of

44 55

traffic, TCP/UDP ports of the source/destination, compliance of TCP/UDP ports to applications and services project design, source/destination IP addresses, DSCP, bandwidth, type and code of messages for ICMP;

generation of events (incidents) at achieving specified thresholds in traffic volume, bandwidth and etc ;

the term of storing the statistical historical data should be at least 2 years.

Gathering statistical information on Netflow is made using the following network sites including, but not limited to, the following:

distribution system interfaces belonging to VRF the IQ , of DVM , SUP , IPG , OTN , RADIO , and NMS serving client network traffic;

Interfaces in network core devices connected on the side of the backup telecommunication channels.

Design solution describes the requirements to Netflow Collector, rectifies the amount and composition of information gathered, provides recommendations on choosing software to be installed in Netflow Collector.

16.10.2 Selecting NETFLOW protocol version Netflow v9 (Netflow v.9) should be used as a protocol to gather traffic statistics; otherwise, it is required to provide justification for implementing another version of Netflow protocol. Network monitoring is using both standard and advanced features (additional data fields) in Netflow v.9 protocol. Metrics of Netflow v.9 protocol is given in Table No 16-10

Table No 16-10

Metrics Note

Period Length The time over which the information is presented in a separate report

Volume - In Bytes The metrics showing the volume of incoming traffic for a particular site or interface in bytes

Volume - Out Bytes The metrics showing the volume of outbound traffic for a particular site or interface in bytes

Number of Packets (Incoming)

Shows the number of incoming packets for a particular site or component

Number of Packets (Outgoing)

It indicates the number of outbound packets for a particular site or component

Number of Flows (Incoming)

Shows the number of incoming flows for a specific site or component

Number of Flows (Outgoing)

It indicates the number of outbound flows for a specific site or component

Sample Count The number of records in the database

Flow Version Version of IP flow protocol (Netflow, Sflow, or IPFIX)

IP Protocol IP protocol used by IP-flow

45 55

Metrics Note

Class of Service Class of Service for IP-flow

Application Name The application interacting with IP-flow

Destination Port The port on the host where IP-flows terminate

Source Port IP Address IP -address of a host generating IP-flows

Destination Port IP address IP -address of host to terminate IP-flows

Source Host NameName of host site generating IP-flows. This metrics corresponds to DNS name of the source-IP-address

Destination Host Name Name of the host site to terminate IP-flows

Source Vlan ID Identifier of VLAN site generating IP-flows

Destination Vlan ID Identifier of VLAN site terminating IP-flows

Source Vlan VLAN generating IP-flows

Destination Vlan VLAN terminating IP-flows

Collector Name Name of the flow collector

Source Site Name Name of the site generating flows

Destination Site Name Name the site terminating flows

Type of Service Type of service applicable to the flowBandwidth - In Megabits per Second (Average, Maximum, and Minimum)

The amount of traffic coming to certain sites per second

Bandwidth - Out Megabits per Second (Average, Maximum, and Minimum)

The volume of traffic leaving specific sites per second

Bandwidth Utilization (Average, Maximum, and Minimum)

The volume of incoming and outgoing traffic passing through certain sites or interfaces (calculated for a specified period of time)

DSCP The field in IP packet header used to classify information transmitted

16.11 Gathering and analyzing Netflow statistics using QoS

Project solution should provide for and describe the possibility of using netflow as a way to control the accuracy in QoS labeling. The netflow collector system should allow generating QoS reports. The project should describe the content of these reports and their use in estimating QoS performance.

46 55

16.12 The requirement on configuration management and hardware configuration images

The system to control and store network equipment configurations should provide for the following functionality (without limitations):

backup and storage of network equipment configurations; backup and storage of software images (IOS, firmware) installed in network

equipment; tracking lifecycle of network equipment software versions, indication of End-of-Sale,

End-of-Life, End-of-Support; track configuration change in network equipment both on a regular basis (against a

schedule) and on occurrence of registering events of changing configuration in network equipment;

verification of network equipment configuration with predetermined templates (the templates should be developed in line with the best practices of network equipment manufacturers);

version control in configurations of network equipment; support of the whole scope of network equipment; generating reports on changes in network equipment configuration visualizing the

changes introduced

16.13 Requirements to the system to control and account for IP addresses

The control and accounting IP address space system should provide for the following functionality:

accounting for allocation of IP addresses; keeping track of statistics of using IP address including, but not limited to the

following: tracking the availability of assigned IP addresses; accounting for information on the date, IP address, hostname, term, basis and

purpose of issuing IP address; accounting for personalized information about who issued and who obtained IP

address assigned; maintaining the availability statistics; the possibility of IP addresses grouping on a functional purpose, territorial jurisdiction

and other parameter basis. generating information on zone changes for DNS server to account for the changes

in IP addressing plan; capabilities of making a global search for IP addresses and mapping information

obtained; displaying statistics on the use of IP addresses in the context of sub-networks; compiling reports on statistics gathered.

16.14 Requirements on integrity of information during incidents and reliability of the system

An option should be provided to have both automatic and manual data backup using tools of developed backup system. The backup system should ensure the following:

full recovery of all services in ISPLS network for any crash using a Recovery Point Objective, RPO no later than in 24 hours;

a capability of using one device to store back-up copies made at different network sites;

centralized operations planning; verification and data compression. Regular checking the integrity of the backup

copies;

47 55

integration with the virus protection system; support to develop the backup copy system using technology SAN (Storage Area

Network); an option should be provided to restore the system using back up device connected

to another server; an option of recovery both the whole file system and separate files and folders; an option to restore user access rights to the file system; the backup system should be able of sending messages about events. Such events

can be as follows: successful completion of the task, the task crashed, and equipment fault detection. The massage should be delivered using various means such as: SNMP, e-mail, pop-up window on computer display, an entry made into the System log;

disaster recovery in monitoring systems; Backup and service recovery should be carried out in accordance with the guidelines of the software manufacturer, manufacturer of equipment and ensure the integrity of the backup data.

16.15 The system should be capable of operating continuously for 24 hours a day, 7 days a week. It is prohibited to stop monitoring subsystem for disabling for operating system maintenance or maintenance of monitoring services. The equipment used in the monitoring and control systems should be maintainable without service interruption.

16.16 Design features of software

"Contractor" is obliged to supply systems and applications proven under operating conditions and requiring only the project-specific configuration."Contractor" should generate the list of new software modules or applications to be developed for the project as well as to develop design parameters for software that provide the legend, proposed architecture of networks, interfaces and other engineering aspects of modules as well as the processes to develop, test, validate, analyze and approve.

16.17 Back-up capabilities of the system, spare parts and warranty

Network management system should be supplied with the following minimum back up capacities. All servers should be delivered having the following:

50% back-up of installed RAM; 50% back-up of installed ROM (data storage system); 50% of free input - output ports; processors having at least 60% of reserve capacities after downloading and

launching all applications even during peak loading time; hardware capacity to install additional 50% memory resources to be used by the

system; hardware capacity to install additional 50% memory resources to be used by the

system;

16.18 Software operating systems on servers and workstations, databases, and network management services should provide for software supply having unlimited licenses to operate additional network elements allowing scaling up the functional capabilities. In order to meet the requirements of reliability (server and network equipment) a set of spare parts tools and accessories (SPTA) should be developed. The project should provide for developing specifications to maintain SPTA based on regional location of the Customer storage facilities.

48 55

16.19 Standardization

Contractor should describe in project documentation the reasonable level of harmonization and standardization of all hardware and software designed taking into account achieving the optimal limit at which its further increase will have a negative impact on the engineering and economic performance of ISPLS network. Together with the tender bid Contractor should provide a detailed specification for products, equipment and materials where standardization applies, the pump station equipment and the software for it based on the optimal level of standardization of 65% providing the feasibility study to implement the solution proposed.

17.0 REQUIREMENTS TO HARDWARE AND VIRTUALIZATION SYSTEM17.1 As ISPLS network will contain a large amount of transmitted and stored information the subsystems to store the data should be also designed. The computing infrastructure (CI) should be a set of interrelated subsystems providing solution of the following functional operations in ISPLS network:

storing the data of the monitoring system; storing the backup copies of virtual machines, DBMS and application data; storing video archives for sufficiently long time.

17.2 The equipment should be connected to an uninterruptible power supply (UPS); it should be controlled using IP-KVM console (having a monitor and track-ball keyboard); it should also have a built-in processor/remote control interface.17.3 Requirements to hardware on the whole depend on requirements to subsystem hardware as described by manufacturer accounting for reliability and redundancy capabilities of the system.17.4 All components of computing infrastructure should be duplicated; a failure in any component should not lead to malfunction or a block in data access. CI should have a modular architecture without a single point of failure (NSPOF). All service operations, including replacement of any active components and adding new CI, should be carried out without interrupting or shut down in CI operations. The composition of each subsystem should comprise a set of software and hardware providing reliable subsystem operations.17.5 All kinds of interaction between the system components should be carried out according to standard physical interfaces, standard protocols and standard data transfer software interfaces as set out in specifications IEEE 802.3 and IEEE 802.2.17.6 CI should consist of three hierarchical levels, each of which is divided into the following subsystems:The data storage level: the data storage subsystem (SS) is designed for failover storage of all ISPLS system data;Network Level Storage Area: is the network storage subsystem (SSS) designed to provide transmission of switchable data between servers and the data storage system.

Performance level:

the server complex subsystem (SCS) designed to provide hardware resources to virtual machines;

virtualization subsystem (VS) is designed to ensure using SCS hardware resources in ISPLS systems.

17.7 Requirements to storage device sub system17.7.1 Data storage subsystem is designed to store, use, backup and restore data.

49 55

17.7.2 The design solution should provide for availability of fault-tolerant disk arrays, one for each process area, MCC and BCC in the storage subsystem (SS); SS should be also available at the sites having dense traffic from surveillance systems.17.7.3 The data storage system should have the following capabilities:

centralized, efficient, and secure data storage; boosting the rate of processing operational information; system expansion in accordance with growing volume of production data and

requirements to productivity and back up capacity of the data storage system without I/O shut down;

Increasing system performance through adding various DSS components without interruption in data accessibility;

the most effectively use of space in racks; supporting expansion of the size of the file system have integrated facilities to access performance

17.7.4 Requirements to SDSS:1) DSSS should support multi protocol access: the technology of Storage Area Network

(SAN - FC; FCoE; iSCSI) and network storage technology NAS (Network Attached Storage - NFS; pNFS; CIFS/SMB);

2) DSSS should be equipped with input output interfaces (no less than 4 of each type): UTA2 (Ethernet 10 Gbps/FCoE, Fibre Channel 16 Gbps), Ethernet 10 Gbps, Ethernet 1 Gbps, SAS 6 Gbps, body type suitable to install in 19 inch cabinet. Expandable IO ports of each type.

3) Configuration support: Active - Active type controller supporting emergency switching and high-availability

DSS having alternate ways to transfer data; controller in Active-Active Metrocluster configuration; two clusters having two switchable or unswitchable sites and switchable FAS DSS;

the total of four sites: cluster from two sites having Active-Active type per site.4) Support of RAID-DP technology.5) Supported operating systems: Windows® 2000, Windows Server® 2003, Windows

Server 2008, Windows Server 2012, Windows XP, Linux®, Oracle® Solaris, AIX, HP-UX, Mac® OS, VMware®, ESX®

6) Snapshot support: no less than 255 000 per controller, integration of snapshot and copy functionality in operating systems and application servers

7) Ability to install Module to improve performance on the basis of Flash Cache technology.

8) Support of Flash Pool technology.9) The initial data storage volume should be determined based on results of FEED survey

to be agreed with the Customer in line with requirements of the services available in ISPLS network.

10) Support quotas on disk space usage for users, groups and specific catalogs;11) Provide for balancing of client connections (using either built-in DSS on the basis of

data on site/controller loading or using means of active network equipment);12) The array should support use of sites having disks of different types (SATA, SAS,

SSD) in one SSD, support of transparent migration of files between them based on customizable policies (hierarchical, tiered storage);

13) The array should maintain the ability to replicate the data between SSD (to implement remote backup);

14) The data storage system should be supplied with a Thing Provisioning (economic allocation of disk space) function and have a license to cover full scope of proposed functionality.

50 55

15) The data storage system should have the remote replication functionality in synchronous and asynchronous mode.

16) The data storage system should support the ability to connect external storage arrays made by other manufacturers and use their (functional) disk capacity without mounting additional equipment;

17) DSS should support Virtual Volumes Storage technology.18) The storage system should have built-in tools to measure performance;

17.7.5 The system should provide for the following compatibility: compatibility with ISPLS DSSS currently used in the network: NetApp FAS 2240-2,

and OS DataONTAP 8.2. DSS should allow connecting the existing DSS with potential virtualization and use of disk capacity in existing DSS as one pool with supplied DSS;

supporting standard back up protocol NDMP; supporting protocols and technology of virtualization (API) for virtualization systems

VAAI,VASA, Site Recovery Manager. The project should describe procedures to update last versions of DSS means in

equipment used

17.8 Requirements to network data storage subsystem

17.8.1 PS connected to ISPLS network are divided into the following types:- Main Control Centre (Onshore facilities in Novorossiisk)- Backup Control Centre, PS Kropotkinskaya- There are two PS in each region (to be determined by the results of FEED survey) for which

fault-tolerant connections of ISPLS system regional storage servers using virtualization and storage solution are designed.

- Regular PS where deployment of virtualization and storage systems is not planned

17.8.2General topology to layout wiring in monitoring systems and MCC ISPLS network is shown in the figure for information

51 55

Существующая сеть

TOPiLO

UID

4 1

A

B

1

2

DC AC

x22

DC AC

x22

87642

531

SID

2 3 41UID

PLAYER

HPProL ia nt

DL580 G7

TOP

TOP

PS 1

PS 2

PS 3

PS 4

10 9 811 7 6 5 4 3 2 1

iLO 3

12

34

UID

1200W

1200W

FAS2240-2FAS2240-2

LNK LNK

0a0b

e 0a

e 0b

e0 c

e0d

LNK LNK

0a0b

e0a

e0b

e 0c

e0dLNK LNK

1a 1b

LNK LN K

1a 1b

73625140 15111410139128 2319221821172016HP StorageWorks 8/24 SAN Switch

NC364 TNC364 TAJ762AAJ762 A

ProLiantDL360p

Gen8

UIDSID

3

4

1

2

5

6 7 8

460WPLC B

94 %

460W

AJ76

2A

87642

531

SID

2 3 41UID

PLAYER

HPProL iant

DL580 G7

TOP

TOP

PS 1

PS 2

PS 3

PS 4

10 9 811 7 6 5 4 3 2 1

iLO 3

12

34

UID

1200W

1200W

NC 36 4TNC 364TAJ 76 2AAJ762A

HPSto rageW orks

MSL4 048Tap e

Li brary

Ready Clean Att ent ion Error

OK

Port A Por t B

LTO3280 Fibre

73625140 15111410139128 2319221821172016HP StorageWorks 8/24 SAN Switch

STATUS

WS- X6524-100FX -MM

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 241

LINK LINK LINK LINK LINK LI NK LI NK LI NK LI NK LINK LINK LINK LINK LINK LI NK LI NK LI NK LI NK LINK LINK LINK LINK LINK LINK

24 PORT 100FX-MMF

STATUS

WS- X6524-100FX -MM

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 241

LINK LINK LINK LINK LINK LI NK LI NK LI NK LI NK LINK LINK LINK LINK LINK LI NK LI NK LI NK LI NK LINK LINK LINK LINK LINK LINK

24 PORT 100FX-MMF

Front

Rear

Front

Rear

Rear

Front

Rear

Front

Rear Front DL360p Gen8 8-SFF

NetApp 2240-2Ethernet1G service1G vMotion1G Management

Fibre Channel8G FibreChannel

AJ762A

The overall topology of proposed integration of MCC and BCC into ISPLS network is shown below for information

FA N STAT

FA N STAT FAN1

FAN2

PS1

PS2

FA ILOK

FA ILOK

FAN1

FAN2

CON SOLEL2

MGMT 0L1

STAT

ID

3

CISCO NEXUS N5548UP

S TAT

ID

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

Существующая сеть

TOPiLO

UID

4 1

A

B

1

2

DC AC

x22

DC AC

x22

87642

531

SID

2 3 41UID

PLAYER

HPPro Lia n t

DL 5 80 G7

TOP

TOP

PS 1

PS 2

PS 3

PS 4

10 9 811 7 6 5 4 3 2 1

iLO 3

12

34

UID

1200W

1200W

FAS2240-2FAS2240-2

LNK LNK

0a0b

e0a

e0b

e0c

e0d

LNK LNK

0a0b

e0a

e0b

e0c

e 0d

NC 364 TNC 364 T

ProLiantDL360pGen8

UIDSID

3

4

1

2

5

6 7 8

460 WPLC B

94 %

460 W

AJ76

2A

87642

531

SID

2 3 41UID

PLAYER

HPPr oL ia nt

DL 5 80 G7

TOP

TOP

PS 1

PS 2

PS 3

PS 4

10 9 811 7 6 5 4 3 2 1

iLO 3

12

34

UID

1200W

1200W

NC 364 TNC 364 T

HPSto ra geW orks

MSL 404 8Tap e

Library

Read y Cle an Atte ntion Erro r

OK

Po rt A Port B

LTO3280 Fibre

STATUS

WS-X6524- 100FX-MM

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 241

LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK

24 PORT 100FX-MMF

STATUS

WS-X6524- 100FX-MM

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 241

LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK

24 PORT 100FX-MMF

DL 580 G7Front

Rear

Front

Rear

Rear

Front

Rear

Front

Rear Front DL360p Gen8 8-SFF

NetApp 2240-2

HP MSL 4048

HP 8/24 SAN Switch

FA N STAT

FA N STAT FAN1

FAN2

PS1

PS2

F AI LOK

FA ILOK

FAN1

FAN2

CON SOLEL 2

MGMT 0L 1

STAT

ID

3

CISCO NEXUS N 5548UP

ST AT

ID

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

PORT 1

10Gb ESFP

PORT 2L A

L A

TX

RX

TX

RX

CN 110 0e

PORT 1

10Gb ESFP

PORT 2L A

L A

TX

RX

TX

RX

CN 110 0e

PORT 1

10Gb ESFP

PORT 2L A

L A

TX

RX

TX

RX

CN 110 0e

PORT 1

10Gb ESFP

PORT 2L A

L A

TX

RX

TX

RX

CN 110 0e

LNK LNK

e1a e1b

LNK LNK

e1a e1b

STATUS

WS-X6524-100FX-MM

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 241

LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK

24 PORT 100FX-MMF

STATUS

WS-X6524-100FX-MM

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 241

LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK

24 PORT 100FX-MMF

ГЦУ

Ethernet1G service1G vMotion1G Management10G Ethernet

Fibre Channel8G FibreChannel

FA N S TA T

FA N S TA T FAN1

FAN2

PS1

PS2

FA IL

OK

FA IL

OK

FAN1

FAN2

CON SOLEL2

MGMT 0L1

STAT

ID

3

CISCO NEXUS N5548UP

S TAT

ID

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

A

B

1

2

DC AC

x22

DC AC

x22

87642

531

SID

2 3 41UID

PLAYER

HPPro Lia n t

DL 5 80 G7

TOP

TOP

PS 1

PS 2

PS 3

PS 4

10 9 811 7 6 5 4 3 2 1

iLO 3

12

34

UID

12 00W

12 00W

FAS2240-2FAS2240-2

LNK LNK

0a0b

e0a

e0b

e0c

e0d

LNK LNK

0a0b

e0 a

e0 b

e0c

e 0d

NC 364 TNC 364 T

87642

531

SID

2 3 41UID

PLAYER

HPPro L ian t

DL5 80 G7

TOP

TOP

PS 1

PS 2

PS 3

PS 4

10 9 811 7 6 5 4 3 2 1

iLO 3

12

34

UID

12 00W

12 00W

NC 364 TNC 364 T

DL 580 G7Front

Rear

Front

Rear

Rear

Front NetApp 2240-2

HP 8/24 SAN Switch

FA N S TA T

FA N S TA T FAN1

FAN2

PS1

PS2

F AI L

OK

FA IL

OK

FAN1

FAN2

C ON SOL EL 2

MGMT 0L 1

S TA T

ID

3

CISCO NEXUS N 5548UP

ST AT

ID

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

PORT 1

10Gb ESFP

PORT 2L A

L A

TX

RX

TX

RX

CN 110 0e

PORT 1

10Gb ESFP

PORT 2L A

L A

TX

RX

TX

RX

CN 110 0e

PORT 1

10Gb ESFP

PORT 2L A

L A

TX

RX

TX

RX

CN 110 0e

PORT 1

10Gb ESFP

PORT 2L A

L A

TX

RX

TX

RX

CN 110 0e

LNK LNK

e1a e1b

LNK LNK

e1a e1b

РЦУ

Project work should include rectifying above network connection architectures, developing existing configurations for network equipment and finding ways to integrate it into proposed design.

17.8.3 The contractor conducts research to determine and substantiate the optimal SAN connectivity designs. The analysis of proposed design should have the following provisions:- consolidation- performance- security- connectivity

52 55

- reliability- manageability- scalability- capacity

Design solution should contain process justification to use proposed SAN technologies. The process solution should monitor the major trends in industry and be up to date for at least 7 years after deployment.Each SSSD should be developed in a manner to provide for future flexible scalability of the storage network resources without compromising requirements on performance and availability. The further expansion of SSSD hardware will be carried out according to Core-Edge (Star) topology.The project should describe the criteria to select the main protocol in the data storage network.

17.8.4 The data storage network should be kept separate from the data transmission network.Requirements to network data storage subsystem are as follows:

availability of mechanisms to protect from locking to provide parallel work in all switch ports;

support of functions such as VSAN & IVR, NPV, Smart Zonning, FCoE, PortChannel,QoS, protection of SAN, SAN control, hardware zoning and software update during operations (ISSU);

support of protocols Fibre Channel having speed no lower than 16 Gb/s.17.8.5 SSSD should provide for data replication between MCC and geographically remote PS to be chosen at design stage.The project should provide for replication of data between repositories hosted in different sites using MetroCluster technology. The solution should provide for network infrastructure failover processing using the following algorithm:- in synchronous regime - by default- if a failure occurs the replication is switched into an asynchronous mode until telecommunication channel is restored.- After the restoration of telecommunication the synchronous mode should be able to recover

17.9 Requirements to server complex subsystem

17.9.1 Server complex Subsystem (SCS) is designed to support all system and application software ensuring ISPLS network services in CI.The equipment based on h86_64 architecture using VmWare virtualization will be used as a hardware (server). Servers should have the form factor the choice of which is based on maximizing the density of computing resources and consolidation in one design server subsystems such as power supplies and fans, as well as infrastructure Ethernet switches, FC and etc. Physically servers are installed in 19" server telecom cabinets. SCSS components should comply with requirements to be compatible with other CI subsystems. 17.9.2 SCSS should be equipped with hardware having adequate capacity to ensure providing ISPLS network services and feature the redundant capacity as required in this document.17.9.3 Subsystem should ensure implementation of the following features:

integrity, unification and compatibility of implemented design, engineering and technological solutions;

53 55

open development architecture; providing standard interfaces and protocols; providing centralized administration; security capabilities to set up round the clock service of service equipment; support for the stack of TCP/IP network protocols; providing for high availability and scalability of services for server components; no single point of failure; availability of in-built solution to manage servers; compatibility with existing server hardware in ISPLS HP Proliant network.

17.9.4 The server complex should be a modular system consisting of a server chassis, servers and switch modules.The servers should be placed in a single chassis.The chassis should use active intelligent fans having airflow optimization, acoustics and power consumption algorithms.The chassis should have a dynamic redistribution of load on the power supply.Power supply and chassis fans should be N + N redundant.Control modules should allow for local control using built-in display combined with the chassis management software.Control modules should support remote control functionality.The chassis should be capable of giving visual indication of failures and warning messages.The chassis should have the software of chassis control on a single media storage including the following: server monitoring and management features, the functions of control guarantee and service contract, BIOS and server driver updates, remote management of virtual KVM functions and virtual medium, rapid deployment features of OS servers from images or from scripts, security functions, vulnerability scanning and patch updates, functions of measurement and control of chassis power supply, the combined control functions to control both physical and virtual servers installed on the chassis.The chassis control software should have the preliminary state monitoring function for at least three components of each server: CPU, memory and hard drives. Replacing components in pre-emergency conditions should be carried out during the warranty period on the server.The amount of servers, efficiency of CPU in each server should comply with requirements set forth to application systems.The amount of memory in each server module should be sufficient to meet requirements of application systems.Each server module should be equipped with a sufficient number of Fibre Channel and Ethernet adapters failover configuration.Each server module should be equipped with two built-in Ethernet server adapters having data rate of at least 1 Gbps; each adapter should support iSCSI protocol; the controller should support it and support Ethernet capability to relieve CPU of processing TCP/IP computing operations (should have a TCP/IP offload engine).The design should be based on using operating systems consistent with requirements set forth to applied software.Servers should be equipped with means to virtualize computing resources;The means to virtualize computing resources should ensure minimizing downtime when hardware failure occurs;Virtualization means of computing resources should be able to ensure the upgrading the data center to provide for disaster recovery;Virtualization of computing resources should ensure even distribution of the load on CPU in virtualized servers;

54 55

Virtualization of computing resources should allow the virtual machine migration from one physical server to another without restarting the guest OS;In case of failure of a physical server, virtualization means of computational resources should provide for ability to start automatically virtual machines installed on the failed server on the other physical server;Virtualization of computing resources should provide for setting to block starting two specific virtual machines on the same physical server;Virtualization of computing resources should include standard components to interact with the backup system;Virtualization of computing resources should provide for the scalability to the following: increasing the number of physical servers without discontinuing operations in the virtual machine environment;Virtualization of computing resources should ensure operations in the virtual computing environments in case server RAM is less than required to operate the media.Resource planning should minimize the number of chassis.The bid proposal should provide servers required to replace critical obsolete servers as well as a dedicated backup and recovery server.

17.9.5 The concept of virtualization subsystemThe stage of designing the virtualization subsystem (VS) should involve following the principle of combining multiple virtual servers on a single physical server; each server should have necessary isolated resources to optimize the use of hardware resources and to save power consumed by the servers.The system should have the following key features:

Virtualization of x86 server resources and their grouping into logical pools that can be allocated with several operational loads;

abstraction of server hardware resources and ensuring their joint use by VM; Centralized initialization, administration and monitoring the network due to

association of the network at the data processing center level; intellectual placement of VM and load balancing on basis of delay in I/O and storage

capacity; high-performance cluster file system optimized for work with VM.

Basic requirements to be met by VS: compatibility with existing system of centralized management VcenterServer. capability of use in VM the disk number to be multiplied as the data volume grows. Availability of Failover VM facilities allowing automatic restart VM from the common

repository in case the physical host server fails. capability of moving VM between physical servers without interrupting its work and

network connections; ability of Hot adding/removing VM devices during operations (processors and RAM

memory); Availability of facilities to ensure virtual infrastructure network security including

control over different types of traffic; Availability of facilities to ensure permanent VM availability which allows keeping a

backup VM copy running on another server to undertake loading instantly in case the main machine fails;

capacity of dynamic relocation of VM storage between arrays without interrupting applications in guest OS;

55 55

Availability of facilities to balance the load on host servers through dynamic migration as well as means to reduce power consumption by bringing idle servers into a standby mode;

Availability of technologies to prioritize VM access to storage, allowing to guarantee service levels for applications on input-output;

Availability of technology to prioritize the traffic of different type within network adapters;

central management means and configuration in host network interaction (Virtual Distributed Switch, Vdistributed Switch );

automatic deployment - deployment of multiple hosts on the fly; load balancing on volumes based on performance parameters of the disk system; developing multiple tiers of storage having own performance characteristics; Compatibility with virtualization means currently used in ISPLS network: Vmware

Vsphere Enterprise Plus. Parameters of virtual machines, their composition, and operating services should be

determined at the stage of FEED survey

SCC VI should ensure compatibility of ISPLS HP Proliant server hardware currently installed in the network

Parameters of virtual machines, their composition, operating services should be determined at the stage FEED survey.

18.0 REQUIREMENTS TO QUALIFICATION AND WORK MANAGEMENT

18.1 Contractor should carry out the work without involving contractors. 18.2 Contractor should involve experts having the following qualifications:

a university degree in natural sciences, IT, telecommunication disciplines; valid Cisco Juniper VmWare Netapp certificates of Professional level and higher or

similar certificates issued by OEM companies supplying equipment for the project; having work experience in consulting and designing no less than 3 years; having a portfolio of projects accomplished indicating contacts or providing feedback

letters on the quality of project work.

18.3 Contractor provides copies of all documents certifying the qualifications, education, and confirming design during the tendering procedures.18.4 Contractor cannot change head design engineers without consent of the Customer.18.5 Customer has the right to discharge any design engineer during project implementation activities without giving any grounds.18.6 Contractor is obliged to replace design engineers without impact on the project implementation time lines.18.7 Contractor is providing a report on the work progress at least once every 2 weeks.18.8 Production meetings to discuss the project progress are held at least once a month.18.9 Contractor ensures generating minutes of meetings and compiles the registry of e-mail correspondence on the project to be delivered to the Customer project completion.Composition and scope of work to develop the system and work stages should comply with GOST 34 and should be defined in the front end engineering survey.18.10 Contractor should develop a program of Customer staff development to train on technology and equipment used in the project and provide for training of three telecommunications engineers.