· web viewrd 34.21.122-87 guidelines on setting up lightning protection in buildings and...

of 74/74
1 74 CJSC CASPIAN PIPELINE CONSORTIUM AGREED APPROVE CPC Security department CPC Telecommunication head ________________________ _______________ K.I. Savchenko "_____"___________ 2016 "_____"_______________ 2016 TECHNOLOGY DEPARTMENT TELECOMMUNICATION GROUP TERMS OF REFERENCE No 01-16/02 To design multiservice information network to ensure security in CPC oil pipeline system

Post on 31-Mar-2018




0 download

Embed Size (px)





CPC Security department

CPC Telecommunication head


_______________ K.I. Savchenko

"_____"___________ 2016

"_____"_______________ 2016




To design multiservice information network to ensure security in CPC oil pipeline system























Autonomous System


Border Gateway Protocol


STP Protocol Frame


Cisco Discovery Protocol


Customer Edge


Control Plane Policing


Class of service


Differentiated Services Code Point


Ethernet over MPLS


Data Transmission Packet Technology


Hot Standby Router Protocol


Interior Gateway Protocol


Internet Protocol


Layer 2


Layer 3


Link Aggregation Control Protocol


Local Area Network


Label Discovery Protocol


Label Forwarding Information Base


Label-switched path


Media Access Control


Message Digest 5


Multichassis Ether-Channel


Multi-protocol Border Gateway Protocol


MultiProtocol Label Switching


Marine Terminal


Maximum Transmission Unit


Non-stop Forwarding


Network Time Protocol


Open Systems Interconnection


Open Shortest Path First


Port Aggregation Protocol


Provider Edge


Policy Feature Card


Private VLAN


Quality of Service


Route Distinguisher


Random Early Detection


Resilient Ethernet Protocol


Request for Comments


Rapid Spanning-Tree Protocol


Synchronous Digital Hierarchy


Shore Facility


Simple Network Management Protocol


Algorithm Shortest Path First


Spanning-Tree Protocol


Switched Virtual Interface


Transmission Control Protocol


Tank Farm


User Datagram Protocol


Universal Serial Bus


Virtual Local Area Network


Virtual Private Network


Virtual Routing and Forwarding


Wide Area Network


Autonomous System


Outband Management Block


Future Pump Station


Main Control Center


Caspian Pipeline Consortium


Main Line Block Valve


Marine Terminal


Pump Station


Tank Farm


Back up Control Center


Integrated System of Pipeline Security


Operating System


Computation Infrastructure


Virtualization Subsystem


2.1 The Terms of Reference herein set out the requirements to design CPC secure multiservice information network specifying the composition and scope of design documentation.

2.2 The telecommunications equipment used in CPC security network is currently out of production. CPC is planning a step by step network upgrading and replacing failed components on the basis of developed project design to provide for continuity of operations in security systems.


The documents listed are an inherent part of technical requirements herein. Except for the changes introduced, the requirements provided in this document suggest using the latest editions of each regulation with addenda in effect on the date of execution of the contract. Work programs covered by the technical requirements herein should be conducted meeting the requirements set forth in applicable provisions of referred documents published (or referred to equivalent in Russian or Kazakhstan standards).

3.1Electronic Industries Association.

TIA/EIA-TSB67 Technical specifications for the data transfer channels under operational testing cables made of non-shielded twisted-pair wires.

TIA/EIA-TSB72 Guiding instructions on developing centralized optic fiber cable networks.

TIA/EIA-568-A TIA/EIA-568-B Standard on telecommunication cable lines to be developed inside industrial buildings.

TIA/EIA-569-A TIA/EIA-569-B Standard on routs and areas to develop telecommunication lines inside industrial buildings.

EIA/TIA570 Standard on telecommunication wire lines to be developed in communal areas and zones having industrial development.

TIA/EIA-606 Standard to set up a telecommunication infrastructure and data transfer in industrial buildings.

TIA/EIA-607 Requirements to earthing and connection of equipment casing inside industrial buildings.

TIA/EIA-TSB75 Methods to lay additional cables horizontally inside working areas having open space design.

3.2Institute of Electrical and Electronics Engineers.

IEEE 802 Local and metropolitan area networks - General provisions and architecture.

IEEE 802.7 Recommended methods to develop broadband local networks.

IEEE 802.8 Engineering consulting group on fiber optics.

3.3ETL testing laboratories.

Package of documents to design cable telecommunication systems to be submitted to ETL

ISO/I EC 11801 14763-1 14763-2 TIA/EIA568A 569A 607 Standard to lay cables inside buildings.

CENELEC EN-50173 European technical requirements to cables.

Recommendation IEC 61754. Fiber Optic Connector Interfaces, IEC 61755-1. Fiber optic connector optical interfaces.

Recommendation IPC -8497-01 (Optoelectronic Assembly and Packaging Technology) Cleaning Methods and Contamination Assessment for Optical Assembly.

3.4GOST, RD, CPC policies.

The documentation pertinent to the project design work to develop SCS and the work itself is carried out in line with requirements set out in GOST series 34 and GOST series 19.

GOST 12.1.030-81 SSBT. Electric safety. Protective earthing, and neutralling.

GOST 464-79 Grounding for the stationary installations of wire telecommunications, repeating stations, wire broadcasting centers and community antenna television systems. Resistance norms.

GOST R 50571.2-94 Indoor electric installations. Part 3. Major parameters.

GOST R 50571.3-94 Indoor electric installations. Part 4. Safety requirements. Electric shock protection systems.

GOST R 50571.10-96 Indoor electric installations. Part 5. Selection and assembly electric equipment. Part 54. Grounding electrode systems and grounding conductors.

GOST 21.110-95 Rules to develop equipment, item and material specifications.

OST 45.88-96 Departmental standardization system. Sequence to develop departmental guiding documents.

OSTN-600-93 Departmental development of engineering norms on structure assembly and setting up telecommunication, radio broadcasting and television broadcasting means.

VSN-332-93 Guidelines on designing electric installation in plants and structures to be used in electric communications, wire telecommunications, radio broadcasting and television broadcasting.

VSN 600-IV-87 Safety rules to follow in assembly of process equipment and electric supply.

VSN 332-93 Guidelines on designing electric installations in plants and structures to be used in electric communications, wire telecommunications, radio broadcasting and television broadcasting.

Text part of the documentation should be developed in line with requirements set forth in RD 50-34.698.90 Information Technology. Methodological guidelines. Set of standards applied to automation systems. Automated systems. Requirements to document composition.

RD CPC 107.03.2015 Specifications to structured cable network systems.

RD 34.21.122-87 Guidelines on setting up lightning protection in buildings and structures

RD 45.091.195-90 Guidelines on designing electric telecommunication complexes. General requirements and norms on equipment, cable and metal structure earthing.

RD CPC Guidelines on operating CPC fiber-optic line.

RD 45.190-2001 Departmental guiding document Element section of fiber-optic line to transfer data Standard user acceptance test program.

RD 45.155-2000 Departmental guiding document Grounding and potential bonding in fiber-optic line equipment installed at wire telecommunications facilities

Rules to apply fiber-optic cables, passive optics devices and gear to fuse-weld optics fibers. Instruction of the Ministry of Information and Telecommunications No 46 dated April 19, 2006

Standards NIST, FSTEC


Customer - CJSC Caspian Pipeline Consortium-R.

Contractor - a company selected to undertake project designing.

ISPLS network - integrated system of CPC secure pipeline network.




1. Project designing and consulting should be done by a company having license and/or other legally established permitting documents to engage into this activity, having work record no shorter than 3 years in this specific area.

0. The Contractor should make an obligatory front end engineering survey involving the visit to the job site of CPC MCC in Novorossiisk through getting permits to access the customs area and preliminary issuing permits, passes and undergoing safety orientations in compliance with norms in effect at the moment of making plans to visit CPC facilities.

0. The scope of front end engineering survey should be sufficient to obtain all required and sufficient information to ensure high quality of design work, to clarify all issues dealing with project implementation.

0. Based on results of the front end engineering survey a certificate should be executed on making survey of facilities to be designed bearing the signature of a responsible engineer from CPC telecommunications group outlining the information gathered.

0. If it is required to conduct additional research in the course of design work these activities will be carried out by the staff of the project design organization without escalating the contract value.

1. During front end engineering survey Contractor should become familiar (without limitation) with the following list of materials and documents:

0. Topological architectural ISPLS network layout;

0. ISPLS network address table;

0. Composition of active network equipment at the network core layer;

0. Composition of active network equipment at the distribution level;

0. Composition of active network equipment at the access level;

0. Composition of server equipment in DVM video surveillance systems;

0. Composition of IQ system to access control server equipment;

0. Composition of EBI server equipment ;

0. ISPLS systems interaction logic;

0. Composition and logic of voice telephone equipment;

0. Composition and logic of ISPLS network monitoring systems and planned expansion;

0. Configuration of all network equipment;

0. Development of existing equipment configurations that are planned to connect to MCC virtualization storage systems;

0. Results of executing verification commands;

0. MDS for the fiber-optic cable line;

0. Parameters of virtual machines, composition, and available services;

1. Requirements to the work scope.

5.3.1 Contractor undertakes the following scope of work on designing and developing process solutions, including (but not limited to) the following:

Front end survey of ISPLS network to be made at the "Customers" facilities;

Detailed analysis of Marine Terminal network configuration (kp 1504) - PS 8 - PS 7 (kp 1353 PS Kropotkin (kp 1237) PS 5 (kp 1138);

Issuing recommendations on configuration of network segments as a separate document;

Provides the functional specification and serves an explanatory note describing design solutions and underlying analysis;

Provides network architectures of PS to be debottlenecked and line pipe shelters indicating the physical circuit to connect equipment;

Provides functional diagrams indicating the logical network interactions;

Develops a migration plan or the program to replace off-spec components with new equipment specifying milestones and risks;

Develops a template configuration of network devices consistent with proposed design;

Carries out standard procedures and methods of pilot tests, factory tests, and acceptance tests;

Makes calculations and provides estimates on power consumption of proposed equipment;

Conducts network analysis accounting for convergence of routing and signaling protocols;

Develops and informs the Customer of the safe rules to handle active equipment proposed;

Compiles the new address plan for the network upgraded;

Makes calculations of the budget of the optics telecommunication lines to connect new equipment;

Offers options to replace the equipment on the basis set forth in draft proposal followed by generating cost estimates on equipment to be installed

Develops a program to train three Customer telecommunication engineers in a training center certified by the manufacturing company

Makes the feasibility study of design solutions




5.3.2 Taking the requirements to operate, maintain, repair and store the equipment as set out in the documentation of the manufacturer and in guiding CPC documents as a base, Contractor develops the following (without limitation):

Guidelines for system user;

Guidelines for system administrator;

Procedures on regular system maintenance;

Procedures on emergency system maintenance;

Guidelines on subsystem installation;

Guidelines for back up copying and data recovery.


6.1 Information on existing backbone telecommunication network

The existing backbone optics telecommunication network is developed in section 1504 - 204 km using Siemens A-DF (ZN) 6x6E9/125 0.36F3.5 + 0.21H18 LG cable.

The fiber optics cable has a capacity of 36 .

The manufacturing company is Siemens.

A schematic diagram of cable cross-section is given in Fig. 1.

Figure No 1

The main parameters and features of design

Optics twisting consists of 6 elements having a central cable member (CCM) of diameter 2 mm.

Attenuation index, no greater than :

0.2 dB/km for a wavelength of 1.55 m,

0.4 dB/km for a wavelength of 1.31 m.

Tensile strength: 3.5 kN.

Temperature range: -40 +50 .

Fiber manufactured by company Corning SMF-20e, complying with standard G.652.D.

Weight, no greater than: 137 kg/km.

Outside diameter: 13.3 mm.

Number of optics fibers: 36.

The parameters of a single optics fiber are given in tables below:

Table No 6.1

Optics guide parameters

Single mode cable without dispersion shift

Core diameter

8.3 m

Optical fiber cladding diameter

125.0 2.0 m

Non-roundness of optical fiber cladding section


Diameter of color optical fiber

250 m

Cladding diameter

245 10 m

Concentricity of core and optical cladding

0.8 m

Beam diameter to channelize mode

9.20 0.50 m at a wavelength of 1310 nm

10.50 1.00 m at a wavelength of 1550 nm

Minimum yield strength

100 000 f-F/sq.inch

Evenness of attenuation

No more than 0.10 dB at 1310 nm or 1550 nm

Maximum dispersion

3.2 ps/nm-km over the range from 1285 to 1330 nm

< 18 ps/nm-km at 1550 nm

Cutoff optic fiber wavelength

< 1260 nm

Zero dispersion wavelength ()

1301.5 nm < < 1321.5 nm

Zero dispersion gradient

< 0.092 ps/nm2-km

Polarized mode dispersion

< 0.5 ps/km1/2

Table No 6.2


Brand OV


Operational wavelength, nm


Attenuation factor, dB/km, no more than:

at a wavelength of 1310 nm


at a wavelength of 1383 nm


at a wavelength of 1550 nm


at a wavelength of 1625 nm


Chromatic dispersion factor, ps/nmkm:

within the wavelength band of (1285-1330) nm


within the wavelength band of (1530-1565) nm


within the wavelength band of (1565-1625) nm


Zero dispersion point, nm


Slope of dispersion curve in zero dispersion wavelength, ps/nm2km, no greater than:

within the wavelength band of (1285-1330) nm


Polarized mode dispersion, ps/m, no more than:

individual fiber


line (20 connected fibers)


Cable cutoff wavelength, nm, no more than


Mode field diameter, m

at a wavelength of 1310 nm


at a wavelength of 1550 nm


Glass geometry

inherent fiber curvature, m


reflecting jacket diameter, m

125.0 0.7

core and jacket eccentricity, m


jacket out-of-roundness, %


CPC is using 36 fiber single mode cable P/N BC01200721 at section 0-204 km as a backbone fiber optics cable; the cable is manufactured by company Teldor LDD-9-06x06-D-ZP-G-FR-BK (SMF-28e) (6 buffer tubings contain 6 cores, the fiber manufactured by Corning SMF-28e, the cable complies with standard G.652.D).

Figure No 2

LDD-9-06x06-D-ZP-G FR-BK

Component number: F90360644B

General design: The cable consists of 36 single mode color coded fibers. The fibers are accommodated inside 6 color coded tubings. The tubings are filled with tixotropic gel and SZ are twisted around the central cable member. The space around the cable core is filled with waterproofing gel. A fiber optics layer is used as an additional strength member. The final element of the cable structure is provided by a light-stabilized flame-retardant polyethylene coating.

Application: Laying into cableways using blowing technique. Long distance telephony, cable TV and transferring data on line pipe facilities.

Outside cable jacket material: FR-PE

Outer diameter: 12.5 mm nom.

Mass: 145 kg/km

Design and materials Buffer material:

Polybutylene terephthalate

Piping diameter: (ID/OD):

2.8 mm/3.0 mm

Color coding:


Central core member:


Piping twisting:


Core members:

optic fibers

Pipings with fibers:


Total pipings:


Total fibers:



Between piping gel

Jacket outer color:



Per request


Standards: IEC 60794, ISO/IEC 11801, TIA/EIA-568

Installation: IEC 60794-1-1 Annex A

Operational features

Maximum admissible tensile force (assembly)

4000 N

Maximum admissible tensile force (operations):

2400 N

Impact strength:

20 Nm

Cyclic impact resistance:

30 cycles

Maximum admissible crushing force:

400 N/cm1

Minimum bending radius (assembly):

20xD mm

Minimum bending radius (operations):

10xD mm

Bending resistivity:

500 cycles

Twisting (L=125 x d):

10 cycles 2

Maximum admissible operational temperature:


Minimum operational temperature:


Maximum admissible cable laying temperature:


Minimum admissible cable laying temperature:


Maximum admissible storage temperature:


Minimum admissible storage temperature:


Light stability:




Optic properties Index

Standard per


IEC 60793-2

Measurement units



@1310 nm



@ 1550 nm



@ 1625 nm




between 1285 and 1330 nm (range O)



between 1460 and 1530 nm (range S)



between 1530 and 1565 nm (range C)



between 1565 and 1625 nm (range L)



Zero dispersion wavelength



Mode field diameter:

@1310 nm



@1550 nm



Critical wavelength



Transmission medium sublayer (separate fiber optic)



Jacket diameter



Core/jacket eccentricity error



Jacket out-of-roundness



Jacket diameter (unpainted)



Check test level



Forced macro bending: 100 turns around a core

Core radius




Max. @ 1550 nm




Max. @ 1625 nm




The backbone telecommunication line connects 16 PS and 98 line pipe valve shelters over a distance of 1504 km.

6.2 Requirements to DWDM system

The designed DWDM system should provide for telecom traffic for all existing CPC communication systems built on the basis of the national DWDM "Volga" developed by Russian company T8.

The main list of services provided using DWDM (solely for preliminary information) is the following:

Corporate network in offices ranging from PS Astrakhan, PS Atyrau, PS 7 up to CPC On-shore facilities (1 Gigabit Ethernet channel per site).

OTN-150 SDH -STM1: one channel between PS.

OTN-2500 SDH -STM16: two channels between PS.

SCADA network 1 Gigabit Ethernet: two channels between PS, in over-lap topology every other PS.

ISPLS network: two channels 10 Gigabit Ethernet between PS.

ISPLS network: two channels 1 Gigabit Ethernet between PS in over-lap topology.

SAN network: up to 8 channels between sites having developed fail-safe virtualization system; the throughput capacity is determined by design based on the service requirements and requirements on redundant capacity

The number of channels in different network segments, the types of interfaces, locations in CPC telecommunication line should be set forth during the design work.

6.2.1 Requirements to DWDM equipment:


Modular design.

Flexibility of the system. Ability to provide a separate optical channel for each user service at a separate wavelength.

Reliable control using a dedicated optical channel having elevated sensitivity.

1 + 1 redundancy or system backing up to provide service reliability

DWDM system should allow a gradual build-up in line and network capacity on as needed basis without deflating investments made.

DWDM equipment should have high reliability: all transponders, amplifiers, systems of power supply should operate in autonomous regime; when an error occurs on one channel the system should continue operating normally; the traffic in adjacent channels should not interrupt. All equipment in the system should support hot-swapping.

DWDM equipment should support all types of digital client interfaces in the range between 100 Mbps and 10 Gbps. It should not require converting the client signal protocol into another protocol and it should be completely transparent to traffic in CPC process networks.

DWDM system should be flexibly adjusted to the Customer needs; it should be able to organize I/O point for required number of channels at a minimum cost; at the same time it should be free from complete signal demultiplexing in fiber and it should pass part of channels through I/O point in transit without regeneration.

6.2.2. DWDM equipment should support remote visual monitoring and remote equalization of the channel spectrum.

6.2.3. DWDM equipment should support the ability to remotely upgrade software installed.

6.2.4. As part of equipment DWDM amplifiers should do the following:

- supporting different stabilization modes: in particular, support stabilization of the gain mode, stabilization of the power output, stabilization of pumping current

- be capable of adjusting the power output

- have a mechanism to compensate the uneven spectral gain

- support the ability to change automatically the stabilization mode

6.2.5 Operational modes and composition of sets of equipment should be designed to provide for consistent operations in telecommunication channels designed in compliance with CPC requirements.


7.1 Requirements to network core layer

The existing security network is an MPLS network providing data transfer using technologies VPN L3, VPN L2 for ISPLS network clients.

The following models of core layer routers installed in operator control rooms are used in each PS.

7.2 Network core hardware.

Specification of the core layer equipment installed at the following facilities Marine Terminal, PS Kropotkin, PS Komsomolskaya, PS Astrakhan, PS Atyrau and PS Tengiz is shown in tables below.

Table 7-1

Physical security network

Security Network (IPSS)


Pump station (server room)

Pump Station (Telecom Room)

Router Cisco 7606 Chassis 6

Cisco 7606 Chassis 6


Processor module RSP720-3C-GE

Route switch Processor RSP720-3C-GE


Line card WS-X6748-GE-TX



Switch board Cisco 7600 ES Line Card, 20xGE SFP with DFC 3C

7600 ES20 Line Card, 20xGE SFP with DFC 3C


Power supply 2700 WDC Power Supply for 7606

2700 WDC Power Supply for 7606


Specification of the core layer equipment installed at the following facilities: Marine Terminal and PS Kropotkin.

Table 7-2

Physical security network

Security Network (IPSS)


Pump station (server room)

Pump Station (Telecom Room)

Router Cisco 7606 Chassis 3

Cisco 7606 Chassis 3


Processor module WS-SUP720-3BXL

Route switch Processor WS-SUP720-3BXL


Memory module Cisco C7600 RSP720 Compact Flash Memory

C7600 RSP720 Compact flash memory


Line card WS-X6548-GE-TX



2700 WDC Power Supply for 7606

2700 WDC Power Supply for 7606


Specification of the core layer equipment installed at the following facilities: PS-3, PS-4, PS-7, PS-2, PS-5, PS-4, PS 4, PS-3, PS-5, PS 4 (kp 390)

Table 7-3

Physical security network

Security Network (IPSS)

Pump station (server room)

Pump Station (Telecom Room)

Router Cisco 7606 Chassis 6 slot RSP720-3C OS included Cisco 7600 Route switch Processor 720 Gbps fabric, PFC3C, GE included 2700 WDC power supply for CISCO 7606

Cisco 7606 Chassis 6 slot RSP720-3C OS included Cisco 7600 Route switch Processor 720 Gbps fabric, PFC3C, GE included 2700 WDC power supply for CISCO 7606


included RSP720-3C-GE


Memory module Cisco C7600 RSP720 Compact Flash Memory //MEM-RSP720-CF256M

C7600 RSP720 Compact flash memory


2700 WDC Power Supply for 7606

2700 WDC Power Supply for 7606


Software Cisco 7600-RSP720 IOS ADVANCED IP SERVICES //S764AIS-12233SRD

Cisco 7600 RSP 720 IOS Advanced IP Services


Switch board Cisco 7600 ES+ Line Card, 20xGE SFP with DFC 3C //7600-ES+20G3C

7600 ES20 Line Card, 20xGE SFP with DFC 3C


7.3. Architecture of existing network core.

A typical physical topology of pump stations is the following; it is shown in scheme No1.

Diagram No 1

7.4. It should be taken into account when designing a new ISPLS network that the core layer architecture provides for a high-speed third layer medium in OSI model.

7.5. Core architecture of the network should provide for the permanent availability level free from single point of failure (SPOF) including, but not limited to the following:

chassis redundancy;

redundancy in power supply for each chassis;

support of hot swap for network equipment modules without service interruption;

use of reserved direct links between pump stations;

use of redundant backup links through a pump station (overlap connections) throughout DWDM infrastructure;

using back-door links for project-defined VPN;

the total convergence of network should be less than 3 seconds;

network availability should be no less than 99.97%

the bandwidth should be used in efficient manner;

the fastest possible data transfer;

transmission of separate traffic flows.

7.6. The network core should comprise two Gigabit Ethernet connections to the backbone high-speed channel. The network core is using switchable D3 up channels to communicate with the network distribution layer. From the standpoint of MPLS network the core network hardware should perform functions of P and PE routers.

7.7. The network core layer should comply with the following requirements and should be capable of supporting the following:

each pump station should have two multi-layer routers having a modular architecture installed;

the network core layer should provide for interaction with the distribution layer equipment;

core network layer should provide for traffic balancing.

high scalability without chassis modifications;

communication channel aggregation technology (protocol 802.1ad, static);

access lists (ACL based on layers L3, L2);

time synch protocol (NTP);

secure management/monitoring protocols (SSH, SNMP v2c, v3);

Syslog protocol event logging;

7.8. These functions should be implemented through application of the following core layer protocols and technologies:

dynamic routing protocol OSPF v2;

Multiprotocol Label Switching for MPLS labels with label transmission using LDP protocol;

transmission of Ethernet services over MPLS network (EoMPLS);

Virtual infrastructure in MPLS networks (MPLS VPN) using MP-BGP protocol;

7.9. OSPF dynamic routing protocol

Designing should provide for use of OSPF dynamic routing protocol accounting for the following features and properties:

control over OSPF Hello and OSPF Dead timers;

mechanism of identification of bilateral accessibility BFD;

maintaining the same type of OSPF networks on interfaces;

use of multiple routs having the same convergence;

mechanisms for optimization of transmission of routing information;

mechanisms of optimization of SPF algorithm;


7.10. Topology

The following types of telecommunication channels should be designed at the core layer:

direct channels between the core layer network equipment;

redundant channels organized by OTN 2500 network (for facilities Marine Terminal and PS Tengiz);

Direct links between the core routers should be used as main channels. Direct channels should be connected using Gigabit Ethernet technology with MTU values and duplex parameters at adjacent interfaces.

Two loops of OTN 2500 network are used as a backup; each loop features a distributed hub. Connectivity technology to switch redundant Fast Ethernet channels having supported MTU values up to 1522 bytes accounting for the link level headers and duplex parameters. It is required to provide for additional aggregation level network equipment in case this is impossible to match the rate of the core layer routers connected to OTN sites.

It is required to provide for an appropriate channel value to prevent Equal Cost routes in redundant channels.

The topology should be connected to ensure that the first core routers in each station are connected to the first OTN ring. The second core router in each station is connected to the second OTN ring.

Two direct point-to-point or point-to-multi-point channels should be designed between the network equipment at the core layer.

OSPF protocol is activated only on interfaces and in networks that connect core layer devices.

It is required to use logic loopback interface having number 0 (loopback0) to identify the routers within OSPF protocol (OSPF Router ID). It allows improving overall stability in protocol performance.

OSPF protocol should be active only on interfaces and in networks connecting devices of the core layer.

7.11. Addressing

Designing should provide for using telecommunication channels between the core equipment ("point to point") using network with mask /29

These channels involve:

direct communication channels between core routers;

networks to connect to backup communication channels;

Provide for using addresses having mask /32 to identify router within OSPF protocol.

Summing routes on the core layer is unacceptable as it affects accurate performance of LDP and MP-BGP protocols. Therefore, there is no need to split different core layers into various OSPF domains. Core layer should belong completely to OSPF Area 0.

7.12. Fast convergence in OSPF

Project should provide for changing the basic characteristics of this protocol. Reduce the convergence time of OSPF protocol in the following ways:

optimize mechanisms to detect line incidents (OSPF Hello and OSPF Dead timers, BFD mechanism, Carrier Delay timer);

optimize performance of SPF algorithm (SPF Throttling and SPF mechanisms);

optimize the transmission of information about the telecommunication status (LSA Throttling, LSA Group Pacing, and LSA Flood Pacing mechanisms).

7.13. Optimizing OSPF timers

During design stage it is required to optimize the timer to send periodic OSPF Hello and OSPF Dead messages.

The default value is equal to seconds respectively. Implying that detecting an incident at the data link level can be as long as 40 seconds. This value is unacceptable and requires a significant reduction.

In standard configuration the minimum value of OSPF Hello timer can be equal to 1 second. OSPF Dead timer value should be at least 4 times greater than OSPF Hello timer or 4 seconds. This is the time required to determine an incident and it is also too long and requires reduction.

The project should provide for the minimum possible values of OSPF Hello and OSPF Dead timers.

7.14. Carrier Delay Timer Optimization

Design should provide for optimization of the Carrier Delay timer to accelerate process convergence when an incident is detected at the OSI model physical level. The timer is set to 0 for all direct link interfaces between core routers. Operational mechanisms of events containment (Half Life Period, Reuse Threshold, Suppress Threshold, Max Suppress, Restart Penalty) should be kept at their default values.

7.15. Implementation of BFD mechanism

Design should provide for detection of failure in bilateral connection of the physical link using BFD mechanisms. The following parameters should be optimized in BFD mechanism performance:

BFD Interval;

Minimum RX;


7.16. Optimizing distribution of the connection status information

It should provide for prevention of generation of a large number of packets sent through optimizing LSA packet dispatch intervals.

It is required to optimize the following timer parameters:

Start-Interval the minimum delay before the onset of generating the first update package;

Hold-Interval the dynamically changeable delay time used to calculate the time to generate update packages. When the second event occurs during the time sending the packet is delayed by the Hold-Interval time. When subsequent events occur the delay will amount to 2*Hold-Interval, 4*Hold-Interval and so on for each event until the delay value becomes equal to the Max-Interval;

Max-Interval is the maximum delay before the onset of generating next LSA packages;

OSPF Group Pacing.

7.17. SPF calculations

It is required to provide for optimization of SPF Throttling mechanism to delay triggering SPF process and, as a result, slow down the procedure of calculation of the shortest route when the network is unstable.

By default the delay in triggering OSPF process is disabled in routers. However, it should be used to ensure better stability in network performance.

7.18. The maximum number of routs

It is required to provide for load balancing across multiple routs having the same value to the network segment. It is required to optimize maximum path parameter.

7.19. Authentication

Authentication should be used to improve OSPF protocol performance security level. Algorithm MD5 - a Digital Signature Algorithm described in RFC1321 - should be used for authentication.

7.20. Recording OSPF system messages

Core layer routers can record the change in OSPF-neighborhood status in the local system message log and send these messages to the Syslog server. This functionality should be enabled in all core switches to facilitate and speed up the search for potential malfunctions and fixing them.

7.21. Optimizing OSPF (incremental OSPF update)

A function of incremental OSPF optimization (ISPF) should be provided as an OSPF protocol optimization option allowing to calculate using SPF algorithm not the whole a graph but only the part showing changes.

The use of ISPF will reduce the operational time of SPF algorithm. The application of ISPF at the core layer routing is justified and necessary as it is required to speed up the network convergence process.

7.22. Label Distribution Protocol (LDP) for Multiprotocol Label Switching (MPLS)

7.22.1. It is required to provide for the label distribution of multiprotocol switching (MPLS) in ISPLS network at the core layer using a dedicated Label Distribution Protocol (LDP).

7.22.2. Functional parameters of LLDP are determined according to the following parameters and properties:

label exchange mode;

protection of sessions;


7.22.3. It is required to analyze the compatibility of solutions with protocols currently used on existing equipment at the stage of designing application of LDP and MPLS.

7.22.4. The results of comparative analysis should include the following details:

the list of supported MPLS standards in existing and designed equipment;

description of Link-layer protocol support in existing and designed equipment;

review of assigning mpls label;

comparative analysis of special MPLS labels, BGP VPN Route distribution and penultimate routers and assess of their impact on overall compatibility of developed design solutions and existing design;

comparative analysis of LSP types used based on deployment of solutions on traffic engineering;

analysis of feasibility of using Constrained Shortest Path First algorithm in LSP computation;

analysis of using Fast Reroute and Link Protection;

7.22.5 Starting LDP

Use Unsolicited Downflow Label Distribution Mode;

label saving mode corresponds to the Liberal Label Retention (LLR) mode;

route control mode corresponds to Independent LSP Control Mode.

7.22.6. Basic LDP configuration

To simplify troubleshooting each router should capable of setting range of the locally used labels. Router ranges should be set equal to NN 0000 to NN 9999, where NN is the number of the facility where the router is installed. A complete list of ranges should be presented in a separate table.

The auto configuration protocol LLDP should not be used. At the same time activation of LLDP protocol is made manually for each core interface. These interfaces include all direct telecommunication link interfaces between core routers.

7.22.7. Router identifier for LDP

The logical loopback interface should be used to identify core routers for LDP protocol (LDP Router-ID); also loopback0 should be used for identification in OSPF protocol.

Determine transport address used by LDP as a Router-ID.

7.22.8. TTL propagation

The Time-To-Live (TTL) should be used as a function to rule out occurrence of routing loops at the core layer at the designing stage.

7.22.9. MPLS MTU

It is required to provide for interface MTU values facing the core layer when a network is set up using MPLS for such technologies as the following:

technology MPLS L2VPN

technology MPLS L3VPN

technology EoMPLS

7.22.10. Controlling label propagation

Provide for support of control of the label propagation mechanisms:

Controlling the Advertisement of Labels;

LDP Inbound Label Binding Filtering.

Both technologies are aimed at reducing the number of label entries in the router memory.

It is required to evaluate the possibility of applying LDP IPv4 FEC filtering. Design solutions should include using ADR automatic connection policers and describing the application rules in the context designed QoS model in ISPLS network.

7. 22.11. Protection of MPLS LDP Session

Provide for protection of LDP sessions for each core layer router. LDP sessions should not terminate to ensure the fastest possible restoration of correct network performance after incident. The timeout to restore the direct telecommunication link between routers is not specified.

7.22.12. Synchronization of LDP-IGP protocols

Provide for synchronization of OSPF routing protocol with LDP protocol in all core routers.

7.22.13. LDP authentication

MD5-authentication should be used to improve security level of LDP protocol performance in ISPLS network.

7.22.14. Recording LDP system messages

It should provide for a record of changes in LDP-neighborhood status into the local system log and send these messages to Syslog server. Also routers allow reporting authentication errors in LDP-sessions. This functionality should be enabled in all core switches to facilitate and accelerate troubleshooting.

It should provide for the activation of collecting statistics in LDP and describe potential limitations.

7.22.15. Convergence of LDP

It should ensure the maximum possible convergence time in LDP protocol. (Unsolicited Downflow Label Distribution Mode, Liberal Label Retention Mode, Independent LSP Control Mode).

Assess configuring LDP timers for hello, HOLD timer, and Keepalive timer messages.

Assess the potential of using strict targeted hello messages for LDP.

Describe application of filtering Inbound Outbound LDP label binding in the process of migration of network segments.

It should provide for the use of BFD for LDP LSPs and describe its operating modes.

It is required to use LDP link protection both for unicast and multicast LSPs.

7.23. Ethernet s over network service transmission technology with multiprotocol label switching (EOMPLS).

Provide for use of EoMPLS (Ethernet over MPLS, EoMPLS) technologies for connecting loops in segment circuits for the main shut-off valve skids at the second level of OSI model.

The project should provide for the following:

possible access load balancing (via EoMPLS) using MPLS core;

estimation of the maximum package size (MTU) for all intermediate links between endpoints and use of TE (RSVP) functions for major core communication channels;

describing methods to place information on VLAN membership in Ethernet frames;

spanning tree and REP (or analog) work for VLAN with Ethernet over MPLS;

limitations of using EoMPLS technology for proposed software version in communication equipment;

- compatibility of equipment used;

- optimization of migration procedures or equipment replacement.

7.24. EoMPLS switching

Requirements to pseudo-conductor to connect circuit segments in MPE blocks:

the possibility of organizing each pseudoconductor on a separate physical interface;

the ability to transmit frames with headlines IEEE 802.1Q;

the potential of Port Mode choosing;

provide for a completely transparent transmission for all second level frames in OSI model;

provide for a single identifier for each pseudoconductor accounting that the first two numbers comply with the numbers of the object code to terminate the segment, and the third number complies with the segment number.

7.25. Design the topology of the ring segments for entire ISPLS network using Resilient Ethernet Protocol (REP) or similar for the chains of blocks in main shut-off valves.

To analyze compatibility of REP protocol in existing equipment with new design solutions for parallel operations and describe migration steps.

7.26 Transparency of dot1q for EoMPLS

Segments of the main chains of shut-off valve blocks in ISPLS network should be designed to be capable of using multiple virtual networks (VLAN). EoMPLS technology should close the segment contours with potential tunneling all required virtual networks. Use this technology due to transparent transmission of Ethernet frames having IEEE 802.1Q headers through pseudoconductor. Port Mode allows sending frames both with IEEE 802.1Q headers and without them.

7.27. Starting MPLS TE technology

Project should provide for using MPLS TE EoMPLS pseudoconductor. Use RSVP as a signaling protocol. EoMPLS should use only the basic core channels.

RSVP protocol is used in all core layer routers; it is only allowed in direct communication channels between routers. At the same time the analysis should be conducted for the maximum available bandwidth in MPLS TE tunnels.

7.28. Starting MPLS TE tunnels

To explicitly specify the traffic rout MPLS TE tunnels should used indicating the network traffic passing parameters. In the future these tunnels should be used for EoMPLS traffic transmission. The amount and direction of the tunnel should be determines by project.


The project should describe the use of the optimal MTU value for EoMPLS-frame with TE tunnel to ensure the necessary level of service.

7.30 The technology of virtual private networks

Multiservice network ISPLS should support L2 and L3 MPLS VPN. The number of L3 MPLS VPN in backbone network, the nature of their interaction should be specified within the feed survey and feed design.

Currently, it is required to provide setting up the following VPN:

VPN (CL) to connect work stations, engineering stations, stations to maintain UPS systems; this VPN should support multicast. It is present in each physical facility;

VPN (DVM): it is required to connect DVM Database Servers, DVM CameraServers, Streamers or Network Cameras. It is present at each facility;

Existing systems limitation:

DVM server should be located in the same subnet with cameras;

the cameras should have the capability of switching (re-registering) with different DVM servers installed in geographically remote PS facilities.

The character of interaction is shown schematically in diagram No 2 to be rectified in design (shown for information only). It should support both the fixed addressing, and automatic assignment of address at connecting to the network.

Scheme No 2. Scheme of interaction between ISPLS systems

(TCP Port 80 (HTTP),TCP Port 135 (DCOM),Configurable DCOM PortsTCP Port 10000 (Event Server)TCP Port 60000 (DVA Configuration)TCP Port 443 (HTTPS DVM Console Logon) NEW!UDP Port 10001 (Multi-monitor ping)TCP Port 135 (DCOM) (for multi-monitor clients only)Multicast Address 226.0.x.x Port 1 (Multicast Live Video)UDP/TCP Port 10001 (Camera Manager Ping and Health Status), TCP Port 135 (DCOM)Configurable DCOM PortsTCP Port 135 (DCOM)Configurable DCOM PortsTCP Port 443 (HTTPS DVM Console Logon NEW!EBIDVM Database Server Station Internet Explorer Streamers or Network Cameras(All supported DVM models)TCP Port 135 (DCOM)Configurable DCOM PortsTCP Port 1433 (SQL Server)TCP Port 10000 (Event Server)UDP/TCP Port 135 (DCOM)Configurable DCOM PortsTCP Port 135 (DCOM)Configurable DCOM PortsTCP Port 10001 (Live Video)TCP Port 10001 (Playback)TCP Port 10001 (VMD Tuning)TCP Port 10010 (Authentication)TCP Port 80 (MJPEG Video)TCP Ports 80 & 554 (MPEG-4 Video) TCP Port 5001 (PTZ for camera control types other than Fixed Camera and Use Streamer Settings)DVM Database Server TCP Port 10000 (Event Server)TCP Port 1433 (SQL Server)Configurable DCOM PortsTCP Port 135 (DCOM)TCP Port 40200 (DVM Service Framework) NEW!TCP Port 12000 (DVM Point Server LinkD) NEW!TCP Port 12001 (DVM Point Server RPC) NEW!TCP Port 12002 (DVM Point Server Notifications) NEW!TCP Port 135 (DCOM),UDP Port 10002 (IntegrityService Ping)Configurable DCOM Ports,TCP Port 1433 (SQL Server)TCP Ports 137, 139, 445TCP Port 60000 (DVA)TCP Port 10001(Video loss alarms) DVM Camera ServerUDP/TCP Port 135 (DCOM), Configurable DCOM PortsTCP Port 10001 (Camera Control)TCP Port 10010 (Authentication)(Custom application/scripts using DVM Object Model))

VPN (IQ) is used to ensure operability of systems to control perimeter access security. Controls access to CPC facilities. It is present at all key locations (PS, MT and shelters).

The project should provide for both L3 VPN, and L2 VPN for such clients. Potentially the need to support multicast will be required. The nature of the interaction is rectified at design stage including interaction with CL VPN, VPN DVM, VPN SUP, and interaction locations at each PS.

VPN IPG is an isolated L 3 VPN set up to control channel generating equipment (optical amplifiers, DWDM). It is present only at PS and in shelters of future PS.

VPN OTN is an isolated L 3 VPN serving systems to control SDH - OTN network, it is present in Marine Terminal and PS Kropotkin.

VPN PHN is a transport L 3 VPN to support ip phones; it is present in all facilities.

VPN RAD is an isolated L 3 VPN to serve systems to support radio telecommunication equipment. It is present only at PS and in radio shelters of future PS.

VPN NMS is an L 3 VPN to control elements at the network access level. It is present in all facilities and it interacts with all VPN.

VPN SUP is a L 3 VPN to connect contractors serving security systems; it is present at all sites and it provides access to VPN DVM. VPN CL. VPN IQ

Project design should determine necessity to set up VPN for virtualization solutions

The project should describe the necessity and implementation of VPN central services and their locations

7.31. The design should account for potential scaling up in the future to develop up to 15 L3 VPN solutions (up to 4 L2 VPN).

7.32. The project should describe MPLS Data Forwarding operations, BGP VPN Route distribution to demonstrate compatibility with existing equipment; in case of finding risks of incompatibility with existing equipment and solution it is required to describe mechanisms of migration and mitigation for the risks identified.

7.33. The project should analyze the interactions between VPN to describe the architecture, taking into account implementation of various topologies (eg, HUB-and-SPOKE, FULL MESH , etc.); a special attention should be paid to analysis of potential traffic asymmetry between L2 L3 VPN having multiple regional points of interactions.

Analyze existing addressing scheme and describe the necessary changes (if required). ISPLS network should use private address space (RFC 1918) ipv4.

7.34. The project should describe the interaction of PE-CE sites both using L3 OSPFv2 protocol and static connections similar to existing design.

7.35. Solutions on Service Level Agreement (SLA) should be provided both for existing equipment and for equipment designed.

7.36. Design solutions on network routing should include the following analysis and description:

the maximum number of routes announced in VPN network and setting relevant restrictions;

backdoor connections between sites for set up in VPNs design using redundant communication lines (e.g., dedicated FastEthernet communication channels in existing SDH ring in OTN -2500);

mechanisms to protect against cycling in routing protocols when backdoor connections are active;

optimization of convergence in routing protocols;

balancing traffic during connection.

7.37 Configuration of MP-BGP Protocol

Provide the following parameters and protocol configuration including but not limited to the following:

autonomous system;

Address family;

Route Distinguisher;

Route Target Community;

Virtual Routing and Forwarding instance;

import/Export policy;

definition of neighbor groups ;

balancing load;

monitoring MP - BGP protocol .

7.37.1 Autonomous system

ISPLS network consists of a single autonomous system comprising all levels of the network core routers. A decimal number 64512 will be used as AS number. This value corresponds to a value in existing network lying within the range of numbers determined by IANA for private use.

All routers in the autonomous system establish direct neighbor adjacency.

7.37.2 Address family

It is required that BGP protocol provides for transmission of routing information of protocol IPv4 in the global routing table. Also, it should provide for transmission of VPNv4 family routes and VPN MDT multicast. This address family should be explicitly specified in BGP protocol configuration, thereby enabling multiprotocol extension in BGP (MP-BGP).

It is also required to provide the parameters of route redistribution in BGP protocol for specific VRF under MP-BGP protocol for multicast addresses family.

7.37.3 Route features

It should provide Route Distinguisher to generate unique VPNv4 prefixes for each VPN.

7.37.4 Import/Export route policies

Provide fully connected topology for each virtual infrastructure through bilateral, symmetrical exchange of routing information for each VPN.

7.37.5 Neighbor groups

It is required to provide for use of one group of neighbors to simplify configuration.

7.37.6 Load Balancing

In order to optimize channel loading and improve convergence time the mechanisms of BGP load balancing should be used.

7.37.7 Interaction with the network management system

The network management system should provide for control of MP-BGP protocol the status using SNMP v2c, v3 protocol

The following SNMP objects that control BGP neighborhood are most critical:





7.37.8 BGP optimisation and convergence

Implement BGP performance optimizations:

change BGP timers and transportation mechanisms TCP at accordance with recommendations RFC 4271;

Fast peering deactivation;

suppression of swinging routs;

Bidirectional Forwarding Detection for BGP;

BGP NextHop Tracking.

7.37.9 It should ensure activation of BGP mechanism to Import VPN based on events

When a change occurs in network mechanism BGP Event-Based VPN Import allows importing information obtained from neighboring routers in VRF much faster than through periodic table scanning.

7.37.10 Provide for BGP security

Use the following BGP security mechanisms:

switch off ibgp (internal bgp) synchronization;

includes registration of changes status in bgp-neighbors;

use of authentication;

harmonization of bgp versions is disabled for acceleration of process start up;

include a limit on the maximum number of prefixes;

loopback interface should be used for ibgp announcements;

turns off the automatic summing prefix announcements;

use next hop-self in re-routers;

avoid availability of excessively detailed routes;

7.37.11 A plane to transfer the data

Provide for using the following mechanisms:

tables CEF;


tables LFIB.

7.37.12 Labels

The range of labels used to label virtual third level infrastructures should be distributed dynamically (dynamic label allocation). Implement the compatibility mode over the label transmission for different network vendors.

7.38. VPLS

In existing network VPLS technology allows combining geographically dispersed LAN segments into a single bridged domain over a packet MPLS network. According to the model to deploy VPLS the backbone network serves as the logical bridge. The topology and signaling in the backbone network are transparent to the customer LAN segments, which are interconnected as if they were connected to a chain of LAN Ethernet switches. The use of VPLS in existing network design is caused by requirement to ensure reliable operations of security applications and accommodation of camcorders and DVM servers, IQ controllers and EBI servers in the same logical subnet.

This restriction is important and requires specific comment in the project documentation.

It is required to evaluate design options, reliability and stability in use; it is also required to describe the mechanisms of protection from broadcast storms or loops.

The following options should be considered:

full mesh VPLS;

H-VPLS for boundary MPLS segments;

single-domain VPLS for full mesh topology;

multidomain VPLS for full mesh topologies at the level of each domain;

multidomain VPLS for H-VPLS configuration of HUB and SPOKE types with HUB redundancy.

It is required to evaluate the risks of traffic asymmetry when these domains have multisite connections to other VPN L3.

On consent of telecommunication group other technologies can be also used.

7.39. Interaction with existing network

The network designed should be connected to the existing network for a period of services migration to ensure continuity in current process operations. All devices in the old network should be accessible from the new network and vice versa.

The design process should take into account the feasibility of stage by stage failed hardware replacement, the project should describe various migration methods and describe the risks of migration procedures and risk mitigation strategy.


8.1 Requirements to distribution level

8.1.1 The existing network has a collapsed core design.

8.1.2 To overcome these operational design limitations the draft design should describe both the use of the new equipment as part of new design of collapsed core and in design of the two-layer approach (PE-CE).

8.1.3 The distribution level should be implemented in the form of a pair of 3 (L3) level switches for the critical pump stations (Onshore Facilities, Kropotkin PS, PS Atyrau, Tengiz PS, PS Astrakhan, PS 3); for other PS the distribution level should involve one switch.

8.1.4 The distribution level should ensure an efficient gathering of routing information as it is leaving the distribution sites to be sent to the core layer using IGP (OSPF v2) protocol.

8.1.5 The distribution level should support the following protocols: OSPF, VLAN, PVLAN, EoMPLS , MPLS, EtherChannel, HSRP, VRRP, GLBP for backup the default gateway, MP-BGP, MPLS-TE , QoS DiffServ.

8.1.6 The distribution level in ISPLS network should meet the following requirements:

the fastest data transfer;

load balancing across multiple active routs to the core;

connection aggregation;

routing between VLAN;

use of active gateway redundancy protocols;

redundancy on chassis;

redundancy on power supply for each chassis;

providing quick convergence of the network when topology changes;

transfer of isolated traffic flows;

connection to the core layer;

connection of remote buildings and access level;

termination of IP traffic from the end directly connected devices and from L2 remote buildings.

8.1.7 These functions should be implemented through applying the following protocols and technologies at the distribution level:

OSPF dynamic routing protocol;

virtual infrastructures (VRF);

Etherchannel technology.

8.1.8 The project is required to provide for the distribution of the interaction level with core, aggregation and access levels. Communication of hardware at the distribution level with other network levels is made using aggregated connection channels using EtherChannel technology. Protocols PAgP and LACP should be used to make a logical interface.

8.1.9 These functions are realised through using the following protocols and technologies at the distribution level:

Etherchannel technology.

8.1.10 Three mechanisms can be used to form EtherChannel channel:

PAgP - a proprietary protocol to concur channels in EtherChannel.

LACP (802.3ad) - a standard multivendor concurrency protocol.

Static ON mode without using concurrency protocols.

8.1.11 Design solutions should describe route summing schemes, schemes of filtering, addressing; the solutions should describe methods to prevent loops both at the L2 protocol level and on L3 protocol level; the description should be given how to set up backdoor connections for project VPN.

8.1.12 Project documentation should include network architectures and functional diagrams.

8.1.13 Design solutions should be justified to use the functional network devices.


9.1 The architecture of access level

ISPLS Network Access Level should provide connection of the end network devices (video servers, video tape drives, video cameras, IQ access controllers, operator workstations, printers, telephones, communication systems management ports, etc.) installed in remote buildings of the pipeline facilities both at PS sites and along the line pipe. The access level is connected to the level of the distribution network using two redundant connections to form a fully-meshed topology.

9.2. The access level in ISPLS network should meet the following requirements:

packet switching;

port security;

PoE support (optional);

connection of network clients using channels and protocols corresponding to Gigabit Ethernet specifications;

design solution should provide for scaling up without replacing the equipment and/or change in architecture provide for each site the port capacity no less than 48 ports;

the network should have a fault-tolerant architecture, eliminating a single point of failure at any level;

the equipment should provide for support in quality of service (QoS) mechanisms in the framework of the whole network;

network segmentation using VLAN technology, including setting up trunk channels of data transfer (802.1q);

network protection at L2 level from appearance of Spanning TreeProtocol (PVST +, Rapid + PVST, MSTP) loops;

support of 10Gbps and 1Gbps interfaces to interact with core/distribution level;

link aggregation technology (protocol 802.1ad, static);

access lists (ACL based on levels L3, L2);

time synch protocol (NTP);

secure management/monitoring protocols (SSH, SNMP v2c, v3);

Syslog protocol event logging;

9.3. Hardware and software components of the access level should provide for the following:

high availability for ISPLS service system users;

use of redundant power supplies and interface hot-swappable modules;

redundant gateways via dual connections to redundant systems (distribution level switches) using GLBP , HSRP and VRRP protocols.

prioritization of critical network traffic using QoS functions. The access level should provide for implementation of priority mechanisms and traffic labeling as close to the network as possible.

additional network protection from unauthorized access should use the following tools: 802.1x protocol, port security, Dynamic ARP Inspection, IP source guard.

Rapid use of protocols PVST + (802.1w) or Multiple STP (MST 802.1s) with the following improvements of the Spanning Tree protocol:

PortFast - It allows the port to skip listening and learning phases

UplinkFast - provides for convergence during 3..5 sec after connection break

BackboneFast - decreases the convergence time to MaxAge value for indirect disruptions

Loop Guard - Excludes the capability of choosing an alternative Root Port when BDPU (Bridge protocol Data Units) data blocks are absent

Root Guard - is preventing capability of using the external switch as a root

BPDU Guard - disables port supporting PortFast functions on receiving BPDU data block

BPDU filter - prevents dispatching or receiving BPDU data blocks through ports supporting PortFast functions

9.4. The project should provide for connection of remote buildings using both L2 and L3 (OSPFv2) with description of routing schemes, summing, filtering, route tagging, setting up backdoor connections for VPN specified in project design.


10.1. ISPLS network is an isolated network without access to Internet, SCADA or the corporate network; however, the network design should provide for security solutions at all levels of the network hierarchy.

10.2 The following technologies and methods should be applied to ensure the necessary level of security to the access switches without limitation:

Port Security;

DPSP Snooping;

Dynamic ARP Inspection;

IP Source Guard;

Storm Control;

security function in STP protocol.

10.3 It is required to use Port Security mechanism in order to limit the number of MAC-addresses on interfaces.

10.4 The project should describe the following for all equipment (projected and existing):

device and service hardening techniques;

main data to ensure security in control - plane and data - plane;

security in routing and switching protocols;

infrastructure security mechanisms;

control over using equipment resources;

mechanisms to protect signaling network device plane;

port mirroring to transfer traffic.

10.5 The project should provide for setting up Cisco ASA firewalls to ensure secure access to network infrastructure monitoring systems in MCC (Onshore Facilities) and BCC (backup control center) in a failover mode. Monitoring and control system is described in the following sections.

10.6 The existing infrastructure allows organizing out-of band management in communication channels for network devices installed indoor the telecommunication buildings at PS. The project should describe a solution on using RS232 ports to provide for access from MCC via existing SDH OTN-2500 telecommunication channels.


11.1 It is required to carry out analysis of traffic transmitted by all systems to ensure subsequent correct labeling and classification when QOS policy is developed.

11.2 The policy of providing quality of service implies the network ability to provide for predictable loss and delay.

11.3 The design should explicitly specify which devices perform QoS on the software and hardware level followed by making risk analysis to ensure compatibility of existing and designed equipment.

11.4 Classify and label the application traffic as close to the source as possible. This implies using the network-wide unified Differential Service principles and incremental maintenance strategies. All connected end devices such as servers, ISPLS workstations, IQ controllers will not be trusted in terms of traffic labeling; however camcorders, video tape drives can support the traffic labeling. The choice of the most optimal strategy to deploy QoS service should be based on analysis of the working documentation provided.

11.5 The unwanted traffic should be restricted as close to the source as possible. This is especially significant for traffic in "denial of service" attacks or unwanted traffic generated by "worms."

11.6 Design should provide for implementation of queue management policy in all sites featuring potential overload regardless of overload frequency. This is especially important for redundant communication channels, in communication channels between switches having overload prerequisites.

11.7 Protect control plane and data plane using Control Plane Policing and using restriction on the network scavenger traffic.

11.8 It is necessary to implement the following mechanisms to meet QoS requirements without limitation:

Classification and labeling allow classifying traffic on the basis of several characteristics such as values in Ethernet, IP and MPLS headers to be followed by labeling relevant packets using a specified value usually located in Type-of-Service byte IPv 4 header, in experimental MPLS header fields or in 802.1p User Priority fields in Ethernet - frame 802.1q. When packets are assigned a specific labeling their traffic class can be easily identified by other network sites.

Overload control allows establishing traffic priorities inside network devices when overload appears. If outlet channel is overloaded the packets designated for telecommunication form a queue. Overload control mechanisms allow redistribute the packets in transmission queue that makes it possible to guarantee the throughput capacity and delay time for certain traffic classes.

Preventing overload is aimed at avoiding overload in reasonable manner before it starts dominating communication through proactive packet rejection when queue exceeds a certain length.

Traffic limitation and control allow restricting incoming or outbound traffic at established "profile" rate level. In case of restrictions traffic exceeding permissible rate is rejected. Control is trying to adjust the traffic that can temporarily exceed a profile rate through queuing the traffic; as a result the overall traffic rate gets smoother and better meets the profile rate due to additional delay.

11.9 Classification and labeling

11.9.1 The network traffic entering DiffServ domain should be classified and made concurrent. Traffic should be classified against many different parameters, such as source address, destination address, or traffic type; the traffic can be attributed to a certain traffic class. Traffic classifiers should accept for processing any DiffServ labeling in packages obtained.

11.9.2 During labeling the switches should analyze incoming packets (frames) and assign an appropriate QoS label ("Internal DSCP "). QoS label determines further QoS actions to be applied to the packet on the route from the input to the output port. QoS label is an internal switch label; it is assigned based on DSCP or CoS packet values and determines the queuing techniques and the packet scheduling actions. In the framework of the project QoS label will coincide with corresponding packet DSCP.

11.9.3 In case other labeling mechanisms are used the project should provide for a complete description of desired functionality in hardware designed.

11.9.4 Network equipment should be able to classify traffic against the following criteria:

Using DSCP field in IP - packets.

Using IP precedence field.

Using 802.1P CoS Ethernet frame field.

Using MPLS EXP field.

Classification should be made using standard or extended IP access lists. Classification should be made based on VLAN ID

Labeling should be made accounting for IP Explicit Congestion Notification (IP ECN) field value.

The labeling should be made on the basis of application signatures using NBAR technology (optional for routers only).

11.9.5 The types of ISPLS network equipment and necessary means of traffic labeling at the inlet port are given in Table No 11.9.5.

Inlet labeling potential

Table No 11.9.5.

Hierarchy level

IP Precedence


802.1p CoS





Access level
















Distribution level







Core layer








The main provisions on labeling traffic in ISPLS network are as follows:

all connected resources are not trusted from the standpoint of traffic labeling except the traffic generated by video cameras and streamers;

the traffic is labeled at the network inlet (at the access level and on the distribution level) using access ip lists with filled ip dscp fields corresponding to table of classes;

traffic labeling for facilities connected is made in port - based and vlan - based modes. Labeling is made individually for each device in "port - based" regime; labeling is made for groups of devices in vlan-based case;

all connections between active network equipment are trusted. Labeling is made for ip - traffic using fields ip dscp, 802.1p, mpls exp;

traffic re-labeling is made from field ip dscp into field mpls exp and vice versa at the boundary mpls - network core;

for transmission along the core mpls ip dscp traffic labeling remains the same and recovers at the outlet of the network mpls - core (pipe mode).

11.10 Classes of traffic pattern

11.10.1 It is required to design ISPLS network traffic labeling pattern on the basis of 8 classes. This labeling is given in Table No 11-10 for informational purposes; it is subject to rectification and adjustment at designing stage.

Traffic classes

Table 11-10


Classification L3

Classification L2





802.1p CoS,







Critical traffic












ISPLS Real Time Traffic






High priority traffic






Normal priority traffic






Low priority traffic






Scavenger traffic

Unclassified (Best Effort)

(Unclassified (without guarantees))





The remaining (non-classified) traffic

11.10.2 This 8-class model should be implemented at all levels, including the network core where traffic labeling is only possible against 3 MPLS EXP bits.

These traffic classes should be processed by queues using different mechanisms; different traffic classes should have different guaranteed bandwidth.

11.11 QOS Domains

ISPLS network should implement 2 QoS domains. One domain should contain MPLS-core, backbone and redundant communication channels. The second domain should contain facilities connected to MPLS-core, namely all PS, Onshore facilities and Tank Farm.

QoS policy should be applied in all interfaces of uplink, downlink, EtherChannel network and others. An important aspect that should be considered when deploying QoS on EtherChannel policy deals with load distribution. It is necessary to distribute the load based on source and destination IP - addresses as this gives a statistically improved load balancing.

Algorithms should be divided using two ways for QoS algorithm configuration on EtherChannel interfaces:

The input algorithms such as trust or labeling and/or restriction policies associated with PortChannel interface.

Output queuing algorithms are applied directly to the (physical) interfaces that make up the EtherChannel package.

11.12 Setting up queues and reset

It is required to use a priority queue to serve real-time traffic; the rest of the traffic is served using CBWFQ mechanism.

Designing should provide for the following functions to set up queues in different models of network equipment:

queue types at outlet;

establishing queue restrictions;

configuration of priority queue;

configuration of standard queues;

configuration of thresholds;

configuration of threshold for queue wred - reset based on service class;

configuration of threshold resetting the last element when a queue is generated on the basis of dscp field value;


12.1 The project should provide for use of synchronization time protocol NTPv4 ensuring authentication of time sources and supporting network equipment.

12.2 The following NTPv4 features should be accounted as a selection criteria:

NTPv 4 is compatible with NTPv 3 in case equipment is used that does not support NTPv 4;

NTPv 4 is an new protocol officially supported by software and network equipment producers;

algorithm MD 5 is applied for server time authentication;

NTPv 4 is identified in RFC 5905: http://tools.ietf.org/html/rfc5905 .

12.3 Solution on system time synchronization

12.3.1 NTP protocol should be used to synchronize the system time in network equipment and equipment of ISPLS security system. It is necessary to implement a two-tier time synchronization scheme.

12.3.2 Two servers should be used as first-level time sources (Stratum 0); they should be located at the following sites:

PS Tengiz, radio shelter;

Onshore facilities, radio shelter.

12.3.3 These servers should set the system time with an accuracy of 10 microseconds. Time synchronization at the Stratum 0 level should be carried out using GPS satellite channels. The local server clock built on crystal oscillator having a temperature compensation (Stratum 12 level) are used as backup time sources. It is not expected that external NTP-servers are used for this purpose.

The second layer (Stratum 1) in the time synchronization system is implemented at the core layer routers. The rest of the network equipment installed at the pipeline facilities is using above sources as a clock.

12.3.4 The project should describe the time synchronization schemes for different levels of network hierarchy.


13.1 The project should provide for mechanisms to protect management and infrastructure planes include the following technologies:

Mechanism to protect Control Plane.

Mechanism to protect Data Plane.

Mechanism to protect Management Plane.

These mechanisms should be implemented for the following network equipment level:




13.2 Traffic Control Plane

The project should traffic policing in control plane and security. Engineering solutions should be implemented through defining traffic classes and restricting rate for each class in traffic processing at the Routing Processor. In accordance with engineering solutions traffic control mechanism is implemented in the Core MPLS Routers and Distribution Routers in locations where the network transfers data.

13.3 The following traffic classes should be adjusted to control the traffic in control plane:

office traffic having dynamic routing protocols;

Management traffic (SSH, Telnet, HTTP, TFTP, FTP, SYSLOG, NTP, DNS, WEB, SNMP);

malicious traffic.

normal traffic;

Default Traffic the rest traffic by default.


14.1.1 The following BGP protocol protection functions (without limitation) should be implemented at the stage of project designing:

bgp neighbor impersonation attack and resetting transmission control protocol;

transmission control protocol session attack using internet control message protocol;

bgp session theft attack;

route "rocking";

route degradation;

unauthorized route announcement;

service rejection due to exhausted resources;

14.1.2 Implement secure BGP model including, but not limited to the following:

iBGP should not contain synchronization (Internal BGP);

disable Fast external failover function to prevent route change due to BGP neighbor temporary disruptions in sending keepalives communication" packets;

enable registering change in BGP neighbor status;

use soft reconfiguration;

use authentication;

disable matching BGP versions to accelerate start process;

block incoming route updates from fraudulent prefixes;

use restriction on the maximum prefix number to avoid inflating Routing Table;

use loopback interface for IBGP announcements;

disable automatic summation prefix announcements;

allow BGP neighbors establishing connection using only port TCP 179;

use prefix lists to restrict announcing only selected prefixes.

14.2 Protocol OSPF v2

14.2.1 The designing should implement the following functions to protect OSPF protocol including, but not limited to the following:

establish MD 5 authentication in each channel;

reject LSA from unknown neighbors;

install external route filters in boundary routers for autonomous system (if available);

provide installation of the latest versions of software with eliminated OSPF vulnerability.


15.1 The project should implement the model of Authentication, Authorization, and Accounting to ensure three consistent independent protection levels to be passed before a user gets controlled access to the active network equipment or network service. AAA model should provide the following functions:

authentication (a set of methods identifying remote user). user ID should be implemented at the level of selecting the users Login and Password.

Authorization (a set of methods to restrict the user access level to operating system installed in the active network equipment and/or network service).

Accounting (a set of methods to gather and export to remote server statistical information on user activity).

15.2 These methods should be implemented both locally and remotely within the framework of Access Control Server (ACS). The design solution should be based on a mixed method where the main user database and policies will be maintained on access control server acting as a distributed disaster-proof cluster whereas the emergency data base (in case ACS servers are not available) will be maintained locally on each device.

15.3 The following event logging categories should be provided in ACS system:

making changes in acs configuration parameters.

logging all events when changes are introduced into acs configuration parameters.

In case a new element is added or a change is made in existing element a detailed information is entered on modified attributes and new values of these attributes.

If the new attributes were not established as a result of change query the log in the journal to audit configuration parameters is not made;

acs administrator access: registration of all events occurring since the administrators access to the system and to moment the administrator logs out. also registered. The type of administrator log out is logged; it can be an explicit request or termination of session after expiry of the maximum idle time. This logbook should are record abortive events of attempted login because of the fact that account is inactive. The journal registers information on a failed login attempt, as well as the reason of login rejection;

entering changes into acs operations: registering all operations requested by administrators including acs activation from the deployment point as the main system, query about full replication, loading software, back-up copying or recovery out of back-up copies, development and recovery of pac, etc .;

changing password of internal user: logging all events dealing with password changes for internal users in all control interfaces;

in addition, the reports of the administrative and operational audit are logged with the local storage. Additionally, a configuration can be made to record these messages into remote event logs;

aaa audit should cover events of successful and unsuccessful authentication radius and tacacs +, events of successful and unsuccessful authentication of command access, events of changing passwords and answers on radius queries;

aaa diagnostics that should involve information about authentication, authorization and account for radius and tacacs + diagnostics queries, as well as attribute requests from radius and information about repository of identity data in the authentication flow. Registering these messages is not mandatory;

diagnostics system covering events of the start and completion of the work in system as well as diagnostic messages dealing with event logs:

diagnostic administration events dealing with command line interface and Web interface;

messages dealing with external server;

local data base messages;

the local service messages;

logging these messages is not mandatory;

system statistics involving information on the system performance and consumption of resources. This information includes the resource consumption of CPU and memory as well as the process status and waiting time to process queries;

user messages such as messages on beginning and termination or change of sessions as well as messages dealing with command accounting; in addition it is possible to configure registration of these messages in the local storage.

15.4 Each registered message should contain the following information:

message code - a unique message code;

category of event registration - determines the category to which the registered message belongs;

severity range - determines the severity range for diagnostic messages.

message class - determines groups of messages having similar content, e.g. messages, dealing with RADIUS, policy or EAP;

text messages - a brief illustrative text message in English;

description - text in English containing description of reasons to register the message, information about the way to eliminate an errors (if applicable) and reference on external resources containing more information;

failure cause (optional) - indicates whether the registered message deals cause of failure.

15.5 The project should provide a description of the licensing model on a device and the application of licenses, procedures to make an automatic backup configuration copies of ACS systems, a description of elements of policies, user roles, hierarchy of regions and network devices, a description of command shell for user roles accounting for privileges. Provide the following roles and privileges:

administrators of all system having full scope of rights and having accesses to all equipment;

regional administrators having full scope of rights and access to regional equipment;

PS administrators h