plnog14: architektura oraz rozwiązywanie problemów na routerach ios-xe - piotr kupisiewicz

© 2008 Cisco Systems, Inc. All rights reserved. Cisco restricted QFP ZBF Implementation 1

Architektura oraz rozwiązywanie problemów

na routerach IOS-XE Piotr Kupisiewicz

Cisco Systems

© 2014 Cisco and/or its affiliates. All rights reserved. BRKCRS-3147 Cisco Public

Agenda

§ ASR 1000 §  ISR Seria 4000 § Dzień z życia pakietu (normalnego pakietu) § Taktyki troubleshootingu § Packet-tracer

ASR 1000 (ASR1K)


5-36 Gbps

Instant On Service Delivery

ASR 1002-X ASR 1004 ASR 1006

10-40 Gbps

10-100 Gbps

ASR 1001/1001-X

2.5-5 Gbps 40-200

Gbps

ASR 1013 IOS-XE

Kompaktowy, mały router

§  Wydajność liniowa od 2.5G do 200G+

§  Ochrona inwestycji poprzez modularną budowę

§  QoS w sprzęcie – do 472 kolejek

Wysoka dostępność i niezawodność

§  Separacja części forwardując od części zarządzającej (Control od Forwarding Plane)

§  Sprzętowa i Softwarowa redundancja

§  Zero-Downtime Upgrade

Dodatkowe usługi

§  Wbudowany Firewall, VPN, DPI, CUBE

§  Kontrola usług dodatkowych poprzez licencję

4

Cisco ASR 1000


ESP

FECP

QFP Crypto Assist.

interconn.

PPE

BQS

ESP

FECP

QFP Crypto Assist.

interconn.

PPE

BQS

5

ASR 1K: Architektura sprzętowa

RP

CPU

interconn. GE switch

SIP

SPA SPA

IOCP SPA Aggreg.

interconn.

RP

CPU


Midplane

SIP

SPA SPA

IOCP SPA Aggreg.

interconn. SIP

SPA SPA

IOCP SPA Aggreg.

interconn.

Active

Active

Stby

Stby

Embedded Service Interconnect aka ESI Bus 11.2 – 40 Gbps Forwarding Bus

Cały ruch “przechodzi” poprzez ESP (QFP), nie poprzez RP !


QFP Subsystem PPE + BQS

TCAM

SPI MUX

Interconnect ASIC

Crypto Engine

FECP CPU

FECP DRAM

BQS Packet DRAM

PPE DRAM

For Your Reference Embedded Services Processor

Prawdziwy sprzęt!


Cisco “Quantum Flow Processor”

•  Packet Processing Engine (QFP-PPE) –  40 Procesorów każdy obsługujący 4 równoległe

wątki. Procesowanie 160 pakietów w tym samym czasie.

–  Do 1.2GHz Tensilica ISA processors + pamięc DRAM

–  Jeden TCAM4 I/F; możliwe łączenie do 1-4 QFP –  Zaprogramowane w C –  Wsparcie sprzętowe dla statystyk, WREDa,

policerów, crypto itd.

•  Buffer/queue subsystem (QFP-BQS) –  Sprzętowa implementacja QoS –  W pełni konfiguralna za pomocą znanego CLI

Multi-Core (40) Packet Processor

Traffic Manager (BQS)

For Your Reference


Crypto

FECP

Architektura ESP

GE, 1Gbps I2C SPA Control SPA Bus

ESI, 11.2Gbps SPA-SPI, 11.2Gbps Hypertransport, 10Gbps Other

RPs RPs RPs ESP SIPs

QFP Complex

TCAM Resource DRAM

Packet Buffer DRAM

Part Len / BW SRAM

SA table DRAM

Dispatcher Packet Buffer

DDRAM

Boot Flash (OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl

Packet Processor Engine

…

PPE1 PPE2 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

Interconnect

SPI Mux

ESP FECP

QFP Crypto Assist.

intercon.

PPE

BQS


ESP

FECP

QFP Crypto Assist.

interconn.

RP

CPU


SIP

SPA SPA

IOCP SPA Aggreg.

interconn.

ASR1K: Architektura Software’owa

RP CPU

IOS Chassis Manager

Forwarding Manager

Linux Kernel

EO

BC

(1 Gbps)

ES

I (10-40 Gbps)

ESP FECP

Linux Kernel

Chassis Manager

Forwarding Manager

QFP Crypto Assist.

µ µ µ

BQS

µ µ µ

Drivers Drivers Drivers

SIP IOCP

Linux Kernel

Chassis Manager

SPA

SPA Driver SPA Driver

SPA Driver

SPA SPA

ES

I (10-40 Gbps)

I2C

ISR Seria 4000 (43xx/44xx)


ISR Seria 4000

ISR 4321 50-100 Mbps

ISR 4331 100-300 Mbps

ISR 4351 200-400 Mbps

ISR 4431 500-1000 Mbps

ISR 4451 1-2Gbps

NEW

NEW

NEW

NEW

Wszystkie ISRy serii 4000 oparte są na IOS-XE

© 2014 Cisco and/or its affiliates. All rights reserved. BRKCRS-3147 Cisco Public 12

ISR 4451-X Architektura Sprzętowa

Data Plane (10 core)


PPE6 PPE7 PPE8 PPE1

0 PPE9

Control Plane (4 cores)

Ctrl SVC1

SVC2 SVC3

FPGE

DRAM

Multi Gigabit Fabric

DSP

SM-X

System FPGA

Peripheral Interconnect

DRAM

Console / Aux

Mgmt Ethernet

Flash

USB

4xPCIe DDR3 4xSGMI

DDR3

1xSGMI

10 Gbps/slot

NIM NIM

NIM

2Gb/slot

SM-X

10 Gbps XAUI


ISR 4451-X Architektura Sprzętowa

Data Plane (10 core)


PPE6 PPE7 PPE8 PPE1

0 PPE9

Control Plane (4 cores)

Ctrl SVC1

SVC2 SVC3

FPGE

DRAM

Multi Gigabit Fabric

DSP

SM-X

System FPGA

Peripheral Interconnect

DRAM

Console / Aux

Mgmt Ethernet

Flash

USB

4xPCIe DDR3 4xSGMI

DDR3

1xSGMI

10 Gbps/slot

NIM NIM

NIM

2Gb/slot

SM-X

10 Gbps XAUI

1 rdzeń - Control Plane Podobnie do RP z ASR1K

3 rdzenie serwisowe

(SourceFire)

10 rdzeni, 1 wątek / core Domyślnie 5 rdzeni fwd 4 pozostałe aktywowane dodatkową licencją

Szyfrowanie danych “in-line” Dane są szyfrowane podczas procesowania przez PPE (w ASR szyfrowanie odbywa się w osobnych chipie)

Brak sprzętowego TCAMu

BQS na dedykowanym rdzeniu Jeden rdzeń jest zawsze zarezerwowany dla BQS (implementacja Software’owa QoS)

For Your Reference


ISR 4451 Layout Systemu

10 core Dataplane 4 Cores Control and

Services Plane

Multi Gig Ethernet Fabric

Dataplane DIMM (left) and

Controlplane DIMM (2x right)

Integrated Services Card

(e.g. DSP)

Front panel PoE power

Compact Flash

1 SW-NIM or Dual HDD Configurable Slot (@ factory only)

For Your Reference


Akronimy

§  RP – Route Processor

§  FP – Forwarding Processor = ESP (Embedded Service Processor)

§  CPP – Cisco Packet Processor Compex= QFP (Quantum Flow Processor)

§  PPE – Packet Processing Engine

§  IOCP – I/O Control Processor

§  FECP – Forwarding Engine Control Processor

§  SPA – Shared Port Adapter

§  SIP – SPA Interface Processor

§  IOSd – IOS image that runs as a process on the RP

§  FMAN – Forwarding manager (FMAN-RP, FMAN-FP)

§  Scbac – FW Session Control Block

§  EOBC = Ethernet Out of Band Channels – Packet Interface for Card to Card Control Traffic

§  IOS-XE (BinOS) = Linux Based Software Infrastructure That Executes on MCP

For Your Reference

Dzień z życia pakietu (normalnego pakietu)


Przepływ w SIP10

…

ESPs

C2W

EV-FC

EV-RP

In ref clocks

Network clocks

SPA Agg.

SPA Aggregation ASIC (Marmot)

Ingress Scheduler

Egress Buffer Status

Ingress Classifier

Egress buffers (per port)

Network clock

distribution

IOCP (SC854x SOC)

…

Ingress buffers (per port)

…

Interconnect

DDRAM


JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl

Reset / Pwr Ctrl

SIP

SPA SPA

IOCP SPA

Aggreg.

intercon.

SPA


Crypto

FECP

Pakiet przychodzący z SIP


QFP Complex

TCAM Resource DRAM

Packet Buffer DRAM

Part Len / BW SRAM

SA table DRAM


DDRAM


JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

Interconnect

SPI Mux

ESP FECP

QFP Crypto Assist.

intercon.

PPE

BQS

PPE2 PPE2


Crypto

FECP

Pakiet jest przypisany do konkretnego rdzenia


QFP Complex

TCAM Resource DRAM

Packet Buffer DRAM

Part Len / BW SRAM

SA table DRAM


DDRAM


JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

PPE2

Interconnect

PPE2

Thread 1

Thread 2

Thread 3

Thread 4


Crypto

FECP


QFP Complex

TCAM Resource DRAM

Packet Buffer DRAM

Part Len / BW SRAM

SA table DRAM


DDRAM


JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

PPE2

Interconnect

PPE2

Thread 1

Thread 2

Thread 4

Thread 3

Pakiet jest przypisany do konkretnego wątku


Crypto

FECP

Pakiet jest obrabiany (FIA)


QFP Complex

TCAM Resource DRAM

Packet Buffer DRAM

Part Len / BW SRAM

SA table DRAM


DDRAM


JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

PPE2

Interconnect

PPE2

Thread 2

Thread 1

Thread 4

Thread 3

PPE2 Thread 3

X-Connect L2 Switch IPv4 IPv6 MPLS

Netflow

Input ACL

NBAR Classify

MQC Classify

…

NAT

PBR

Dialer IDLE Rst

URD

IP Unicast

IP Multicast

Packet For Us

Netflow

NAT

NBAR Classify

…

MQC Policing

MAC Accounting

Output ACL

Input FIA Output FIA


Crypto

FECP

Pakiet jest obrabiany (FIA)


QFP Complex

TCAM Resource DRAM

Packet Buffer DRAM

Part Len / BW SRAM

SA table DRAM


DDRAM


JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…


PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

PPE2

Thread 1

Thread 2

Thread 4

Thread 3

Interconnect


Netflow

BGP Accounting

NBAR Classify

MQC Classify

…

NAT

PBR

Dialer IDLE Rst

URD

IP Unicast

IP Multicast

Packet For Us

Netflow

NAT

NBAR Classify

…

MQC Policing

WRED

Output ACL

Input FIA Output FIA

PPE2 Thread 3


Crypto

FECP

Pakiet przekazany jest do QoSa potem do SIP


QFP Complex

TCAM Resource DRAM

Packet Buffer DRAM

Part Len / BW SRAM

SA table DRAM


DDRAM


JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

Interconnect

SPI Mux

PPE2 PPE2


Pakiet wychodzi z SIP

ESPs

C2W

EV-FC

EV-RP

In ref clocks

Network clocks

SPA Agg.

SPA Aggregation ASIC (Marmot)

Ingress Scheduler

Egress Buffer Status

Ingress Classifier

Egress buffers (per port)

Network clock

distribution

IOCP (SC854x SOC)

…

Ingress buffers (per port)

…

Interconnect

DDRAM


JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl

Reset / Pwr Ctrl

SIP

SPA SPA

IOCP SPA

Aggreg.

intercon.

SPA

Taktyki Troubleshootingu


Codzienność …

Pakiet nie dotarł do hosta docelowego! Co się stało?

Dlaczego to się stało ?


Używanie statystyk do znalezienia gdzie pakiet jest gubiony

SPA

§  show interfaces <interface-name>

§  show interfaces <interface-name> accounting

§  show interfaces <interface-name> stats

SIP

§  show platform hardware port <slot/card/port> plim statistics

§  show platform hardware subslot {slot/card} plim statistics

§  show platform hardware slot {slot} plim statistics

§  show platform hardware slot {0|1|2} plim status internal

§  show platform hardware slot {0|1|2} serdes statistics

RP

§  show platform hardware slot {r0|r1} serdes statistics

§  show platform software infrastructure lsmpi

ESP

§  show platform hardware slot {f0|f1} serdes statistics

§  show platform hardware slot {f0|f1} serdes statistics internal

§  show platform hardware qfp active bqs 0 ipm mapping

§  show platform hardware qfp active bqs 0 ipm statistics channel all

§  show platform hardware qfp active bqs 0 opm mapping

§  show platform hardware qfp active bqs 0 opm statistics channel all

§  show platform hardware qfp active statistics drop [detail]

§  show platform hardware qfp active interface if-name <Interface-name> statistics

§  show platform hardware qfp active infrastructure punt statistics type per-cause | exclude _0_

§  show platform hardware qfp active infrastructure punt statistics type punt-drop | exclude _0_

§  show platform hardware qfp active infrastructure punt statistics type inject-drop | exclude _0_

§  show platform hardware qfp active infrastructure punt statistics type global-drop | exclude _0_

§  show platform hardware qfp active infrastructure bqs queue output default all

§  show platform hardware qfp active infrastructure bqs queue output recycle all

Bardzo mało praktyczne…


Taktyki Troubleshootingu

IOS Control Plane •  ACL + show access-list,… •  show interface / ip route / bgp …

Platform Control Plane •  ESP “stuff” •  “show platform”

Data Plane •  ESP “stuff” •  Dalsza część “show platform”

Z góry na dół

Ciężkie

Dobrze znane

Zmieńmy to!

Hardcore

Troubleshooting ruchu 29


The Embedded Packet Capture

Device# show monitor capture mycap buffer dump 0 0000: 01005E00 00020000 0C07AC1D 080045C0 ..^...........E. 0010: 00300000 00000111 CFDC091D 0002E000 .0.............. 0020: 000207C1 07C1001C 802A0000 10030AFA .........*...... 0030: 1D006369 73636F00 0000091D 0001 ..example....... 1 0000: 01005E00 0002001B 2BF69280 080046C0 ..^.....+.....F. 0010: 00200000 00000102 44170000 0000E000 . ......D....... 0020: 00019404 00001700 E8FF0000 0000 .............. 2 0000: 01005E00 0002001B 2BF68680 080045C0 ..^.....+.....E. 0010: 00300000 00000111 CFDB091D 0003E000 .0.............. 0020: 000207C1 07C1001C 88B50000 08030A6E ...............n 0030: 1D006369 73636F00 0000091D 0001 ..example.......

Device# monitor capture mycap start Device# monitor capture mycap access-list v4acl Device# monitor capture mycap limit duration 1000 Device# monitor capture mycap interface GigabitEthernet 0/0/1 both Device# monitor capture mycap buffer circular size 10 Device# monitor capture mycap start Device# monitor capture mycap export tftp://10.1.88.9/mycap.pcap Device# monitor capture mycap stop

Pokazuję czy pakiet dotarł do naszego urzadzenia oraz czy nasze urządzenie pakiet

wysłało Pokazuję jak pakiet wygląda (export do

PCAPa) Nie mówi nam co stało się z pakietem!

Świetne narzędzie, ale nie zawsze wystarcza …

http://www.cisco.com/en/US/docs/ios-xml/ios/epc/configuration/xe-3s/asr1000/nm-packet-capture-xe.html

IOS 3.7


Crypto

FECP

The Packet Tracer and FIA Debugger


QFP Complex

TCAM Resource DRAM

Packet Buffer DRAM

Part Len / BW SRAM

SA table DRAM


DDRAM


JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

PPE2

Interconnect

PPE2

Thread 2

Thread 1

Thread 4

Thread 3 PPE2 Thread 3


Input ACL

MQC Classify

NAT

PBR

IP Unicast

Output ACL

NAT

Encaps

Crypto

Input FIA Output FIA Pak Match ?

Packet # 16

Input ACL

MQC Classify

NAT

PBR

Output ACL

NAT

Encaps

Crypto

Opcjonalnie można śledzić co się dzieję bardzo dogłębnie. Aż do poziomu

każdego kroku w tablicy FIA.

Jakiego pakietu szukamy ?

IOS 3.10

Statystyki oraz przeprowadzone akcje będą

zalogowane (pakiet upuszczony “dropped”,

pakiet przesłany do RP itd.)

DEMO ;-)


DEMO: Topologia sieci

ASR1000

? ?

PC2

SRV 172.16.140.100

PC1

PC3

GE 2 GE 1

PC5: 192.168.20.102


Packet-Trace: Configuration Commands

§  The Pactrac (Packet Tracer) shows us what happens to a series of packets –  True inspection of IOS XE packet forwarding flow

§  debug platform packet-trace enable –  Enables accounting –  Required for all levels of inspection

§  debug platform packet-trace packet <pkt-num> \ [fia-trace | summary-only] [circular] [data-size <data-size>]

–  Required for any per-packet data capture (e.g. necessary for packet copy to function) –  Specifies maximum number of packets maintained at one time (<pkt-num>) –  Always enables capture of summary data or only summary data (summary-only) –  Captures feature path data by default –  Optionally performs FIA trace (fia-trace) in addition to path data capture –  Allows specifying the size of the path data buffers (defaults to 2048)

For Your Reference



§  debug platform packet-trace copy packet {in | out | both} [L2 | L3 | L4] [size <num-bytes>]

–  Enables copy of the ingress and/or egress packets –  Optionally allows specifying where to start copy of the packet (L2 is default) –  Optionally allows specifying the maximum number of octets to copy (64 is

default)

Available XE3.11 and forward

§  debug platform packet-trace drop [code <code-num>] –  Enables retention only for dropped packets –  Optionally allows specifying retaining packets for a specific drop code –  Can be used without global/interface conditions to capture drop events*

*Drop event capture means the only the drop itself is traced not the life of the packet, but, it still allows capture of summary data, tuple data and the packet to help refine conditions or provide clues to the next debug step.

For Your Reference



§  clear platform packet-trace statistics –  Clears any collected statistics and data buffers –  Tracing must be stopped first (debug platform condition stop)

§  clear platform packet-trace configuration –  Removes all debug platform packet-trace commands

§  clear platform condition all –  Removes all debug platform condition and debug platform packet-trace

commands

For Your Reference



Packet-trace relies on the conditional infra to determine which packets are interesting. The condition infra provides the ability to filter by protocol, IP address and mask, ACL, interface and direction. A complete discussion of conditions is not made here but some illustrative examples are:

§  debug platform condition ingress –  Checks all incoming packets on all interfaces for all protocols

§  debug platform condition interface g0/0/0 ipv4 ingress –  Checks all IPv4 packets arriving on interface g0/0/0

§  debug platform condition interface g0/0/0 ipv4 access-list FOO ingress

–  Checks incoming IPv4 packets on interface g0/0/0 that match access-list FOO

Conditions are activated or de-activated using debug platform condition start or debug platform condition stop respectively.

For Your Reference



NOTA BENE!!!!!

Conditions define what the filters are and when the filters are applied to a packet. For example, debug platform condition interface g0/0/0 egress means that a packet will be identified as a match when it reaches the output FIA on interface g0/0/0 so any packet-processing that took place from ingress up to that point is missed.

Best Practice

It is highly recommended to use ingress conditions for pactrac to get the most complete and meaningful data. Egress conditions can be used but just be aware of the limitation above.

For Your Reference


Packet-Trace: Configuration Example

The following shows how one would trace the most recent 128 packets entering GigabitEthernet0/0/0 including FIA trace and a copy of up to the first 2048 octets of the input packet.

debug platform condition interface g0/0/0 ingress

debug platform packet-trace enable

debug platform packet-trace packet 128 fia-trace circular

debug platform packet-trace copy packet input size 2048

debug platform condition start

<…wait until you’ve captured the packets you think you want…>

debug platform condition stop

For Your Reference


Packet-Trace: Configuration Highlights

§  Pactrac buffers consume QFP DRAM –  Be mindful of how much memory a config needs and how much memory is

available

§  Configure as much detail as you want…more detail…more performance impact for matched packets

§  Each pactrac “config” change will temporarily disables pactrac and clears counts/buffers –  “Cheap” way of ‘debug plat cond stop’, ‘clear plat pack stats’ and ‘debug

plat cond start’

§  Some configs require a ‘stop’ in order to display summary or per packet data –  Currently circular and drop tracing

§  Conditions define where and when filters are applied to a packet

For Your Reference


Packet-Trace: Show Commands

Show commands are used to display pactrac configuration and each level of data: §  show platform packet-trace configuration

–  Displays packet-trace configuration including any defaults

§  show platform packet-trace statistics –  Displays accounting data for all pactrac packets

§  show platform packet-trace summary –  Displays summary data for the number of packets specified by debug platform

packet-trace packet

§  show platform packet-trace packet { all | <pkt-num>} [decode]* –  Displays all path data for all packets or the packet specified –  Decode attempts to display packets captured by debug platform packet-trace

copy in user friendly way –  * decode was introduced in XE3.11

§  NOTE: only a few protocol headers are supported initially (ARPA, IP, TCP, UDP, ICMP)

For Your Reference


Example of Packet-Trace Configuration

For Your Reference


Example of Packet-Trace Accounting

For Your Reference


Example of Packet-Trace Summary

in0/0/rp:0 is how the ESP sees the RP

For Your Reference


Example of Packet-Trace Packet Details

For Your Reference


Example of Clearing Packet-Trace Stats

For Your Reference

Sprawdzanie jak zajęty jest system … 47


Sprawdzanie jak zajęty jest system

show processes cpu pokazuję tylko obciążenie RP!

ASR1K# show platform hardware qfp active datapath utilization CPP 0: Subdev 0 5 secs 1 min 5 min 60 min Input: Priority (pps) 22 19 18 19 (bps) 12736 11368 10592 11048 Non-Priority (pps) 21 17 17 17 (bps) 21624 17320 16624 16240 Total (pps) 43 36 35 36 (bps) 34360 28688 27216 27288 Output: Priority (pps) 20 18 17 18 (bps) 16112 14664 13608 14272 Non-Priority (pps) 14 10 10 10 (bps) 25128 14800 14552 14576 Total (pps) 34 28 27 28 (bps) 41240 29464 28160 28848 Processing: Load (pct) 0 0 0 0 …

Podsumowując

49


50

Nowa strategia troubleshootingu!

IOS Control Plane •  show interface, show ip route, show

bgp … •  Feature debugging

Platform Control Plane •  Unified show commands •  Platform show commands •  Future: control plane conditional

debugging

Data Plane •  Packet Tracer •  Forwarding plane conditional

debugging •  Embedded Packet Capture

Ciężkie

Proste!

Dobrze znane

Dziękuję ! [email protected]

51

plnog14: architektura oraz rozwiązywanie problemów na routerach ios-xe - piotr kupisiewicz

Internet