node controller
DESCRIPTION
ControllerTRANSCRIPT
Node Controller Traffic overview of the master
module
• GGSN-C Node Controller/master module – Load balancing - distributes traffic over GGSN-C, GGSN-U/I
and GGSN-T – Supervises the GGSN PICs and takes action if there is a
failure. – Admission control – Imsi/Msisdn tree – Local IP pool – MS Route handling – O&M handling – SBR – Charging – QoS implementation – etc
? GTPCD
GCCP GUCP IPC OAM
Core
APP
RE Proxy
GTP Basic
LIH
DHCP RADIUS
FSAD SAC
CHARGING
GTP
ANP Mgr NCT LI TDM
MASTER PDP
SrvLib junos_lib RPD XMLParse XMLTok
gtpcd libraries
SSA/ SNMP
librpd libirsd pwc
JunOS libraries
Application
Services
Basic
Framework
Support Layer
SBCC
DBP
PRSI PPSI
SACC
DAPP
DATA Data
? GTPCD
GCCP GUCP IPC OAM
Core
APP
RE Proxy
GTP Basic
LIH
DHCP RADIUS
FSAD SAC
CHARGING
GTP
ANP Mgr NCT LI TDM
MASTER PDP
SrvLib junos_lib RPD XMLParse XMLTok
gtpcd libraries
SSA/ SNMP
librpd libirsd pwc
JunOS libraries
Application
Services
Basic
Framework
Support Layer
SBCC
DBP
PRSI PPSI
SACC
DAPP
DATA Data
Components in Node Controller
• GTPCD_MULTICORE – set a compilation time • Core loop
– Gccp – Configuration – Statistics – U/T/SC handling – Etc
Node controller – thread view Common data
Legacy Coreloop ”thread”
GNSP threads
GTP v0/v1 threads
Admission control threads.
Producer – Consumer Queues
SGSN SC 1 thread on each CPU core on M120
M20/120 M120 only
Get Read lock
Get Read lock
Get Write lock
GCCP SC, U/T PIC etc.
Threads and locks
Adm. Ctrl.
MT_IPC
GTP v97 v99
From SGSN
To/from SC
MtIpc Send
MtIpc Send
callback
UDP
TCP
Core
GTP
Adm Ctrl
W
R
t
W
R
R R
MT_IPC
Traffic flow in PDP create GTP-C/NC RPD SGSN
FSAD GTP-C/SC GTP-U/I
GCCP_FWD_PDP_ CREATE_IND
GCCP_ADMCTL_ ACQUIRE_REQ
GCCP_ADMCTL_ ACQUIRE_RSP
rpd_cb_rt_add ()
GUCP_SESSION_ OPEN_REQ
GUCP_SESSION_ OPEN_RSP
GTP Create PDP Response
GTP Create PDP Request
rpd_rt_add (ifl,rtid,MS slice)
SGSN calling GTP-C/NC
GTP Create PDP Request
NcGtpThread_Run(..) 1. Reads GTP packet from socket 2. Basic packet sanity check 3. Determines if create or echo request or drop the packet
ProcessCreatePdpRequest(..) 1. Decodes apn, imsi, msisdn, nsapi and pdp_address
2. Adds imsi in table (only shadow entry at the moment) and msisdn
3. Check for max pdp per apn (avoid Radius overflow)
4. Gets a C-Pic to run on (least loaded) 5. Gets a charging id
SendFwdPdpCreateInd(..)
1. Build a GNSP message 2. Send to designated C-Pic
master_gtpthread.c: Running on thread
GTP v1 or v0
Admission acquire GTP-C/NC
AdmCtrl_Req(..) 1. Update load from slave (used previously)
2. Use of T-pic check -> get one. 3. Check max contexts on node 4. Check max context per APN 5. Handle QoS 6. Check IP address 7. Get a U-pic 8. Sends back to calling C-pic.
GCCP_ADMCTL_ ACQUIRE_REQ
adm_ctrl.c:
AdmCtrl_ReqCb(..) 3 threads handling IPC, packet FWD on M120
Running on 2 threads on M120
AdmCtrl_ScheduleWorker(..)
QoS i.e. 8 7 6 5 4 3 2 1
Quality of service IEI octet 1
Length of quality of service IE Octet 2
0 0 spare
Delay class
Reliability class
octet 3
Peak
throughput 0
spare
Precedence class
octet 4
0 0 0 spare
Mean throughput
octet 5
Traffic Class Delivery order Delivery of erroneous SDU Octet 6
Maximum SDU size Octet 7
Maximum bit rate for uplink Octet 8
Maximum bit rate for downlink Octet 9
Residual BER SDU error ratio Octet 10
Transfer delay Traffic Handling priority Octet 11
Guaranteed bit rate for uplink
Octet 12
Guaranteed bit rate for downlink Octet 13
0 0 0 spare
Signal-ling Indicat-ion
Source Statistics Descriptor Octet 14
Maximum bit rate for downlink (extended) Octet 15
Guaranteed bit rate for downlink (extended) Octet 16
Maximum bit rate for uplink (extended) Octet 17
Guaranteed bit rate for uplink (extended) Octet 18
DSCP DSCP
DSCP
SGSN
SGSN SGSN
SGSN
SGSN
SGSN
QoS DSCP (Diff Serv Code Point)
• Used on both uplink and downlink IP packets • Dependent upon neqotiated QoS:
– Traffic class – Transfer delay – SDU error ratio
and the APN configuration. DSCP bits are set in the Gi and Gn IP headers.
DSCP cont.
• DiffServ is a coarse-grained, class-based mechanism for traffic management.
• DiffServ operates on the principle of traffic classification, where each data packet is placed into a limited number of traffic classes.
• DiffServ relies on a mechanism to classify and mark packets as belonging to a specific class.
DSCP uplink packet marking
Admission release GTP-C/NC
AdmCtrl_Rel(..) 1. Update load from slave 2. Update load on DSCP. 3. Decrease the number of PDP in slice. 4. Decreate PDP counter on U-pic
statistic , APN and total GGSN 5. Release PDP address 6. Sends release resp back to calling C-
pic.
GCCP_ADMCTL_ RELEASE_REQ
Running on 2 threads on M120
AdmCtrl_ScheduleWorker(..)
adm_ctrl.c:
AdmCtrl_ReqCb(..) 3 threads handling IPC, packet FWD on M120
Local pool
• Support for fast ip address allocation from subnets for the apn
• Clean up ip addresses for a specific slave • No reuse of ip addresses before for a
configured time • At U pic limit response, flag not reuse the
address
Soak timer
Adm release request
Soak timer Soak timer Soak timer
used soaked
Adm Create request
released Available
• Three bitfields used to secure this (per IP address) – Used 0 1 1 1 0
Released 1 -> 1 -> 0 -> 1 -> 1 Soaked 1 1 1 0 1
• Config options, functions used
– Master_Wa.Config.Pdp.bIpReuseTimeout – “no-address-reuse-quarantine” – “address-reuse-timeout” – 10 - 1800s. – Master_Wa.Config.Pdp.noReleaseAddrOnFullGtpu – “pdp1 no-address-release-on-exhausted-
gtpu” • pReleaseInfo->wNoSoakOnPdpIp • pReleaseInfo->noResourcesOnGtpu
Local pool structure
• Batch 128 bits (addresses) • Slice 1024 addresses
Address pool workarea List of GTP-SCs
Hash table of APNs
Batches of 128 bits
Hash table of APNs
Full slice tree
Available slice tree
Released batch bitset list List of address ranges
Batches of 128 bits Slice tree
Bit 1: used Id Bit 2: Release Id Bit 3: Soaking Id
IdRange_t
idSlice_t IdSoakBatch_t
idSlice_t
IdRange_t
IdSubSource_IdBatch_t -> available Id
IdSubSource_idSlice_t
IdSubSource_IdRange_t
subSource_t
IdSubSource_IdSource_t
MasterAddrPool_t
IdSource_t
Calculate address and id
• Id (in relation to slice) = ((slice no) * (SliceSize)) + (BatchNum * BatchSize) + BatchId
• Address = Subnet.StartAddr + Id (ipv4)
• To find batch => – bitPos = Id – ((slice no) * (SliceSize)) – BatchNum = bitPos / BatchSize – BatchId = bitPos – BatchNum * BatchSize
Release / Failure scenarios
• Adm Ctrl Release – adt_toggleAddr (setOff = TRUE, noSoak?, NoCb = TRUE)
• Adm Ctrl Request failure (U limit, other error) – adt_toggleAddr (setOff = TRUE, noSoak?, NoCb = TRUE)
• Session controller failure – toggleId called directly (setOff = TRUE, noSoak = FALSE,
NoCb = FALSE)
• Node Controller failure – session controller takeover – adt_toggleAddr (setOff = FALSE, noSoak = FALSE, NoCb =
TRUE)
Important implementation details • Find and create an address - Master_adtAllocAddr( U32 ApnIndex, Slave_Info_t *pSlave,
InAddr2_t *pAddr, U8* reqSubnetName )
• Remove an address - Master_adtFreeAddr -> adt_toggleAddr -> ap_toggleId
• The actual toggle of the id for the address - ap_toggleId(Apnmgt_ApnEntry_t *pApn, U8 slaveIndex, U32 id, U8 masterRangeIndex, U8 setOff, U8 noSoak, bool bNoCb)
• Background task every soak timer call, next soak timer start after the previous is done - ap_BTSoakTimerAction -> ap_loopSoakBatch( IdSoakBatch_t * pIdSB, idSlice_t * pSlice)
• MAP_MASTER_INDEX_TO_APNMGT_INDEX(Master_index, pApn) Master_Wa.pApnTable[pApn->ApnIndex]->Master_to_Apnmgt_Mapping[Master_index]
• Master_Wa.pApnTable[pApn->ApnIndex]->Range[index]
• adt_convertIdToAddr
• Mutex_Lock(Master_Wa.pIdPoolMtx);
GTP-U load balancing and slice handing
• Steps – IP address selected – Find range from apn – Find slice in slice tree from apn range - RwMutex_LockRead(pRangeInfo-
>pRange->SliceTreeMtx); • If slice is not found
– Create a new slice – Find least loaded Upic - Master_GetLeastLoadedGtpu – admCtrl_AssignSliceToGtpu
• Assign slice to Upic • Add Upic index to slice • Add slice to SliceTree on range • Add route for slice - Master_AddSliceRoute -> Rpd_AddRoute
• If existing just find the U pic – Check that U still is active and okay
• Respond back to session controller
Conclusions and Questions