ddb phase1-2 explanation and toubleshooting guide

7/28/2019 DDB Phase1-2 Explanation and Toubleshooting Guide

1/16

1.0 DDB PHASE 1.......................................................................................................2

1.1 INTRODUCTION.................................................................................................. 2

1.2 DEFINITIONS.......................................................................................................2

1.2.1 DDB Quiet Period......................................................................................... 2

1.2.2 OAM DDB audit............................................................................................ 3

1.2.3 DDB real-time checksums............................................................................3

1.2.4 New DDB statistics.......................................................................................3

1.2.5 DDB wild write audit.................................................................................... 4

1.2.6 DDB Timers..................................................................................................4

1.3 DDB debugging.................................................................................................. 5

1.3.1 Description of DDB send-msg debug commands table................................5

1.3.2 dbg-ddb command.......................................................................................6

1.3.3 send-msg to dbg-ddb command mapping...................................................7

2.0 DDB PHASE 2........................................................................................................ 9

2.1 Mechanism DIFFERENCES WITH DDB phase 1...................................................9

2.2 DDB wild write audit..........................................................................................9

2.3 NEW DDB Timers.............................................................................................10

3.0 METHODS OF RESOLUTION OF DDB ISSUES ......................................................113.1 Adding a secondary route by using a dummy route to trigger a route state

change .................................................................................................................. 11

3.2 CARDS report DDB MISMATCH checksum FOR LINKSET table .......................13

3.3 CARDS report DDB MISMATCH checksum for LInk table ..............................14

1


2/16

1.0 DDB PHASE 1

1.1 INTRODUCTION

The accuracy of the DDB checksums is determined based on the DDB idle period

collected from the MTP cards. If all idle periods reported are greater than the DDBquiet period, the checksums are deemed to be accurate.

The consistency of the DDB is determined based on the cross-reference of allchecksums on all cards for mistmatch. The evaluation of the DDB consistency is onlyachieved if the DDB checksums are deemed accurate.

The DDB audit is automatic and periodic. The user may also manually trigger anaudit. The user may also use DDBAUDTIMER SS7OPTS option to enable anddisable the background audit.

If any inconsistencies are detected by the OAM as a result of a background audit ormanual audit, a system alarm is raised in-order for the user to take the appropriatesteps to remedy the DDB corruption. The alarm is cleared after an audit with noinconsistencies (either background or manual).

The MTP cards (LIM and SCCP), calculate and record DDB checksums in real-time asDDB updates are applied.

When an audit request is received from the OAM, each MTP card collect all of itsDDB checksums and send them in a reply to the OAM card.

In addition to providing the OAM with the DDB checksums, the MTP cards, wouldalso attach a time stamp representing the time elapsed in milliseconds since the lastDDB update was processed (DDB idle period).

The MTP cards implement new DDB statistics. The statistics measure the DDBupdate rates. Their intent is to help in quantifying the DDB update traffic.

The new DDB statistics are not EAGLE measurements and are not saved when acard reboots.

1.2 DEFINITIONS

1.2.1 DDB Quiet Period

The DDB quiet period is defined as the minimum DDB idle time (no DDB updatesapplied) after which it is safe to assume that all DDB updates in the system havebeen processed and no outstanding in-flight multi-cast update exists.

The DDB quiet period shall be greater than the sum of:

1. The maximum time it takes a multicast to reach all cards

2


3/16

2. The maximum time for a DDB multicast to be processed by the receiving cardwhen there are no other updates to process.

3. The maximum time for a broadcast to be received and processed by a card.

The addition of the broadcast time is due to the fact that the DDB audit uses IMTbroadcast to collect checksum data. Since the broadcast maybe sent on both A&B

IMT buses, a card may receive duplicate audit requests while another card may missone of the two. The quiet period as defined above insures that if a card replies toany of the two broadcasts it receives, the data reported stays valid (see timingdiagram below).

1.2.2 OAM DDB audit

The DDB audit mechanism is triggered either:1. Manually, using a aud-data command (refer to section 3.4.2)

2. Periodically by a background process.

The audit process consists of an audit message broadcast[1] (using IMT broadcastcapability) by the active OAM card to all cards in the system.Upon reception of the audit request, each MTP card in the system, would read thereal-time DDB checksums for each table and save them into a reply message.Non MTP cards in the system shall ignore the DDB audit message and shall notoutput any error message.

1.2.3 DDB real-time checksums

Currently, the DDB checksums are calculated for each DDB table when requested bythe user (OAM card).

The DDB checksum for each table is a calculated as a sum of each entry checksum.Each entry checksum is a Polynomial checksum.

The current computation of DDB checksums is time consuming.The DDB checksums shall be updated in real-time as follow: when a DDB entry isupdated, the polynomial checksum is re-calculated and the overall DDB tablechecksum is adjusted accordingly based on the old and new entry checksums.

All DDB tables shall be expanded to include the entry DDB checksum.

1.2.4 New DDB statistics

New DDB statistics are proposed for implementation on MTP cards. These are stand-

alone statistics and are not measurements.These statistics are implemented on the MTP cards as counters saved in thememory. They are reported with each DDB audit reply and on demand via the dbg-ddb command (see section 1.3.2).

[1] True group broadcast feature is expected to be implemented in the future when all MTP cards will become a part

of the MTP broadcast list. When this feature becomes available, individual cards and GPLs will not need to filter anddiscard broadcast messages they dont need. Also, the group broadcast feature will have the COM processor send

only one copy of the message to the APPL, hence obviating the need to discard duplicates in the future

3


4/16

The statistics are saved on the OAM with a time stamp when reported. Even thoughthe checksums in the audit replies maybe discarded due to a short DDB idle period,the DDB statistics are valid and shall be saved in memory.

The following DDB statistics shall be implemented:

1. The number of DDB updates received and applied. These counters shall besynchronized during the DDL phase.

2. The average DDB updates per 100 ms.

3. The maximum DDB updates per 100 ms.

4. The average DDB idle period.

5. The maximum and minimum DDB idle period.

1.2.5 DDB wild write audit

Since the DDB checksums are updated in real-time, a wild write to any DDB tablewould result in the DDB checksum(s) to be out of sync.

A wild write may be caused by:

1. A DDB update which does not update the checksums

2. A memory violation caused by a software error

An audit process is implemented on all MTP cards.The process is periodic and time sliced. When triggered the processes runs on a 5ms time-slice.Every time-slice the process computes a maximum number of DDB entrychecksums.

1.2.6 DDB Timers

Timer Description Value(s)

DDB backgroundaudit timer

Time interval between consecutive DDB audits 5 to 1440minsDefault: 10mins

DDB audit retry

timer

Time between retries when DDB idle period

threshold is not met.There shall be only 3 retries.

2, 5, 10

secs

DDB audit repliestimer

Time within which all audit replies are received. 2000 ms

DDB wild writeaudit timer

Timer interval between consecutive Wild writeaudits

60 mins

DDB wild writetime-slice

Time intervals between each time-slice of DDBwild write audit process

5 ms

4


5/16

DDB Quiet period Default value for DDB quiet period 500 ms

1.3 DDB DEBUGGING

Currently, EAGLE provides the user with a multitude of send-msg commands thatmay help in debugging the dynamic database.

The existing debug functionality offers the user commands to access individual DDBlink, link-set and route entries.

In addition the user is provided with commands to audit DDB at a debug level asfollow:

Per card DDB table audit: For a given source card and a reference card,the source card sends an audit request for the specified table. The referencecard responds with the checksum for the table.

Per entry DDB table audit: If the source cards in the above audit (per card

audit) detect a table checksum mismatch, it audits every entry in the tablewith the reference card. This process starts at entry index zero. The sourcecard sends a per entry audit request to the reference card with the table IDand starting index. The reference card calculates as many entry checksumsas it can pack in a reply.

The existing debug send-msg commands are consolidated into one dbg-ddbcommand. The new dbg-ddb command provides an easy to use wrapper. Forexample, in order to display a DDB route or link entry, the user is required toprovide the entry index if the send-msg command is used. The new dbg-ddbcommand allows the user to avoid the time consuming task of figuring out entryindexes and simply provide the link location or route DPC to display a specific entry.

The table below describes the send message commands consolidated by the newdbg-ddb command.

1.3.1 Description of DDB send-msg debug commands table

Dest.Appl.

Function

Description

He1 H83 This command is used to display an entry within a cards DDBtables.

The DDB tables supported are: Link, Link-set and Route

Parameters:

D0 : DDB table (0:Link, 1:Link-set, 2: Route)

D1 : Least significant byte of the table entry index

D2: Most significant byte of the table entry index

He1 H86 Card to System DDB table audit. (Unicast).

This command audits a specified DDB table.

A per table audit message is issued to a selected reference MTP

5


6/16

card. The card selection is based on an internal list. Every timethis command is invoked the next card in the list is selected.

If the process of the per table audit reply leads to no checksummismatches, a new per table audit is sent to the next card in thelist.

If a mismatch is detected, a per entry audit process is startedwith the reference card. When this process ends, the total numberof mismatches and the first entry mismatch are displayed.

Parameters:


He1 H85 Card to System DDB table audit. (Broadcast)

This command audits a specified DDB table.

per table audit messages are sent to every MTP card in thesystem. The messages are sent 10 at once every 20ms.

The per table audit replies are processed and the checksums forthe reference cards are stored.

When all per table audit replies are received, processing isstarted. If a table checksum mismatch is found, a per entry auditis started with the reference card that reported the mismatch.

Parameters:


He1 H80 Card to card table audit.

This command does a per entry audit for the specified table with

the reference card specified.Parameters:


D1 : Reference card IMT address

1.3.2 dbg-ddb command

This is a new command used to debug DDB.

Parameter M/O DescriptionLOC M Card location: the location of the card where the command

would be sent.TBL M DDB table:

Values: lnk, ls, rte

Note: Only Links, Link-sets and route DDB tables are

6


7/16

supported. There is no support for the remaining DDB tables.

ACTION M disp: Displays a DDB table entrystats: Sends a message to the card to display the DDBstatistics.audit: Audits a specific DDB table

AUDTYPE O Audit type: Using unicast or multicast

Values: MC, UCDefault: MC

TIDX O DDB table index: optional. Used only when action=disp

Mutually exclusive with options: LSN, RLOC,LINK and DPCLSN O Link-set name.

Used only when action=disp and tbl=ls

Mutually exclusive with options: TIDX, RLOC,LINK and DPCRLOC O Reference card location.

Used only when action=disp and tbl=lnk or when action=AUDand audtype=UC

Mutually exclusive with options: TIDX ,LSN and DPCLINK O Link (A..B31)

Used only when action=disp and tbl=lnk

Mutually exclusive with options: TIDX , LSN and DPCDPC O Destination point code.

Used only when action=disp and tbl=rte

Mutually exclusive with options: TIDX,LSN, RLOC and LINK

The following table shows the translation between the dbg-ddb command and thecurrently implemented send-msg commands.

1.3.3 send-msg to dbg-ddb command mapping

DBG-DDB command DS DA F D0 D1 D2

Tbl=LNK:action=DISP:TIDX=n 1 He1 H83

0 n-LO1

n-HI2

Tbl=LNK:action=DISP:RLOC=x:LINK=yThe RLOC and LINK shall be converted into alink index (n).

1 He1 H83

0 n-LO

n-HI

Tbl=LS:action=DISP:TIDX=n 1 He1 H83

1 n-LO

n-HI

Tbl=LS:action=DISP:LSN=The link-set name shall be converted into a

1 He1 H83

1 n-LO

n-HI

1

2

7


8/16

link-set index (n).Tbl=RTE:action=DISP:TIDX=n 1 He1 H8

32 n-

LOn-HI

Tbl=RTE:action=DISP:DPC=a-b-cThe DPC shall be converted into a route index(n)

1 He1 H83

2 n-LO

n-HI

Tbl=LNK:action=AUD:audtype=MC 1 He1 H85

0

Tbl=LS:action=AUD:audtype=MC 1 He1 H85

1

Tbl=RTE:action=AUD:audtype=MC 1 He1 H85

2

Tbl=LNK:action=AUD:audtype=UC 1 He1 H86

0

Tbl=LS:action=AUD:audtype=UC 1 He1 H86

1

Tbl=RTE:action=AUD:audtype=UC 1 He1 H86

2

Tbl=LNK:action=AUD:audtype=UC:RLOC=xThe alternative location shall be translatedinto an IMT address (A)

1 He1 H80

0 A

Tbl=LS:action=AUD:audtype=UC:RLOC=xThe alternative location shall be translatedinto an IMT address (A)

1 He1 H80

1 A

Tbl=RTE:action=AUD:audtype=UC:RLOC=xThe alternative location shall be translatedinto an IMT address (A)

1 He1 H80

2 A

8


9/16

2.0DDB PHASE 2

2.1 MECHANISM DIFFERENCES WITH DDB PHASE 1

In DDB Phase 1, when any DDB variable is updated, a checksum would be calculatedfor the particular entry and the table right away - e.g. if a link state changes, the linkentry and the link table checksum would be immediately calculated. Forperformance reasons and for maintainability, DDB Phase 2 implements abackground DDB checksum task that re-calculates checksums for the entry, and thetable after the fact.

In DDB Phase 2 (like in DDB Phase 1), the quiet time requirement must be met foran audit calculation to come up with a result of either consistent or inconsistent.

An additional check will be done on per card basis (which will be described in moredetail later.) If DDB Update in progress is TRUE, either DDB checksums have notbeen calculated completely on that card by the new DDB checksum task, or the(existing) SS7 TSRC task has not completed evaluating all routes in the route table.

The first check (quiet time requirement) is mandatory and is required to be met.Once this is satisfied, the second check is used to determine whether a cards DDBchecksums can/should be used for overall DDB checksum calculations anddetermination of DDB state.

The DDB audit task that monitors and determines DDB state has the following twomajor functionality:

1. Dirty Bit audit (checksum calculation) (new in DDB Phase 2).

2. Wild Write Audit (WWA) (Updated the Phase-1 implementation)

2.2 DDB WILD WRITE AUDIT

WWA has been modified in phase to run in everytimeslice with-in the audit task. Themajor functionality of WWA is as follows:

1. WWA will only work when the table under monitor is clean i.e. tabledirty bit has not been set.

2. Check for checksum discrepancy for each table and table entry andrectify the checksum

3. In case WWA finds any table entry with the dirty bit as set then WWAwill skip the entries for the dirty audit task.

When a checksum is identified to be incorrect and is updated by a wild write audit,the user may want to know that a DDB inconsistency reported on a card was due to

9


10/16

a wild write (rather than any other cause, like missed multicast). Hence in DDBPhase 2, each MTP card will store the last 10 entries in chronologically reverse orderwhich were updated by wild write audit on that card. These in-memory entries willget erased when the card reboots. The user can query these wild write entries byissuing the command: dbg-ddb=:action=wwa (new parameter).Following information will be stored for each such entry.

1. On card time stamp (milliseconds since card initialized) when the checksumdetected and updated. (At this time, the current date and time are notaccessible readily on the MTP cards, in the future, this card specific timestamp could be replaced by a date and time).

2. Table ID/name (e.g. Link table, Linkset table, Route table etc.)

3. Entry ID (For tables which have no entries, only one table, the entry ID fieldwill be blank)

4. Original checksum

5. Checksum after update by wild write audit task

2.3 NEW DDB TIMERS

Timer Description Value(s)

DDB backgroundaudit timer

No change in Phase 2: Time interval betweenconsecutive DDB audits

5 to 1440minsDefault: 10mins

DDB audit retrytimer No change in Phase 2: Time between retrieswhen DDB idle period threshold is not met.There shall be only 3 retries.

2, 5, 10secs

DDB audit repliestimer

Changed from 2000 ms to 4000 ms in DDBPhase 2. Time within which all audit replies arereceived. (This should account for STH pollperiod)

4000 ms

DDB Audit timeslice

Time intervals between each time-slice of DDBaudit process

5 ms

10


11/16

3.0METHODS OF RESOLUTION OF DDB ISSUES

When a system report DDB inconsistencies t he first thing to check in the system are

the IMT busses statistics in order to make sure they are clean. This point is reallyimportant since multicast updates resulting from network activity transit via the IMTbusses and any outstanding issue on this part of the system may lead some cards tomiss the updates and OAM to report DDB inconsistencies.

Starting release 41.1 when a checksum is identified to be incorrect and is updatedby a wild write audit, the user may want to know that a DDB inconsistency reportedon a card was due to a wild write (rather than any other cause, like missedmulticast).

In the case one or a group of cards miss a DDB update related to a network statechange then the counter collecting the total number of update misses is

incremented. This counter can be retrieve either via a send-msg or a debug ddbcommand. The parameters to use in the dbg-ddb command is depending on thetype of DDB update missed (Route/link/linkset ). For more details refer to paragraph1.3.

The more DDB updates one or a group of card would miss and the less chance thecard would have to get a consistent DDB. The number of misses for a card coulddecrease if the same state change leading a DDB update to be multicast to othercards occurs and this time it is not missed by the card with its DDB inconsistent.

This means that a card needs the same state changes that have been previouslymissed to recur in order for the total number of misses to abate till value 0 isreached and when this happens then the card would have its DDB consistent.

Example 1 Consider the following scenario a card is missing for some reasons a DDBupdates which are multicast through the IMT and then got out of sync. In this casebased on this the new checksums computed for this card would be different thanother cards which have received all updates and then this card would be reported ashaving a DDB inconsistent.

To bring the card back the consistent state there can be many ways. Such as firstfinding out where the inconsistency is, such as if the inconsistency is in routetable etc. Then there can be steps to rectify this problem. The below explains whichoption can be used in order to resolve DDB issues.

3.1 ADDING A SECONDARY ROUTE BY USING A DUMMY ROUTE TO TRIGGER A ROUTESTATE CHANGE

There is method to clean DDB checksum inconsistencies for route table checksums. This method is not tobe communicated to the customer and has to be applied by experienced TAC engineer.

11


12/16

The method consists of adding a secondary route (or a route with the lowest cost if more than one routeto the DPC already exist) using an APC which is in a prohibited status to a DPC. The DPC is given bythe send-message result as displayed by the below command. The use of the prohibited APC to createthe route does not disturb the current behavior of the system. When the route is created then the secondstep consists of deleting the route just after. When this done then run the send-msg again which shouldgive another PC to which the method is to be applied and the totalMisses should decrement each time.Perform this while the send-message does not report that the route status is clean.

Example:

With a send message the command would be run on card 1317 which report a ddb checksum issue forRTE table.send-msg:ds=1:da=h'e1:f=h'86:d0=2:loc=1317

With a dbg-ddb command by using card 1314 as a reference card in which no DDB checksum issue isreported

Dbg-ddb :loc=1317 :Tbl=RTE:action=AUD:audtype=UC:RLOC=1314

The commands will provide a total misses number and the first PC to which the route using a dummylinkset will be added

paristk1 09-01-06 11:53:27 FWT EAGLE

[1317]Card->System (Tbl:2) Diff:Card[4217] LclCksm:h'65ad RmtCksm:h'63ad

TblAuditDone:TotalMisses:h'1 FirstMiss:h'661

;paristk1 09-01-06 11:53:27 FWT EAGLE

[1317] [Route:h'661] Chksum h'b426 at h'76e04a (42 bytes)

PC: 6-039-6 LstRt:0 CmbRt:6 Dyn:h'1 TFC:0

Xlst:0 MOBR:2 NAdj:1 3 3 3 3 3 NmTFR:0PrevStat:1

SRT:h'727db2 (26 bytes) AKT:h'896afc (12 bytes)

;

paristk1 09-01-06 11:53:27 FWT EAGLE

[4217] [Route:h'65c] Chksum h'b226 at h'876d298 (42 bytes)

12


13/16

PC: 6-039-6 LstRt:0 CmbRt:6 Dyn:h'1 TFC:0

Xlst:0 MOBR:0 NAdj:1 3 3 3 3 3 NmTFR:0PrevStat:1

SRT:h'86ac7b8 (26 bytes) AKT:h'8744030 (12 bytes)

;

2/The next step should be adding a route to that PC 6-039-6 using a linkset which has its APCprohibited. The addition of the new secondary route will trigger a route state change for that point codeand then a DDB update will be sent to the card 1317.

3/ the route should now be deleted and the above send-message/dbg-ddb run again to get the new PCto which the route should be added/deleted in order to continue decrementing the number of DDB updatemissed and clear the DDB inconsistency.

As the total number of DDB update misses is 1 then this one would decrement to 0 as the card 1317 hasnow received the missed update. The card 1317 gets now its DDB consistent and DDB alarm is cleared.Note that this method is useful when the number of misses is quite reasonable otherwise it may take a

long time to trigger all DDB updated missed and clear the DDB inconsistency.

3.2 CARDS REPORT DDB MISMATCH CHECKSUM FOR LINKSET TABLE

The method is quite similar to the one explained in section 3.1.

For example if some cards report DDB checksum mismatches then the method ofresolution consists of running DBG-DDB command or via send-msg to identify thelinkset from which the cards missed DDB updates. Actually the debug commandstill reports an adjacent point code number from which the linkset can be deducted.

The below is an example of debug command that can be run to figure out the pointcode number in question and the total number of DDB update misses.

The debug command use a reference card in loc 1314 and the card reporting theDDB mismatch for the linksets. The point code is 867-aa and the total number ofDDB update missed by card 1313 is 1.

> Dbg-ddb:loc=1313:Tbl=ls:action=AUD:audtype=UC:RLOC=1314

tstpe0akb1 09-10-20 14:31:26 IST EAGLE5 41.0.0-62.33.0Dbg-ddb:loc=1313:Tbl=ls:action=AUD:audtype=UC:RLOC=1314Command entered at terminal #19.

;Command Accepted - Processing

tstpe0akb1 09-10-20 14:31:26 IST EAGLE5 41.0.0-62.33.0User Message sent to location 1313.

;

13


14/16

Command Executedtstpe0akb1 09-10-20 14:31:27 IST EAGLE5 41.0.0-62.33.0

Card[1313]->card[1314] (Tbl:1) TblAuditDone:TotalMisses:h'1 FirstMiss:h'1;

tstpe0akb1 09-10-20 14:31:27 IST EAGLE5 41.0.0-62.33.0[1313] [Linkset:1] Chksum h'aa68 at h'8d23787 (167 bytes)Assign:1 Avail:1 APC: 00867-aa ITUNVar:0

;tstpe0akb1 09-10-20 14:31:27 IST EAGLE5 41.0.0-62.33.0

[1314] [Linkset:1] Chksum h'f287 at h'8d23787 (167 bytes)Assign:1 Avail:1 APC: 00867-aa ITUNVar:0

;

In order to resolve this issue then the user need to find out which linkset uses APC867-aa and next during a maintenance window deactivate all the links and activatethem again to trigger a linkset state change. When this happens then DDB updatemissed would be generated and total number of misses decremented. If the totalnumber of misses is different of 0 then this needs to be repeated until value 0 is

reached and the DDB inconsistency and DDB alarm should be cleared.

In some cases the full linkset deactivation/reactivation may not besufficient to trigger the expected DDB update and the linkset may need tobe cancelled/re-created as an alternate option.

Modify a linket parameter (which will not impact the normal functioning of thelinkset) and then revert it. (for example you may assign a gateway screening set intest mode to the linkset and then remove). Do aud-data:type=ddb:display=all toverify the DDB status

3.3 CARDS REPORT DDB MISMATCH CHECKSUM FOR LINK TABLE

The resolution of DDB issues related to link tables is similar to the one explained insection 3.2

The cards reporting this issue are found via a rept-stat-ddb in full mode then a ddb-ddb command may be run to the card reporting DDB mismatch for link table as perthe below example. This command will provide the link ID and the card from which astate change has been missed and the total number of misses. The link ID ismapped with link port as follows: A :0 , B:1, A1:2, B1:3, and so on

Dbg-ddb :loc=1317 :Tbl=LNK:action=AUD:audtype=UC:

blmptss 08-10-07 15:40:08 FST EAGLE 41.0.0-62.33.0

[1317]Card->System (Tbl:0) Diff:Card[4101] LclCksm:h'e5e8 RmtCksm:h'6e8

TblAuditDone:TotalMisses:h'1 FirstMiss:h'1c

;

14


15/16

blmptss 08-10-07 15:40:08 FST EAGLE 41.0.0-62.33.0

[1317] [Link:x1c] Chksum h'3f52 at h'7b5c60 (14 bytes)

PortId:h'656 (Card:2114 Link h'6)

Slc:1 Stat:h'1 LsId:h'6 Class:0

Status:Avl

In this case a link deactivation/activation of 2114:A3 should trigger the missed DDBupdate and decremented the total number of misses from 1 to 0.

15


16/16

16

ddb phase1-2 explanation and toubleshooting guide

Documents