progress database admin 1 jeffrey a. brown - technical support consultant (jefbrown@nxtrend.com)

Progress Database Admin

1

Jeffrey A. Brown - Technical SupportConsultant (jefbrown@nxtrend.com)

2

Introduction:

Agenda:

• Lecture - Database Concepts

• Lunch

• Database Workshop (hands-on)

3

NxTrend & Progress:

• NxTrend & Progress

• NxTrend runs on products we sell.

4

What Is Progress?

• Not an O/S

• Database engine

• 4GL Programming language

5

Licensing

• Enterprise DB

• Appserver

• 4GL

• Provision (PC Client side)

• Client Networking

• Query/Results (being replaced by SX.explorer)

• Merant (ODBC)

6

Progress Database Concepts:

• Physical Parts of a Database– .db - database– .bi – before image file– .lg – log file– .lk – lock file (only there when Broker is up)

7

.db

.bi

.lg8

Starting The Database Broker

• Unix Process

• Shared memory– virtual tables

• lock file (.lk)

9

.db

_mprosrv

.bi

.lg10

_mprosrv

Shared Memory

.db

.bi

.lg10

Shared Memory Virtual tables

User id Lock TTY PID

11

u l t p

_mprosrv

Shared Memory Virtual tables

.db

.bi

.lg12

.lk.lg

u l t p

_mprosrv

Shared Memory Virtual tables

.db

.bi

12

User Login

• Unix Process

• Logs into virtual tables

• Logged in the log file (.lg)

13

u l t p

User 1

_mprosrv

Shared Memory Virtual tables

.db

.bi

.lk.lg14

u l t p

User 1

_mprosrv

Shared Memory Virtual tables_progres

.db

.lk.lg

.bi

14

u l t p

User 1

_mprosrv

Shared Memory Virtual tables_progres

u1

.db

.lk.lg

.bi

14

Shared Memory Virtual tables

User id Lock TTY PID

User 1 tty/01 12345

15

u l t p

User 1

_mprosrv

Shared Memory Virtual tables_progres

u1

.db

.lk.lg

.bi

16

u l t p

User 1

_mprosrv

Shared Memory Virtual tables_progres

u1

.db

.lk.lg

.bi

16

Data-Flow Discussion

• Data gets pulled into memory– User reads from memory– User never read from the database

17

.db

a b c

u l t p

User 1

_mprosrv

Shared Memory Virtual tables_progres

u1

.lk.lg

.bi

18

a b c u l t p

User 1

_mprosrv

Shared Memory Virtual tables_progres

u1

.db

a b c

.lk.lg

.bi

18

a b ca b c

u l t p

User 1

_mprosrv

Shared Memory Virtual tables_progres

u1

.db

a b c

.lk.lg

.bi

18

Data-Flow Discussion

• User Updates data– lock table

19

a b ca b d

u l t p

User 1

_mprosrv

c

Shared Memory Virtual tables_progres

u1

.db

a b c

.lk.lg

.bi

20

a b da b d

u l t p

User 1

_mprosrv

Shared Memory Virtual tables_progres

u1

.db

a b c

.lk.lg

.bi

20

Data-Flow Discussion

• Before-Image File– BIW– Data gets written to disk (.bi file)

21

.lk.lg

.bi

a b da b d

u l t p

User 1

_mprosrv

Shared Memory Virtual tables_progres

u1

.db

a b c

22

a b da b d

u l t p

User 1

biw

_mprosrv

Shared Memory Virtual tables_progres

u1

.db

a b c

.lk.lg

.bi

22

a b da b d

u l t p

User 1

biw

_mprosrv

bt

Shared Memory Virtual tables_progres

u1

.db

a b c

.lk.lg

.bi

22

a b da b d

c

u l t p

User 1

biw

_mprosrv

bt

Shared Memory Virtual tables_progres

u1

.db

a b c

.lk.lg

.bi

22

a b da b d

c

u l t p

User 1

biw

_mprosrv

bt

Shared Memory Virtual tables_progres

u1

.db

a b c

.lk.lg

.bi

22

a b da b d

c d

u l t p

User 1

biw

_mprosrv

bt

Shared Memory Virtual tables_progres

u1

.db

a b c

.lk.lg

.bi

22

a b da b d

c d

u l t p

User 1

biw

_mprosrv

bt et

Shared Memory Virtual tables_progres

u1

.db

a b c

.lk.lg

.bi

22

Data-Flow Discussion

• Review

23

a b da b d

c d

u l t p

User 1

biw

_mprosrv

bt et

Shared Memory Virtual tables_progres

u1

.db

a b c

.lk.lg

.bi

24

a b da b d

c d

u l t p

User 1

biw

_mprosrv

bt et

Shared Memory Virtual tables_progres

u1

.db

a b c

.lk.lg

.bi

24

a b da b d

c d

u l t p

User 1

biw

_mprosrv

bt et

Shared Memory Virtual tables_progres

u1

.db

a b c

.lk.lg

.bi

24

a b da b d

c d

u l t p

User 1

biw

_mprosrv

bt et

Shared Memory Virtual tables_progres

u1

.db

a b c

.lk.lg

.bi

24

a b da b d

c d

u l t p

User 1

biw

_mprosrv

bt et

Shared Memory Virtual tables_progres

u1

.db

a b c

.lk.lg

.bi

24

Data-Flow Discussion

• After-Imaging File– AIW

25

.ai

a b da b d

c d

u l t p

User 1

biw

_mprosrv

bt et

Shared Memory Virtual tables_progres

u1

.db

a b c

.lk.lg

.bi

26

a b da b d

c d

u l t p

User 1

biw

_mprosrv

aiw

bt et

Shared Memory Virtual tables_progres

u1

.db

a b c

.lk.lg

.bi

.ai

26

Data-Flow Discussion

• Asynchronous Page Writer– APW– Data gets written to .db

27

.db

a b d

a b da b d

c d

u l t p

User 1

biw

apw

_mprosrv

aiw

bt et

Shared Memory Virtual tables_progres

u1

.lk.lg

.bi

.ai

28

Data-Flow Discussion

• Final Review

29

a b da b d

c d

u l t p

User 1

biw

_mprosrv

bt et

Shared Memory Virtual tables_progres

u1

apw aiw

.db

a b d

.lk.lg

.bi

.ai

30

a b da b d

c d

u l t p

User 1

biw

_mprosrv

bt et

Shared Memory Virtual tables_progres

u1

apw aiw

.db

a b d

.lk.lg

.bi

.ai

30

a b da b d

c d

u l t p

User 1

biw

_mprosrv

bt et

Shared Memory Virtual tables_progres

u1

apw aiw

.db

a b d

.lk.lg

.bi

.ai

30

a b da b d

c d

u l t p

User 1

biw

_mprosrv

bt et

Shared Memory Virtual tables_progres

u1

apw aiw

.db

a b d

.lk.lg

.bi

.ai

30

a b da b d

c d

u l t p

User 1

biw

apw

_mprosrv

aiw

bt et

Shared Memory Virtual tables_progres

u1

.db

a b d

.lk.lg

.bi

.ai

30

a b da b d

c d

u l t p

User 1

biw

apw

_mprosrv

aiw

bt et

Shared Memory Virtual tables_progres

u1

.db

a b d

.lk.lg

.bi

.ai

30

a b da b d

c d

u l t p

User 1

biw

apw

_mprosrv

aiw

bt et

Shared Memory Virtual tables_progres

u1

.db

a b d

.lk.lg

.bi

.ai

30

SX.enterprise Concepts

• Client/server vs self serving clients

• Appserver

31

User 1

_mprosrv

Shared Memory

prowin32

.db

32

User 1

_mprosrv

Shared Memory

prowin32Server

.db

32

User 1

_mprosrv

Shared Memory

prowin32Server

.db

32

User 1

_mprosrv

Shared Memory

prowin32Server

AppServer

.db

32

Appserver Discussion

33

User 1

AppServer Database

Appserver Discussion

34

User requests customer address

34

User 1

AppServer Database

Appserver Discussion

34

User requests a price for a specific customer

34

User 1

AppServer Database

Appserver Discussion

34

User 1

AppServer Database

Appserver Discussion

34

User 1

AppServer Database

Appserver Discussion

34

Appserver Discussion - Summary

• Users are connected to Appserver and Database• Small requests do not use Appserver• Larger requests go through Appserver• Need to have large pipe between

Database Server and Appserver Server

35

Disconnecting Users - shutuser script

• Using shutuser

• Risks in shutting a user out

• Never use kill -9

36

User 1

_mprosrv

_progresu l t p

Shared Memory Virtual tables

u1

.db

37

User 1

_mprosrv

prowin32Server

u l t pShared Memory Virtual tables

u1

.db

38

Database Files & Crash Recovery

• .bi file

• when brokers start after crash, rebuilding shared memory not flushed to disk

• when truncate, flushes to disk

• .lg file

39

Power Outage Scenario #1

• Data Pulled into memory

• User changes data

• Data not written to .bi file

40

a b da b d

u l t p

User 1

biw

apw

_mprosrv

aiw

Shared Memory Virtual tables_progres

u1

.db

a b c

.lk.lg

.bi

.ai

41

Loss of Power #1

41

User 1

.db

a b c

.lk.lg

.bi

.ai

41

Power Restored #1

What happens to data?

41

u l t p

User 1

biw

apw

_mprosrv

aiw

Shared Memory Virtual tables_progres

u1

.db

a b c

.lk.lg

.bi

.ai

41

Power Outage Scenario #2

• Data pulled into Memory

• User changes data

• Data is being written to Before-Image file

• No “end transaction” written

42

a b da b d

c d

u l t p

User 1

biw

apw

_mprosrv

aiw

bt

Shared Memory Virtual tables_progres

u1

.db

a b c

.lk.lg

.bi

.ai

43

Loss of Power #2

43

User 1

c dbt

.db

a b c

.lk.lg

.bi

.ai

43

Power Restored #2

What happens to data?

43

u l t p

User 1

biw

apw

_mprosrv

aiw

Shared Memory Virtual tables_progres

u1

.db

a b c

.lk.lg

.bi

.ai

43

Power Outage Scenario #3

• Data pulled into Memory

• User changes data

• Data is being written to Before-Image file

• “end transaction” written

44

a b da b d

c d

u l t p

User 1

biw

apw

_mprosrv

aiw

bt

Shared Memory Virtual tables_progres

u1

et

.db

a b c

.lk.lg

.bi

.ai

45

Loss of Power #3

45

User 1

c dbt et

.db

a b c

.lk.lg

.bi

.ai

45

Power Restored #3

What happens to data?

45

u l t p

User 1

biw

apw

_mprosrv

aiw

Shared Memory Virtual tables_progres

u1

.db

a b d

.lk.lg

.bi

.ai

45

Crash Recovery - Summary

• Check the .lg file

• Truncate .bi file after crash

• Check last 3 minutes of work

46

NxTrend Installation Standards:

• File Systems

• /db (striped filesystem)

• /bi (separate disk)

• /rd (striped filesystem)

47

Directories: /db

• /db/nxt.db

• /db/nxt.bi

• /db/nxt.lg

• /db/nxt.lk

48

Directories: /db/sort

• /db/sort/*.DBI

• /db/sort/*.lbi

• /db/sort/*.srt

49

Directories: /bi

• /bi/nxt.bi

** Only if /bi can be put on it’s own dedicated spindle

50

Directories: /rd

• /rd/bin

• /rd/opsys/ (*.pf, *.sh)

• /rd/dlc

• /rd/dlc/bin

• /rd/src/ (*.p, *.i, *.h)

• /rd/cust/ (*.p, *.i, *.h)

• /rd/exec/ (*.r)

• /rd/lib/nxt.pl

51

Progress Libraries

52

Index .r files

Progress Libraries

53

Progress Libraries

• Replacing files

• Adding files

53

Index .r files

Progress Libraries

53

Index .r files

Progress Libraries

53

Index .r files

Progress Libraries

53

Index .r files

Progress Libraries

53

Review Directories and Libraries:

• /db

• /bi

• /rd

• nxt.pl

54

NxTrend’s Expectation Of A Database Admin:

• Maintenance

• Crash Recovery

• Troubleshooting

• Scheduled Work

55

Database Admin Maintenance:

• Disconnecting users

• Verify Backup Logs

• Monitor/maintain extent structure – prostats.log

• Purging Database log files

• Promon

• Maintaining Scripts

• Maintain .pf files

• Maintain library file

56

Database Admin Crash Recovery:

• Brokers – start & stop

• Removing .lk files

• Clearing shared memory

• Removing processes

• Disconnecting users

• Truncating BI files

• Error messages in log files

57

Database Admin Scheduled Work:

• Dump/loads

• Progress upgrades/patches

58

Progress Parameter

Discussion:• Database Broker Parameters

• Client Parameters

• BI file parameters for Truncate

59

Database Broker Parameters

/rd/opsys/nxtdb.pf

• –bibufs 30

• –spin 1

• –directio

• –B 3750

• –L 50000

• –n 120

• –db /db/nxt

• –g /bi/nxt.bi

• –H <hostname>

• –S <service name>

• –N TCP

• –minport

• –maxport

60

Client Parameters /rd/opsys/enterprise.pf

• –mmax 512

• –q

• –T /db/tmp

• –t (not needed on PC client)

• –h (not needed for single database)

• –db /db/nxt

• –g /bi/nxt.bi

61

BI File Parameters For Truncate

/rd/bin/truncate.bi

• –bi 1024 (Cluster)

• –biblocksize 16 (Block)

• –G 0

• –g /bi/nxt.bi

Cluster

Block

.bi file

62

Report Management:

• Report Manager– rptmgr– rptmgr– rptmgr

• Report Scheduler– rptsch.p

• rptrun.p

• rptrun.p

• rptrun.p

• New rptmgr hybrid (SX.enterprise)– rptmgr1– rptmgr2– rptmgr3

63

Report Management:

• Report Manager

• Start/Stop

• Troubleshooting

• rptlog in sasc.printdir– rptmgr.err in /usr/tmp– user definable parameters

64

Database Workshop

Working With Databases (hands-on)

65

Working with databases

• start broker

• identify database broker– shared memory– Unix process– lock file

• stop broker

66

Working with databases

• emergency shutdown

• log file

• progress editor (errors messages)

67

Working with databases

• start APW

• start BIW

68

Working with databases

• crash recovery

• truncate bi file

69

Single-Volume vs Multi-Volume

70

Single-Volume vs Multi-Volume

• .db, .bi • .db, .d1, .d2...

• .b1, .b2...

• no -g option needed

71

Progress File Size Limit

2GB72

Single-Volume vs Multi-Volume

• .db, .bi

• 2GB• .db, .d1, .d2...

• .b1, .b2...

• no -g option needed

• 256 @ 2GB

73

ExtentsVariable vs Fixed

74

Variable Extents

Variable Extents

New Record

75

Variable Extents

Variable Extents

New Record

OS

75

Variable Extents

Variable Extents

New Record

75

Variable Extents

Variable Extents

New Record

75

Variable Extents

Variable Extents

New Record

OS

75

Variable Extents

Variable Extents

New Record

75

Variable Extents

Variable Extents

75

Fixed Extents

Fixed Extents

76

Fixed Extents

Fixed Extents

New Record

76

Fixed Extents

Fixed Extents

76

Single-Volume vs Multi-Volume

• .db, .bi

• 2GB

• 2 step writes

• .db, .d1, .d2...

• .b1, .b2...

• No -g option needed

• 256 @ 2GB

• Single step writes

77

Inode Locking

78

Inode Locking

78

Single-Volume vs Multi-Volume

• .db, .bi

• 2GB

• 2 step writes

• One inode lock

• .db, .d1, .d2...

• .b1, .b2...

• No -g option needed

• 256 @ 2GB

• Single step writes

• Many inodes to lock

79

Single-Volume vs Multi-Volume - Summary

• NxTrend recommends using Multi-Volume• Multi-Volume is easier to administer• Multi-Volume supports larger databases• Multi-Volume has faster writes• Multi-Volume can have more than one

write at once

80

Record Management

• High Water Mark

• RM Blocks

• RM Chain

• Free Chain

• Empty Blocks

81

High Water Mark

Free Chain

Empty Blocks

RM ChainRM Blocks

Record Management

82

High Water Mark

Free Chain

Empty Blocks

RM ChainRM Blocks

Record ManagementIndexed

82

High Water Mark

Free Chain

Empty Blocks

RM ChainRM Blocks

Record Management

Indexed

82

High Water Mark

Free Chain

Empty Blocks

RM ChainRM Blocks

Record Management

82

High Water Mark

Free Chain

Empty Blocks

RM ChainRM Blocks

Record Management

82

Record Management Scenario #1

• New Record

• Checks first 3 RM Chain blocks

• Record fits

• New Record fills RM Chain block

• Block gets taken off RM Chain

83

Record Management

Free Chain

Empty Blocks

RM ChainRM Blocks

NewRecord

84

Record Management

Free Chain

Empty Blocks

RM ChainRM Blocks

84

Record Management

Free Chain

Empty Blocks

RM ChainRM Blocks

84

Record Management

Free Chain

Empty Blocks

RM ChainRM Blocks

84

Record Management

Free Chain

Empty Blocks

RM ChainRM Blocks

84

How full is full?

• Approx. 93% full

• So that record can grow in same block

• Database may grow after dump and load

85

Record Management Scenario #2

• New Record

• Checks first 3 RM Chain blocks

• Record does Not fit

• Block pulled from Free Chain

• New Record gets put on RM Chain

• First 3 RM blocks get moved to end of Chain

86

Record Management

Free Chain

Empty Blocks

RM ChainRM Blocks

NewRecord

87

Record Management

Free Chain

Empty Blocks

RM ChainRM Blocks

87

Record Management

Free Chain

Empty Blocks

RM ChainRM Blocks

87

Record Management

Free Chain

Empty Blocks

RM ChainRM Blocks

87

Record Management

Free Chain

Empty Blocks

RM ChainRM Blocks

87

Record Management

Free Chain

Empty Blocks

RM ChainRM Blocks

87

Record Management

Free Chain

Empty Blocks

RM ChainRM Blocks

1 2 3

87

Record Management

Free Chain

Empty Blocks

RM ChainRM Blocks

1 2 3

87

Record Management Scenario #3

• New Record

• Checks first 3 RM Chain blocks

• Record does Not fit

• No blocks left on Free Chain

• Block pulled from Empty Blocks

• New Record gets put on RM Chain

• First 3 RM blocks get moved to end of Chain

88

Record Management

Free Chain

Empty Blocks

RM ChainRM Blocks

NewRecord

89

Record Management

Free Chain

Empty Blocks

RM ChainRM Blocks

89

Record Management

Free Chain

Empty Blocks

RM ChainRM Blocks

89

Record Management

Free Chain

Empty Blocks

RM ChainRM Blocks

89

Record Management

Free Chain

Empty Blocks

RM ChainRM Blocks

89

Record Management

Free Chain

Empty Blocks

RM ChainRM Blocks

89

Record Management

Free Chain

Empty Blocks

RM ChainRM Blocks

1 2 3

89

Record Management

Free Chain

Empty Blocks

RM ChainRM Blocks

1 2 3

89

Record Management - Summary

• New Records only check first 3 RM Chain blocks• First 3 RM Chain blocks get moved to end of chain• If you delete data, you cannot guarantee space will

be reused• Full Blocks are approx. 93% full

90

Dump & Load Exercise: (hands on)

• What is a dump & load

• m&m analogy

• Why do a dump & load

• How often should you dump & load

• Enable VST

91

Dump & Load

92

Dump & Load

92

Dump & Load

93

Review:

• Question & Answer Session

94

Questions

?

?

?

?

?

?

?

?

?

?

?

?

95

Progress Data Flow Exercise

• DB

• Shared Memory

• Auto-Server

• User

• BIW

• BI

• APW

96