colleague system maintenance

Colleague System Maintenance

File System Administration & Maintenance – Part One

Cinda GoffChuck Hauser2005-10-26

Presentation Conventions Names (files, users, daemons) are usually in bold:

/etc/syslog.conf System dependent or variable items are usually in italics:

/var/sadm/patch/patchnumber/log File entries and output are in mono-spaced type:> root 8036 c Tue Apr 26 23:59:00 2005 < root 8036 c Tue Apr 26 23:59:59 2005

marks a line wrapped to fit on the slide: mv Solaris_8_Recommended_log

Solaris_8_Recommended_log.yyyymmdd marks a horizontal tab (09 hex) Reference OE is Solaris 8 Reference UniData is 6.0

Introduction

Why maintain files UniData File Maintenance and Utilities Account File Cleanup Transaction Log Cleanup Status File Purging File Rebuilds

UniData Maintenance Utilities

Datatel’s WUFA - Weekly Udt File Analysis (free)

Fitzgerald & Long’s FAST - File Analysis and Sizing Tool (license required)

WUFA & FAST Primary Functions

Report file storage Recommend optimum block size and

modulo selection Automate file resizing

WUFA

Reports on and resizes both static and dynamic files

Support recently added for dictionaries

WUFA Recommendations

Leave all files static upon initial installation Use weekly WUFA run to evaluate

whether some files should be redefined as dynamic

For more information see AnswerNet document 147.992: Whether to use Static vs Dynamic UniData Files in Colleague.

WUFA Processing

Performs thorough system analysis on both static and dynamic files

Provides data file, UDT_GUIDE, for statistical queries

Creates DATATEL.RESIZE.FILES paragraph for file resizing

Running WUFA

WUFA can be run with users on system DATATEL.RESIZE.FILES must be run in

a quiet account Use cron to automate WUFA execution

Automating WUFA with cron

Create for each account: shell script for cron execution Datatel crontab entry to run shell script VOC paragraphs to run WUFA

Datatel wufascript crontab entry

Login as datatel If the EDITOR environment variable is not set or

exported, give the command: EDITOR=vi ; export EDITOR

Giving this command to edit crontab: crontab –e

Add the following line to run the wufa script every night, Monday through Thursday:

0 22 * * 1-4 /export/home/datatel/wufascript_collive

Unix Script to run WUFA Use vi (or another editor) to create a unix shell script that runs a UniData

paragraph for the live remote account.

cd /export/home/datatelvi wufascript_collive

Make sure the cd command in the script points to the appropriate account:

#!/usr/bin/kshumask 007cd /datatel/live/colliveUDTBIN=/usr/ud60/binUDTHOME=/usr/ud60export UDTBIN UDTHOME$UDTBIN/udt<<ineofWUFALOineof#EOF

Variant WUFA Shell Script (part 1)

This version will accept an argument specifying the account to run in; the default account is /datatel/live/collive. Example:

# wufa.ksh colconv

#!/usr/bin/ksh

# @(#)wufa.ksh 1.5 @(#)

# @(#) Source: cis:/opt/local/sbin/SCCS/s.wufa.ksh 1.5 05/10/20 11:42:54 @(#)

# @(#) Last update: 05/10/20 11:42:56 @(#)

#

# Local Variables

PROG="`basename $0`"

DEFAULT_DATATELDIR="/datatel/live/collive"

DATATELDIR=""


# Get UniData Variables

. /usr/local/udtparams

# Only datatel should run this. UID is probably 100,

# but to be safe check user name.

UNAME=`id | sed -e 's/).*$//' -e 's/^uid=.*(//'`

if [ $UNAME != "datatel" ]

then

printf "$PROG: stopping, should be run by user datatel.\n"

exit 1

fi

Use the dot command to load the current UniData value for UDTBIN, umask, etc.


if [ "$1" ]

then

case "$1"

in

*live ) DATATELDIR="/datatel/live/collive" ;;

*test ) DATATELDIR="/datatel/work/coltest" ;;

*conv ) DATATELDIR="/datatel/work/colconv" ;;

*LIVE17 ) DATATELDIR="/datatel/release/LIVE17/INSTALL" ;;

*TEST17 ) DATATELDIR="/datatel/release/TEST17/INSTALL" ;;

*DEBUG17 ) DATATELDIR="/datatel/release/DEBUG17/INSTALL" ;;

* ) printf "$PROG: stopping, bad datatel directory ($1) specified!\n" ; exit 2 ;;

esac

else

DATATELDIR="$DEFAULT_DATATELDIR"

fi

Test for a script argument. If there is one, choose the correct account. If there is no argument, use the default account (collive).


if [ ! -d $DATATELDIR ]

then

printf "$PROG: stopping, directory $DATATELDIR does not exist!\n"

exit 3

fi

cd $DATATELDIR

$UDTBIN/udt<<INEOF

WUFA

LO

INEOF

logger -p user.info "WUFA run on $DATATELDIR"

# End /opt/local/sbin/SCCS/s.wufa.ksh

WUFA VOC paragraphs

Create a VOC paragraph in each account that will run WUFA and send the results to a hold file.

Create additional VOC paragraphs for two more hold files that will list any damaged files and any files in level 2 overflow.

VOC paragraph to run WUFA

Top of "WUFA" in "VOC", 11 lines, 167 characters.

001: PA

002: SETPTR ,,,,,3,BANNER UNIQUE WUFA,BRIEF

003: WEEKLY.UDT.FILE.ANALYSIS

004: SETPTR ,,,,,3,BANNER UNIQUE DAMAGED,BRIEF

005: XNC.DAMAGED.FILES

006: SETPTR ,,,,,3,BANNER UNIQUE OVER2,BRIEF

007: XNC.OVERFLOW.FILES

Bottom

VOC Entry for Damaged Files

Top of "XNC.DAMAGED.FILES" in "VOC", 2 lines,103 characters.

001: S002: LIST UDT_GUIDE BY FILE_NAME WITH

DATATEL.DAMAGED GT "" FILE_NAME FMT "25" STATUS DATATEL.DAMAGED LPTR

Bottom.

VOC Entry for Overflow 2 Files

Top of "XNC.OVERFLOW.FILES" in "VOC", 2 lines, 177 characters.

001: S

002: LIST UDT_GUIDE BY FILE_NAMEFILE_NAME FMT "25L" STATUS OVERFLOW2DATATEL.DAMAGED ID.SUP WITHDATATEL.DAMAGED GT "" OR WITH OVERFLOW2 GT 1 BLOCKSIZE NEW_BLOCKSIZE FILETYPE LPTR

Bottom.

Resizing Files – MEMRESIZE WUFA creates a DATATEL.RESIZE.FILES paragraph

that assumes a VOC entry called MEMRESIZE. MEMRESIZE executes the unix level memresize

command. Create a MEMRESIZE entry in both remote and install

accounts:

AE VOC MEMRESIZE Top of "MEMRESIZE" in "VOC", 2 lines, 77 characters. *--: P001: PA002: !/usr/ud60/bin/memresize <<I2,file>>

<<I3,modulo>> <<C4,ARG>> memory 32678

Bottom.

Resizing Files

Run the DATATAL.RESIZE.FILES paragraph on a weekly basis.

Exclude files from being resized by putting them in a RESIZE.EXCLUDE savedlist.

The VOC file is always excluded from DATATEL.RESIZE.FILES.

Datatel resizescript crontab entry

Login as datatel If the EDITOR environment variable is not set or

exported, give the command: EDITOR=vi ; export EDITOR

Giving this command to edit crontab: crontab –e

Add the following line to run the script every Saturday morning:

0 2 * * 6 /export/home/datatel/resizescript_collive

Unix Script to run MEMRESIZE Use vi (or another editor) to create a unix shell script that runs a UniData

paragraph for the live remote account.

cd /export/home/datatelvi resizescript_collive

Make sure the cd command in the script points to the appropriate account:

#!/usr/bin/kshumask 007cd /datatel/live/colliveUDTBIN=/usr/ud60/binUDTHOME=/usr/ud60export UDTBIN UDTHOME$UDTBIN/udt<<ineofDATATEL.RESIZE.FILES

LOineof#EOF

Resizing the VOC File

The DATATEL.RESIZE.FILES paragraph will not resize the VOC file

VOC files should always be resized manually. The account should always be quiet with no user

having access to the account. The steps are performed either from an account

colon prompt or at the Unix prompt in the account directory.

Steps for Resizing the VOC (Part 1)

From the colon/SHEL prompt, create a new file with the proper modulo and blocksize as recommended by WUFA:

CREATE.FILE VOC.NEW 179,2 Copy records from the original VOC file to

the new VOC:

COPY FROM VOC TO VOC.NEW ALL

Steps for Resizing the VOC (Part 2)

From the Unix prompt in the account directory, rename the VOC files and remove the dictionary of the new VOC:

# mv VOC VOC.OLD# mv VOC.NEW VOC# rm D_VOC.NEW

From the colon/SHEL prompt, delete the VOC entry for VOC.NEW:

DELETE VOC VOC.NEW

Application/Account File Cleanup

appl.PPROCESS purge (UTJP) Transaction Log Cleanup EDX.STATUS purge ELF.TRANSACTIONS purge Express Load Purge Repository (ELPR) appl.PRINTERS

Transaction Log Cleanup - UTTP

UTTP can only be run against files for which transaction logging has been set up locally.

If no transaction logging has been set up, this process does not need to be run on a regular basis.

appl.PPROCESS Purge (UTJP)

Access the UTJP screen from the appropriate application in a remote account.

UTJP allows selection by date and time of process, by username, or by mnemonic.

EDX Status Archive Purge (EDSP)

The EDX phantom creates a record in the EDX archive (the EDX.STATUS file) for each DMI transaction.

The records stay in this file until you delete them. Use the EDX Status Archive Purge (EDSP) screen to delete records from the EDX archive .

ELF.TRANSACTIONS Purge (EPRG)

The ELF.TRANSACTIONS file is purged with the EPRG screen.

This screen allows you to purge batches that were imported into Colleague through the ELF routines.

These records are not archived, so once they are purged, they are gone.

Express Load File Purge (ELPR)

The Express Load Purge Repository (ELPR) process purges:

unneeded IMAGE directories VOC paragraphs data backup repositories for successfully

installed software updates

ELPR Screen

You can specify whether or not to purge each item.

In the bottom part of the screen, you can select the software update batches for purging by entering Yes in the Purge field (the default is “No”).

ELPR Comments

Remember that the software updates repository may be shared by more than one installation account.

If it is shared by more than one installation account, make sure you don't purge the IMAGE directories from this software updates repository until you have loaded those updates into the other installation accounts sharing this repository.

ELPR Issue

There is a known issue with ELPR if you have many items to purge.

See AnswerNet document 8870.89 - S_MIO_OPEN errors on ELPR for a workaround.

Files Purged at the Colon Prompt

appl.PRINTERS VOC RUN.#### files _PH_ COMO Output _HOLD_ Records SAVEDLISTS PARAMETERS

JSPARAMS X.SECURITY.LOG AE_COMS WWW.STATE &

WWW.TOKENS

appl.PRINTERS

Records in appl.PRINTERS beginning with SETPTR can be safely purged.

This should be done in a quiet account. Run this command for each application: SELECT appl.PRINTERS WITH @ID LIKE ‘SETPTR…’

If the above command returns records, execute this command: DELETE appl.PRINTERS

VOC Process Entries

Some Colleague processes create entries in the VOC that can be cleaned up.

The entries should be removed in a quiet account. Run this command to remove RUN.#### files:

SELECT VOC @ID LIKE “’RUN.’…” AND WITH F1 UNLIKE “C…”

If the above command returns records, execute this command: DELETE VOC

Express Load VOC Entries

The Express Load process creates various entries in the VOC of both the install and remote accounts.

Because multiple naming conventions were used in the past for patches, there are multiple steps for safely removing the various VOC entries.

After the initial cleanup, all of the following steps may not be needed.

Steps to Remove Express Load Entries

Select the _17014_UNIX records:

SELECT VOC WITH @ID LIKE …_17014_UNIX… If the above command returns records, execute this

command: DELETE VOC Select the _170_U records:

SELECT VOC WITH @ID LIKE …_170_U… If the above command returns records, execute this

command: DELETE VOC

Steps to Remove Express Load Entries continued

Select the _4178_U records:

SELECT VOC WITH @ID LIKE …_4178_U_… If the above command returns records, execute

this command: DELETE VOC

_PH_ / _HOLD_ / SAVEDLISTS Directory Files

These are directory files in each account that are used by various UniData and Colleague processes.

Most of the data in these files can be purged on a periodic basis, but each college should set local naming conventions and retention standards.

_PH_ Records

The _PH_ file stores records of all processes executed in background mode and the output for COMO commands.

COMO output records start with an O_ and the name assigned when starting the como.

_PH_ Background Record Keys

Each background record has a multi-part key: ID of the VOC record executed in background

mode The internal representation of the time the

process was executed The internal representation of the data on which

the process was executed

The _HOLD_ File

The HOLD file is the database file in which Envision Run Time writes report output for processing by the BROWSE utility.

Some records are keyed by the date and time the user sent report output to the HOLD file.

Other records have IDs that are strings users entered.

SAVEDLISTS File

The SAVEDLISTS file stores any created list the user has saved using the SAVE LIST command.

Envision Run Time also temporarily stores lists of records ids for use in procedures in the SAVEDLISTS file

savedlists File

The savedlists file (lower case) is a workfile that is used when selecting on multivalues.

All data in this file is transient and can be deleted in a quiet account.

PH, HOLD, and SAVEDLISTS Clean Up

Multiple ways to clear data in these files. Method used depends on a college’s

standards and procedures.

The CLEAR.FILE method for PH, HOLD, and SAVEDLISTS Accounts

The UniData CLEAR.FILE command can be used to clear all records in the file:CLEAR.FILE filename

The CLEAR.ACCOUNT command issues a CLEAR.FILE command for the _PH_ and _HOLD_ files.

The SELECT and DELETE method for PH, HOLD, and SAVEDLISTS Accounts

Use the UniData SELECT and DELETE commands to selectively delete records in the files.

This method uses local naming conventions to build a list of records to be deleted.

The example assumes that users append ‘PERM’ to the end of any records to be kept.

SELECT fname WITH @ID UNLIKE …PERM

SAVE.LIST fname.DEL

EDIT.LIST fname.DEL GET.LIST fname.DEL

DELETE fname

SELECT and DELETE Example for PH, HOLD, and SAVEDLISTS Accounts

To review and manually remove anything that should not be deleted.

The Unix find and rm method for PH, HOLD, and SAVEDLISTS Accounts

Because these files are UniData DIR files, the records can be manipulated using standard Unix commands.

This example uses find and rm to remove HOLD files that are older than 30 days:

find /datatel/live/collive/_HOLD_ -mtime +30 –exec rm {} \;

Clearing the PARAMETERS File

The PARAMETERS file is a work file that can grow rapidly.

It can be safely cleared in a quiet account:CLEAR.FILE PARAMETERS

Clearing the JSPARAMS File

JSPARAMS is a Runtime Variable file. The records result from the Application

Peripheral Options Default Screen. The file holds ‘remembered’ user responses. Records can be purged in a quiet account, but

do not use CLEAR.FILE to purge records!

Example Clearing of JSPARAMS

The key structure for JSPARAMS is:PROCESS_FILENAME_OPERATOR_DATE_TIME

Select the JSPARAMS records:

SELECT JSPARAMS WITH @ID LIKE

‘..._ ..._..._...’ If the above command returns records, execute

this command: DELETE JSPARAMS

The X.SECURITY.LOG File

The NCCCS modification for verb security logs access to verbs in the X.SECURITY.LOG file.

This file can grow large if there are many users with colon level or Query Builder access.

Purge this file by selecting records by date: SELECT X.SECURITY.LOG WITH XSL.DATE LE ’01/01/04’

If the above command returns records, execute this command: DELETE X.SECURITY.LOG

Clearing the AE_COMS File

The AE_COMS file should be deleted monthly in a quiet account.

Purge this file by selecting records by date:

SELECT AE_COMS WITH @ID LIKE ’com…’’pre…’ If the above command returns records, execute

this command: DELETE AE_COMS

WWW.STATE and WWW.TOKENS Files

WWW.STATE and WWW.TOKENS files are self-maintaining as of DMI release 3.3 and higher.

If data is found in these files can be safely removed in a quiet account.

If WebAdvisor is used in the account, it must be inaccessible during the file purge.

Use CLEAR.FILE:CLEAR.FILE WWW.STATECLEAR.FILE WWW.TOKENS

GUIDE_ (Guide Files)

When guide is run to fix damaged files, a series of files beginning with GUIDE_ are created in the account directory (ex. /datatel/live/collive).

These files need to be deleted regularly:

cd /datatel/live/collive

rm GUIDE*

Security File Maintenance

Occasionally the appl.SECLASS and appl.PRCS.CTL files will get out of sync in regard to security classes

The BUILD.SECURITY functions will clear out the appropriate fields in the appl.PRCS.CTL file and rebuild based on the appl.SECLASS definitions

Rebuilding appl.PRCS.CTRL

All applications should be rebuilt in the this order:BUILD.SECURITY UTBUILD.SECURITY COREBUILD.SECURITY CFBUILD.SECURITY STBUILD.SECURITY HR

These commands are issued at the colon, or as an alternative, you may run BSEC from each application.

Colleague System Maintenance Schedule

Frequency Level Task

Daily OS System backup, either full or incremental

Weekly UniData File system maintenance using WUFA; maintain any files that need manual intervention due to damage or level 2 overflow

Monthly OS Cleanup operating system logs and document as required by state auditors. Check sulog, wtmpx, utmpx, etc.

Monthly UniData Remove any RUN.#### files in the VOC of each remote account

Monthly UniData Cleanup the _PH_, _HOLD_, SAVEDLISTS, and savedlists Files


Frequency Level TaskMonthly UniData Clean up the PARAMETERS file

Monthly Colleague Clean up the appl.PPROCESS file using the UTJP screen in each application

Monthly Colleague Clean up the transaction logging files in each application using the UTTP screen

Monthly Colleague Clean up EDX.STATUS in the UT application using the EDSP screen

Quarterly Colleague Clean up ELF.TRANSACTIONS in each application using the EPRG screen

Quarterly Colleague Clean up Express Load files in each install account using the ELPR screen


Frequency Level Task

Quarterly Colleague Remove any patch files in the VOC of each remote and install account

Quarterly Colleague Unix backup to supplement a third-party backup strategy

Quarterly Colleague Clean up the appl.PRINTERS file by deleting all records with @ID beginning with SETPTR

Quarterly UniData Clean up the AE_COMS file

Qaurterly UniData Clean up the JSPARAMS file

Quarterly UniData Clean up the X.SECURITY.LOG file


Frequency Level TaskQuarterly UniData Clean up the WWW.STATE and

WWW.TOKENS files

Quarterly UniData Remove any guide GUIDE_ files form the Unix directory of each remote account

Quarterly Colleague Rebuild security classes for each application using the BUILD.SECURITY function

colleague system maintenance

Documents