celerra automatic log collection tool

Document Template

( EMC2 Corporation CONFIDENTIAL AND PROPRIETARY9/11/01

Celerra Automatic Log collection tool

Revision 1.0[November 1, 2006]

Confidential and Proprietary

Copyright 2003 EMC Corporation. All rights reserved.

Printed 2/6/2006EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

2Revision History

3CELERRA AUTOMATIC LOG/DUMP COLLECTION AND TRANSFER

3Celerra Automatic Log Collection and Transfer Feature:

3The purpose of the tool is two-fold:

3A Few Anticipated Benefits of the Auto Log Collection & Transfer Capability:

4Summary of Configuration and Default Behavior:

4DEPLOYING, INSTALLING, CONFIGURING, TESTING AUTO COLLECT TOOL:

5HOW TO DETERMINE IF AUTO-COLLECT HAS BEEN INSTALLED ON CONTROL STATION:

5CONTROL STATION INSTALLATION & CONFIGURATION:

10UNINSTALLING FEATURE:

10DEFAULT LOCATION OF LOGS AND DUMPS WHEN AUTO-TRANSFER IS EITHER NOT ENABLED, OR NOT ABLE TO FTP OUT OF THE SYSTEM:

10AUTOMATICCOLLECTION SCRIPT USAGE:

11VERIFYING FEATURE FUNCTIONALITY:

11TESTING & VERIFYING AUTOMATIC LOG/DUMP COLLECTION AND TRANSFER:

12The sys_log illustrates what occurs:

13TRANSFER DISABLED, TESTING AUTO LOG & DUMP COLLECTION:

14MANUALLY INITIATING LOG COLLECTION:

Revision History

DateRevisionChange Description

Nov 1,20061.0

May 15, 2006 rev 6 thm

CELERRA AUTOMATIC LOG/DUMP COLLECTION AND TRANSFER

Celerra Automatic Log Collection and Transfer Feature:

As part of an ongoing EMC effort to reduce administrative overhead for both EMC support personnel and Customers, the Auto Log Collection and Transfer tool was developed as a way to automate the collection of logs and dumpfiles when CallHome events, such as Server Panics, are generated on the Celerra.

The purpose of the tool is two-fold:

1) To automatically collect all appropriate logs and/or dumpfiles associated with a triggering Event, at the time that a CallHome is produced

2) To automatically transfer the Logs and Dumpfiles to EMC for analysis and root cause, if configured to do so.

A Few Anticipated Benefits of the Auto Log Collection & Transfer Capability:

1.) Since logs can quickly wrap and overwrite critical data, it is anticipated that automating what was previously a completely manual process, often involving multiple EMC and Customer resources, would be greatly streamlined by offering automated collection and delivery of all the information necessary to root cause the CallHome event.

2.) Nearly realtime transfer of logs and dumpfiles (if a panic occurred) to an external EMC FTP Server (if configured and allowed to do so), or redirection of the logs and dumps to a location on the Celerra, or redirection of logs and dumps to an internal FTP site of the Customers choosing, for later transfer to EMC.

3.) If the automatic transfer feature is not enabled, the automatic log collection portion of the feature still greatly reduces the complexity of log and dumpfile collection by automatically collecting logs & dumps and compressing them to a designated location on the Control Station, conserving and managing diskspace as part of its logic.

4.) Outside of certain timers and controls built into the feature, the auto log collection and transfer feature offers nearly unlimited capability in delivering logs and dumpfiles to EMC, or to an internal staging site, without human intervention after initial setup.

5.) Ease of installation and configuration. The automatic collection and transfer system can be setup and placed in operation in 15 minutes or less.

Summary of Configuration and Default Behavior:

1. By default, (5) sets of log collections are saved in /nas/var/log, but can be modified via cfg file

2. By default, (2) panic dumps are gzipped and saved in /nas/var/dump, but can be modified via cfg file

3. By default, (10) dump header files are saved in /nas/var/dump

4. Up to (2) simultaneous transfer sessions can be started

5. Auto collection script will not run for events that occur within 15 minutes of the previous collection event, except for Server Panics

6. Transfer feature is disabled by default and requires initial configuration of /nas/site/automaticcollection.cfg file. Run /nas/tools/automaticcollection enable | -disable to toggle on or off.

7. Run /nas/tools/automaticcollection enable | -disable to toggle on or off.8. To disable or re-enable both the Collection and Transfer features at once, from the CLI:

# /nas/tools/automaticcollection -disable -all

automaticcollection[25450]: The automatic collection and transfer feature has been disabled.

automaticcollection[25450]: The automatic transfer feature has been disabled.

# /nas/tools /automaticcollection -enable -all

automaticcollection[17544]: The automatic collection and transfer feature has been enabled.

automaticcollection[17544]: The automatic transfer feature has been enabled

9. How to manually run the Log Collection getlogs script, Dump Collection retrievedump script, and Transferlogs script, respectively:

/nas/tools/automaticcollection getlogs (gathers logs and transfers, if transfer feature enabled)/nas/tools/automaticcollection retrievedump s 2 | 3 | | n (extracts most recent dump from the indicated data mover slot_# , and transfers if the transfer feature is enabled)

/nas/tools/automaticcollection transferlogs (transfers only most recent set of logs found in /nas/var/log)

DEPLOYING, INSTALLING, CONFIGURING, TESTING AUTO COLLECT TOOL:

DEPLOYMENT OPTIONS:

1. Stage file on EMC FTP Server and download directly to Control Station, if an internet route exists

2. Otherwise, email file to customer and have uploaded to Control Station

3. Or, email to CE or EMC Field representative for uploading to Control Station

4. Or, have Support persons upload file directly to Control Station from desktop using ProComm, or some other terminal transfer program

Uploaded File Name & Size:

-rw-r--r-- 1 root root 19612 May 11 14:04 logcoltrans.tar.gz

HOW TO DETERMINE IF AUTO-COLLECT HAS BEEN INSTALLED ON CONTROL STATION:

( # nas_event -L -i

Loaded config. files:

1: /nas/sys/nas_eventlog.cfg

2: /nas/http/webui/etc/web_client_eventlog.cfg

3: /nas/site/mail_events.cfg

4: /nas/site/trap_events.cfg

5: /nas/jserver/event_config/events.cfg

Note: Above output shows that the automaticcollection tool has not been installed

# nas_event -L -i






5: /nas/site/automaticcollection-cs.cfg (Auto log collection already loaded

6: /nas/site/automaticcollection-ui.cfg (Auto log collection already loaded


Note: Above output shows that the automaticcollection tool has been installed and loaded on the CS, the version of this tool that gets installed with NAS Code 5.4.21.4 and 5.5.22.0 and higher does not use these event configuration files. Instead it is part of the native NAS event handling configuration file, and will use facility 141 LogCollect to handle its events.CONTROL STATION INSTALLATION & CONFIGURATION:

( 1. Dial into Control Station, login as nasadmin, and su to root user

( 2. Download or Upload logcoltrans.tar.gz file to desired directory on Control Station:

#ftp ftp.emc.com

Name (ftp.emc.com:nasadmin): ftp

Password: [email protected]

ftp>cd /incoming/autocollect

ftp> bin

200 Type set to I.

ftp> hash

Hash mark printing on (1024 bytes/hash mark).

ftp> get logcoltrans.tar.gz

local: logcoltrans.tar.gz remote: logcoltrans.tar.gz

ftp>quit

( 3. Unzip and untar the logcoltrans.tar.gz file:# tar -zxvf logcoltrans.tar.gz

logcoltrans/

logcoltrans/automaticcollection

logcoltrans/transfer_support_materials

logcoltrans/collect_support_materials.auto

logcoltrans/automaticcollection-cs.cfg

logcoltrans/automaticcollection-ui.cfg

logcoltrans/recover_slot.auto

logcoltrans/INSTALL

logcoltrans/UNINSTALL

logcoltrans/README

Note: Creates a new directory called logcoltrans in the directory that the unzip and untar was conducted( 4. Go to new logcoltrans directory and review the README file before running INSTALL script:

a.) #cd /home/nasadmin/logcoltrans;ls -la

-rwxr-xr-x 1 root root 16668 May 2 18:07 automaticcollection

-rw-r--r-- 1 root root 3631 May 3 16:03 automaticcollection-cs.cfg

-rw-r--r-- 1 root root 583 May 3 16:04 automaticcollection-ui.cfg

-rwxr-xr-x 1 root root 30885 May 2 18:09 collect_support_materials

-rwxr-xr-x 1 root root 2691 Apr 15 23:10 INSTALL

-rw-r--r-- 1 root root 917 Mar 1 16:55 README

-rwxr-xr-x 1 root root 9298 May 2 18:07 recover_slot.auto

-rwxr-xr-x 1 root root 14769 May 2 18:10 transfer_support_materials

-rwxr-xr-x 1 root root 1673 Apr 11 07:48 UNINSTALL

# cat README

This package can be installed on 5.4.18.3 or later system. The package will add automatic log collection and transfer feature into the system, and it will allow a CS to automatically collect log/dump when certain event occurs. If the CS has external network connection, it will automatically ftp the result to the server and location as configured in /nas/site/automaticcollection.cfg file. The transfer feature is disabled by default, and it has to be enabled via command /nas/tools/automaticcollection -Enable. Prior to enabling the feature, please configure the /nas/site/automaticcollection.cfg file to your liking. By default, ftp.emc.com is the host, and directory is /incoming/.

To install this feature:

Run ./INSTALL script. You have to make sure the NAS service is running.

To uninsstall this feature:

Run ./UNINSTALL script. You have to make sure the NAS service is running.

b.) Check to make sure NAS_MCD is running before installing:

# ps -ef |grep nas_mcd

root 1182 1 0 Apr10 ? 00:00:00 /nasmcd/nas_mcd -h /nasmcd /nas/





root 1724 1439 0 Apr10 ? 00:00:00 /nasmcd/nas_mcd -h /nasmcd /nas/( 5. Install Auto Log Collection Utility:# ./INSTALL

This package can be installed on 5.4.18.3 or later system. The package will add automatic log collection and transfer feature into the system, and it will allow a CS to automatically collect log/dump when certain event occurs. If the CS has external network connection, it will automatically ftp the result to the server and location as configured in /nas/site/automaticcollection.cfg file. The transfer feature is disabled by default, and it has to be enabled via command /nas/tools/automaticcollection -Enable. Prior to enabling the feature, please configure the /nas/site/automaticcollection.cfg file to your liking. By default, ftp.emc.com is the host, and directory is /incoming/.

Starting copying files....

EventLog : will load /nas/site/automaticcollection-cs.cfg... done

EventLog : will load /nas/site/automaticcollection-ui.cfg... done

Generate configuration file...

# This is the configuration file for automaticcollection.

# Anything after the # character on that line is ignored. Blank

# lines are also ignored. Whitespace at the begining and end of each

# line is also ignored.

# To get back to the default configuration file, simply remove this

# file (or rename it, of course), and a new one will be created the

# next time automaticcollection runs. You may change this file if you wish,

# and if you do so, automaticcollection will not recreate it.

# However, you will not recieve any updates to this file should you

# choose to upgrade your control station software later.

# Don't modify the following line unless instructed by authorized

# service personnel.

# OFF

# Uncomment the following line to disable the transfer feature.

Disable

# Uncomment the following line to control the number of copies of

# support materials you would like to keep in /nas/var/log directory.

# We strongly suggest to keep the default value. The default is 5.

# MaxCopyLog 5

# Uncomment the following line to control the number of copies of

# panic dumps you would like to keep in /nas/var/dump directory.


# MaxCopyDump 2

# Uncomment the following line to control the number of simultaneous

# transfer you will allow from the Control Station. The default is

# 2, and it should not go any lower.

# MaxTransfer 2

# The following configurations are used by the transfer process.

# Please do not modify these unless instructed by authorized service

# personnel.

host 168.159.4.20

user anonymous

pass [email protected]

# Modify the following line to specify the remote directory where

# the materials should be stored. The default is on the FTP server

# /incoming/APM00040301591 directory.

remote_directory /incoming/APM00040301591

Done.

( 6. Verify that Auto-Collect is Loaded, Functional & has Transfer Disabled:

a.) # nas_event -L -i







6: /nas/site/automaticcollection-cs.cfg

7: /nas/site/automaticcollection-ui.cfg

b.) # /nas/tools/automaticcollection -Info

Transfer feature is DISABLED( 7. To use the auto-transfer feature, configure the automaticcollection.cfg file first:

# cd /nas/site

# vi automaticcollection.cfg

# Uncomment the following line to disable the transfer feature.

Disable


MaxCopyDump 2

# personnel.

host 168.159.4.20

user anonymous

pass [email protected] (May need to edit this line with correct Serial number

# Modify the following line to specify the remote directory where

# the materials should be stored. The default is on the FTP server

# /incoming/APM00040301591 directory.

remote_directory /incoming/APM00040301591 (May need to edit this line with correct Serial number

Note: Should only need to edit the above two highlighted lines when configuring the Transfer feature. Use /nas/sys/callhome.config to derive the correct APM serial number. It is recommended to configure the correct serial number, even if the Transfer feature is not going to be enabled. By default, the automaticcollection.cfg file will be configured with the Serial number from /nas/sys/callhome.config or by the Control Station name.

# head /nas/sys/callhome.config

modem phone number: 15003

site name: laip1 Celerra PVT / SVT Lab - EMC Lab @ 171 South St

serial number:APM00040301591( 8. To Enable & Verify the Automatic Transfer Script:

# /nas/tools/automaticcollection -Enable

automaticcollection[17117]: The automatic transfer feature has been enabled

# tail /nas/log/sys_log

May 11 15:05:31 2006 OTHERS:6:1300 automaticcollection[17117]: The automatic transfer feature has been enabled.

Note: Even if accidentally enabled, and there is no valid FTP path off the system, the script will still kick-off for the designated CallHome events and collect the appropriate logs and dumps.

UNINSTALLING FEATURE:

# ./UNINSTALL

Starting deleting files....

EventLog : will unload /nas/site/automaticcollection-cs.cfg... done

EventLog : will unload /nas/site/automaticcollection-ui.cfg... done

Done.

# nas_event -L -i





DEFAULT LOCATION OF LOGS AND DUMPS WHEN AUTO-TRANSFER IS EITHER NOT ENABLED, OR NOT ABLE TO FTP OUT OF THE SYSTEM:

Dumps are extracted from one of the (3) full dump_slots on LUN 0 backend, and gzipped to /nas/var/dump

Logs are executed from collect_support_materials and deposited to /nas/var/log

Review sys_log for details of Collection activity

AUTOMATICCOLLECTION SCRIPT USAGE:

# /nas/tools/automaticcollection -help

usage: automaticcollection

| -help

| -getlogs [ -filename ]

| -retrievedump -slot [-getlogs] [ -filename ] [ -local ]

| -transferlogs

| -config

| -disable

| -enable

| -info

Options:

-help - this help screen

-getlogs - Run collect script

-retrievedump - Run transfer script for dump

-transferlogs - Run transfer script for log

-filename - Include additional info in the name, such as clarify case number

-local - Write the dump to a local dir

-slot - The slotnum to retrieve dump

-config - Display the current configuration

-disable - Turn off the Transfer feature

-enable - Turn on the Transfer feature

-info - Check the state of Transfer feature

VERIFYING FEATURE FUNCTIONALITY:

TESTING & VERIFYING AUTOMATIC LOG/DUMP COLLECTION AND TRANSFER:

( 1. Force a Server panic dump, which will trigger the automaticcollection & transfer scripts:

# date;.server_config server_2 -v "panic auto log dump collect and transfer test"

Fri Apr 14 10:56:17 EDT 2006

server_2 : commands processed: 0

Error 4020: server_2 : Input/output error

Timed out

Note: Server panicks and fails over to standby server

# nas_server -l

id type acl slot groupID state name

1 4 1000 2 2 server_2.faulted.server_3

2 1 1000 3 0 server_2

( 2. Checking Progress of Autocollection and Transfer Scripts:# ls -la /nas/var/dump

prw-r--r-- 1 root root 4096 Apr 14 10:58 dump_060414_1056.dump.gz

prw-r--r-- 1 root root 0 Apr 14 10:58 dump_APM00024300874.060414_1056.dump.gz

-rw-r--r-- 1 root root 1175 Apr 14 10:57 header_APM00024300874.060414_1056.txt (crash header)

Note: Shows dumpfile being extracted from backend to /nas/var/dump and is being redirected directly to FTP server where it is gzipped and the transfer of the dump is completed. When the transfer feature is Enabled, the dumpfile is not deposited in /nas/var/dump. If the transfer feature was Disabled, then the panic dump would be gzipped directly to the /nas/var/dump directory. Notice that a separate file is created that stores the crash header of the dumpfile, as called by dump_slot F 0. Partial dumps will not have a crash header or be extracted by the automatic collection tool.

# ps -eafl |grep -i automatic

000 S root 5019 2290 0 69 0 - 579 wait4 10:56 ? 00:00:00

/bin/bash /nas/tools/automaticcollection -retrievedump -s 2

000 S root 5049 5019 0 69 0 - 575 wait4 10:56 ? 00:00:00

/bin/sh /nas/tools/transfer_support_materials -config /nas/site/automaticcollect

000 S root 5625 5049 0 69 0 - 487 pipe_w 10:57 ? 00:00:00

/usr/bin/ncftpput -V -m -f /nas/site/automaticcollection.cfg /incoming/APM000243

000 S root 6561 2290 0 69 0 - 579 wait4 10:59 ? 00:00:00

/bin/bash /nas/tools/automaticcollection getlogs

Note: Processes that run on the Control Station during the automatic collection and transfer when it involves a dumpfile. The first thing that occurs for a Panic dump is the transfer script starts, retrieves dump and transfers to the FTP site. Then the collection script starts to gather the logs.

The sys_log illustrates what occurs:

/nas/log/sys_log

Apr 14 10:56:22 2006 BoxMonitor:2:302 Slot 2 has panicked

Apr 14 10:56:36 2006 OTHERS:7:1202 Slot 2 dump is ready

Apr 14 10:56:37 2006 141:6:100 Transfer script has started.

Apr 14 10:59:29 2006 141:6:0 Collection script has started.

Apr 14 11:04:18 2006 141:6:1 Collection script has finished successfully.

Apr 14 11:11:15 2006 141:6:101 Transfer script has finished successfully. Check host 168.159.4.20 for dump saved as dump_APM00024300874.060414_1056.dump and dump header file /nas/var/dump/header_APM00024300874.060414_1056.txt in /incoming/APM00024300874.( 3. Verifying logs on FTP Server & in local /nas/var/log directory:257 "/incoming/APM00024300874" is current directory.

ftp> ls -la

-r-------- 1 u3 g3 478807681 Apr 14 15:15 dump_APM00024300874.060414_1056.dump.gz

-r-------- 1 u3 g3 1175 Apr 14 15:01 header_APM00024300874.060414_1056.txt

226 Transfer complete.

# ls la /nas/var/log

-rw-r--r-- 1 root root 4644295 Apr 14 11:03 support_materials_APM00024300874.060414_1059.tar.gz

Note: The crash_header file is only created for dump_slot 0. Test successful and transfer completed in 18 minutes.

TRANSFER DISABLED, TESTING AUTO LOG & DUMP COLLECTION:

( 1. Disable the automatic FTP Transfer feature:

# /nas/tools/automaticcollection -Disable

The transfer feature has been disabled

# tail /nas/log/sys_log

Apr 14 11:14:25 2006 141:6:301 The transfer feature has been disabled.

( 2. Initiating Panic Dump:

# date;.server_config server_2 -v "panic testing auto collect logs and dumps only"

Fri Apr 14 13:03:17 EDT 2006

server_2 : commands processed: 0

Error 4020: server_2 : Input/output error

Timed out

( 3. Server fails over, Autocollect gathers dumpfile, logs & places on /nas/var/dump & /nas/var/log, respectively:

# nas_server -l

id type acl slot groupID state name

1 4 1000 2 2 server_2.faulted.server_3

2 1 1000 3 0 server_2

# ps -eafl |grep -i automatic

000 S root 9674 2290 0 69 0 - 579 wait4 13:03 ? 00:00:00

/bin/bash /nas/tools/automaticcollection -retrievedump -s 2

000 S root 9701 9674 0 69 0 - 575 wait4 13:03 ? 00:00:00

/bin/sh /nas/tools/transfer_support_materials -config /nas/site/automaticcollect

# ls -la /nas/var/dump

prw-r--r-- 1 root root 4096 Apr 14 13:04 dump_060414_1303.dump.gz (Dump extraction in progress)

-rw-r--r-- 1 root root 15007744 Apr 14 13:04 dump_APM00024300874.060414_1303.dump.gz

-rw-r--r-- 1 root root 1176 Apr 14 13:03 header_APM00024300874.060414_1303.txt

# ls -la /nas/var/dump

-rw-r--r-- 1 root root 416846064 Apr 14 13:15 dump_APM00024300874.060414_1303.dump.gz (Dump completed)

# ls la /nas/var/log

-rw-r--r-- 1 root root 4693505 Apr 14 13:11 support_materials_APM00024300874.060414_1306.tar.gz

/nas/log/sys_log

Apr 14 13:03:22 2006 BoxMonitor:2:302 Slot 2 has panicked

Apr 14 13:03:36 2006 OTHERS:7:1202 Slot 2 dump is ready

Apr 14 13:03:37 2006 141:6:100 Transfer script has started.

Apr 14 13:06:37 2006 141:6:0 Collection script has started.

Apr 14 13:07:44 2006 OTHERS:7:1 nasdb_backup: done

Apr 14 13:11:17 2006 141:6:1 Collection script has finished successfully.

Apr 14 13:15:26 2006 141:3:101 Please upload the dump /nas/var/dump/dump_APM00024300874.060414_1303.dump.gz and dump header file /nas/var/dump/header_APM00024300874.060414_1303.txt to EMC Support and delete the dump because automatic transfer feature is disabled, or there is a problem c

MANUALLY INITIATING LOG COLLECTION:

( 1. Start Manual Log Collection:

Note: -getlogs will run the collection script and transfer logs if the transfer feature is enabled, as well as write the logs to /nas/var/log. If Transfer is disabled, will only collect and deposit the logs to /nas/var/log.

#/nas/tools/automaticcollection getlogs Running Master Script

/bin/ls: /nas/var/log/support_materials*: No such file or directory

collect_support_materials[30758]: The collection script revision 2.7.1 has started.

Collecting /nas/log/*, and /nas/jserver/logs

Collecting output from server_log

Collecting output from internal commands

Collecting event log configuration files

Collecting other files from /nas, /nas/site, /nas/sys,

/nas/rdf, and /nas/dos

Collecting files from .etc dir of each DM

Collecting /var logs

Collecting upgrade logs

Collecting /etc files

Collecting /http/logs and /tomcat/logs

Collecting Celerra Manager tasks

Collecting cron files

Collecting Control Station process information and versions

Collecting /nas/jserver/debug_of_core* files

Now running material collection script for longer running commands.

Collecting complete nas dir listing

Collecting output from nas commands

Collecting output from other CS commands

Zipping archive for single file output

Material Collection File:

/nas/var/log/support_materials_APM00040301591.060511_1434.tar.gz has been generated.

**********************************************************************

Please include file /nas/var/log/support_materials_APM00040301591.060511_1434.tar.gz

with materials submitted to EMC for problem investigation.

**********************************************************************

collect_support_materials[30758]: The collection script has finished successfully.

( 2. Verifying Results of Log Collection:# ls la /nas/var/log

-rw-r--r-- 1 root root 3810432 May 11 14:37 support_materials_APM00040301591.060511_1434.tar.gz

/nas/log/sys_log

May 11 14:34:23 2006 OTHERS:6:1001 collect_support_materials[30758]: The collection script revision 2.7.1 has started.

May 11 14:38:02 2006 OTHERS:6:1002 collect_support_materials[30758]: The collection script has finished successfully.

May 11 14:38:03 2006 OTHERS:6:1100 transfer_support_materials[5003]: The transfer script has started.

May 11 14:38:17 2006 OTHERS:6:1105 transfer_support_materials[5003]: The transfer script has finished successfully. Your authorized service provider should check host 168.159.4.20 for the log saved as /nas/var/log/support_materials_APM00040301591.060511_1434.tar.gz at /incoming/APM00040301

ftp> cd incoming/APM00040301591

ftp> ls -la

-r-------- 1 g3 3810432 May 11 18:47 support_materials_APM00040301591.060511_1434.tar.gz

EMBED Word.Picture.8

Draft

_1178590623.doc

EMC NAS Organization

NAS Service Readiness

EMC2 Corporation

celerra automatic log collection tool

Documents