ccpr workshop introduction to the cluster july 13, 2006
TRANSCRIPT
Outline
What is a cluster? How can CCPR’s cluster help you? Using the cluster
• Connecting from Windows
• Unix commands
• Submitting jobs
• Managing jobs
What is a cluster?
Think of a cluster as having two parts:• Hardware: a collection of computers
• Cluster management system: software for coordinating hardware with a collection of computational tasks (batch jobs)
CCPR’s Cluster: Management System
Jobs are submitted to the cluster via the cluster management system• An advanced batch system
• Efficiently allocates resources to jobs
• Remembers job requirements
• Remembers node specs and availability
• Schedules jobs according to requested resources and priority
• Users can easily view jobs
How does this help your research?
Easy access to software and data Faster processors More RAM Easy to share data, programs, etc. with
colleagues via the cluster More advanced users: parallel processing,
compilers, etc. Your pc is available for other work when you
submit a job to the cluster Submit a job and forget about it
Secure Shell
Secure Shell software (2 pieces)1. SSH Secure Shell Client (Telnet)
2. SSH Secure File Transfer Client (FTP).
• Download software from
www.ssc.ucla.edu Computing Security
Secure Shell: Connecting to lexis from Windows
Connect to lexis via SSH Secure Shell Client
• Startup SSH Secure Shell/File Transfer
• Click Quick Connect
• Enter:• Host Name: lexis.ccpr.ucla.edu
• User Name: your lexis account name
• Port Number: 22
• Click connect and enter password Can add “profile” for repeated use Same process for SSH Secure File Transfer Client
Samba: Transferring files from pc to lexis
Analagous to “mapping a drive” Allows user to view lexis files via
Windows Explorer interface Must use vpn when you’re off campus.
Download from• http://www.bol.ucla.edu/services/vpn/
Step-by-step instructions:• http://lexis.ccpr.ucla.edu/using/samba/
Samba: Transferring files from pc to lexis Open Windows Explorer on the local machine. Select Tools->Map Network Drive In the "Folder" text box enter:
\\lexis.ccpr.ucla.edu\your-user-name Click "Reconnect at login" if you want the drive to be
mounted each time the computer starts up. Click the "different user name" link to enter your user
name and password for the connection. Click "OK" then click the "Finish" button on the Map
Network Drive window.
Unix commands
passwd Change lexis password
smbpasswd Change samba password
pwd Print working directory
cd Change directory
mv filename1 filename2 Rename/move filename1 to filename2
cp filename1 filename2 Copy filename1 to filename2
More Unix Commands
cat file Display contents of file to screen
cat file1 >> file2 Concatenate file1 and file2
more file Display contents of file to screen
ls List files
ls –lah List files in current directory (options l=LONG, a=ALL, h = HUMAN-READABLE
More Unix commands
du –h Disk usage (option h=HUMAN-READABLE)
.forward forward mail to another account
gzip Compress file
gunzip Expand compressed file
ln –s actual link Create symbolic link
which name search for name
Getting help in Unix
man command list help for command
man –k command
keyword search for command
whatis command brief description of command
apropos keyword
list commands with keyword in their description
Unix Commands: Using Aliases
Alias• Create an alias for commonly used
commands
• Example: alias ls=“ls –lah”• Anytime you type ls, you will see the ls command
with options lah (l=long, a=all, h=human-readable)
• Edit .aliases file to create and remove aliases
Unix commands: Pipes and grep
history displays a “history” of previously entered commands
history | more history, but one page at a time
history | grep cat sends output of history to commnd “grep cat”, which displays only lines containing “cat”
history | wc Counts number of newlines, words, and bytes contained in output of history
Editing Files on Lexis
Pico – easiest editor. Open and type. Commands listed at bottom of editor.
Vi – more advanced. You can move around quickly, do copy/paste, search, etc, but keystrokes required for everything.
Emacs – most advanced
Submitting Stata and SAS jobs Submit jobs by typing qjob, qstata, qsas,
qmatlab at Unix prompt You will then be prompted for:
• filename
• memory (number in megs, ex. 1050)
• if memory > 1 gig -> Over 2 hours (y/n),
• job type (for qjob only)
• submit (y/n) Alternatively, submit all options without being
prompted:• qstata sample.do 500 y
Files produced by qstata
filename.log - This is the log file from Stata. The output of your entire Stata job is caputured in this file.
filename.do.cmd - This is the Command file that is submitted to the cluster.
filename.do.joblog - This file will capture the output of your CMD file and any errors that may occur (Stata output is not captured by this log.)
Job Management
RAM Compute time requested
Job sent to…
<1 GB n/a Any node, no time constraints
>2 GB n/a >4-GB node
1-2 GB <2 hours “2-hour” node, terminates after 2 hours
1-2 GB >2 hours >4-GB node
Job Management
qstat or qstatus to view overall cluster usage
qstat –j jobnumber for information particular to your job
qdel jobnumber to kill your job
Working interactively - qinteractive
To work interactively, type “qinteractive” at prompt
Default RAM allotted is 256 • Request x megs by typing “qinteractive x”
Please use interactive nodes responsibly• Limited number of spaces
• Testing only
• Logoff promptly