linux shell & shell scripting - i - ashok r....
TRANSCRIPT
“IBS574-ComputationalBiology&Bioinformatics”
Spring2018,Tuesday(01/30),2.00-4.00PM
Linuxshell&shellscripting-I
AshokR.DinasarapuPh.DScientist,Bioinformatics
Dept.ofHumanGenetics,EmoryUniversity,Atlanta
11/30/2018,2-4PM
• KernelisthecorethatprovidesbasicservicesforallotherpartsoftheOS.
• ShellistheoutermostpartofanOSthatinteractswithusercommands• commandlineinterpreter• sh:bash(bourneagainshell),csh,ksh,dash
• TerminalemulatorgivesusaccesstotheShell• xterm,rxvt,genome-terminal• console=physicalterminal
2
Kernel,Shell&TerminalfrequentlyusedtermsinUNIX-likeoperatingsystems(OS)
1/30/2018,2-4PM
3
Kernel,Shell&TerminalfrequentlyusedtermsinUNIX-likeoperatingsystems(OS)
terminal
1/30/2018,2-4PM
• Shellpromptwillusuallyinclude
[root@machinename~]# /root
[user_name@machinename~]$/home/user_name
Console/Terminal
4
Linuxfilesystemhierarchy1/30/2018,2-4PM
5
Easiestwaytodothisis…
Starttyping!
1/30/2018,2-4PM
• SSHallowsyoutoconnecttoyourserversecurelyandperformLinuxcommand-lineoperations.
Alternatively,https://blnx1.emory.edu:22443/
user_name@blnx1:~$
• ~meansyourhomedirectory(/home/user_name) • SSHfromWindows:downloadPuTTY
6
ConnectingServerviaSSH
1/30/2018,2-4PM
• WhatisthedefaultShellforblnx1?Usage:ls–la/bin/*sh /bin/sh->dash
• Whatisthedefaultterminalemulator?Usage:ls-la/usr/bin/*terminal*
/usr/bin/gnome-terminal
• Whatisphysicalterminal?
7
Console/Terminal
1/30/2018,2-4PM
Somesimplecommands:• env (currentenvironment)• date (fordate)• cal (forcalendar)• dforfree (diskspaceormemory)• exit (closeterminal/session)
Usage:env (thenpressenterkey)
8
Console/Terminal
1/30/2018,2-4PM
NavigationofthefilesystemCommands:pwd,ls,cd
• pwd (print/currentworkingdir)• ls (listingdirectoriesandfiles)
ls-l (longlistingformat)ls-la (withhiddenfiles)
Usage:pwd /home/user_name
Usage:ls-l91/30/2018,2-4PM
• mkdir makeasub-directory
Usage:mkdirdocument
mkdir-pdocument/resumemkdir-pdocument/{resume,cv,address}
• cdchangedirectory
Usage:cddocument/resume (enterdir)
cd../.. (returntopreviousdirectory) 10
NavigationofthefilesystemCommands:pwd,ls,cd,mkdir
1/30/2018,2-4PM
cd~ (changetohomedir)cd (changetohomedir)
cd/ (changetorootdir)
cddocument (relativepath)cd~/document (absolutepath)cd/home/user_name/document
*with$HOMEenvironmentvariable11
NavigationofthefilesystemCommands:pwd,ls,cd
1/30/2018,2-4PM
Ifadirectorynamehaswhitespace–
mkdir‘MyData’
Usage:cdMy\Data cd"MyData” cd'MyData'
12
NavigationofthefilesystemCommands:pwd,ls,cd,mkdir
Anon-quotedbackslash,\,isusedasanescapecharacterinbash.
1/30/2018,2-4PM
Chooseatexteditor:emacs,Vim
Usage:vihome.txt
INSERTmode:presskeyslikeiORa&starttyping."i"willletyouinserttextjustbeforethecursor."I"insertstextatthebeginningofthecurrentline."a"willletyouinserttextjustafterthecursor,and"A"willletyoutypeattheendofthecurrentline.
Create/Edittextfiles
131/30/2018,2-4PM
Typethefollowingtext:
Create/Edittextfiles
Street:201DowmanDriveCity:AtlantaState:GeorgiaCountry:USAZip:30322
141/30/2018,2-4PM
SAVEmode:pressesckeyAND:q!fornottosaveOR
:xtosaveandexit.
15
HowTo:SavefileinViTextEditor
1/30/2018,2-4PM
Alternatively,use:wqforwriteandquit(saveandexit)
Chooseatexteditor:emacs,Vim
Usage:vicollege.txt
Create/Edittextfiles
16
Street:954GatewoodRdCity:AtlantaState:GeorgiaCountry:USAZip:30329
1/30/2018,2-4PM
Listingfiles,directories&it’scontents
17
• Listfilesrecursively Usage:ls-R
document/resumedocument/cvdocument/addresshome.txtcollege.txtMyData
1/30/2018,2-4PM
Listingfiles,directories&it’scontents
18
CopyMoveSymlink
• Listfilesrecursively Usage:ls-R
document/resumedocument/cvdocument/addresshome.txtcollege.txtMyData
1/30/2018,2-4PM
• Copyafile
Usage:cphome.txtdocument/address/
• Copyallfiles
Usage:cp*.txtdocuments/address/
“cp”–keepstheoriginalfile&makesaduplicateofit
19
CAREFUL!Ifthedestinationfilealreadyexists,itwillbeoverwrittenwithoutaconfirmationprompt.Use“cp-i”interactiveoptionforaprompt.
1/30/2018,2-4PM
• Moveafile
Usage:mvhome.txtdocument/cv/
• Moveallfiles
Usage:mv*.txtdocument/cv/
Renamefiles&directoriesusing“mv”movecommand
20
CAREFUL!Ifthedestinationfilealreadyexists,itwillbeoverwrittenwithoutaconfirmationprompt.Use“mv-n”interactiveoptionforaprompt.
1/30/2018,2-4PM
• ln:createshardlink,defaultln~/document/cv/home.txt~/document/resume/
• ln-s:symboliclinkln-s~/document/cv/college.txtdocument/resume/
Symboliclinkisaspecialkindoffilethatpointstoanotherfile
211/30/2018,2-4PM
• view/read Usage:cathome.txt
• concatenate&viewUsage:cathome.txtcollege.txt
• concatenate&redirectionUsage:cathome.txtcollege.txt>add.txt
Thecatcommandisusedfor:
22
WARNING:ifadd.txtalreadyexists,itwillbeover-written.Use>>toappendtheoutputtotheexistingfile.
1/30/2018,2-4PM
• less(readwithcursor)Usage:lessadd.txt
• more(readpagebypage)Usage:moreadd.txt
“moreorless”:terminalpagercommandsinUnix-likesystems
23
Theprogramlessormoredoesnotrequirethewholefiletobeloadedinmemorytoviewpartsofit.
1/30/2018,2-4PM
• Viewthefirst5lines Usage:head–n5add.txt
• Viewthelast4linesUsage:tail–n4add.txt
head&tailletsyouseethefirst&lastfewlinesofatextfile
24
Bydefault,headortailreturnsthefirstorlast10lines,respectivelyofeachfilethatisprovidedtoit.
1/30/2018,2-4PM
• sendingdatafromoneprogramtoanother Usage:cathome.txt|less
• Redirectingtoafile Usage:head-2home.txt>lines.txt
wc-l<home.txt>count.txt
“pipe&redirect”
251/30/2018,2-4PM
Whatdoes“>/dev/null2>&1”mean?
• ArithmeticinPOSIXshellsisdonewith$anddoubleparentheses:
echo$((2+2)) STDIN (<0):keyboard STDOUT (>1):screen STDERR (>2):screen
echo$((2+2))>result.txt
(STDOUTredirectedintoafile,result.txt)
echo$((2+2))>/dev/null2>&1
(TolistenonlytoSTDERR(2>&1),STDOUTredirectedinto/dev/null&STDERRredirectedtoSTDOUT).
261/30/2018,2-4PM
Whatdoes“>/dev/null2>&1”mean?
• ArithmeticinPOSIXshellsisdonewith$anddoubleparentheses:
echo$((2+2))>/dev/tty STDIN (<0):keyboard STDOUT (>1):screen STDERR (>2):screen
echo$((2+2))>result.txt
(STDOUTredirectedintoafile,result.txt)
echo$((2+2))>/dev/null2>&1
(TolistenonlytoSTDERR(2>&1),STDOUTredirectedinto/dev/null&STDERRredirectedtoSTDOUT).
27
/dev/ttyisaspecialfile,representingtheterminalforthecurrentprocessUse,ps–a(toseeprocessesattachedtottys)InLinux,theconsoleappearsasseveralterminals(ttys)
1/30/2018,2-4PM
28
Createafile(test_error.sh)withthefollowingtwolinesofcode#!/bin/shecho$((2+'x'))Runas./test_error.sh./test_error.sh2>/dev/tty./test_error.sh2>error_file
1/30/2018,2-4PM
• Searchafileforkeywords
Usage:grepStateadd.txt
• ignorecaseoption Usage:grepstateadd.txt grep-istateadd.txt
SearchingaPATTERNwithgrep
29
"globalregularexpressionprint"
Regularexpressionsareusedtosearchandmanipulatethetext,basedonthepatterns[Beginningofline(^);Endoftheline($)]
1/30/2018,2-4PM
1 India2 USA3 Ireland4 UK5 Canada
30
“jointwofiles”
Usage:joincountry.txtcity.txt>file.txt
Usage:vicountry.txt
Usage:vicity.txt
1 NewDelhi2 WashingtonDC3 Dublin4 London5 Toronto
• Joinstherowsoftwofileswhichshareacommonfieldofdata.
1/30/2018,2-4PM
• Threemaintypeofpermissions• Read(r),write(w),andexecute(x)• 3typesofusers(user,group&other)
Usage:ls-l
Changethepermissionsoffiles
311/30/2018,2-4PM
• Threemaintypeofpermissions• Read(r),write(w),andexecute(x)• 3typesofusers(user,group&other)
Usage:chmodu=rwx,g=rx,o=rhome.txt
Changethepermissionsoffiles
321/30/2018,2-4PM
• Threemaintypeofpermissions• Read(r),write(w),andexecute(x)• 3typesofusers(user,group&other)
Usage:chmodu=rwx,g=rx,o=rhome.txt
Changethepermissionsoffiles
33
Usage:chmod754home.txt
4standsfor"read",2standsfor"write”,1standsfor"execute",&0standsfor"nopermission"
1/30/2018,2-4PM
34
Createafile(test_error.sh)withthefollowingtwolinesofcode#!/bin/shecho$((2+'x'))Runas(chmod+xtest_error.sh)./test_error.sh./test_error.sh2>/dev/tty./test_error.sh2>error_file
1/30/2018,2-4PM
• rmremovesfilesblindly,withnoconceptof'trash’!!!
• Removeafile Usage:rmdocument/cv/add.txt
• RemovedirectoryrecursivelyalongwithallofitscontentsUsage:rm-rdocument
Deletefiles&directories
35
Write-protectedfilesprompttheuserforaconfirmation(withayandann)beforeremoval.Use-f(i.e.,force)toremoveallspecifiedfiles,whetherwrite-protectedornot.
1/30/2018,2-4PM
HowtosetupcommandaliasesinLinux
• Openyour.bashrcvi~/.bashrc
• Addthealiasaliasu="cd..;ls”
• PathadjustmentsexportPATH="$PATH:/<here_new_path>”
• Reloadthe“.bashrc”file source~/.bashrc
361/30/2018,2-4PM
Listofarchiveand/orcompressionformats
Fileextension Officialname Description.tar Tapearchive Archiving.bz2 bzip2 Compression.gz gzip Compression.tar.gz(.tgz) tarwithgzip both.tar.bz2 tarwithbzip2 both.zip ZIP both
DataarchivinginLinux
371/30/2018,2-4PM
• Tocompressasinglefile(.gz)
Usage:gziphome.txt
• Archive&compress
Usage:tar-jcvfhome.tar.bz2home.txt
j:usebzip2compress(z:usegzipcompress)c:compressv:verbosef:file
Data/Filecompression
381/30/2018,2-4PM
wgethttps://github.com/samtools/samtools/releases/download/1.3.1/samtools-1.3.1.tar.bz2
• Viewthecontentsofatar.bz2file
Usage:tar-jtvfsamtools-1.3.1.tar.bz2
• ExtractthecontentsUsage:tar-jxvfsamtools-1.3.1.tar.bz2
Data/Filedecompression
39
t:listthecontents;x:extractthecontentsofanarchive
Downloadfileusingwget
1/30/2018,2-4PM
WhatisSAMTOOLS?
40
• SAMTOOLSisasuiteofprogramstohandlehigh-throughputsequencingdata.
• UsesSAM(SequenceAlignment/Map)/BAM(BinaryAlignment/Map)files.
• VariantCallFormat(VCF),mostlikelystoredinacompressedmanner.samtoolsmpileup|\
bcftoolscall>name.vcf1/30/2018,2-4PM
41
• CompressingVCFfileswithBGZipandindexingitwithTabixisthestandardwayVCFfilesarestored.
• bgzip–Blockcompression/decompressionutility.
• tabix–GenericindexerforTAB-delimitedgenomepositionfiles.
• DownloadBGZip’pedVCFfilefromhttp://vat.gersteinlab.org/datasets.php
VCFfilecompressionwithBGZip&indexingitwithTabix
1/30/2018,2-4PM
VCFfilecompressionwithBGZip&indexingitwithTabix
42
• View/Readcompressedfilegzip-cdCEU.low_coverage.2010_07.indel.genotypes.vcf.gz|head-20zcatCEU.low_coverage.2010_07.indel.genotypes.vcf.gz|head-20
• Decompress,gzip(compress)&indexgunzipCEU.low_coverage.2010_07.indel.genotypes.vcf.gzgzipCEU.low_coverage.2010_07.indel.genotypes.vcftabixCEU.low_coverage.2010_07.indel.genotypes.vcf.gzNotaBGZFfile:CEU.low_coverage.2010_07.indel.genotypes.vcf.gztbx_index_buildfailed:CEU.low_coverage.2010_07.indel.genotypes.vcf.gz
• Decompress,bgzip(compress)&indexgunzipCEU.low_coverage.2010_07.indel.genotypes.vcf.gzbgzipCEU.low_coverage.2010_07.indel.genotypes.vcftabixCEU.low_coverage.2010_07.indel.genotypes.vcf.gzCEU.low_coverage.2010_07.indel.genotypes.vcf.gz.tbi1/30/2018,2-4PM
43
Rsync(RemoteSync)commandforcopyingandsynchronizingfiles&directoriesremotelyaswellaslocallyinLinux/Unixsystems
rsync–av~/script/[email protected]:~/[email protected]'spassword:buildingfilelist...doneCorrPlot.Rsent3310bytesreceived42bytes103.14bytes/sectotalsizeis3177speedupis0.95
1/30/2018,2-4PM
44
Thank you
Practice Makes Perfect
1/30/2018,2-4PM