unix text tools - masaryk universitypary/texttools.pdf · 2008-07-24 · unix text tools tradition...
TRANSCRIPT
![Page 1: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/1.jpg)
Unix Text ToolsEuropean Masters in Language & Speech
Pavel Rychly
Faculty of InformaticsMasaryk University
Brno, Czech Republic
24 July, 2008
Pavel Rychly Unix Text Tools
![Page 2: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/2.jpg)
Unix Text Tools Tradition
Unix has tools for text processing from the very beginning(1970s)Small, simple tools, each tool doing only one operationPipe (pipeline): powerful mechanism to combine tools
Pavel Rychly Unix Text Tools
![Page 3: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/3.jpg)
Short Description of Basic Text Tools
cat concatenate files and print on the standard outputhead output the first part (few lines) of files
tail output the last part (few lines) of filessort sort lines of text filesuniq remove duplicate lines from a sorted file
comm compare two sorted files line by linewc print the number of newlines, words, and bytes in
filescut remove sections (columns) from each line of filesjoin join lines of two files on a common field
paste merge lines of filestr translate or delete characters
Pavel Rychly Unix Text Tools
![Page 4: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/4.jpg)
Short Description of Basic Text Tools
egrep prints lines matching a pattern(g)awk pattern scanning and processing language
sed stream editor, use for substring replacementuse perl -p for extended regular expressions
Pavel Rychly Unix Text Tools
![Page 5: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/5.jpg)
Text Tools Documentation
info run info and select from a menu or run directly:info coreutilsinfo head, info sort, . . .info gawk
manman 7 regexman grep, man awk, man tail, . . .
–help most tools display a short help message on the--help option
sort --help, uniq --help, . . .
Pavel Rychly Unix Text Tools
![Page 6: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/6.jpg)
Unix Text Tools PackagesWhere to find it
set of system toolsdifferent sets and different features/options on each UnixtypeGNU textutilsGNU coreutils – textutils + shellutils + fileutilsother GNU packages: grep, sed, gawk
installed on all Linux machineson Windows: install mingw32/cygwin, then coreutils, grep,. . .
Pavel Rychly Unix Text Tools
![Page 7: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/7.jpg)
Unix Text Tools PackagesWhere to find it
set of system toolsdifferent sets and different features/options on each UnixtypeGNU textutilsGNU coreutils – textutils + shellutils + fileutilsother GNU packages: grep, sed, gawkinstalled on all Linux machineson Windows: install mingw32/cygwin, then coreutils, grep,. . .
Pavel Rychly Unix Text Tools
![Page 8: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/8.jpg)
Text Tools Usage
command line tools – enter command in a terminal(console) windowcommand name followed by options and argumentsoptions start with -quote spaces and metacharacters: ’, ”, $redirect input and output from/to files using <,>
use | less to only display a result without saving
Pavel Rychly Unix Text Tools
![Page 9: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/9.jpg)
Text Tools Example 1
task Convert plain text file to a vertical text.input plain.txt
output plain.vertsolutions
tr -s ’ ’ ’\n’ <plain.txt >plain.vert
tr -sc a-zA-Z0-9 ’\n’ <plain.txt >plain.vert
perl -ne ’print "$&\n" while /(\w+|[ˆ\w\s]+)/g’ \plain.txt >plain.vert
Pavel Rychly Unix Text Tools
![Page 10: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/10.jpg)
Text Tools Example 1
task Convert plain text file to a vertical text.input plain.txt
output plain.vertsolutions
tr -s ’ ’ ’\n’ <plain.txt >plain.vert
tr -sc a-zA-Z0-9 ’\n’ <plain.txt >plain.vert
perl -ne ’print "$&\n" while /(\w+|[ˆ\w\s]+)/g’ \plain.txt >plain.vert
Pavel Rychly Unix Text Tools
![Page 11: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/11.jpg)
Text Tools Example 1
task Convert plain text file to a vertical text.input plain.txt
output plain.vertsolutions
tr -s ’ ’ ’\n’ <plain.txt >plain.vert
tr -sc a-zA-Z0-9 ’\n’ <plain.txt >plain.vert
perl -ne ’print "$&\n" while /(\w+|[ˆ\w\s]+)/g’ \plain.txt >plain.vert
Pavel Rychly Unix Text Tools
![Page 12: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/12.jpg)
Text Tools Example 1
task Convert plain text file to a vertical text.input plain.txt
output plain.vertsolutions
tr -s ’ ’ ’\n’ <plain.txt >plain.vert
tr -sc a-zA-Z0-9 ’\n’ <plain.txt >plain.vert
perl -ne ’print "$&\n" while /(\w+|[ˆ\w\s]+)/g’ \plain.txt >plain.vert
Pavel Rychly Unix Text Tools
![Page 13: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/13.jpg)
Text Tools Example 2
task Create a word listinput vertical text
output list of all unique words with frequenciessolutions
sort plain.vert | uniq -c >dictsort plain.vert | uniq -c | sort -rn | head -10
Pavel Rychly Unix Text Tools
![Page 14: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/14.jpg)
Text Tools Example 2
task Create a word listinput vertical text
output list of all unique words with frequenciessolutions
sort plain.vert | uniq -c >dictsort plain.vert | uniq -c | sort -rn | head -10
Pavel Rychly Unix Text Tools
![Page 15: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/15.jpg)
Text Tools Example 3
task Corpus/list sizeinput vertical text/word list
output number of tokens/different wordssolutions
wc -l plain.vertwc -l dictgrep -c -i ’ˆ[a-z0-9]*$’ plain.vert
Pavel Rychly Unix Text Tools
![Page 16: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/16.jpg)
Text Tools Example 3
task Corpus/list sizeinput vertical text/word list
output number of tokens/different wordssolutions
wc -l plain.vertwc -l dictgrep -c -i ’ˆ[a-z0-9]*$’ plain.vert
Pavel Rychly Unix Text Tools
![Page 17: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/17.jpg)
Text Tools Example 4
task Create a list of bigramsinput vertical text
output list of bigramssolution
tail +2 plain.vert |paste - plain.vert \|sort |uniq -c >bigram
Pavel Rychly Unix Text Tools
![Page 18: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/18.jpg)
Text Tools Example 4
task Create a list of bigramsinput vertical text
output list of bigramssolution
tail +2 plain.vert |paste - plain.vert \|sort |uniq -c >bigram
Pavel Rychly Unix Text Tools
![Page 19: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/19.jpg)
Text Tools Example 5
task Filteringinput word list
output selected values from word listsolutions
grep ’ˆ[0-9]*$’ dictawk ’$1 > 100’ dict
Pavel Rychly Unix Text Tools
![Page 20: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/20.jpg)
Text Tools Example 5
task Filteringinput word list
output selected values from word listsolutions
grep ’ˆ[0-9]*$’ dictawk ’$1 > 100’ dict
Pavel Rychly Unix Text Tools
![Page 21: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/21.jpg)
Text Tools Debuging
data driven programmingcut the pipline and display partial resultstry single command with a test input
Pavel Rychly Unix Text Tools
![Page 22: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/22.jpg)
Text Tools Exercise
task Find all words from a word list differing withs/z alternation only:apologize/apologise
solutions
tr s z < dict | sort |uniq -d >szaltern
Pavel Rychly Unix Text Tools
![Page 23: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/23.jpg)
Text Tools Exercise
task Find all words from a word list differing withs/z alternation only:apologize/apologise
solutions
tr s z < dict | sort |uniq -d >szaltern
Pavel Rychly Unix Text Tools
![Page 24: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/24.jpg)
Text Tools Exercises
Find all words from a word list differing withs/z alternation only,and each alternation has higher frequency than 50
and display their frequencesFind all words which occurs in the word listonly with capital letter (names).
Pavel Rychly Unix Text Tools
![Page 25: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/25.jpg)
Text Tools Exercises
Find all words from a word list differing withs/z alternation only,and each alternation has higher frequency than 50and display their frequences
Find all words which occurs in the word listonly with capital letter (names).
Pavel Rychly Unix Text Tools
![Page 26: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/26.jpg)
Text Tools Exercises
Find all words from a word list differing withs/z alternation only,and each alternation has higher frequency than 50and display their frequencesFind all words which occurs in the word listonly with capital letter (names).
Pavel Rychly Unix Text Tools
![Page 27: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/27.jpg)
Summary
Use simple Unix text tools for processing text files andcomputation of global statistics.Use a powerful graphical user interface for local corpusexploration:
Sketch Engine: www.sketchengine.co.ukManatee/Bonito: www.textforge.cz
Pavel Rychly Unix Text Tools
![Page 28: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/28.jpg)
Summary
Use simple Unix text tools for processing text files andcomputation of global statistics.
Use a powerful graphical user interface for local corpusexploration:
Sketch Engine: www.sketchengine.co.ukManatee/Bonito: www.textforge.cz
Pavel Rychly Unix Text Tools
![Page 29: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/29.jpg)
Summary
Use simple Unix text tools for processing text files andcomputation of global statistics.Use a powerful graphical user interface for local corpusexploration:
Sketch Engine: www.sketchengine.co.ukManatee/Bonito: www.textforge.cz
Pavel Rychly Unix Text Tools
![Page 30: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/30.jpg)
Summary
Use simple Unix text tools for processing text files andcomputation of global statistics.Use a powerful graphical user interface for local corpusexploration:
Sketch Engine: www.sketchengine.co.uk
Manatee/Bonito: www.textforge.cz
Pavel Rychly Unix Text Tools
![Page 31: Unix Text Tools - Masaryk Universitypary/texttools.pdf · 2008-07-24 · Unix Text Tools Tradition Unix has tools for text processing from the very beginning (1970s) Small, simple](https://reader034.vdocuments.mx/reader034/viewer/2022042114/5e9164c6adc3c334aa75a293/html5/thumbnails/31.jpg)
Summary
Use simple Unix text tools for processing text files andcomputation of global statistics.Use a powerful graphical user interface for local corpusexploration:
Sketch Engine: www.sketchengine.co.ukManatee/Bonito: www.textforge.cz
Pavel Rychly Unix Text Tools