rolando v. raqueñowednesday, june 10, 2015 statistics function implementation traditional...

Rolando V. Raqueño Tuesday, April 18, 2023

Statistics Function Implementation

Traditional Programming Approach


minimum.pro

function minimum, xn = n_elements(x)answer = x(0)for i=1L, n-1 do begin

if( answer gt x(i) ) then $answer = x(i)

endforreturn, answer

end


sum.pro

function sum, x

n = n_elements( x )

answer = 0

for i = 0, n-1 do begin

answer = answer + x(i)

endfor

return, answer

end


Overflow Problems

• What if the array is an integer array?

• Sums will have a maximum limit based on the number of bits specified by the integer type.


Corrected sum.pro

function sum, x

n = n_elements( x )

answer = 0.0D

for i = 0L, n-1 do begin

answer = answer + x(i)

endfor

return, answer

end


mean.pro

function mean, x

n = n_elements(x)

answer = sum(x) / n

return, answer

end


Integer Problems

IDL> b=9/5

IDL> print,b

1


Integer Problems

IDL> b=9/double(5)

IDL> print,b

1.8000000


Corrected mean.pro

function mean, x

n = n_elements(x)

answer = sum(x) / double(n)

return, answer

end


sum_squares.pro

function sum_squares, x

squares = x^2.0

answer= sum( squares )

return, answer

end


variance.pro

function variance, x

n = double(n_elements(x))

answer=(sum_squares(x)-(sum(x)^2.0/n))/(n-1)

return, answer

end


sd.pro

function sd, x

answer = sqrt(variance(x))

return, answer

end


Shell Programming


UNIX Tip of the Day

• Directory maneuvering commands pushd, popd, and, dirs

% cd /usr/tmp% pwd/usr/tmp% pushd ~rvrpci% pwd/cis/staff/rvrpci


UNIX Tip of the Day

% dirs

/cis/staff/rvrpci /usr/tmp

% pushd

% pwd

/usr/tmp

% dirs

/usr/tmp /cis/staff/rvrpci

% pushd /usr/local/bin


UNIX Tip of the Day

% dirs

/usr/local/bin /usr/tmp

/cis/staff/rvrpci

% pwd

/usr/local/bin

% pushd

% dirs

/usr/tmp /usr/local/bin

/cis/staff/rvrpci


UNIX Tip of the Day% dirs


/cis/staff/rvrpci

% pwd

/usr/tmp

% pushd +2

% pwd

/cis/staff/rvrpci


UNIX Tip of the Day% dirs

/cis/staff/rvrpci /usr/tmp

/usr/local/bin

% popd

% dirs



IMPORTANT UNIX Concepts• Environment and Shell Variables

– These allow you to customize your UNIX environment

– They are different in terms of their SCOPE• SCOPE determines the visibility of a variable


Other IMPORTANT UNIX Concepts• Environment Variable

– Examples are TERM and DISPLAY– Set a particular variable to a value by using the setenv command

– You can print the value of a particular variable or all the environment variable using the printenv command


% Environment Variables %• Examples

– To set environment variables

% setenv TERM vt100% setenv DOG Goofy– print out the terminal type

% printenv TERM vt100– print out all environment variables

% printenv


Shell Variables

• Shell variables are similar to Environment variables except they have a limited scope, i.e., they exist only in the shell which they are defined.

• Environment variables on the other hand, exist in all its children shells

• To illustrate this concept, let us look at the following example


Environment vs. Shell Variables% set prompt = "Parent Shell > "Parent Shell > setenv DOG GoofyParent Shell > set mouse=MickeyParent Shell > printenv DOGGoofyParent Shell > echo $mouseMickeyParent Shell > xterm &

(YOU SHOULD NOW HAVE A NEW xterm WINDOW)THIS IS KNOWN AS

“SPAWNING A NEW (OR CHILD) PROCESS”


Environment vs. Shell Variables(IN THE NEW xterm WINDOW, DO THE FOLLOWING)

% set prompt = "Child Shell > "Child Shell > printenv DOGGoofyChild Shell > echo $mousemouse: Undefined variable.


Environment vs. Shell VariablesChild Shell > setenv DOG PlutoChild Shell > set mouse=MinnieChild Shell > printenv DOGPlutoChild Shell > echo $mouseMinnieChild Shell > exit

(THE xterm WINDOW SHOULD NOW GO AWAY - THIS PROCESS HAS NOW BEEN KILLED)


Environment vs. Shell VariablesParent Shell >Parent Shell > printenv DOGGoofyParent Shell > echo $mouseMickeyParent Shell >


Environment & Shell Variables

• Why is this important?– UNIX uses Environment and Shell Variables

control a number of processes– Customizes your working environment– Variables used for UNIX Scripts

• They are typically defined and initialized in your .login and .cshrc files


Useful Shell Variables

filec#Allows file completionpath #List of command

directoriescdpath #List of candidate

directories to cd intohistory #Number of commands to

remember


What is shell programming?

• Shell programming – automate a set of UNIX commands.– Just like any programming language– “wrappers”

• black box a customized collection of UNIX commands.

– Example of shell programs.login.cshrc


.login file

set path=($HOME/bin /usr/local/bin \/usr/ucb /usr/sbin /bin /usr/bin \/usr/bin/X11 .)stty dec newtset -I -Qset mail=/usr/spool/mail/$USERset editmode = emacsumask 077biff ndate


.cshrc file

if ($?prompt) then

set notify

set history = 100

set savehist = 100

alias pd pushd

alias pop popd

alias vt100 "set term = vt100"

endif


When these files are executed?

.cshrc – is automatically executed when you start a new

shell

.login – only gets executed once when you first login

Can be re-executed by giving the source command% source .cshrc


Other useful .login and .cshrc entries

set filec

set cdpath=(~ ~rvrpci/pub ~/mythesis)

Other common entries

set path=( $path /usr/local/bin)

set path=(/usr/local/bin $path)


User defined shell program

• Determine name of command

• Determine input, output, and option arguments

• Determine UNIX commands to execute

• Establish error trapping

• Make shell program executable


A simple shell program

• dd command to swap bytes

% dd if=input.dat of=output.dat bs=2 conv=swab

• Very difficult to remember• Very little utility to non-UNIX geeks

(normal people)


We would rather see...

% swap_bytes input.dat output.dat


Special Shell Variables Set


$0 $1 $2$argv[1] $argv[2]command


Another Special Shell Variables


$#argv

Indicates how many arguments are present

In this case, 2


shell program swap_bytes

#!/bin/csh -f

dd if=$1 of=$2 bs=2 conv=swab


Making swap_bytes shell script executable

% ls -l swap_bytes

-rw------- ... swap_bytes

% chmod u+x swap_bytes

% ls -l swap_bytes

-rwx------ ... swap_bytes


To run swap_bytes

• swap_bytes becomes just another unix command!



Limitation of swap_bytes

• No error trapping

• Should give usage when typing command

% swap_bytes

usage: swap_bytes input_file output_file


Improvement to swap_bytes

#!/bin/csh -f if ( $#argv != 2 ) then

echo "usage: $0 input_file output_file"

exit 1

endif

dd if=$1 of=$2 bs=2 conv=swab


Commad exit status

• By convention

exit 0

Indicates successful command completion

exit 1 (or non-zero)

Indicates some error condition


Informational message from swap_bytes

• UNIX style informational message

% swap_bytes

usage: swap_bytes input_file output_file


Interactive swap_bytes

• If you want a “friendlier” shell program– Have it query the user for the inputs

• Another special shell variable can be used

$<


Interactive swap_bytes

#!/bin/csh -f

if ( $#argv != 2 ) then

echo -n "Please enter the input file> "

set input=$<

echo -n "Please enter the output file> "

set output=$<

endif

dd if=$input of=$output bs=2 conv=swab


Interactive swap_bytes example

• User simply types the command

% swap_bytes

Please enter the input file> input.dat

Please enter the output file> output.dat


UNIX Quotes


A note about quotes in UNIX

% set a=ls

% echo a

% echo $a

% set b=“$a”

% echo $b

% set b=‘$a’

% echo $b

% set b=`$a`

% echo $b


A note about shell variables

• Shell variables can also double up as arrays

• Using the previous example,

% echo $b

% echo $b[1]

% echo $#b

% echo $b[$#b]


A more complex shell program

• In pbmplus utilities, rawtopgm conversion existspgmtoraw conversion does not

• A version of pgmtoraw in a programming language like C– Time consuming– Program will likely be used infrequently

• Solution: shell program


pgmtoraw shell script design

• Define input and output files

• Figure out dimensions of input image

• Determine number of bytes for input image

• Determine number of bytes for header

• Need to strip out the header bytes

• Write out headerless image data


Define input and output files pgmtoraw

#!/bin/csh -fset command_name=$0set number_args=$#argvif( $number_args != 1 ) then echo “usage:$command_name input_file_name” exit 1endif...


Dimensions of input image ( pnmfile)

% more test_data.pgmP23 32551 2 34 5 67 8 9

% pnmfile test_data.pgmtest_data.pgm: PGM plain, 3 by 3 maxval 255


pgmtoraw (continued)

set input_file=$1

set pnm_info = `pnmfile $input_file`

set image_type = $pnm_info[2]

set data_type = $pnm_info[3]

set width = $pnm_info[4]

set height = $pnm_info[6]

set maxval = $pnm_info[8]

set pixel_bytes = 1

@ image_bytes = $width * $height


pgmtoraw (continued)

set file_info=`wc -c $input_file`

set bytes_in_file = $file_info[1]

@ header = $bytes_in_file - $image_bytes

dd if=$input_file bs=$pixel_bytes skip=$header


Resulting pgmtoraw utility

• Uses existing routines to obtain informationpnmfile

wc

dd

• Functional tool written in 20 command lines


Current Limitations of Resulting pgmtoraw utility

• No check between “ASCII” vs. “RAW” pgmif( data_type == ‘plain,’) ...

• No provisions for multibyte per pixel case– Use pnmfile results to check for above cases– endian case needs to be addressed for multibyte

case

• Above conditions can be addressed by suite of UNIX commands


Shell Scripts Wrappers and IDL

• Another utility formerly missing in pbmplusjpegtopnm or pnmtojpeg

• IDL has jpeg read/write capability– Create a “wrapper” that makes an idl

program pbmplus-like


pnmtojpeg.pro

pro pnmtojpeg, input_file, output_file

read_ppm, input_file, image

write_jpeg, output_file, image

end


Usage of pnmtojpeg.pro in IDL

IDL> pnmtojpeg,‘image.pnm’,’image.jpg’


Usage of pnmtojpeg.pro in IDL

IDL> pnmtojpeg,‘image.pnm’,’image.jpg’

• For IDL to automatically find pnmtojpeg.pro

– It must be in the current working directory

– Directory containing pnmtojpeg.pro must be defined in the ENVIRONMENT VARIABLE•IDL_PATH


IDL_PATH environment variable

setenv IDL_DIR /cis/common/rsi/idl_5

setenv IDL_PATH \+$IDL_DIR/lib:\+$IDL_DIR/examples:

\+/dirs/common/rsi/idl_5:\+/dirs/common/lib/idl:\+~/lib/idl


pnmtojpeg.csh

#!/bin/csh -f

echo pnmtojpeg “,” “’”$1”’” “,” “’”$2”’” | idl


Usage of pnmtojpeg.csh

% pnmtojpeg.csh image.pnm image.jpg


Limitation of pnmtojpeg.csh

• Does not conform to pbmplus piping, i.e.,% tifftopnm file.tif | pnmscale 2.0 > new_file.pnm

• No error trapping


Usage cases of pnmtojpeg(no command line arguments)

% tifftopnm file.tif | pnmscale 2.0 | pnmtojpeg > new_file.jpg


Usage cases of pnmtojpeg(1 command line argument)

% pnmtojpeg image.pnm > image.jpg


Usage cases of pnmtojpeg(2 command line arguments)

% pnmtojpeg image.pnm image.jpg


Yet another wrapper pnmtojpeg

#!/bin/csh -f

# If user interrupts process, jump to stop

onintr stop

# $0 is the command name

# $#argv is the number of arguments

# $$ is the process id


Code for no argument case

if($#argv == 0) then

set input_file = /usr/tmp/$0_input_$$

set output_file = /usr/tmp/$0_output_$$

cat > $input_file

pnmtojpeg.csh $input_file $output_file

cat $output_file

.

.

.


Code for 1 argument case

else if($#argv ==1) then

set input_file = $1

set output_file = /usr/tmp/$0_output_$$

pnmtojpeg.csh $input_file $output_file

cat $output_file

.

.

.


Code for 2 argument caseelse set input_file = $1 set output_file = $2 pnmtojpeg.csh $input_file $output_fileendif

#clean up when finishedstop: rm -f /usr/tmp/$0_input_$$ rm -f /usr/tmp/$0_output_$$


pnmtojpeg summary

• Produced a “new” pbmplus utility• Used UNIX shell scripting

– Argument handling– Scratch space /usr/tmp– Process id handling– Clean up

• Integrated IDL program and commands• 21 lines of UNIX commands


Summary

• The “dot” files

• Basics of Shell Scripting

• Special Shell Variables

• Seamless integration of UNIX to other utilities (IDL)


Other Shell Constructs to keep in mind

• foreach• while• case

rolando v. raqueñowednesday, june 10, 2015 statistics function implementation traditional...

Documents

answer end slide

elementsx answer

variable slide

elements x answer

dirs usrtmp usrlocalbin

answer gt

pwd cisstaffrvrpci slide

pushd usrlocalbin slide