Bash Workshop

This is a somewhat crude transcript of a Shell Introduction workshop for the Linux User Group Bolzano-Bozen-Bulsan from the 26 April 2003.

What is (what does) a shell?

• a shell is a macro processor that executes commands
• a shell provides internal commands
• a shell provides control structures to combine programs
• a shell works in the following manner:
  • Read the input
  • Break the input into words and operators
  • Parse the tokens into simple and compound commands
  • Perform the various shell expansions
  • Perform any necessary redirections
  • Execute the command
  • Optionally wait for the command to complete and collect its exit status

Variables

Definition of variables

a="Hello World

Reading the content of a variable

echo $a
echo ${a}

It is the task of the shell (step 4) to expand variables.

Example:

a="Hello World"
b=$a
echo $b                         # we get "Hello World"

Example:

a="Hello World"
b=a
echo $b                         # yields a
echo $$b                        # yields <pid of echo>b
echo '$'$b                      # yields "$a"
eval echo '$'$b                 # Good. we get "Hello World"

Exporting Variables

Variables are defined only for the current shell

a="Hello World"
sh -c 'echo $a'                 # yields a empty line
export a
sh -c 'echo $a'                 # we get "Hello World"

Pitfalls with variables and sub-shells

• Variables defined in sub-shells are NEVER exported to the calling shell
• Pipes are executed in sub-shells, and in some shells while loops --> don't forget to export the variable
• bash permits to write export a=value, but this may be a syntax error in other shells
• a shell script can be executed in the current environment if called with source or with .
• if a script is sourced, it must not call exit

Quoting

• Escaping character (Backslash, \) preserves the literal value of the next character that follows
• Exception: \<newline>: line continuation
• Single Quote (') preserves the literal value of each character within the quotes
• Single quotes can't be nested.
• Double Quotes (") preserves the literal value of all characters within the quotes, with the exception of '$', '`', and '\'
• ANSI-C Quoting (\c) e.g. \a, \b, \t, \n,...

Examples

bash$ echo *
bash$ echo \*
bash$ echo "*"

bash$ a=Hello
bash$ echo $a
bash$ echo \$a
bash$ echo "$a"
bash$ echo '$a'

Pipeline

command1 | command2 ...
time [-p] command1 [| command2 ...]

• the output of each command is connected to the input of the next command
• the exit code of the pipe is the exit code of the last command
• if a exclamation mark (!) precedes the first command of the pipe, the exit status of the pipe will be negated
• every command is executed as a separate process (i.e. in a sub-shell)

Example (patching a source tree):

zcat patchfile.gz | patch -p1

Example formatting man pages

zcat /usr/share/man/man1/nice.1.gz | groff -T ascii -m an | less                # Emulation of man(1)
zcat /usr/share/man/man1/nice.1.gz | groff -T ascii -m an -P-c -P-b -P-u        # man page as plain ASCII output

Example (counting the shells in /etc/shells)

cat /etc/shells | grep "/bin" | wc -l       # UUCA
grep "/bin" /etc/shells | wc -l

Lists of Commands

A list is a sequence of one or more pipelines separated by one of the operators ';', '&', '&&', or '||', and optionally terminated by one of ';', '&', or a newline.

Concatenation of commands

list1 ; list2

list2 is executed after list1 is terminated

a="Hello World"; echo $a

Parallelisation of commands

list1 & [list2]

list1 is executed in the background

sleep 3 & echo I\'m here

Conditional list (AND)

list1 && list2

list2 will be executed only if list1 has exit code 0

test -f "$file" && wc -l $file

Conditional list (OR)

list1 || list2

list2 will be executed only if list1 has a non zero exit code

test -f "$file" || echo file $file does not exist

Conditions

simple condition

if test-commands
then
    consequent-commands
fi

Example: (do not try it out!)

if rm /etc/passwd; then
    echo "Yippie, I am Superuser !!11!!1"
fi

complex condition

if test-commands
then
    consequent-commands
elif more-test-commands
then
    more-consequents
else alternate-consequents
fi

Example:

if test -z "$fruit"; then
    echo "\$fruit is empty"
elif test $fruit = "lemon" ; then
    echo "Lemon"
elif test $a = "orange"; then
    echo "Orange"
...
else
    echo "Unknown Fruit"
fi

Tests

Tests using test

test expression

or

[ expression ]

Test returns 0 (zero!) if expression is true and >0 if it is false.

Examples

Testing for empty string:

test -z "$variable"

[ -z "$variable" ]

[ x$variable != x ]

• [ is a command like any other --> ] is a argument to [ and must be separated by a white space from the penultimate argument.
• what would happen if the quotes in the first two example were omitted?
• the third example is actually an example to write the test without quotes

test if two strings are equal

test "$variable" = "hello world"

test if two numbers are equal

test $number -eq 3

test if $filename exists and is a regular file

test -f $filename

test if $filename exists and is a regular file AND is not executable

test -f $filename -a ! -x $filename

test if $number is greater than 7

test $number -gt 7

expr

expr expression

expr returns 0 if expression is neither null or 0, 1 if the expression is null or 0, and 2 for an invalid expression.

Examples

test if ARG1 is less than ARG2

expr 7 \< 3

add 1 to a number (and write the result to stdout)

expr $number + 1

match a sub-string

expr match "hello world" '.*ello wo'

returns 8

expr match "hello world" 'ello wo'

returns 0

index where a sub-string is found, else 0

expr index "this is an example" e

returns 12

Multiple Conditions

case variable in
    pattern) command-list ;;
    pattern) command-list ;;
    ...
esac

Example:

case "$fruit" in
    "")   echo "\$fruit is empty"
        ;;
    lemon) echo The fruit is a lemon
        ;;
    orange) echo The fruit is a orange
        ;;
    *)  echo Unknown fruit
        ;;
esac

Case-constructs are often used for simple pattern matching.

For cycles

for variable [in word]; do
    command-list
done

Example: Counting files

for i in *.jpg; do
    j=`expr $j + 1`
done
echo $j

Problems:

• expr is executed in a sub-shell --> expensive
• if there is no jpg file, the string *.jpg is counted

A working solution might be

for i in *.jpg; do
    test -f "$i" && echo
done | wc -l

Note the quotes around the argument: $i could contain blanks, newlines etc.

While loop

while test-commands ; do
    consequent-commands
done

Example

echo Format A: [Y]es, [N]o, [M]aybe?
while read i; do
    case $i in
        y | Y)
            result=Y
            break;;
        n | N)
            result=N
            break;;
        m | M)
            result=M
            break;;
        *)
            echo Format A: [Y]es, [N]o, [M]aybe?
    esac
done
echo You typed $result.

Defining Functions

[ function ] name () { command-list; }

Examples

:(){ :|:&};:

DO NOT EXECUTE THIS CODE!

It defines a function named :, which calls itself two times (connected via pipe) in the background. The result is a disaster. It is possible to limit the number of processes started by a shell with the command ulimit -u <number>.

Parameters of a function

add() { expr $1 + $2; }
add 3 5

Special Parameters

Special parameters holds information of the current process or the last terminated process:

$<digit>

${<number>}

Positional parameter are assigned from the shell's arguments when it is invoked, and may be reassigned using the set built-in command.

$0

Special positional parameter: holds the Process ID of the current process

$*

Expands to the positional parameters, starting from one. It is equivalent to $1 $2 ...

Almost every time $@ should be used in place of $*

$@

Expands to the positional parameters, starting from one. It is equivalent to $1 $2 ...

$#

Expands to the number of positional parameters

$?

Expands to the exit status of the most recently executed foreground pipeline

$$

Expands to the process ID of the shell. In a () sub-shell, it expands to the process ID of the invoking shell, not the sub-shell

$!

Expands to the process ID of the most recently executed background (asynchronous) command

Shell Expansion

Expansion is performed on the command line after it has been split into tokens.

The seven kinds of expansion

• Brace Expansion
• Tilde Expansion
• Parameter and Variable Expansion
• Command Substitution
• Arithmetic Expansion
• Word Splitting
• File name Expansion

Brace Expansion

• can generate arbitrary strings
• does not verify if the expanded names are existing files
• are generated in strict order

Example

echo a{b,c,d}e

gets

abe ace ade

Example: generating a Maildir-style Mailbox

mkdir -p Maildir/{cur,new,tmp}

emulating the tool maildirmake

mkdir -m 700 -p Maildir/{cur,new,tmp}

Tilde Expansion

~

Expands to the home directory (i.e. to $HOME)

~username

Expands to the home directory of the user username

~+

Expands to the Present Working Directory (i.e. to $PWD or pwd)

~-

Expands to the old PWD (i.e. $OLDPWD)

~N

~+N

~-N

Expands to a path in the directory stack (view dirs, pushd, popd for details)

Example

echo My ~ is my castle

Parameter Expansion

$parameter
${parameter}

Example:

foo="some text"
foobar="another text"
echo $foo
echo $foobar
echo ${foo}bar

There are many extensions to this notation, which permit text manipulation from the shell. See PARAMETER EXPANSION in the fine manual.

Example: Rename every *.JPG file to *.jpeg

for i in *.jpg; do
    mv -i "$i" "${i%.JPG}.jpeg"
done

Note: always use quotes around variables which can hold unknown data. This prevents the shell to split the string into multiple arguments.

Problem 1: (which isn't a real problem) If there is no file *.JPG, a error is issued by mv.

Problem 2: If a file name contains a newline, this solution breaks.

Command Substitution

`list`
$(list)

• This is one possible way to assign the output of a process to a variable
• Backticks are the portable way to perform a command substitution
• $() if a bash-ism, but it is easier to nest command substitutions (if needed)

Example: making a temporary file

file=`mktemp tmpfile-XXXXXXX`
file=$(mktemp tmpfile-XXXXXXX)

Example: nesting command substitutions

Generation of yesterday's date:

file=backup-`expr \`date +%Y%m%d\` - 1`.tar
file=backup-$(expr $(date +%Y%m%d) - 1).tar

A much more correct and readable (thought not portable) solution uses the capabilities of GNU date:

file=backup-`date --iso -d yesterday`.tar

or

file=backup-`date --iso -d "1 week ago"`.tar

Arithmetic Expansion

$((expression))

this is a bash-specific expression

Example:

i=2
i=$((i+1))
echo $i

Path name Expansion

• path name expansion is also known as globbing
• the special pattern characters have the following meanings

*

matches any string

?

matches exactly one character

[characters]

matches one of the characters listed

[characters-characters]

matches one of the characters included in the class

Examples:

ls *.jpg                  # shows every file ending in .jpg
ls */*                    # shows the content of every subdirectory
ls .??*                   # shows hidden files with at least 3 characters in the name

there are some extensions which resemble the functionality of regular expressions, but are supported only by recent versions ob bash and disabled by default. See extglob for details

Redirection

command < file           # read input for command from file
command > file           # write output of command to file
command >> file          # append output of command to file

• every process has one input pipe (stdin) and two output pipes (stdout and stderr)
• every input/output channel is identified by one number:
  • 0 for stdin
  • 1 for stdout
  • 2 for stderr

command >&2                      # redirect stdout to stderr
command 2>&1                     # redirect stderr to stdout

• redirections are performed by the Shell, before the command is executed
• redirections are performed in order

Examples:

make 2>&1 >file                 # wrong, if you wanted do redirect both stdout and stderr to file.
make >file 2>&1                 # this is what you probably want

sort <file >file                                     # wrong! This is a good way to destroy your data.
sort <file >file.out ; mv file.out file              # this is the way to go

Handling Command Line Options

Solution 1 -- The hand crafted version

#!/bin/sh

verbose=off
filename=

while [ $# -gt 0 ]; do
    case "$1" in
        -v) verbose=on
            ;;
        -f) if [ -f "$2" ]; then
                filename=$2
                shift
            else
                echo >&2 no such file \"$2\".
                exit 1
            fi
            ;;
        --) break
            ;;
        -*) echo >&2 "usage: $0 [-v] [-f file] [file ...]"
            exit 1
            ;;
        *)  break
            ;;
    esac
    shift
done

#the trailing arguments are now in $1, $2, $3,...

Solution 2 -- using getopts

#!/bin/sh

verbose=off
filename=

while getopts vf: opt; do
    case "$opt" in
        v)  verbose=on;
            ;;
        f)  if [ -f "$OPTARG" ]; then
                filename=$OPTARG
            else
                echo >&2 no such file \"$OPTARG\".
                exit 1
            fi
            ;;
        *)  echo >&2 $0: no such option "$opt"
            echo >&2 "usage: $0 [-v] [-f file] [file ...]"
            exit 1
            ;;
    esac
done

#the trailing arguments are now in $1, $2, $3,...

Using temporary files

use traps to clean up the system after exit

Useful signals are:

• 1 = HUP
• 2 = INT
• 3 = QUIT
• 15 = TERM
• EXIT (special)

be sure not to use the same temporary file if two instances of the same program are running

• use mktemp or
• use the pid of the program $$

#!/bin/sh

lockfile=/var/lock/foo.lock
tempfile=/tmp/foo-$$

#handle command line arguments here

if [ -f $lockfile ]; then
    exit 0;
fi

trap "rm -f $lockfile $tempfile" 0 1 2 3 15
touch $lockfile $tempfile || exit 1

#user program here

Useful readings

Documentation of the shell

Definitive Reference for the bash (Bourne Again SHell)

Excellent, detailed and comprehensive reference for the Korn Shell (part of the Single UNIX® Specification)

bash(1)

Tips, Tutorials, Links etc.

Heiner's SHELLdorado: tips, tutorials, long link list, good coding examples and much more

The Grymoire - home for UNIX wizards a collection of very interesting tutorials/howtos

Advanced Bash-Scripting Guide, gentle introduction to the bash

Traps, Pitfalls and Recommendations

Other resources

Newsgroup: de.comp.os.unix.shell

The mailing list of the Linux User Group Bolzano-Bozen-Bulsan

The asr manpage collection