Unix script - 14 hints and tips

Unix is full of text manipulating utilities, some of the more powerfulof which we will now discuss in this section of this tutorial. The significance of this, is that virtually everything under Unix istext. Virtually anything you can think of is controlled by either a textfile, or by a command-line-interface (CLI). The only thing you can'tautomate using a shell script is a GUI-only utility or feature. Andunder Unix, there aren't too many of them!

You may have heard it said, that, with *nix, "everything is a file" - it's true.

We have a few subsections here ... what follows is general advice, hints and tips.

CGI Scripting

Exit Codes and flow control

Simple Expect replacement

Using trap to know when you've been interrupted - such as a CTRL-C, etc.

Workaround for the 'echo -n' vs 'echo \c' dichotomy

Documented Example of a real-life script I wrote - it helps users configure a SpeedTouch modem, and is available at http://speedtouchconf.sourceforge.net/.

We have already shown above a use of thesimple but effective cut command. We shall discuss a few examples here some of the more common external programs to be used.

grep is an extremely useful utility for the shell scriptprogrammer.
An example of grep would be:

#!/bin/sh
steves=`grep -i steve /etc/passwd | cut -d: -f1`
echo "All users with the word \"steve\" in their passwd"
echo "Entries are: $steves"

This script looks fine if there's only one match. However, if there are two lines in /etc/passwd with the word "steve" in them,then the interactive shell will display:

$> grep -i steve /etc/passwd 
steve:x:5062:509:Steve Parker:/home/steve:/bin/bash
fred:x:5068:512:Fred Stevens:/home/fred:/bin/bash
$> grep -i steve /etc/passwd |cut -d: -f1
steve
fred

But the script will display:

Entries are: steve fred

By putting the result into a variable we have changed the NEWLINEs into spaces; the sh manpage tells us that the first character in $IFS will beused for this purpose. IFS is <space><tab><cr> by default.Maybe though we wanted to keep the NEWLINEs: It could look better if we made the spaces into NEWLINEs.... This is a job for tr:

#!/bin/sh
steves=`grep -i steve /etc/passwd | cut -d: -f1`
echo "All users with the word \"steve\" in their passwd"
echo "Entries are: "
echo "$steves" | tr ' ' '\012'

Note that tr translated the spaces into octalcharacter 012 (NEWLINE).
Another common use of tr is its use of range... it can converttext to upper or lower case, for example:

#!/bin/sh
steves=`grep -i steve /etc/passwd | cut -d: -f1`
echo "All users with the word \"steve\" in their passwd"
echo "Entries are: "
echo "$steves" | tr ' ' '\012' | tr '[a-z]' '[A-Z]'

Here we have added a translation of [a-z] to [A-Z]. Note that there areexactly the same number of values in the range a-z as A-Z. This can thentranslate any character falling into the ASCII range a-z into A-Z ... inother words, converting lowercase letters into uppercase. tris actually cleverer than this: tr [:lower:] [:upper:]would do the job just as well, and possibly more readably. It's also not as portable; not every tr can do this.

Cheating

Those who can't ... cheat

There is nothing wrong with cheating! Some things the shell just isn'tvery good at. Two useful tools are sed and awk.Whilst these are two hugely powerful utilities, which can be used as mini-programming languages in their own right, they are often used in shellscripts for very simple, specific reasons.

Whilst this means that the system has to load a largeish executable (52k forsed and 110k for awk), which is a nasty thingto do, the reason a good workman doesn't blame his tools, is that a goodworkman uses the right tools in the first place.
So let me introduce these two, with very simple uses.

Cheating with awk

Consider wc, which counts the number of characters, lines,and words in a text file. Its output is:

$ wc hex2env.c
        102     189     2306    hex2env.c

If we want to get the number of lines into a variable, simply using:

NO_LINES=`wc -l file`

which would read in the whole line.
Because the output is space-padded, we can't reliably get the number 102into the string. Instead, we use the fact that awk workssimilarly to scanf in C - it strips unwanted whitespace. Itputs these into variables $1 $2 $3 etc. So we use thiscontstruct:

NO_LINES=`wc -l file | awk '{ print $1 }'`

The variable NO_LINES is now 102.

Cheating with sed

Another handy utility is sed - the stream editor. Perl is verygood at dealing with regular expressions, the shell isn't. So we canquickly use the s/from/to/g construct by invoking sed.For example:

sed s/eth0/eth1/g file1 >  file2

changes every instance of eth0 in file1 to eth1 in file2.
If we were only changing a single character, tr would bethe tool to use, being smaller and therefore faster to load.
Another thing that tr can't do, is remove characters froma file:

echo ${SOMETHING} | sed s/"bad word"//g

This removes the phrase "bad word" from the variable ${SOMETHING}.It may be tempting to say, "But grep can do that!" - grep onlydeals with whole lines. Consider the file:

This line is okay.
This line contains a bad word. Treat with care.
This line is fine, too.

Grep would remove the whole second line, leaving onlya two-line file; sed would change the file to read:

This line is okay.
This line contains a . Treat with care.
This line is fine, too.

Telnet hint

This is a useful technique that I picked up fromSun's Explorer utility. Although telnet is not used on servers any longer, it is still used by some network devices, such as terminal concentrators and the like.By creatinga script such as this, your own script, or from a command line, you can run:

$ ./telnet1.sh | telnet

I have had a few people ask me about this, and have tended to point them towardsthe expect suite of code, which is pretty complex and bulky; this code should be prettyportable amongst systems (so long as they've got egrep). If it doesn't work on yoursystem, try using GNU grep with the -q switch, or a proprietary grep and direct to /dev/null.Still a lot easier than installing expect, though.

telnet1.sh

#!/bin/sh
host=127.0.0.1
port=23
login=steve
passwd=hellothere
cmd="ls /tmp"

echo open ${host} ${port}
sleep 1
echo ${login}
sleep 1
echo ${passwd}
sleep 1
echo ${cmd}
sleep 1
echo exit

However, Sun add some clever error-checking code (note that the variablesyou could set and export from your current shell or shell script, to avoidstoring passwords in readable files):

$ ./telnet2.sh | telnet > file1

telnet2.sh

#!/bin/sh
# telnet2.sh | telnet > FILE1 
host=127.0.0.1
port=23
login=steve
passwd=hellothere
cmd="ls /tmp"
timeout=3
file=file1
prompt="$"

echo open ${host} ${port}
sleep 1
tout=${timeout}
while [ "${tout}" -ge 0 ]
do
    if tail -1 "${file}" 2>/dev/null | egrep -e "login:" > /dev/null
    then
        echo "${login}"
        sleep 1
        tout=-5
        continue
    else
        sleep 1
        tout=`expr ${tout} - 1`
    fi
done

if [ "${tout}" -ne "-5" ]; then
  exit 1
fi

tout=${timeout}
while [ "${tout}" -ge 0 ]
do
    if tail -1 "${file}" 2>/dev/null | egrep -e "Password:" > /dev/null
    then
        echo "${passwd}"
        sleep 1
        tout=-5
        continue
    else
      if tail -1 "${file}" 2>/dev/null | egrep -e "${prompt}" > /dev/null
      then
        tout=-5
      else
        sleep 1
        tout=`expr ${tout} - 1`
      fi
    fi
done

if [ "${tout}" -ne "-5" ]; then
  exit 1
fi

> ${file}

echo ${cmd}
sleep 1
echo exit

Note that with this version, the output is grabbed to file1, and that this file is actuallyused by the script to check on its progress. I have added "> ${file}" so that the outputreceived into the file is just the output of the command, not the logging-in process too.