Exploring the latest version of the great Bourne-again shell

A Good Bash

By Bernhard Bablok and Nils Magnus

yewkeo, 123RF

Despite recent competition from powerful alternatives such as Zsh [1], the Bourne-again shell (Bash) [2] is still the king of the hill on the Linux console. Users can use Bash interactively, and it also serves as a simple yet practical scripting language. Bash is part of the backbone of any working Linux system - all the more reason to investigate the benefits of upgrading to the new Bash 4 release, which appeared February 2009.

Why or Why Not?

On production systems, you might want to consider whether it is really necessary to upgrade to Bash 4. The major distributions will eventually spread the new version through their own updates, so the new Bash will reach you someday whether you download and install it or not. Programmers and many power users, on the other hand, like to embrace the goodies a new version offers as quickly as possible.

If you want to get a head start on new Bash features that are making their way to the next generation of Linux systems, you'll enjoy spending some quality time with Bash 4.

Table 1 provides a summary of some important new features; for a complete list, check out the NEWS file in the Bash documentation. Here, we highlight some of the most important changes.

At the Command Line

Command-line users will appreciate a few inconspicuous, but very useful, extensions debuting with the latest Bash. For example, the   string expands to a list of files and paths below the current working directory in a fashion similar to the external find command. However, users need to enable the feature by issuing the command shopt -s globstar.

The developers have now adopted a more user friendly approach to what used to be one of the greatest mysteries of the Bourne shell: the way standard error message output is redirected. Instead of the 2>&1 1>file mantra, users can now use &>> file to redirect both error and standard output into a file. The |& shortcut, which redirects the standard error for a command to a pipe, is another useful addition.

Associative Arrays

One popular myth is that Bash scripts create too many processes, which ultimately affects performance. But many of the more simple applications once used with Bash, including sed, grep, basename, or dirname are no longer necessary; Bash handles these tasks just as quickly as any other scripting language with on-board tools. Despite these improvements, Bash programmers cast many an envious glance at Perl and Python, both of which have more versatile data structures.

In Version 4, Bash finally adds associative arrays to its existing single-dimensional arrays. For many coders, this change in itself is reason enough to move to the new version because it provides a more elegant approach to many problems. For example, developers can use arbitrary strings as indices in associative arrays, rather than just as integers. Listing 1 gives an example.

01 #!/bin/bash
02
03 declare -A name
04
05 name["Linus"]="Torvalds"
06 name["Bill"]="Gates"
07 name["Steve"]="Jobs"
08 name["George W."]="Bush"
09
10 # output all values:
11 echo "Values: ${name[@]}"
12
13 # output all keys:
14 echo "Key: ${!name[@]}"
15
16 # Access individual values
17 for v in "${!name[@]}"; do
18   echo "$v ${name[$v]}"
19 done

Listing 2 is a script that sorts files in directories according to their properties - for example, the creation date. The script has two fundamental problems: First, it does not work with directories that contain blanks, and second, it might process individual directories multiple times.

01 #!/bin/bash
02
03 dirs=""
04
05 for f in "$@"; do
06    d=`getDir "$f"`
07    mkdir -p "$d"
08    mv -f "$f" "$d"
09    dirs="$dirs $d"
10 done
11
12 for d in $dirs; do
13    createIndex "$d"
14 done

Previous Bash versions have taken different approaches to solving this issue. In Bash 3.2, programmers can store quoted directory names in strings or an array. A script could prevent double processing by searching in the string - or by a slow linear search in the array. Neither of these approaches is particularly elegant.

Bash 4 handles this task far more simply (see Listing 3). The directory name serves as a key; the value itself is of no interest. As of line 12, the loop iterates over all the keys. The special construction with ampersands in double quotes does what it does anywhere in Bash: It tells the shell to process the values as individual tokens. The solution thus works with blanks in the target directories.

01 #!/bin/bash
02
03 declare -A dirs
04
05 for f in "$@"; do
06    d=`getDir "$f"`
07    mkdir -p "$d"
08    mv -f "$f" "$d"
09    dirs[$d]=1
10 done
11
12 for d in "${!dirs[@]}"; do
13    createIndex "$d"
14 done

File Handling

Listing 4 contains a typical programming pattern. The while loop in lines 5 through 8 parses the lines from an input file one after another, storing the results in an array. This construction occurs frequently, but unfortunately, it is imperfect. If the last line does not end with a newline character, the loop will not store the line. The built-in read command does not perform the assignment until the line is terminated.

01 #!/bin/bash
02
03 inputFile="$1"
04 i=0
05 while read line; do
06    lines[$i]="$line"
07    let i++
08 done < "$inputFile"
09
10 # Ongoing processing of the array lines ...

The new Bash version not only saves programmers some typing, it provides a cleaner implementation. Instead of the while loop, a single line is all it takes (see line 4 in Listing 5). The mapfile command additionally uses the readarray alias, which describes the purpose more aptly.

01 #!/bin/bash
02
03 inputFile="$1"
04 mapfile -n 0 lines < "$inputFile"
05
06 # Ongoing processing of the array lines  ...

The mapfile command can do even more. If you are interested in more detail, type help mapfile or visit the man page. The command can parse multiple lines in a single pass and process these lines one by one using a callback function. Unfortunately, the developers did not provide an especially elegant implementation of this function. Bash 4 calls the callback before parsing and not after. This approach gives the script a callback before the shell has actually read anything, but loses a callback after the last line. In spite of this complication, the function is still useful for parsing complete files.

Upper and Lower Case

If you have ever had to implement a robust approach to processing configuration files or user input, you will be aware of the following issue: Is the value in the configuration file or the result of a read operation YES, yes, Yes, true, TRUE, or True? Up to and including Bash 3.2, script authors used a call to tr to convert the value unambiguously to upper or lower case.

Bash 4 takes a far easier approach. The declare -u Varname instruction automatically converts all assignments to the Varname variable to upper case. The similar declare -l function converts to lower case. These conversion tools save the programmer the trouble of calling an external program and improve the script's performance. If you just want to convert once, rather than globally, you can use the new parameter extension:

foo=yes
echo ${foo^}
echo ${foo^^}

The first command converts the first letter to uppercase, thus outputting Yes. The second command converts all the letters and thus returns a result of YES.

If you need to convert to lowercase letters, you can use:

foo=YES
echo ${foo,}
echo ${foo,,}

The ${Var^Pattern} syntax supports even more complex replacements.

Co-Processes and Lists of Values

Most hardware manufacturers today offer multiple core CPUs, and techniques for exploiting multi-core functionality are starting to appear. Bash 4 allows programmers to launch what are referred to as co-processes:

coproc pipes command

This command returns the descriptors for standard input and output for the command in the pipes[0] and pipes[1] variables. The main Bash process uses them to communicate with the co-process, which is particularly useful for shell scripts in parallel processing [5].

Automatic value lists, or brace expansions, have been around for a while, but most users don't know about them: The call to echo {5..15} counts from the first to the last number. Many Bash programmers still use a call to seq for this:

echo $(seq -s " " 5 15)

The seq command is not just slower, the result is also more difficult to read. But if you need to sort this kind of output, you have to contend with the drawback of losing the leading zeros. Bash 4 shell users can now write

echo {05..15}

and, if needed, receive two-digit values with leading zeros as the return value.

The new version of the standard Linux shell offers some useful new features for programmers and command-line fans. Despite minor incompatibilities, the Bash maintainer encourages users to upgrade (see the "Interview with Bash 4 Developer Chet Ramey" box).

The new version of Bash is not exactly lean and mean - the binary now weighs in at 730KB compared with 590KB in the previous version. It is measurably, but not noticeably, slower; the slower execution times will hardly be a factor on today's hardware.

If you want to use the new functions today, you need full control of your target environment to avoid having to replace the whole Bash environment. In some scripts, you might want to query the BASH_VERSION or BASH_VERSINFO environmental variables just to be on the safe side.

Let the script terminate gracefully and output an error message if the conditions for Bash 4 are not fulfilled.