Simple Bash Scripts for Lazy People | Part 5: When to Choose Bash

This is part 5 of a five-part series:

• Part One has examples for common daily tasks in Git.
• Part Two, similar examples for Rails.
• Part Three, miscellaneous cases.
• Part Four dissects an example of a failed attempt at a useful script.
• Part Five concludes with a brief discussion of when to use Bash as opposed to some other scripting language.

TL;DR for Part 5:

• If your task involves only filesystem and OS-level tasks, Bash is probably a good fit
• Math! ((foo += 1)) or ((foo = foo+100))
• shellcheck.net is your friend

When To Use Bash

A common response I get when talking about Bash scripts with workmates is “why not use $language?”, where $language is often Ruby or Perl. Here’s my general thinking on that:

Perl is a great fit for doing text processing. I have a use case with one project where I need to parse a listing of files on a file system and update an SQLite database. Perl’s a natural here.

I’d use Ruby (or Python, or Perl, or PHP, or R, etc. — whatever was the dominant language on the project) where the problem required other functionality of a high-level language, like connecting to a third-party service or data manipulation.

Bash’s natural fit is where everything you’re doing is related to the OS & file system.

If the tasks is “get x from foo and put it to bar”, that’s scp, and there’s no reason to use a language that just has scp wrapped as a native function. In the use case I mentioned for Perl, that script is actually part of a process that works like this:

1. cron on source host runs a Bash script that executes a find and stores a gzip of the output at canonical location
2. cron on destination host runs a Bash script to scp and unzip the file, then runs the Perl script to parse the file and update the database

Here’s another use case for Bash where a lot of folks would use something else:

A web application has a feature where users can request a zip archive of a large number of files. These can be hundreds of megs in size so it’s not realistic to generate them synchronously with page requests.

The web app enqueues the request. cron runs a Bash script that gets a lock (since it has no way of knowing how long it will run), pulls the queue from the database, and for each item makes the corresponding zip file, stores it at a user-accessible location, and emails the user telling them that their archive is ready.

This script has 14 functions for things like outputting log entries consistently, normalizing or escaping IDs that could be inconsistent between data sources, preparing the email content, sending the email, generating the archive, and so on.

For a lot of people a scripting language would be the natural first choice. But every single thing it’s doing is an OS-level action. Even the database interaction is just a function of passing text to a DB client, which is all the scripting language’s wrappers would be doing.

Here’s an example (where client-specific things are redacted with foo-bar-baz stuff).

Also note the math using the (( var_name math_expression )) syntax, which is crazy useful.

function fetch_files {
  foo_id="$1";
  files_tmp="$2";

  echo "$(log_time)" "copying files to ${files_tmp}"

  # format %s for timestamp integer
  start_time="$(date +'%s')";
  nfiles=0;
  while read -r file; do
    file="$(escape_path "$file")";
    scp -q user@host.com:"${file}" "${files_tmp}"/;
    (( nfiles += 1 ));
  # $bar_id and $baz_id from global scope; TODO: fix this
  done < <(transcript_files "$foo_id" "$bar_id" "$baz_id");
  end_time="$(date +'%s')";
  (( elapsed_time = "$end_time" - "$start_time" ));
  echo "$(log_time)" "${elapsed_time} seconds for ${nfiles} files";
  return_status=0;
  if [[ "$nfiles" -eq 0 ]]; then
    return_status=100;
  fi
  return "$return_status";
}

The database interaction comes with the transcript_files function:

function transcript_files {
  foo_id="$1";
  bar_id="$2";
  baz_id="$3";

  # we'll be concatenating these values into SQL (DANGER, WILL ROBINSON!)
  # so cursory security check even though the bar ID is supposed to
  # be enforced as an int in the database
  if [[ ! "$bar_id" =~ ^[0-9]+$ ]]; then
    echo "$(log_time) BAR ID '${bar_id}' IS NOT INTEGER - what's going on with the DB and how did this get there?" >&2;
    exit 255;
  fi

  if [[ ! "$foo_id" =~ ^[0-9]+[-_][0-9]+$ ]]; then
    echo "$(log_time) FOO ID '${foo_id}' NOT FORMATTED AS EXPECTED - what's going on with the DB and how did this get there?" >&2;
    exit 255;
  fi

  if [[ ! "$baz_id" =~ ^[0-9]+[-_][0-9]+$ ]]; then
    echo "$(log_time) BAZ ID '${baz_id}' NOT FORMATTED AS EXPECTED - what's going on with the DB and how did this get there?" >&2;
    exit 255;
  fi

  transcript_key="$(echo "${baz_id}" "${bar_id}" | sed -e 's/-/_/g')";
  transcript_dir="${transcript_key} Transcript";

  sql="SELECT f.full_path
    FROM files f
    JOIN directories d ON f.directory_id = d.id
    WHERE d.name LIKE \"${transcript_dir}\"
    AND f.name LIKE \"%.pdf\"
    AND f.bar_id = ${bar_id}
    AND d.bar_id = ${bar_id}";

  echo "$sql" | sqlite3 -noheader "$DB_PATH";
}

Obviously with another language you could make a prepared statement, or use an ORM, but you’d still have to test for and log bad data, so I don’t think it would be less code. The files we care about are guaranteed to live in a directory called “$transcript_key Transcript“, and we have the full, absolute path to those files in the database. transcript_files ultimately invokes the sqlite3 client with a query that returns a single column, so the output is a list of file names, which is what fetch_files needs to execute the scp commands to move the files from the remote host to the host on which the archive needs to be available to web users.

I’m dubious using a higher-level language buys you anything other than some syntax sugar. Unless you’re relying on some feature of the language to do something you can’t easily do on the command line, there’s just no point.

Final Thoughts

I’ll give the last word here to the last word in bash scripting, shellcheck.net. Run every single script you write through there, and you will learn heaps and heaps and heaps every time!