on
Debugging bash scripts
Unlike many programming languages, shell(bash) scripting has no official tool for debugging. Besides the classic “prints”, I was not aware that bash has some built-in mechanisms for tracing, until I decided to “read the manual”. As I was struggling with understanding some large shell scripts (that I did not write myself), I found some valuable information just by running man bash.
To my surprise, bash does have a debugger mode/profile, which can be seen on the very first page of the manual:
--debugger
Arrange for the debugger profile to be executed before the shell starts. Turns on extended debugging mode
nn(see the description of the extdebug option to the shopt builtin below).
What does --debugger do? Well, it turns on the extended debugging mode, which is equivalent to setting a shell option called extdebug. A search for the keyword extdebug can give us the answer:
extdebug
If set at shell invocation, or in a shell startup file, arrange to execute the debugger profile be‐
fore the shell starts, identical to the --debugger option. If set after invocation, behavior in‐
tended for use by debuggers is enabled:
1. The -F option to the declare builtin displays the source file name and line number correspond‐
ing to each function name supplied as an argument.
2. If the command run by the DEBUG trap returns a non-zero value, the next command is skipped and
not executed.
3. If the command run by the DEBUG trap returns a value of 2, and the shell is executing in a
subroutine (a shell function or a shell script executed by the . or source builtins), the
shell simulates a call to return.
4. BASH_ARGC and BASH_ARGV are updated as described in their descriptions above.
5. Function tracing is enabled: command substitution, shell functions, and subshells invoked with
( command ) inherit the DEBUG and RETURN traps.
6. Error tracing is enabled: command substitution, shell functions, and subshells invoked with (
command ) inherit the ERR trap.
The features 4-6 caught my interest. By doing more research, I found out that a shell option can be toggled either using the shopt built-in command or using +/-O (+ is for untoggling the option and - is for toggling, weird right?) when launching the bash executable. Our option can be set using shopt -s extdebug from a script.
The extdebug option is tightly related to the trap command that allows setting event listeners on different types of events and signals. The DEBUG trap in particular is executed before every command is executed:
If a sigspec is DEBUG, the command arg is executed before every simple command, for command, case command, select command, every arithmetic for command, and before the first command executes in a shell function (see SHELL GRAMMAR above).
I did more digging for BASH_ARGC and BASH_ARGV, and I found more than I was looking for in the shell variables section:
BASH_ARGC
An array variable whose values are the number of parameters in each frame of the current bash execution call
stack. The number of parameters to the current subroutine (shell function or script executed with . or
source) is at the top of the stack. When a subroutine is executed, the number of parameters passed is pushed
onto BASH_ARGC. The shell sets BASH_ARGC only when in extended debugging mode (see the description of the
extdebug option to the shopt builtin below). Setting extdebug after the shell has started to execute a
script, or referencing this variable when extdebug is not set, may result in inconsistent values.
BASH_ARGV
An array variable containing all of the parameters in the current bash execution call stack. The final pa‐
rameter of the last subroutine call is at the top of the stack; the first parameter of the initial call is at
the bottom. When a subroutine is executed, the parameters supplied are pushed onto BASH_ARGV. The shell
sets BASH_ARGV only when in extended debugging mode (see the description of the extdebug option to the shopt
builtin below). Setting extdebug after the shell has started to execute a script, or referencing this vari‐
able when extdebug is not set, may result in inconsistent values.
BASH_LINENO
An array variable whose members are the line numbers in source files where each corresponding member of FUNC‐
NAME was invoked. ${BASH_LINENO[$i]} is the line number in the source file (${BASH_SOURCE[$i+1]}) where
${FUNCNAME[$i]} was called (or ${BASH_LINENO[$i-1]} if referenced within another shell function). Use LINENO
to obtain the current line number.
BASH_COMMAND
The command currently being executed or about to be executed, unless the shell is executing a command as the
result of a trap, in which case it is the command executing at the time of the trap. If BASH_COMMAND is un‐
set, it loses its special properties, even if it is subsequently reset.
FUNCNAME
An array variable containing the names of all shell functions currently in the execution call stack. The el‐
ement with index 0 is the name of any currently-executing shell function. The bottom-most element (the one
with the highest index) is "main". This variable exists only when a shell function is executing. Assign‐
ments to FUNCNAME have no effect. If FUNCNAME is unset, it loses its special properties, even if it is sub‐
sequently reset.
This variable can be used with BASH_LINENO and BASH_SOURCE. Each element of FUNCNAME has corresponding ele‐
ments in BASH_LINENO and BASH_SOURCE to describe the call stack. For instance, ${FUNCNAME[$i]} was called
from the file ${BASH_SOURCE[$i+1]} at line number ${BASH_LINENO[$i]}. The caller builtin displays the cur‐
rent call stack using this information.
The above environment variables provide us with some contextual information about the line where the command is executed within the script file, the current function name as well as the function arguments if there are any. We have now enough information to establish some debugging infrastructure of our own.
By creating a debug trap that prints the above variables, we can instrument any script to display those informations while running:
shopt -s extdebug
_debug_() {
echo -e "----------- DEBUG INFO ------------------\n"
echo -e "line $BASH_LINENO, executing: $(echo -e $BASH_COMMAND | tr -d '\r\n')\n"
echo -e "current function call stack:\n"
for func_name in ${FUNCNAME[@]}; do
if [ "$func_name" != "_debug_" ]; then
echo -e "$func_name"
fi
done
echo -e "current function arguments: ${BASH_ARGV[*]}"
echo -e "\n------------------\n"
}
trap '_debug_' DEBUGPasting those statements in every script we want to debug can be tedious, so we can create a wrapper script that accepts the script to debug as an argument:
echo "debugging script $1: $@"
shopt -s extdebug
script_args=$@
_debug_() {
echo -e "----------- DEBUG INFO ------------------\n"
echo -e "line $BASH_LINENO, executing: $(echo -e $BASH_COMMAND | tr -d '\r\n')\n"
echo -e "current function call stack:"
for func_name in ${FUNCNAME[@]}; do
if [ "$func_name" != "_debug_" ] && [ "$func_name" != "source" ]; then
echo -e "$func_name"
fi
done
echo "current function arguments: "
# we need to do some processing here to avoid printing the script args
# and also the source function args
for arg in ${BASH_ARGV[@]}; do
found=false
for script_arg in $script_args; do
if [ "$arg" == "$script_arg" ]; then
found=true
break
fi
done
if [ "$found" != "true" ]; then
echo -e "$arg"
fi
done
echo -e "\n------------------\n"
}
trap '_debug_' DEBUG
#we need to pass parameters as well, in case the script is taking parameters
#$0 is the name of the executed script and $1 is the name of our input script
#our parameters start from index 2
source $1 ${@:2}Last but not least, we can also add some colors to our debugging information to distinguish between the script output and the debug output.
In the bash manual (section PROMPTING), it is mentioned that an escape AINSI character can be used by inserting a \e. AINSI escape characters are used to control the color, fonts,…etc in terminals. We need to put our debug text between two escape sequences. To set the color of our debug output to red, we can add \e[31m (the red code is 31m) to the beginning of our first print, and \e[0 (special sequence to indicate reset of the original attributes) to the end of the last print.
_debug_() {
echo -e "\e[31m----------- DEBUG INFO ------------------\n"
// other stuff...
echo -e "\n------------------\e[0m\n"
}
Something important that we should not forget is output redirection. We should make sure that our debug output is redirected to standard error to avoid interfering with the debugged script output. In case the debugged script is interacting with standard input/output, debugging should have no side effects. In the REDIRECTION section in the bash manual, it is mentioned that redirection from standard output to standard error can be done by adding 1>&2 at the end of a command, and therefore, we need then to add 1>&2 to each of our echo commands:
echo "debugging script $1: $@" 1>&2
shopt -s extdebug
script_args=$@
_debug_() {
echo -e "\e[31m----------- DEBUG INFO ------------------\n" 1>&2
echo -e "line $BASH_LINENO, executing: $(echo -e $BASH_COMMAND | tr -d '\r\n')\n" 1>&2
echo -e "current function call stack:" 1>&2
for func_name in ${FUNCNAME[@]}; do
if [ "$func_name" != "_debug_" ] && [ "$func_name" != "source" ]; then
echo -e "$func_name" 1>&2
fi
done
echo "current function arguments: "
for arg in ${BASH_ARGV[@]}; do
found=false
for script_arg in $script_args; do
if [ "$arg" == "$script_arg" ]; then
found=true
break
fi
done
if [ "$found" != "true" ]; then
echo -e "$arg" 1>&2
fi
done
echo -e "\n------------------\e[0m\n" 1>&2
}
trap '_debug_' DEBUG
source $1 ${@:2}Et voilà! We now have a debugging tool that we can use for tracing any bash script: debug.sh the_script_to_debug.sh