shellspec

BDD style unit testing framework for POSIX compliant shell script.

Let’s test your shell script!

• Introduction
  • Features
  • Supported shells
  • Requirements
• Tutorial
  • Installation
    • Web installer
    • Package manager
    • Others (archive / make / manual)
  • Use with Docker
  • Getting started
• shellspec command
  • Usage
  • Configure default options
  • Special environment variables
  • Parallel execution
  • Random execution
  • Reporter / Generator
  • Ranges / Filters
  • Coverage
  • Profiler
  • Task runner
• Project directory
  • .shellspec
  • .shellspec-local
  • report/
  • coverage/
  • spec/
  • banner
  • spec_helper.sh
  • support/
• Specfile
  • Example
  • Translation process
  • DSL
    • Describe, Context - example group
    • It, Example, Specify, Todo - example
    • When - evaluation
    • The - expectation
    • Skip, Pending - skip and pending example
    • Include - include a shell script
    • Set - set shell option
    • Path, File, Dir - path alias
    • Data - input data for evaluation
    • Parameters - parameterized example
    • subject, modifier, matcher
      • Custom matcher
  • Hooks
    • Before, After - example hook
    • BeforeCall, AfterCall - call evaluation hook
    • BeforeRun, AfterRun - run evaluation hook
  • Directive
    • %const (%) - constant definition
    • %text - embedded text
    • %puts (%-), %putsn (%=) - put string
    • %logger
    • %data
  • Mock and Stub
  • Testing a single file script
    • Sourced Return
    • Intercept
• For developers
• Version history

Shellspec was developed as a cross-platform testing tool for developing POSIX-compliant shell scripts that work in many environments. It not only works on PC but also in restricted environments like cloud and embedded OS. It provides first-class features equivalent to other language testing tools. Of course shellspec is tested by shellspec.

• Works with all POSIX compliant shells (dash, bash, zsh, ksh, busybox, etc...)
• Implemented by shell scripts with minimal dependencies (use only a few basic POSIX-compliant commands)
• BDD style specfile interpretable as a shell script
• Supports nestable blocks with scope like lexical scope
• Easy to mock and stub in cooperation with scope
• Skip / pending of the examples, and support of easy-to-skip "x" known as "xit"
• Hooks before / after examples
• Parameterized examples for Data-Driven tests
• Execution filtering by line number, id, focus, tag and example name
• Parallel execution, random ordered execution and dry-run execution
• Modern reporting (colorized, failed line number, progress / documentation / TAP / JUnit formatter)
• Coverage (Kcov integration) and Profiler
• Friendly with Docker and CI
• Builtin simple task runner
• Extensible architecture (custom matcher, custom formatter, etc...)

dash, bash, ksh, mksh, oksh, pdksh, zsh, posh, yash, busybox (ash), bosh, pbosh

Tested Platforms

Tested version details

Shellspec is implemented by a pure shell script and uses only shell builtins and a few basic POSIX-compliant commands to support widely environments (except kcov for optional coverage).

Currently used external (not shell builtins) commands:

• cat, date, ls, mkdir, od (or not POSIX hexdump), rm, sleep, sort, time
• ps (used to autodetect the current shell in environments which do not implement procfs)
• ln, mv (used only when generating coverage report)
• kill, printf (used but almost shell builtins)

Install the latest release version.

curl -fsSL https://git.io/shellspec | sh

or

wget -O- https://git.io/shellspec | sh

curl -fsSL https://git.io/shellspec | sh -s 0.19.1

curl -fsSL https://git.io/shellspec | sh -s -- --switch

curl -fsSL https://git.io/shellspec | sh -s 0.18.0 --switch

$ curl -fsSL https://git.io/shellspec | sh -s -- --help
Usage: [sudo] ./install.sh [VERSION] [OPTIONS...]
  or : wget -O- https://git.io/shellspec | [sudo] sh
  or : wget -O- https://git.io/shellspec | [sudo] sh -s -- [OPTIONS...]
  or : wget -O- https://git.io/shellspec | [sudo] sh -s VERSION [OPTIONS...]
  or : curl -fsSL https://git.io/shellspec | [sudo] sh
  or : curl -fsSL https://git.io/shellspec | [sudo] sh -s -- [OPTIONS...]
  or : curl -fsSL https://git.io/shellspec | [sudo] sh -s VERSION [OPTIONS...]

VERSION:
  Specify install version and method

  e.g
    1.0.0           Install 1.0.0 from git
    master          Install master from git
    1.0.0.tar.gz    Install 1.0.0 from tar.gz archive
    .               Install from local directory

OPTIONS:
  -p, --prefix PREFIX   Specify prefix                 [default: $HOME]
  -b, --bin BIN         Specify bin directory          [default: <PREFIX>/bin]
  -d, --dir DIR         Specify installation directory [default: <PREFIX>/lib/shellspec]
  -s, --switch          Switch version (requires installed via git)
  -l, --list            List available versions (tags)
      --pre             Include pre-release
      --fetch FETCH     Force command to use when install from archive (curl or wget)
  -y, --yes             Automatic yes to prompts
  -h, --help            You're looking at it

# Install the latest stable version
$ aura -A shellspec

# Install from master branch
$ basher install shellspec/shellspec

# To specify a version (example: 0.19.1)
$ basher install shellspec/shellspec@0.19.1

# Install from master branch
$ bpkg install shellspec/shellspec

# To specify a version (example: 0.19.1)
$ bpkg install shellspec/shellspec@0.19.1

sudo make install

make install PREFIX=$HOME

sudo make uninstall

make uninstall PREFIX=$HOME

$ cd /SOME/WHERE/TO/INSTALL
$ git clone https://github.com/shellspec/shellspec.git
$ ln -s /SOME/WHERE/TO/INSTALL/shellspec/shellspec /EXECUTABLE/PATH/
# (e.g. /EXECUTABLE/PATH/ = /usr/local/bin/, $HOME/bin/)

$ cd /SOME/WHERE/TO/INSTALL
$ wget https://github.com/shellspec/shellspec/archive/{VERSION}.tar.gz
$ tar xzvf shellspec-{VERSION}.tar.gz

$ ln -s /SOME/WHERE/TO/INSTALL/shellspec-{VERSION}/shellspec /EXECUTABLE/PATH/
# (e.g. /EXECUTABLE/PATH/ = /usr/local/bin/, $HOME/bin/)

$ cat<<'HERE'>/EXECUTABLE/PATH/shellspec
#!/bin/sh
exec /SOME/WHERE/TO/INSTALL/shellspec/shellspec "$@"
HERE
$ chmod +x /EXECUTABLE/PATH/shellspec

You can run shellspec without installation by using Docker. The shellspec and specfiles run in a Docker container.

See How to use shellspec with Docker.

Just create your project directory and run shellspec --init to setup to your project

# Create your project directory, for example "hello".
$ mkdir hello
$ cd hello

# Initialize
$ shellspec --init
  create   .shellspec
  create   spec/spec_helper.sh
  create   spec/hello_spec.sh # sample

# Write your first specfile (of course you can use your favorite editor)
$ cat<<'HERE'>spec/hello_spec.sh
Describe 'hello.sh'
  Include lib/hello.sh
  It 'says hello'
    When call hello shellspec
    The output should equal 'Hello shellspec!'
  End
End
HERE

# Create lib/hello.sh
$ mkdir lib
$ touch lib/hello.sh

# It will fail because the hello function is not implemented.
$ shellspec

# Write hello function
$ cat<<'HERE'>lib/hello.sh
hello() {
  echo "Hello ${1}!"
}
HERE

# It will success!
$ shellspec

Usage: shellspec [options] [files or directories]

  -s, --shell SHELL                   Specify a path of shell [default: current shell]
      --[no-]fail-fast[=COUNT]        Abort the run after a certain number of failures [default: 1]
      --[no-]fail-no-examples         Fail if no examples found [default: disabled]
      --[no-]fail-low-coverage        Fail on low coverage [default: disabled]
                                        The coverage threshold is specified by the coverage option
  -r, --require MODULE                Require a file
  -e, --env NAME=VALUE                Set environment variable
      --env-from ENV-SCRIPT           Set environment variable from script file
      --random TYPE[:SEED]            Run examples by the specified random type
                                        [none]      run in the defined order [default]
                                        [specfiles] randomize the order of specfiles
                                        [examples]  randomize the order of examples (slow)
  -j, --jobs JOBS                     Number of parallel jobs to run (0 jobs means disabled)
      --[no-]warning-as-failure       Treat warning as failure [default: enabled]
      --dry-run                       Print the formatter output without running any examples
      --keep-tempdir                  Do not cleanup temporary directory [default: disabled]

  **** Output ****

      --[no-]banner                   Show banner if exist 'spec/banner' [default: enabled]
  -f, --format FORMATTER              Choose a formatter to use for display
                                        [p]rogress      dots [default]
                                        [d]ocumentation group and example names
                                        [t]ap           TAP format
                                        [j]unit         JUnit XML
                                                        (Require --profile for time attributre)
                                        [null]          do not display anything
                                        [debug]         for developer
                                        custom formatter name
  -o, --output GENERATOR              Choose a generator(s) to generate a report file(s)
                                        You can use the same name as FORMATTER
                                        Multiple options can be specified [default: not specified]
      --force-color, --force-colour   Force the output to be in color, even if the output is not a TTY
      --no-color, --no-colour         Force the output to not be in color, even if the output is a TTY
      --skip-message VERBOSITY        Mute skip message
                                        [verbose]  do not mute any messages [default]
                                        [moderate] mute repeated messages
                                        [quiet]    mute repeated messages and non-temporarily messages
  -p  --[no-]profile                  Enable profiling of examples and list the slowest examples
      --profile-limit N               List the top N slowest examples (default: 10)

  **** Ranges / Filters ****

    You can select examples range to run by appending the line numbers or id to the filename

      shellspec path/to/a_spec.sh:10:20     # Run the groups or examples that includes lines 10 and 20
      shellspec path/to/a_spec.sh:@1-5:@1-6 # Run the 5th and 6th groups/examples defined in the 1st group

    You can filter examples to run with the following options

      --focus                         Run focused groups / examples only
                                        To focus, prepend 'f' to groups / examples in specfiles
                                        e.g. Describe -> fDescribe, It -> fIt
      --pattern PATTERN               Load files matching pattern [default: "*_spec.sh"]
      --example PATTERN               Run examples whose names include PATTERN
      --tag TAG[:VALUE]               Run examples with the specified TAG
      --default-path PATH             Set the default path where shellspec looks for examples [defualt: "spec"]

  **** Coverage ****

      --[no-]kcov                     Enable coverage using kcov [default: disabled]
                                        Note: Requires kcov and bash, parallel execution is ignored.
      --kcov-path PATH                Specify kcov path [default: kcov]
      --kcov-options OPTIONS          Kcov options to overwrite and add (coverage limits, coveralls id, etc...)
                                        Default specified options:
                                          --exclude-pattern=/spec/,/coverage/,/report/
                                          --include-path=.
                                          --include-pattern=.sh
                                          --path-strip-level=1
                                        To include files without extension, specify --include-pattern
                                        without '.sh' and filter with --include-*/--exclude-* options

  **** Utility ****

      --init                          Initialize your project with shellspec
      --count                         Count the number of specfiles and examples
      --list LIST                     List the specfiles / examples
                                        [specfiles]       list the specfiles
                                        [examples]        list the examples with id
                                        [examples:id]     alias for examples
                                        [examples:lineno] list the examples with lineno
                                        [debug]           for developer
                                        affected by --random but TYPE is ignored
      --syntax-check                  Syntax check of the specfiles without running any examples
      --translate                     Output translated specfile
      --task [TASK]                   Run task. If TASK is not specified, show the task list
  -v, --version                       Display the version
  -h, --help                          You're looking at it

To change the default options for the shellspec command, create an options file. Files are read in the order shown below, options defined last take precedence.

1. $XDG_CONFIG_HOME/shellspec/options
2. $HOME/.shellspec
3. ./.shellspec
4. ./.shellspec-local (Do not store in VCS such as git)

Special environment variables understood by shellspec begin with SHELLSPEC_. They can be overridden with a custom script using the --env-from option.

Todo: descriptions of many special environment variables.

You can use parallel execution for fast test with --jobs option. Parallel jobs are executed per specfile. So it is necessary to separate the specfile for effective parallel execution.

You can randomize the execution order to detect troubles due to the test execution order. If SEED is specified, the execution order is deterministic.

You can specify one reporter (output to stdout) and multiple generators (output to a file). Currently builtin formatters are progress, documentation, tap, junit, null, debug.

You can execute specified spec only. It can be specified by line number, example id, example name, tag and focus. To focus, prepend f to groups / examples in specfiles (e.g. Describe -> fDescribe, It -> fIt).

Shellspec is integrated with coverage for ease of use. It works with the default settings, but you may need to adjust options to make it more accurate.

kcov is required to use coverage.

• How to install kcov.
• Sample of coverage report.

Be aware that bash is the only shell which can be used for coverage analysis.

When the --profile option is specified, the profiler is enabled and lists the slow examples.

You can run the task with --task option.

Typical directory structure.

Project directory
├─ .shellspec
├─ .shellspec-local
├─ report/
├─ coverage/
│
├─ bin/
│   ├─ executable_shell_script
│              :
├─ lib/
│   ├─ your_shell_script_library.sh
│              :
├─ libexec/
│   ├─ executable_utilities
│              :
├─ spec/
│   ├─ banner
│   ├─ spec_helper.sh
│   ├─ support/
│   ├─ your_shell_script_library_spec.sh
│              :

Project default options for shellspec command.

Override project default options (Do not store in VCS such as git).

Directory where the generator outputs reports.

Directory where kcov outputs coverage reports.

Directory where you create specfiles.

If spec/banner file exists, the banner is shown when the shellspec command is executed. Disable that behavior with the --no-banner option.

The spec_helper.sh loaded by the --require spec_helper option. This file is used to prepare for running examples, to define custom matchers, etc.

This directory is used to store files for custom matchers, tasks, etc.

The best place to learn how to write a specfile is the sample/spec directory. You should take a look at it ! (Those samples include failure examples on purpose.)

Describe 'sample' # example group
  Describe 'bc command'
    add() { echo "$1 + $2" | bc; }

    It 'performs addition' # example
      When call add 2 3 # evaluation
      The output should eq 5  # expectation
    End
  End

  Describe 'implemented by shell function'
    Include ./mylib.sh # add() function defined

    It 'performs addition'
      When call add 2 3
      The output should eq 5
    End
  End
End

The specfile is a valid shell script, but a translation process is performed to implement the scope, line number etc. Each example group block and example block is translated to commands in a subshell. Therefore changes inside those blocks do not affect the outside of the block. In other words it realizes local variables and local functions in the specfile. This is very useful for describing a structured spec. If you are interested in how to translate, use the --translate option.

You can write a structured example by using the Describe, Context block. Example groups can be nested. They can contain example groups or examples. Each example group runs in a subshell.

You can describe how code behaves by using the It, Example, Specify block. It is composed by a maximum of one evaluation and multiple expectations. Todo is a one liner empty example.

Defines the action for verification. The evaluation begins with When. Only one evaluation can be defined for each example.

When call echo hello world
 |    |    |
 |    |    +-- The rest is action for verification
 |    +-- The evaluation type `call` calls a function or external command.
 +-- The evaluation keyword

There are two types of evaluation, When call and When run. and When run has sub types of command, script and source.

See more details of Evaluation

Defines the verification. The expectation begins with The.

Verifies the subject with the matcher.

The output should equal 4
 |    |           |
 |    |           +-- The `equal` matcher verifies the subject value is 4.
 |    +-- The `output` subject uses the stdout as a subject for verification.
 +-- The expectation keyword

You can reverse the verification with should not.

The output should not equal 4

You can use the modifier to modify the subject.

The line 2 of output should equal 4
    |
    +-- The `line` modifier use the specified line 2 of output as subject.

The modifier is chainable.

The word 1 of line 2 of output should equal 4

You can use ordinal numbers.

The second line of output should equal 4

Shellspec supports language chains like chai.js. It only improves readability, does not affect the expectation:

• a
• an
• as
• the

The following two sentences have the same meaning:

The first word of second line of output should valid number

The first word of the second line of output should valid as a number

You can skip an example by using the Skip keyword. If you want to skip only in some cases, use a conditional skip Skip if. You can also use Pending to indicate that the example needs to be implemented. You can temporary skip Describe, Context, Example, Specify, It blocks by prefixing with x (xDescribe, xContext, xExample, xSpecify, xIt).

Include the shell script file to test.

Set shell option before executing each example. The shell option name is the long name of set or the name of shopt:

Set 'errexit:off' 'noglob:on'

TODO

You can use the Data Helper which inputs data from stdin for evaluation. The input data is specified after #| in the Data block.

Describe 'Data helper'
  Example 'provide with Data helper block style'
    Data
      #|item1 123
      #|item2 456
      #|item3 789
    End
    When call awk '{total+=$2} END{print total}'
    The output should eq 1368
  End
End

You can Data Driven Test (aka Parameterized Test) with Parameters.

Note: Multiple Parameters definitions are merged.

Describe 'example'
  Parameters
    "#1" 1 2 3
    "#2" 1 2 3
  End

  Example "example $1"
    When call echo "$(($2 + $3))"
    The output should eq "$4"
  End
End

The following four styles are supported.

# block style (default: same as Parameters)
Parameters:block
  "#1" 1 2 3
  "#2" 1 2 3
End

# value style
Parameters:value foo bar baz

# matrix style
Parameters:matrix
  foo bar
  1 2
  # expanded as follows
  #   foo 1
  #   foo 2
  #   bar 1
  #   bar 2
End

# dynamic style
#   Only %data directive can be used within Parameters:dynamic block.
#   You can not call function or accessing variable defined within specfile.
#   You can refer to variables defined with %const.
Parameters:dynamic
  for i in 1 2 3; do
    %data "#$i" 1 2 3
  done
End

There are more subjects, modifiers, matchers. please refer to the References

Shellspec has an extensible architecture. So you can create custom matchers, custom modifiers, custom formatters, etc.

See sample/spec/support/custom_matcher.sh for custom matcher.

You can define before / after hooks by using Before, After. The hooks are called for each example.

You can define before / after call hooks by using BeforeCall, AfterCall. The hooks are called before or after a "call evaluation".

You can define before / after run hooks by using BeforeRun, AfterRun. The hooks are called before or after a "run evaluation".

These hooks are executed in the same subshell as the "run evaluation". So you can mock/stub the function before run. And you can access a variable for evaluation after run.

%const (% is short hand) directive defines a constant value. The characters which can be used for variable names are uppercase letters [A-Z], digits [0-9] and underscore _ only. It can not be defined inside an example group nor an example.

The value is evaluated during the specfile translation process. So you can access shellspec variables, but you can not access variable or function in the specfile.

This feature assumed use with conditional skip. The conditional skip may runs outside of the examples. As a result, sometime you may need variables defined outside of the examples.

You can use the %text directive instead of an hard-to-use heredoc with indented code. The input data is specified after #|.

Describe '%text directive'
  It 'outputs texts'
    output() {
      echo "start" # you can write code here
      %text
      #|aaa
      #|bbb
      #|ccc
      echo "end" # you can write code here
    }

    When call output
    The line 1 of output should eq 'start'
    The line 2 of output should eq 'aaa'
    The line 3 of output should eq 'bbb'
    The line 4 of output should eq "ccc"
    The line 5 of output should eq 'end'
  End
End

%puts (put string) and %putsn (put string with newline) can be used instead of (not portable) echo. Unlike echo, it does not interpret escape sequences regardless of the shell. %- is an alias of %puts, %= is an alias of %putsn.

Output log to $SHELLSPEC_LOGFILE (default: /dev/tty) for debugging.

See Parameters.

Currently, shellspec does not provide any special function for mocking / stubbing. But redefining a shell function can override existing shell function or external command. It can be used for mocking / stubbing.

Remember that Describe, Context, It, Example, Specify blocks run in a subshell. When going out of the block, redefined functions are restored.

Describe 'mock stub sample'
  unixtime() { date +%s; }
  get_next_day() { echo $(($(unixtime) + 86400)); }

  Example 'redefine date command'
    date() { echo 1546268400; }
    When call get_next_day
    The stdout should eq 1546354800
  End

  Example 'use the date command'
    # date is not redefined because this is another subshell
    When call unixtime
    The stdout should not eq 1546268400
  End
End

Shell scripts are often made up of a single file. Shellspec provides two ways of testing a single shell script.

This is a method for testing functions defined in shell scripts. Loading a script with Include defines a __SOURCED__ variable available in the sourced script. If the __SOURCED__ variable is defined, return in your shell script process.

#!/bin/sh
# hello.sh

hello() { echo "Hello $1"; }

${__SOURCED__:+return}

hello "$1"

Describe "sample"
  Include "./hello.sh"
  Example "hello test"
    When call hello world
    The output should eq "Hello world"
  End
End

This is a method to mock/stub functions and commands when executing shell scripts. By placing intercept points in your script, you can call the hooks defined in specfile.

#!/bin/sh
# today.sh

test || __() { :; }

__ begin __

date +"%A, %B %d, %Y"

Describe "sample"
  Intercept begin
  __begin__() {
    date() {
      export LANG=C
      command date "$@" --date="2019-07-19"
    }
  }
  Example "today test"
    When run source ./today.sh
    The output should eq "Friday, July 19, 2019"
  End
End

If you want to know shellspec architecture and self test, see CONTRIBUTING.md

See Changelog