runbtf

The runbtf script is a framework for the control of test case execution. The execution of test case/suite variants and the parallel execution is inherently supported.

Test Cases, Test Suites and Test Collections

A test case is comprised of a directory with the main test case file with name: ‘TestCase.sh’ and other necessary artifacts which are necessary for the test execution. The name of a test case is the relative path from the containing entity to the main test case file. The test case file contains the necessary definitions and the script code to execute the test.

A test suite is a collection of test cases, test suites and artifacts to prepare and finalize the execution of the suite. The directory sub tree of the test suite may have an arbitrary depth. A test suite is defined through a directory with the main suite file with name: ‘TestSuite.sh’ The name of a test suite is the relative path from the containing entity to the main test suite file. The test suite file contains the necessary definitions and the script code to execute the test suite preparation and suite finalization. The test suite file may contain common suite code (you must export functions, that are to be used in sub-suites or test cases). Test cases may exists without a suite.

One or more test suites and/or test cases form a Test Collection. A Test Collection is defined through a directory. A test collection may have a test properties file TestProperties.sh which should contain the definition of variables and properties which may be variable in different test environments. The name of the test properties file may be changed by a command line parameter (–properties).

Common used script code may be placed in separate script files, which must be imported during test run (use the import function).

Test cases must not be nested in other test case directories. All path names of test cases and suites must not contain any white space characters.

Test Cases and Test Suites may define variants. This allows the repeated execution of the artifact with changed parameter sets.

Execution Environment

The test framework starts with the analysis of the input directory (directory list) (option -i|–directory).

If no list with case wildcards is given as command line parameter, all found test suites and test cases which are not marked as ‘skipped’ property are executed. In this case all suites (also empty suites) are executed.

If a cases list is given from the command line, all test cases with match the cases list are executed (pattern match). Additionally all suites are executed, which are necessary to reatch the cases with a pattern match. In this mode suites that do not include an active case are not executed.

There is always an enveloping ‘dummy’ suite, which is always executed.

All generated artifacts are stored in a sub-directory of the workdir (option -w|–workdir) for further analysis. The sub-directory name is composed of the actual date and time when the test case execution starts.

A summary is printed after test case execution.

Test Case File ‘TestCase.sh’ and Test Suite File ‘TestSuite.sh’

These files have in general two sections: The preamble and a script code section. Both sections may be empty.

The preamble defines variables which are necessary before execution of appropriate artifacts starts. A preamble statement starts with the character sequence ‘#–’. A preamble line may be continued after a single \ before nl. The continuation line must start also with sequence ‘#–’

The script code section is a bash script. In the script section, you can define required code for the initialization and the custom functions for the test preparation, for the test step execution and the test finalization.

In general all collections, suites and test cases are executed in 4 phases:

The script code of the main body is executed during initialization of the Test Suite or of the Test Case.

In the preparation phase of Test Case or Suite, functions that are defined through the optional variables and functions:

During execution phase of the Test Collection the framework iterates recursively though all defined Test Suites.

During execution phase of the Test Suite the framework iterates though all Test Cases and then all defined Sub-Suites are executed. Test Case execution may use parallel execution. Test Suites are always sequentially executed.

During execution phase of the Test Case the framework iterates sequentially though all Test Cases Steps. Test steps are defined:

In the finalization phase of the artifact, functions that are defined through the optional variables and functions:

If the variables TTPR_noPrepsSuite TTPR_noPrepsCase TTPR_noFinsSuite TTPR_noFinsCase are set to a non empty value the preparation and the finalization of the appropriate artifact is supressed.

Test Property File TestProperties.sh

This file may contain global property and variable definitions. This file should no contain script code. This file is intended to store stuff which may change when the test collection is executed in different environments. The default name of this file is ‘TestProperties.sh’ and it is expected in the Test collection directory. An alternative file name may be assigned with command line parameter –properties or the environment TTRO_propertyFiles is evaluated. The command line option overwrites environment and the default. The properties file is a bash script.

Test Tools/Modules

If your test collection requires special functions, you must import the appropriate script module in the initialization part of a Test Suite or Test Case file. The test Tools Script may define user defined variables, properties and functions. The defined functions in a Tools Script must be exported like:

export -f fname

The defined tool artifacs are available in nested Suites and nested cases. Once a function has been defined, it can not be re-defined in a nested element.

This modules are searched in the directory of the current TestCase/TestSuite, in all directories of enclosing suites, in the directory of the Test Collection and in the bin-directory.

Test File Preamble

The definition of the variables and properties must have the form: #–name=value No spaces are allowed between name ‘=’ and value. The assignement requires the same quoting as a reqular bash assignement. The assignment can use a continuation line if the line ends with an escaped newline character (backlslash before newline) The continuation line must also start with #– The preamble may define the variants of the test artifact and in case of a test case, the timeout value and boolean property exclusiveExecution for the test case.

Test Collection, Test Case and Test Suite variants

The variants of cases, suites and collections are defined in the preamble of the ‘TestCase.sh’ or the ‘TestSuite.sh’ file. The appropriate file must have either no variant variable, a variantCount variable or a variantList variable.

The variantCount must be in the form: #–variantCount=number

The variantList must be a space separated list of identifiers: #–variantList=’space separated list of variant identifiers’ An identifier should be composed from following characters : 0-9a-zA-Z-_ Make sure that the variant identifiers can be easily used in pattern matching expressions. Thus avoid characters: *?[]-:!^

Test Case timeouts

The test case execution is superviced and when the timeout (TTPR_timeout in seconds) is reached, the job is killed with SIGTERM (15). If the job still runs after additional time (TTPR_additionalTime), the job is killed with SIGKILL (9).

The test case timeout can be controlled with property TTPR_timeout. If this property is not set, the value 240 is used. The additional time is controlled property TTPR_additionalTime. If this property is not set, the vaue 60 is used.

Each test Case can define an individual test Case timeout. This can be done in the Test Case File Preamble like: #–timeout=600

The individual test case timeout is used if the value is greater than TTPR_timeout (if exists) or if the value is greater than 240.

Test Case exclusive execution property

If the preamble contains a line #–exclusive=true the test case is always executed as a single case and no parallel execution is done.

Reserved Variable Name Ranges

Variables used for the framework have special prefixes.

Test Framework Variables and Properties

Variables with the prefix TT_, TTRO_, TTPR_ or TTPRN_ are treated as global definitions and they are exported from Test Collection to Test Suite and from Test Suite to Test Case.

In the script code section, variables and properties can be assigned with function setVar ‘name’ “value”.

Property Variables

Property variables are not changed once they have been defined. Re-definition of property variables will be ignored. An pure assignment to a property in a test suite/case script may cause a script failure. Use function setVar instead. The name of a property must be prefixed with TTPR_ or TTPRN_

Empty values are considered a defined value for properties with prefix TTPR_ and can not be overwritten. Empty values are considered a undefined value for properties with prefix TTPRN_ and can be overwritten.

NOTE: Prefer the TTPR_ version for varables which can have different values. NOTE: The TTPRN_ version is used as switch which can be used only once to become true.

Simple Global Variables and Global Read-only Variables

Global variables may be defined in the script code section of the test artifacts. Simple variables and can be re-written in suite- or test-case-script and must have the prefix TT_. Read-only variables can not be re-written once they have been defined and must have the prefix TTRO_. In script code use function setVar to define such a variable. To re-write a global variable (TT_) a plain assignment is sufficient. A re-write of an read-only variable will cause a script/test failure.

Trueness and Falseness

Logical variables with the semantics of an boolean are considered ‘true’ if these variables are set to something different than the empty value (null). An empty (null) variable or an unset variable is considered ‘false’. Care must be taken if a variable is unset. In general the usage of an unset variable will cause a script failure. Use function ‘isExisting’ or ‘isNotExisting’ to avoid script aborts.

Some properties are designed that the existence of the property indicates the trueness.

Accepted Environment

Debug and Verbose

The testframe may print verbose information and debug information or both. The verbosity may be enabled with command line options. Additionally the verbosity can be controlled with existence of the properties:

NOTE: The check if an existing variable is empty or not is much faster then the check against existance of an variable. Therefore we use here the empty value an consider it as unset property.

Variables Used

Variables Provided

Special Script Execution Options

To maintain the correctness of the test execution all scripts are executed with special options set:

errexit: Exit immediately if a pipeline (which may consist of a single simple command), a sub-shell command enclosed in parentheses, or one of the commands executed as part of a command list enclosed by braces exits with a non-zero status. The shell does not exit if the command that fails is part of the command list immediately following a while or until keyword, part of the test following the if or elif reserved words, part of any command executed in a && or || list except the command following the final && or ||, any command in a pipeline but the last, or if the command’s return value is being inverted with !.

pipefail: If set, the return value of a pipeline is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands in the pipeline exit successfully.

posix: Change the behavior of bash where the default operation differs from the POSIX standard to match the standard.

nounset: Treat unset variables and parameters other than the special parameters “@” and “*” as an error when performing parameter expansion. If expansion is attempted on an unset variable or parameter, the shell prints an error message, and exits with a non-zero status.

nullglob: bash allows patterns which match no files to expand to a null string, rather than themselves.

globstar: The pattern ** used in a path-name expansion context will match all files and zero or more directories and sub-directories. If the pattern is followed by a /, only directories and sub-directories match

Note: The options errexit and nounset are disabled during test case and test suite finalization.

If a test case requires the execution of a command that fails intentionally, you should use one of the functions: echoExecuteAndIntercept - echo command and parameters; execute command guarded; return value in TTTT_result echoExecuteAndIntercept2, echoExecuteInterceptAndSuccess, - echo command and parameters; execute command guarded; return value in TTTT_result; expect cmd success set failure otherwise echoExecuteInterceptAndError, - echo command and parameters; execute command guarded; return value in TTTT_result; expect cmd error set failure otherwise executeAndLog - echo command and parameters; execute command guarded; return value in TTTT_result; - save command output into file executeLogAndSuccess - echo command and parameters; execute command guarded; return value in TTTT_result; save command output into file; expect cmd success set failure otherwise executeLogAndError - echo command and parameters; execute command guarded; return value in TTTT_result; save command output into file; expect cmd error set failure otherwise compileAndIntercept, compileInterceptAndSuccess, compileInterceptAndError, submitJobAndIntercept.

Test Case Result Failures and Errors

All test case preparation steps must return success. If an case preparation step returns a non zero value, the execution stopps immediately and the all case finalization steps are executed. The case is reported as an error case.

All test case steps must return success. If an case step returns a non zero value, the execution stopps immediately and all case finalization steps are executed. The case is reported as error case.

During test case step excution one can set an arbitrary failure text with function setFailure ‘failure test’. This prevents further test steps from execution and the finalization steps are executed. The case is reported as failure case. (The triggering of an failure during test case preparation is possible but not intended.)

If a test case function is returns a non zero return code the case is counted as error.

To signal the success of a test case just leave all test preparation steps and test steps with success ‘return 0’.

The test frame environment attempts to execute the test finalization functions in case of error and in case of failure. During execution of case finalization, the bash options errexit and nounset are unset. Thus errors during command execution are ignored and access to non existing variables return the null value. Failure conditions set during finalization are ignored.

Test Suite Result and Errors

All test suite preparation steps must return success. If an suite preparation step returns a non zero value, the execution stopps immediately and the all suite finalization steps are executed. The suite is reported as an error suite.

All test cases of the suite are executed, failures and errors are counted. Depending on the setup, the test case execution may happen in parallel.

All sub suites are executed, errors are counted.

The test frame environment attempts to execute the test suite finalization functions in any case. During execution of case finalization, the bash options errexit and nounset are unset. Thus errors during command execution are ignored and access to non existing variables return the null value.

Failure conditions set during test suite execution is not possible and triggers an error.

Skip Test Cases - Category Control

A test case or a test suite is skipped if the function setSkip is called during initialization phase of the artifact. This function sets the property TTPRN_skip with the supplied non-empty reason string.

A test case is skipped if the property TTPRN_skip is defined and not empty. This property may be set :

Alternatively the existence of an file SKIP in the Test Case/Suite directory inhibits the execution of all variants of the Case/Suite.

The function ‘setCategory’ defines the categories of a Test Case or Test Suite. If the function ‘setCategory’ is not called during Case initialization the Case has the default category ‘default’. If the function ‘setCategory’ is not called during Suite initialization the Suite has no category. If the function ‘setCategory’ is called with an empty parameter list, all catagories are cleaned. The categories are checked before the Case- or Suite- preparation is executed. The run-categories of the a test run can be defined with command line parameter -c|–category VALUE. The run-categories are considered to be patterns. If one of the run-category pattern matches any of the categories of the artifact, the Case/Suite is executed. Otherwise it is skipped. A test Case or Suite without a defined categorie is always executed independently from the run-categories. If no run-category pattern is entered, all Cases and Suite are executed, regardless of the defined categories. If the run-caegory ‘default’ is specified, all Cases and Suites are executed that have no explicit category set.

Category Control is disables if a test case wildcard list was entered from command line.

Sequence Control

The Test Collection, each Test Suite variant and each Test Case variant are executed in an own environment. The global variables and properties (TT.. variables) are inherited from Test Collection to Suite and to Case.

The test execution is done in the following order:

The Test Collection:

The Test Suite:

The Test Case