The Xpress Mosel language has evolved considerably over more than 20 years since its first release in 2001. Whilst continuously building out the available optimization features by adding support for new Solver features or new solvers (such as Constraint Programming with Xpress Kalis, Robust Optimization, Nonlinear Optimization, and most recently, Global Optimization), Mosel is increasingly used for the implementation of large-scale projects and general programming tasks. Therefore, recent releases of Xpress Mosel have built out the programming capabilities of the language along with providing support for new solver features.
An important step within the development of large-scale projects is the creation of a suitable test suite that can be run regularly, preferably in a fully automated fashion to make sure that updates to the model itself or in any of the underlying software components do not cause any unintended changes to the behavior or performance of a program. For testing Mosel projects we have developed the ‘moseltest’ tool that is distributed as an open source component (see https://github.com/fico-xpress/mosel). Starting with Xpress Release 9.5 (Mosel 6.8) this tool is also included as an executable in the Xpress distribution build. In this blog we shall provide an overview of the functionality of this tool and provide guidance on its use, including some examples of how to write Mosel test programs and tests for Insight apps implemented with Mosel.
The moseltest tool
The moseltest tool is a Mosel program that provides a general framework for testing Mosel using Mosel source files, C programs or Java programs - the latter two can, for example, employ the XPRM or XPRD libraries to execute Mosel models, and through C programs it is also possible to test functionality of the Mosel Native Interface.
The moseltest tool can also be used for testing pre-compiled models, packages or applications (in the form of zip archives) and optionally collect coverage statistics during the execution of the tests.
For its execution moseltest only requires a valid Xpress installation and, if the tests are written in C or Java, the corresponding development tools (that is, C compiler and make or Java compiler and runtime respectively).
The input to moseltest is either a single file or a directory, specified via the SRCDIR parameter:
moseltest SRCDIR=mytest.mos
The specified directory will be traversed including all its subdirectories and files are processed in alphabetical order. A subset of tests can be selected via inclusion or exclusion list patterns of file names (parameters TINCL and TEXCL) or via labels (parameters ONLYLABELS and SKIPLABELS) that need to be defined via tags within the individual test files. For example, the following command will process all tests in the subdirectories ‘unit’ and ‘dataio’ under the directory ‘alltests’ that are not excluded by any pattern listed in the file ‘toexclude.txt’ and that do not define the tag ‘label’ with the value ‘odbc’:
moseltest SRCDIR=alltests TINCL='unit/*,dataio/*' TEXCL='@toexclude.txt' SKIPLABELS='odbc'
A test is either a single file (extensions .mos, .c, .java, or .mcf) or an archive (.tar, .tar.gz, .tgz or .zip) or a subdirectory with a name ending with ‘.dir’. If an archive or a ‘.dir’ subdirectory contains several executable files the main file to be executed as the test must be named 'main.mos', 'main.c', 'main.java' or 'main.mcf'. Files with the extension ‘.mcf’ are moseltestconfiguration files, that is, text files that specify a list of moseltest tags (see below). In particular, they must state the file to be run via the ‘model’ tag, typically along with several other configuration settings, such as model parameters (tag ‘parms’) to exercise different cases within a program, and possibly also other components or programs to be built or run before or after executing the test program (using tags such as ‘build’, ‘runbefore’, ‘runafter’).
Configuration options
By default, any specified test program is run and an exit status of 0 is interpreted as a success. This behaviour can be changed using tags embedded into the source program file. Tag lines are of the form:
!*! <tagname><sep><value>
Where 'tagname' is the name of the tag, 'sep' a separator (either a space, ':' or '=') and 'value' the associated value. These tag lines can be directly incorporated into Mosel files (they are valid comments) but for C and Java files they have to be included in comments. For instance:
/*
!*! skip_host:winhost
!*! build mytestpkg.bim
*/
Tags such as ‘outexpect’, ‘errexpect’, ‘exitcode’, ‘runstat’ serve to check that output or results generated by a test program take the expected values. Other tags such as ‘build’, ‘componly’, ‘module’, ‘runbefore’ configure the build or execution process of a test. It is also possible to specify a selection of target platforms for a test (for example by using ‘only_sys’, ‘skip_host’) or define labels (tag ‘label’) that serve for the selection of tests via the parameters ONLYLABELS and SKIPLABELS of the moseltest program. The latter is useful to specify a subset of tests when working on a specific feature. For a full list of available tags please refer to the section ‘moseltest tags’ in the documentation of moseltest.
Parallel execution of tests can be enabled by setting the parameter NBW to a value greater than 1, corresponding to the number of Mosel instances to use for running the tests. The concurrent execution of tests may change the order of when results become available, however, by default moseltest will delay the display of results such that the order in the report does not depend on the number of workers. This behaviour can be changed via the model parameter ASYNCOUT. Where required, the execution order of tests in a concurrent setting can be controlled via special tags such as 'after' and 'sync' that can be applied to groups of files.
Embedding Mosel tests into CI/CD pipelines
The default output by the moseltest program are reports in text format. Its output format can be adapted for embedding it into CI/CD pipelines (Teamcity, Jenkins) by configuring alternative output formats such as TeamCity Service Messages (RFMT=’TC’), TAP-14 (RFMT='TAP') or TAP-13 (RFMT='TAP13') reports as well as Junit XML reports (RFMT='JUNIT'). In all cases the report is sent to the standard output stream of the program and includes output and error logs of failing tests (or for all tests if SHOWLOGS is set). Additionally, the log size can be restricted via MAXLOGLEN. It is also possible to specify an execution time limit per test (parameter MAXWAIT).
The moseltest program can be used to produce coverage statistics for a list of Mosel files defined via the parameter COVLST. Besides outputting coverage statistics in Mosel text format (.cov files), alternative reports can be generated for the ‘lcov’ tool or in Cobertura XML file format, this selection is configured via the parameter COVREP. The following image is an example of the visualization of Mosel coverage results obtained through ‘genhtml’ of the ‘lcov’ tool. All file and directory names in this visualization are clickable, allowing users to drill down to the level of individual lines in their Mosel source code.
Writing tests
In order to be able to test individual portions of your Mosel code it is recommended that you organize your source code in a modular fashion, preferably by using libraries (Mosel packages), that is, defining subroutines for every individual task.
From your test programs, these subroutines can then be invoked with different data sets (typically configured via the subroutine arguments) in order to cover all different cases that might occur.
As shown in the program example below, Mosel ‘assert’ statements can be used in combination with moseltest tags – remember to specify option ‘keepassert’ in the test program in order to apply the assertions with standard compilation. (Note also the syntax used for the ‘outexpect’ tag: the value of the ‘outexpect’ and ‘errexpect’ tags employs regular expression syntax, any characters with special meaning therefore need to be masked with backslash characters.)
Testing Insight Mosel apps
The moseltest tool can be used for testing the Mosel portion of Insight Mosel apps. Insight apps are typically structured into subroutines that implement the execution modes, for example:
Any Insight Mosel (or Python) app code run outside of Insight will default to the Insight execution mode ‘NONE’, indicating that the model is run standalone. However, for testing other execution modes offline you can work with the mminsighttest mode as shown in the following code snippet. When using this test mode functionality, any scenario data that might be required by a particular execution mode can be provided by specifying a data input callback function via the insightsetcbpopulate subroutine.
The code snippet above shows how the testmode can be enabled and selected via a model parameter. A second parameter might serve for specifying the companion file, as shown above. The input data dump can be produced from the ‘LOAD’ mode when running the Insight app in test mode as shown in the first code snippet. That is, with moseltest we can test an Insight app explapp.zip by defining a series of .mcf files, firstly to create the required data dump:
!*! model:explapp.zip
!*! save usrinputdump.txt
!*! parms:TESTMODE=LOAD
And subsequently employ the input data dump saved during this ‘LOAD’ mode test run to perform test runs of ‘RUN’-type execution modes via a second .mcf file, such as:
!*! model:explapp.zip
!*! restore usrinputdump.txt
!*! parms:TESTMODE=runsomething,CFILE=testcfg.xml
!*! outexpect:Report uploaded
In addition to processing the complete app archive with moseltest, it is recommended to write more detailed test procedures by structuring the app source into packages that can be invoked from separate test programs to validate the specific behaviour of individual subroutines with given test data sets.
With this blog we are initiating a new miniseries of blogs about Mosel tools and system functionality available through Mosel. Look out for further installments of this miniseries where we plan to cover Mosel functionality for code analysis and debugging, tools for preprocessing, building, and deployment of Mosel programs, and support of specific system functionality such as compression or solver console commands.
Updated 1 year ago
Version 3.0