The FICO® Xpress Mosel language has evolved considerably over more than 20 years since its first release in 2001. Whilst continuously building out the available optimization features by adding support for new Solver features or new solvers (such as Constraint Programming with Xpress Kalis, Robust Optimization, Nonlinear Optimization, and most recently, Global Optimization), Mosel is increasingly used for the implementation of large-scale projects and general programming tasks. Therefore, recent releases of Xpress Mosel have built out the programming capabilities of the language.
Having looked into the ‘moseltest’ tool in the previous blog of this miniseries on Mosel tools and utilities, we now want to explore some further utilities that are available with the Mosel distribution, in particular to support the deployment and packaging of Mosel projects, along with functionality for compression and encryption. The new Mosel package ‘makeutil’ that has been released with Mosel 6.10 (Xpress Release 9.8) provides some significant simplifications for routine programming tasks. In this context, we also remind the reader how to work with system and solver console commands from within a Mosel model.
Compression (zip/tar) and encoding
The Mosel distribution has a set of functionalities for compressing data:
This sequence of commands findfiles+newzip is aggregated to a single command zip with the new makeutil package. Note that the ‘zip’ command will automatically produce some logging output recording the action that it has performed (none if archive exists and all files are unchanged), by default this will be:
And similarly, the combination of findfiles+newtar is now aggregated into a single command tar.
Binary data format and encodings
For the representation of data in memory the user has various choices:
Encryption (mmssl)
Encoding and encryption are transformations of data servicing two different purposes:
Examples of mmssl functionality:
Besides the simplified forms of zip and tar commands that we have seen earlier, the recently introduced package makeutil also provides high-level versions of typical file and directory access routines, such as creating or copying directories or individual files. Its main purpose is to ease build processes, but the routines are equally usable in other contexts.
System and Solver console commands
In addition to its built-in system functionality, Mosel allows developers to work with system commands to invoke other software from a Mosel model. For example, with the following command we can execute a Mosel model from within another model, also passing some runtime parameter settings – contrary to the compile/load/run commands of mmjobs, model runs started this way are not treated as submodels, they inherit system environment settings and any additional settings made via the setenv routine, which might be useful in the context of testing systems. Note: after the execution of a system command, Mosel programs should always invoke getsysstat in order to check whether the command was executed successfully:
Calls to external software can be combined with in-memory data transfer through memory pipes (see pipe) as shown in this example that outputs result data to gnuplot for creating a PNG or PDF format graphic (optgrid.mos).
The subroutine command of the mmxprs module makes it possible to embed Xpress Solver console commands into a Mosel model, complementing the functionality that is directly available through mmxprs. For example, after an LP problem solve in Mosel has reported infeasibility, the following sequence could be used to generate IIS and write out the first IIS to a text file (see the documentation of the iis command for further details) – note the use of backquotes to specify a multiline string:
The mmxprs ‘command’ functionality also allows users to run matrices through Mosel (see this example).
Security aspect of system commands: the execution of Mosel models can be made secure by applying restrictions, such as limiting file access to the working directory, disabling file system read or write access entirely, or entirely disabling the execution of system commands (see the discussion of restrictions in the Mosel Language Reference Manual). For the latter case, exceptions can be stated to explicitly allow the execution of specific external programs. All security-related setup of Mosel is configurable via environment variables (these settings cannot be changed from the currently executing model itself).
Deployment and packaging: bpack, mosdeploy
The standard way of distributing Mosel programs to end users consists in distributing compiled Mosel files (BIM format files). This file format has the advantage of being portable across platforms, the binary format protects your IP and it also supports increased security through optional encryption and signing of the compiled code.
However, applications rarely consist of just a single file – what if you wish to package some additional files along with your BIM file? One way is to include such additional files directly into the BIM file via the ‘datablock’ functionality, such as demonstrated through the bpack utility. When compiling the model source, the contents of the ‘datablock’ gets included into the generated binary model file, and it is extracted when running this model.
Alternatively, Mosel’s ‘deploy’ module can be used to generate executables, optionally including additional files. An example is the compilation command of Mosel’s online doc generation component ‘moseldoc’ (open source at https://github.com/fico-xpress/mosel, and also included readily compiled as executable in Xpress distributions):
With this command, the compiled model along with some HTML style files (included via ‘datablock’) is embedded into a C program for which an executable is generated. Note that the I/O driver deploy.exe requires a C compiler to be installed. In the absence of a C compiler, compiled Mosel models can be turned into self-extracting shell scripts with the help of the mosdeploy.mos utility.
Conclusion
This blog about Mosel utilities for compression, encryption, build processes and deployment forms part of a miniseries of blogs about Mosel tools, utilities and system functionality available through Mosel. The first blog in this series discussed the Mosel testing tool. Look out for further installments of this miniseries where we plan to cover Mosel functionality for preprocessing, code analysis and debugging.
To try out Mosel yourself: download the free Community Edition of FICO Xpress or (NEW!) run Mosel examples online via Python Notebooks.
Subscribe to the FICO Xpress Optimization Blog for more content on Mosel, optimization, and the latest Xpress features.
Having looked into the ‘moseltest’ tool in the previous blog of this miniseries on Mosel tools and utilities, we now want to explore some further utilities that are available with the Mosel distribution, in particular to support the deployment and packaging of Mosel projects, along with functionality for compression and encryption. The new Mosel package ‘makeutil’ that has been released with Mosel 6.10 (Xpress Release 9.8) provides some significant simplifications for routine programming tasks. In this context, we also remind the reader how to work with system and solver console commands from within a Mosel model.
Compression (zip/tar) and encoding
The Mosel distribution has a set of functionalities for compressing data:
- zip and tar commands have been available through mmsystem since early Mosel releases, with newly added aggregated forms through makeutil.
- Several compression I/O drivers are made available through the zlib module – the whitepaper on Mosel I/O drivers documents a detailed example how further compression libraries can be added to the Mosel language via a user module, the corresponding module code is available here.
This sequence of commands findfiles+newzip is aggregated to a single command zip with the new makeutil package. Note that the ‘zip’ command will automatically produce some logging output recording the action that it has performed (none if archive exists and all files are unchanged), by default this will be:
And similarly, the combination of findfiles+newtar is now aggregated into a single command tar.
Binary data format and encodings
For the representation of data in memory the user has various choices:
- Text-format data:
- Model entity data can simply be output ‘as is’, using Mosel’s default text format. Some care needs to be taken with respect to floating point values (see parameters realfmt and txtztol).
- Alternatively, using Mosel’s binary format (I/O driver ‘bin:’) results in a more compact representation and preserves full precision of floating point values. Mosel’s C, Java, and Python APIs also support this binary format to ease the exchange of data in memory between a model and its host program: some examples are discussed in the Mosel I/O whitepaper.
- Binary data (such as the output of a compression algorithm like the previous example):
- Keeping the original binary format results in the most compact representation, however, for storing or transmitting binary information it might sometimes be preferrable to transform the encoding to a more restrictive set of ASCII characters: this can be achieved by applying the hexadecimal or base64 encodings of the module mmssl (I/O drivers mmssl.hex and mmssl.base64) or ascii85 encoding provided by the module deploy (I/O driver deploy.ascii85) – note that ascii85 encoding generally requires fewer characters/space than the two other encodings, in particular, this is the encoding used by the PDF document standard.
Encryption (mmssl)
Encoding and encryption are transformations of data servicing two different purposes:
- Encoding denotes the transformation of data into another format for processing, transmitting, or storing it more efficiently.
- Encryption transforms readable data into an unreadable format, ensuring that only authorized parties can access the original data; thus maintaining data security, integrity, and confidentiality.
Examples of mmssl functionality:
- Creating a hash value for a file to validate that it remains unchanged:
- Signing a file in order to control its origin:
- Simple encryption with a password (file):
- Create a new RSA key pair and apply this key to produce an encrypted compiled model: a model file can be compiled using a list of public keys, with the effect that only the owners of one of the corresponding private keys will be able to load and run the resulting BIM file. (These compilation options are also available through the Mosel APIs and from the Mosel command line):
- BIM files can also be signed, with a signature check on loading the model file in order to control their origin (loading fails if signature is not validated).
Besides the simplified forms of zip and tar commands that we have seen earlier, the recently introduced package makeutil also provides high-level versions of typical file and directory access routines, such as creating or copying directories or individual files. Its main purpose is to ease build processes, but the routines are equally usable in other contexts.
- Compilation of models, packages, and modules (DSOs): bim2bat, bim2exe, bim2sh, c2dso, c2exe, c2obj, mos2bat, mos2bim, mos2exe, mos2run, mos2sh, obj2dso, obj2exe
- Execution of commands and Mosel models: cmdsubst, runcmd, runcmddir, runmos, runmosdir
- Handling files and directories: basename, cat, cleandirs, cleanfiles, cp, dirname, findexec, fixdst, fregreplace, fsort
- Configuration of environment variables: addenvpath, delenvpath, dumpconfig
- Accessing information about files and directories: checkdir, checkfile, getbimdeps, getbimversion, isbimfile, isdir, isemptydir, islink, isnewer, isregfile, issubdir, pathexists
- Compression and TAR commands: gzip, tar, zip
System and Solver console commands
In addition to its built-in system functionality, Mosel allows developers to work with system commands to invoke other software from a Mosel model. For example, with the following command we can execute a Mosel model from within another model, also passing some runtime parameter settings – contrary to the compile/load/run commands of mmjobs, model runs started this way are not treated as submodels, they inherit system environment settings and any additional settings made via the setenv routine, which might be useful in the context of testing systems. Note: after the execution of a system command, Mosel programs should always invoke getsysstat in order to check whether the command was executed successfully:
Calls to external software can be combined with in-memory data transfer through memory pipes (see pipe) as shown in this example that outputs result data to gnuplot for creating a PNG or PDF format graphic (optgrid.mos).
The subroutine command of the mmxprs module makes it possible to embed Xpress Solver console commands into a Mosel model, complementing the functionality that is directly available through mmxprs. For example, after an LP problem solve in Mosel has reported infeasibility, the following sequence could be used to generate IIS and write out the first IIS to a text file (see the documentation of the iis command for further details) – note the use of backquotes to specify a multiline string:
The mmxprs ‘command’ functionality also allows users to run matrices through Mosel (see this example).
Security aspect of system commands: the execution of Mosel models can be made secure by applying restrictions, such as limiting file access to the working directory, disabling file system read or write access entirely, or entirely disabling the execution of system commands (see the discussion of restrictions in the Mosel Language Reference Manual). For the latter case, exceptions can be stated to explicitly allow the execution of specific external programs. All security-related setup of Mosel is configurable via environment variables (these settings cannot be changed from the currently executing model itself).
Deployment and packaging: bpack, mosdeploy
The standard way of distributing Mosel programs to end users consists in distributing compiled Mosel files (BIM format files). This file format has the advantage of being portable across platforms, the binary format protects your IP and it also supports increased security through optional encryption and signing of the compiled code.
However, applications rarely consist of just a single file – what if you wish to package some additional files along with your BIM file? One way is to include such additional files directly into the BIM file via the ‘datablock’ functionality, such as demonstrated through the bpack utility. When compiling the model source, the contents of the ‘datablock’ gets included into the generated binary model file, and it is extracted when running this model.
Alternatively, Mosel’s ‘deploy’ module can be used to generate executables, optionally including additional files. An example is the compilation command of Mosel’s online doc generation component ‘moseldoc’ (open source at https://github.com/fico-xpress/mosel, and also included readily compiled as executable in Xpress distributions):
With this command, the compiled model along with some HTML style files (included via ‘datablock’) is embedded into a C program for which an executable is generated. Note that the I/O driver deploy.exe requires a C compiler to be installed. In the absence of a C compiler, compiled Mosel models can be turned into self-extracting shell scripts with the help of the mosdeploy.mos utility.
Conclusion
This blog about Mosel utilities for compression, encryption, build processes and deployment forms part of a miniseries of blogs about Mosel tools, utilities and system functionality available through Mosel. The first blog in this series discussed the Mosel testing tool. Look out for further installments of this miniseries where we plan to cover Mosel functionality for preprocessing, code analysis and debugging.
To try out Mosel yourself: download the free Community Edition of FICO Xpress or (NEW!) run Mosel examples online via Python Notebooks.
Subscribe to the FICO Xpress Optimization Blog for more content on Mosel, optimization, and the latest Xpress features.
Updated 3 months ago
Version 3.0