Managing data effectively is crucial for running applications that handle large volumes of data. It enhances performance by ensuring efficient data retrieval and processing, which is essential for maintaining high throughput. Proper data management avoids redundancies and maintains the integrity of data, reducing the risk of errors and inconsistencies that can affect application functionality. Well-managed systems can scale more easily, ensuring that applications continue to perform well even as the volume of data increases. Fast and reliable access to data also improves the overall user experience by making applications more responsive.
If you are using Xpress Insight as a user-friendly interface for building analytical models and deploying them as business applications, you may have been in the situation where a large amount of data (entities) has been populated for performing a certain execution mode that only requires part of these data. With the 5.11 release of Xpress Insight, lightweight execution modes are available via new functions (Python and Xpress Mosel) to populate and capture a partial set of model entities for scenario execution. This improves performance visibly when the scenario data set is large and the execution mode only operates on a small subset of the data, leading to a considerably lower computational effort required to execute that mode.
In this blog post, we demonstrate how to set up a Lightweight custom execution mode for the Portfolio Optimization application distributed with the Insight Developer Kit (IDK) of Xpress Insight that comes with theFICO Xpress community license. You can find the Python and Mosel versions of the original app in the respective directories of the IDK:
IDK\examples\insight\basic_apps\python\portfolio_optimization
IDK\examples\insight\basic_apps\mosel\portfolio_optimization\complete_app
Conversely, the capture method allows the developer to specify a list of entities to be saved back to the Insight scenario at the end of the current execution mode. The specified entities must be a RESULT or updateable-input entity. Again, this method cannot be used within a mode declared with clear_input=True(such as the default LOAD mode), since in those execution modes all input entities must be captured and this behavior cannot be changed.
In Python, both populate and capture methods accept an entity_filter argument, which includes a user-defined function that will be called for each entity, and entities will be captured for which the function returned True.
In Xpress Mosel, lightweight execution can be similarly configured via the insightpopulate and insightcapture subroutines. The model adaptations are equivalent to the ones described for Python, except that in Mosel there is no need to explicitly configure the model to allow partial population of entities since the user can simply state a list of entities as an optional argument to insightpopulateto indicate for which entities data needs to be retrieved.
Now we lay out a step-by-step guide on how to create a custom lightweight execution mode for the Portfolio Optimization application example app implemented with Python. In this application, a dataset of shares and corresponding fields (Expected return, if high risk, if from North America,…) is loaded before an optimization model computes the optimal budget allocation. The goal of a new custom RUN mode for our example is to calculate the average expected return for high-risk versus low-risk shares in the input dataset. We will save the calculated values in newly specified Insight entities of type RESULT, and print them in the view tab within the app. This custom mode is triggered by the app user via a new button appearing in the user interface.
Extending on the Portfolio Optimization example app, start by opening the Python source file named application.py (located in the python_source folder) using Xpress Workbench. When in editing mode, in the AppConfig decorator, set the attribute partial_populate=Trueto allow further execution modes to populate the only a subset of the model entities.
Next, declare two new scalar entities of type RESULT (named ReturnHighRisk and ReturnLowRisk) within the Insight app class, before the definition of the execution modes, to store the value for the new calculated average return amongst high-risk and low-risk share, respectively.
Thereafter, still within the Insight app class, create a new custom RUN mode as shown in the code snippet below. The xpressinsight.ExecMode decorator is used for this purpose.
Inside the new function named my_custom_run_mode, the self.insight.populate statement indicates that only the columns “Return” and “HighRisk” of the “Shares” dataframe (can be indicated either as Shares_Return or Shares.Return) should be populated for this execution mode, since that is the only data column needed for the calculation. In this case, the index set associated with the dataframe is also populated along with the entity. This call avoids populating all other unnecessary entities at the start of the execution mode, making it lightweight. After the calculation of both average returns using the Python native mean() function on a sliced dataframe, the new entities are saved back into the Insight scenario via self.insight.capture. This call avoids all other RESULT entities from being captured, that is, only the specified entity value will be updated and all other entities remain at their previous state.
Since the app has been configured to allow partial population, all RUN execution modes must explicitly populate the INPUT entities needed for that mode. In the default RUN mode (which includes the execution of the optimization model), this is done by calling the populate method as shown below. Since we need to populate all declared INPUT entities for this mode, simply call the function without passing any arguments.
At the end of the RUN execution mode, insert the ‘capture’ method to save all but the new entities ReturnHighRisk and ReturnLowRisk back to the Insight scenario so they keep their previous value, ensuring that the two RUN execution modes remain independent. The entity_filter argument is used to retrieve all entities whose name does not match either of the new entities’ name, as shown in the line highlighted in yellow in the code snippet below.
In the VDL file portfolio.vdl, located in the client_resources folder, start by creating a second VDL action group to execute the new custom mode named “CUSTOM_RUN” inside the vdl-header section.
Then, after the first row the vdl-section, add a new row and, inside a new column with the heading “Partial Results”, insert a labelled button that triggers the new execution mode and two containers. Each container includes two text components for printing the description and the value of the new entities, respectively.
Next, click on the Publish button in Xpress Workbench to publish the app to Xpress Insight and open it with your favorite web browser. After a scenario is added to the shelf and then loaded, click on the new button in the partial results section to execute the new custom mode and obtain the value of the new entities (21.4 $ and 7.0 $), as depicted at the bottom of the picture below.
At this point, all other RESULT entities are undefined. After executing the default RUN mode to populate all other RESULT entities by clicking on the “RUN OPTIMIZATION” button, one can obtain the results without affecting the values of the two new entities. Note that in case there are changes to the input data, all RESULT entities will have their values cleared, including the newly created entities.
View a video demonstration on the “Portfolio Optimization - Lightweight execution example (Python)”
In the source file named portfolio.mos (located in the source folder of the Mosel version of the Portfolio Optimization app), declare the two new RESULT entities by using the corresponding annotations, as shown below.
The new custom execution mode and its description are declared by adding the corresponding annotations to the subroutine implementing this execution mode as shown in the next code snippet. Within this subroutine the function insightpopulate is called, passing it the list of INPUT entities that are needed for the new execution mode. The function calcavg, defined separately outside of the run mode, calculates the average return for an array of shares whose HighRisk values match the intended value (true or false). Then, the function insightcapture is used to save back the new entity values to the Insight scenario.
Then, in the default RUN mode, call insightpopulate at the start of the mode…
…and insightcapture for the RESULT entities modified in this mode, as shown in the picture below.
Since the custom mode and the two new entities have been given the same names as in the Python version, the adaptations described beforehand for the VDL file portfolio.vdlremain exactly the same as what we have seen earlier for the Python version of the app. By publishing the app, you will be able to experience exactly the same behavior as with the Python version.
View a video demonstration on the “Portfolio Optimization - Lightweight execution example (Mosel)”
The source files of the example apps shown in this example can be downloaded for Python and Mosel.
Effective data management is essential for applications handling large data volumes, as it boosts performance, ensures data accuracy, and supports scalability. This leads to faster, more responsive applications and a better overall user experience. With Xpress Insight 5.11, new lightweight execution modes allow for partial data population of execution modes, significantly improving performance when only a small subset of data is needed for a certain task. This avoids loading redundant data, reducing the computational effort and promoting efficiency when handling large scenario data sets.
Stay tuned to learn more about applying the newest features of FICO Xpress Insight. Until then, you are welcome to try the Community Edition and implement this tutorial on top of the Portfolio Optimization example app that forms part of the Insight developer kit download.
If you are using Xpress Insight as a user-friendly interface for building analytical models and deploying them as business applications, you may have been in the situation where a large amount of data (entities) has been populated for performing a certain execution mode that only requires part of these data. With the 5.11 release of Xpress Insight, lightweight execution modes are available via new functions (Python and Xpress Mosel) to populate and capture a partial set of model entities for scenario execution. This improves performance visibly when the scenario data set is large and the execution mode only operates on a small subset of the data, leading to a considerably lower computational effort required to execute that mode.
In this blog post, we demonstrate how to set up a Lightweight custom execution mode for the Portfolio Optimization application distributed with the Insight Developer Kit (IDK) of Xpress Insight that comes with theFICO Xpress community license. You can find the Python and Mosel versions of the original app in the respective directories of the IDK:
IDK\examples\insight\basic_apps\python\portfolio_optimization
IDK\examples\insight\basic_apps\mosel\portfolio_optimization\complete_app
The methods populate/capture (Python) and insighpopulate/insightcapture (Mosel)
Since the Xpress Insight 5.11 release in July 2024, two new AppInterface methods are available within the xpressinsightPython package: populate and capture. The populate method allows the developer to specify which entities from the current scenario are to be populated for the current Insight app execution mode. The method can be called from any execution mode declared with clear_input=False, which is the programmed behaviour for the default RUN and custom execution modes. Additionally, the AppConfig decorator must set the partial_populate attribute to True to turn off the automatic population of all entities (default behaviour), and allow entities to be populated through the populate function instead.Conversely, the capture method allows the developer to specify a list of entities to be saved back to the Insight scenario at the end of the current execution mode. The specified entities must be a RESULT or updateable-input entity. Again, this method cannot be used within a mode declared with clear_input=True(such as the default LOAD mode), since in those execution modes all input entities must be captured and this behavior cannot be changed.
In Python, both populate and capture methods accept an entity_filter argument, which includes a user-defined function that will be called for each entity, and entities will be captured for which the function returned True.
In Xpress Mosel, lightweight execution can be similarly configured via the insightpopulate and insightcapture subroutines. The model adaptations are equivalent to the ones described for Python, except that in Mosel there is no need to explicitly configure the model to allow partial population of entities since the user can simply state a list of entities as an optional argument to insightpopulateto indicate for which entities data needs to be retrieved.
Setting up lighweight execution for the Portfolio Optimization example app (Python)
Now we lay out a step-by-step guide on how to create a custom lightweight execution mode for the Portfolio Optimization application example app implemented with Python. In this application, a dataset of shares and corresponding fields (Expected return, if high risk, if from North America,…) is loaded before an optimization model computes the optimal budget allocation. The goal of a new custom RUN mode for our example is to calculate the average expected return for high-risk versus low-risk shares in the input dataset. We will save the calculated values in newly specified Insight entities of type RESULT, and print them in the view tab within the app. This custom mode is triggered by the app user via a new button appearing in the user interface.Extending on the Portfolio Optimization example app, start by opening the Python source file named application.py (located in the python_source folder) using Xpress Workbench. When in editing mode, in the AppConfig decorator, set the attribute partial_populate=Trueto allow further execution modes to populate the only a subset of the model entities.
Next, declare two new scalar entities of type RESULT (named ReturnHighRisk and ReturnLowRisk) within the Insight app class, before the definition of the execution modes, to store the value for the new calculated average return amongst high-risk and low-risk share, respectively.
Thereafter, still within the Insight app class, create a new custom RUN mode as shown in the code snippet below. The xpressinsight.ExecMode decorator is used for this purpose.
Inside the new function named my_custom_run_mode, the self.insight.populate statement indicates that only the columns “Return” and “HighRisk” of the “Shares” dataframe (can be indicated either as Shares_Return or Shares.Return) should be populated for this execution mode, since that is the only data column needed for the calculation. In this case, the index set associated with the dataframe is also populated along with the entity. This call avoids populating all other unnecessary entities at the start of the execution mode, making it lightweight. After the calculation of both average returns using the Python native mean() function on a sliced dataframe, the new entities are saved back into the Insight scenario via self.insight.capture. This call avoids all other RESULT entities from being captured, that is, only the specified entity value will be updated and all other entities remain at their previous state.
Since the app has been configured to allow partial population, all RUN execution modes must explicitly populate the INPUT entities needed for that mode. In the default RUN mode (which includes the execution of the optimization model), this is done by calling the populate method as shown below. Since we need to populate all declared INPUT entities for this mode, simply call the function without passing any arguments.
At the end of the RUN execution mode, insert the ‘capture’ method to save all but the new entities ReturnHighRisk and ReturnLowRisk back to the Insight scenario so they keep their previous value, ensuring that the two RUN execution modes remain independent. The entity_filter argument is used to retrieve all entities whose name does not match either of the new entities’ name, as shown in the line highlighted in yellow in the code snippet below.
In the VDL file portfolio.vdl, located in the client_resources folder, start by creating a second VDL action group to execute the new custom mode named “CUSTOM_RUN” inside the vdl-header section.
Then, after the first row the vdl-section, add a new row and, inside a new column with the heading “Partial Results”, insert a labelled button that triggers the new execution mode and two containers. Each container includes two text components for printing the description and the value of the new entities, respectively.
Next, click on the Publish button in Xpress Workbench to publish the app to Xpress Insight and open it with your favorite web browser. After a scenario is added to the shelf and then loaded, click on the new button in the partial results section to execute the new custom mode and obtain the value of the new entities (21.4 $ and 7.0 $), as depicted at the bottom of the picture below.
At this point, all other RESULT entities are undefined. After executing the default RUN mode to populate all other RESULT entities by clicking on the “RUN OPTIMIZATION” button, one can obtain the results without affecting the values of the two new entities. Note that in case there are changes to the input data, all RESULT entities will have their values cleared, including the newly created entities.
View a video demonstration on the “Portfolio Optimization - Lightweight execution example (Python)”
Setting up lighweight execution for the Portfolio Optimization example app (Mosel)
In the source file named portfolio.mos (located in the source folder of the Mosel version of the Portfolio Optimization app), declare the two new RESULT entities by using the corresponding annotations, as shown below.The new custom execution mode and its description are declared by adding the corresponding annotations to the subroutine implementing this execution mode as shown in the next code snippet. Within this subroutine the function insightpopulate is called, passing it the list of INPUT entities that are needed for the new execution mode. The function calcavg, defined separately outside of the run mode, calculates the average return for an array of shares whose HighRisk values match the intended value (true or false). Then, the function insightcapture is used to save back the new entity values to the Insight scenario.
Then, in the default RUN mode, call insightpopulate at the start of the mode…
…and insightcapture for the RESULT entities modified in this mode, as shown in the picture below.
Since the custom mode and the two new entities have been given the same names as in the Python version, the adaptations described beforehand for the VDL file portfolio.vdlremain exactly the same as what we have seen earlier for the Python version of the app. By publishing the app, you will be able to experience exactly the same behavior as with the Python version.
View a video demonstration on the “Portfolio Optimization - Lightweight execution example (Mosel)”
The source files of the example apps shown in this example can be downloaded for Python and Mosel.
Effective data management is essential for applications handling large data volumes, as it boosts performance, ensures data accuracy, and supports scalability. This leads to faster, more responsive applications and a better overall user experience. With Xpress Insight 5.11, new lightweight execution modes allow for partial data population of execution modes, significantly improving performance when only a small subset of data is needed for a certain task. This avoids loading redundant data, reducing the computational effort and promoting efficiency when handling large scenario data sets.
Stay tuned to learn more about applying the newest features of FICO Xpress Insight. Until then, you are welcome to try the Community Edition and implement this tutorial on top of the Portfolio Optimization example app that forms part of the Insight developer kit download.
Updated 2 years ago
Version 3.0