Machine Learning and Optimization are two disciplines that can greatly benefit from each other. This blog explores one type of integration and is brought to you by one of our expert Xpress users, Nidhi Sawhney, Principal Data Scientist at SAP. A version of it has also been published here: SAP Community Blogs.
Introduction
In many business areas the users need to find the right set of parameters or business tactics to ensure they can achieve their targets. This problem is resolved in many cases by trial and error by the very experienced users and relies on their ability to forecast or project how the changes they make have an impact on their bottom-lines. For a large part the problem of forecasting can be and is usually solved via Machine Learning (ML) trained on their historic data. However, in cases when you need to first provide the right (or "optimal") input to achieve your optimal target the problem is only partially solved. The business users still need to figure out the right set of inputs to provide to the machine learning model to get the desired forecast.
The technique described here has much broader applicability but if you are like me, it helps to see it applied for a concrete business problem. In this blog post we will address the problem by applying it to the real-world problem that occurs in Trade Promotion Optimization for Consumer Goods industry. In this context we need to find the right price reduction tactics to ensure the optimal uplift in volumes and other derived targets like net sales, profitability and ROI.
The focus of this blog post is the use of black box optimization techniques when the predicted uplift is defined by inferring from a pre-trained machine learning model. The machine learning model itself has been trained on customer specific data and understands the price-volume elasticity or in this case specifically the price reduction volume uplift elasticity.
The optimization model provides decision support to the business user (account managers in this chosen context) by exploring the space of different price points and temporary price reduction tactics like discount percentages vs absolute dollars off, ‘Buy 1 Get 1’, Multi-Buy etc. All of these plus the communication and display mechanisms have a significant impact on the projected uplifts even when the financial incentive value is the same to the end user. For example, consumer behaviour is very different if they see a promotion of 2$ off on a product normally priced at 20$ off vs 10% off. Also, the impact is influenced by whether the end user receives the communication of the promotion via a front-page ad (typically more expensive for the business but produces a much higher uplift) vs a coupon page or in-store communication when the consumer enters the outlet.
The ML infused optimization model, OptML, is deployed and served via components from SAP's Business Technology Platform to ensure performance and scalability requirements from the business and IT users at optimal TCO (total cost of ownership).
Optimization
The business problem of deciding between the innumerable price reduction tactics can be solved via a mathematical optimization model. The choice of price reduction variables can be modelled by continuous, integer or binary variables. In addition, there can be constraints on sets of price points that are acceptable, for example, prices can only be $x.49 or $x.99. These influence whether the value variable can be continuous or is restricted to integers.
Here is an example of a set of such variables
The mathematical model then has constraints to ensure the variables behave in the desired way. Here is a set of constraints to ensure the solution meets business needs.
Now we have defined the variables and constraints that can model the business problem. However, if we use typical mathematical programming techniques like Linear or Mixed Integer Programming (LP or MIP) we have to describe the objective function as a linear expression of the variables. The end user wants to find the variables which would give the maximum uplift or its derived targets. The uplift itself is known only via an ML model and is a function of the variables of the optimization model. We do not have an explicit linear model that can provide the uplift in terms of the variables. This is where Black-Box Optimization techniques come to the rescue, instead of defining the objective as a linear expression we provide it to the model via User-defined functions.
User-defined Functions
We use the user-defined functionality provided by FICO Xpress.
The library provides functionality for expressing both the objective function and additional non-linear constraints via user-defined functions.
For our use-case the user defined function needs to provide the uplift or other derived targets based on user selection as business users want to evaluate different strategies and tactics depending on the product, retailer, current market dynamics. Here is pseudo-code snippet showing the use of a user-defined function:
One thing to bear in mind when defining the user-defined function is that such a function may be called for values that result from relaxations of the constraints. In our example, binary variables are not necessarily all at integral when the obj_func callback is called, that is, we need to give the optimizer values so that it can figure out what would be a good direction to move the variables. One way to do this is via returning a linear combination of the uplift values evaluated for all binary variables. This way any fractional variables will get fraction of the uplift till the optimizer reaches optimality.
User-defined Constraints
In addition to objective function being user-defined additional constraints can also be added in a similar manner in case there are nonlinear constraints for which the user can only provide evaluations during the iterations. For example, if we wanted to ensure that a minimum profitability (which itself is a function of uplift) is guaranteed even when the user defines the target to be pure uplift, we can add an additional constraint which returns the profitability metric during the optimizer iterations and ensures it is bounded from below by a minimum.
In our example cp_func returns the same value as if the obj_func was called for target 'PROFTABILITY'.
Optimal Solutions
The optimizer evaluates different options for promotional incentives and converges on the optimal solution.
If the objective is purely uplift (incremental POS volume) or incremental Gross Sales (GES in the figure below) the optimizer chooses incentives which do not account for the additional investment required to achieve those targets.
An example optimizer solution development across the iterations and considering metrics like incremental uplift (POS), the optimal promotion is to give $3 off as discount.
However, if you now want to consider the investment needed and optimize for profitability the optimal solution might look for reduced discount that maximizes uplift as increasing discount increases the investment the manufacturer has to do by reducing his own margins and that of the retailer. The retailers would only agree to reduced prices if they can still guarantee their minimum margins, which in turn means the manufacturer has to give the retailer additional discount based on scanned units or bulk investments to ensure retailer still goes for the promotion.
For the above example, now the optimal solution is to give only $0.50 off in discount, this reduces the uplift compared to the previous solution but ensures highest profit.
The optimizer can provide one single proposal which is optimized towards a particular target or a proposal weighted across multiple targets. It can also provide a selection of top performing proposals (promotional offers in this case) and allow the business user to select the best based on their current strategy. These proposal offers are used by the business users for benefit analysis.
Benefit Analysis
Ideally the account managers want to find options that are best across all targets, but this is usually impossible as promotions with higher uplifts tend to be expensive and lower the ROI for the manufacturer and so the account managers find the best strategy for the product and retailer by analyzing the trade-offs of volumes vs profitability. The account manager needs to choose if they want to go for sales volume with first offer proposal (proposal 10 in the figure below) of ‘Save $3’ or drive profitability with second proposal of ‘Price Reduction’ (proposal 12).
In some cases the solutions are easier for decision making for the account managers, for example for the proposals below the top 2 proposals with BOGO (‘Buy 1/Get 1 Free’) and Price Reduction (with Feature Page) are competitive, with BOGO being better for Gross Sales (GES) and Price Reduction being highest on Net Sales and Profitability metrics (CP).
With the optimized decision support system, the account managers can now do the job they are really good at. They can focus on the strategic decision making instead of trying out multiple options manually while still feeling they have not really explored the whole space of options that are out there.
Having a set of optimally selected proposals that account for the price elasticity seen from their historic data the account managers can do the next step of portfolio optimization (you guessed it, it’s yet another mathematical optimization problem, perhaps for another blog post). This helps them ensure overall profitability across all the products and retailers without exceeding their allocated budgets and providing their expertise and learnings from current market trends.
The account managers can take decisions grounded with learnings from historical promotions and optimized for their target while respecting the restrictions from their business partners (for example retailer margins), budget constraints, and consumer acceptable price points.
Architecture
To allow for dynamic scalability the following architecture was chosen for the above use-case. SAP's Business Technology Platform provides the UI, persistency on HANA Cloud, and runtime via Kyma. As multiple business users need to run multiple scenarios for different products the system needs to dynamically scale up (and gracefully scale down after) during the promotional planning periods. This is enabled by scaling horizontally on Kyma as additional pods come up when the optimizer runs increase.
Conclusion
In many business areas the complete potential of data can only be unlocked by utilizing the learnings from predictive models to create prescriptive models which provide high-value decision support to the business users. This may seem a challenging exercise at the outset but with the right set of ingredients this is achievable. The maths and the technology are available to realize the benefits in a timely and cost-efficient manner. This is achieved by empowering the experts with OptML decision support that combines Machine Learning and Mathematical Optimization together.
A version of this blog post has been published here: SAP Community Blogs
Introduction
In many business areas the users need to find the right set of parameters or business tactics to ensure they can achieve their targets. This problem is resolved in many cases by trial and error by the very experienced users and relies on their ability to forecast or project how the changes they make have an impact on their bottom-lines. For a large part the problem of forecasting can be and is usually solved via Machine Learning (ML) trained on their historic data. However, in cases when you need to first provide the right (or "optimal") input to achieve your optimal target the problem is only partially solved. The business users still need to figure out the right set of inputs to provide to the machine learning model to get the desired forecast.
The technique described here has much broader applicability but if you are like me, it helps to see it applied for a concrete business problem. In this blog post we will address the problem by applying it to the real-world problem that occurs in Trade Promotion Optimization for Consumer Goods industry. In this context we need to find the right price reduction tactics to ensure the optimal uplift in volumes and other derived targets like net sales, profitability and ROI.
The focus of this blog post is the use of black box optimization techniques when the predicted uplift is defined by inferring from a pre-trained machine learning model. The machine learning model itself has been trained on customer specific data and understands the price-volume elasticity or in this case specifically the price reduction volume uplift elasticity.
The optimization model provides decision support to the business user (account managers in this chosen context) by exploring the space of different price points and temporary price reduction tactics like discount percentages vs absolute dollars off, ‘Buy 1 Get 1’, Multi-Buy etc. All of these plus the communication and display mechanisms have a significant impact on the projected uplifts even when the financial incentive value is the same to the end user. For example, consumer behaviour is very different if they see a promotion of 2$ off on a product normally priced at 20$ off vs 10% off. Also, the impact is influenced by whether the end user receives the communication of the promotion via a front-page ad (typically more expensive for the business but produces a much higher uplift) vs a coupon page or in-store communication when the consumer enters the outlet.
The ML infused optimization model, OptML, is deployed and served via components from SAP's Business Technology Platform to ensure performance and scalability requirements from the business and IT users at optimal TCO (total cost of ownership).
Optimization
The business problem of deciding between the innumerable price reduction tactics can be solved via a mathematical optimization model. The choice of price reduction variables can be modelled by continuous, integer or binary variables. In addition, there can be constraints on sets of price points that are acceptable, for example, prices can only be $x.49 or $x.99. These influence whether the value variable can be continuous or is restricted to integers.
Here is an example of a set of such variables
The mathematical model then has constraints to ensure the variables behave in the desired way. Here is a set of constraints to ensure the solution meets business needs.
Now we have defined the variables and constraints that can model the business problem. However, if we use typical mathematical programming techniques like Linear or Mixed Integer Programming (LP or MIP) we have to describe the objective function as a linear expression of the variables. The end user wants to find the variables which would give the maximum uplift or its derived targets. The uplift itself is known only via an ML model and is a function of the variables of the optimization model. We do not have an explicit linear model that can provide the uplift in terms of the variables. This is where Black-Box Optimization techniques come to the rescue, instead of defining the objective as a linear expression we provide it to the model via User-defined functions.
User-defined Functions
We use the user-defined functionality provided by FICO Xpress.
The library provides functionality for expressing both the objective function and additional non-linear constraints via user-defined functions.
For our use-case the user defined function needs to provide the uplift or other derived targets based on user selection as business users want to evaluate different strategies and tactics depending on the product, retailer, current market dynamics. Here is pseudo-code snippet showing the use of a user-defined function:
One thing to bear in mind when defining the user-defined function is that such a function may be called for values that result from relaxations of the constraints. In our example, binary variables are not necessarily all at integral when the obj_func callback is called, that is, we need to give the optimizer values so that it can figure out what would be a good direction to move the variables. One way to do this is via returning a linear combination of the uplift values evaluated for all binary variables. This way any fractional variables will get fraction of the uplift till the optimizer reaches optimality.
User-defined Constraints
In addition to objective function being user-defined additional constraints can also be added in a similar manner in case there are nonlinear constraints for which the user can only provide evaluations during the iterations. For example, if we wanted to ensure that a minimum profitability (which itself is a function of uplift) is guaranteed even when the user defines the target to be pure uplift, we can add an additional constraint which returns the profitability metric during the optimizer iterations and ensures it is bounded from below by a minimum.
In our example cp_func returns the same value as if the obj_func was called for target 'PROFTABILITY'.
Optimal Solutions
The optimizer evaluates different options for promotional incentives and converges on the optimal solution.
If the objective is purely uplift (incremental POS volume) or incremental Gross Sales (GES in the figure below) the optimizer chooses incentives which do not account for the additional investment required to achieve those targets.
An example optimizer solution development across the iterations and considering metrics like incremental uplift (POS), the optimal promotion is to give $3 off as discount.
However, if you now want to consider the investment needed and optimize for profitability the optimal solution might look for reduced discount that maximizes uplift as increasing discount increases the investment the manufacturer has to do by reducing his own margins and that of the retailer. The retailers would only agree to reduced prices if they can still guarantee their minimum margins, which in turn means the manufacturer has to give the retailer additional discount based on scanned units or bulk investments to ensure retailer still goes for the promotion.
For the above example, now the optimal solution is to give only $0.50 off in discount, this reduces the uplift compared to the previous solution but ensures highest profit.
The optimizer can provide one single proposal which is optimized towards a particular target or a proposal weighted across multiple targets. It can also provide a selection of top performing proposals (promotional offers in this case) and allow the business user to select the best based on their current strategy. These proposal offers are used by the business users for benefit analysis.
Benefit Analysis
Ideally the account managers want to find options that are best across all targets, but this is usually impossible as promotions with higher uplifts tend to be expensive and lower the ROI for the manufacturer and so the account managers find the best strategy for the product and retailer by analyzing the trade-offs of volumes vs profitability. The account manager needs to choose if they want to go for sales volume with first offer proposal (proposal 10 in the figure below) of ‘Save $3’ or drive profitability with second proposal of ‘Price Reduction’ (proposal 12).
In some cases the solutions are easier for decision making for the account managers, for example for the proposals below the top 2 proposals with BOGO (‘Buy 1/Get 1 Free’) and Price Reduction (with Feature Page) are competitive, with BOGO being better for Gross Sales (GES) and Price Reduction being highest on Net Sales and Profitability metrics (CP).
With the optimized decision support system, the account managers can now do the job they are really good at. They can focus on the strategic decision making instead of trying out multiple options manually while still feeling they have not really explored the whole space of options that are out there.
Having a set of optimally selected proposals that account for the price elasticity seen from their historic data the account managers can do the next step of portfolio optimization (you guessed it, it’s yet another mathematical optimization problem, perhaps for another blog post). This helps them ensure overall profitability across all the products and retailers without exceeding their allocated budgets and providing their expertise and learnings from current market trends.
The account managers can take decisions grounded with learnings from historical promotions and optimized for their target while respecting the restrictions from their business partners (for example retailer margins), budget constraints, and consumer acceptable price points.
Architecture
To allow for dynamic scalability the following architecture was chosen for the above use-case. SAP's Business Technology Platform provides the UI, persistency on HANA Cloud, and runtime via Kyma. As multiple business users need to run multiple scenarios for different products the system needs to dynamically scale up (and gracefully scale down after) during the promotional planning periods. This is enabled by scaling horizontally on Kyma as additional pods come up when the optimizer runs increase.
Conclusion
In many business areas the complete potential of data can only be unlocked by utilizing the learnings from predictive models to create prescriptive models which provide high-value decision support to the business users. This may seem a challenging exercise at the outset but with the right set of ingredients this is achievable. The maths and the technology are available to realize the benefits in a timely and cost-efficient manner. This is achieved by empowering the experts with OptML decision support that combines Machine Learning and Mathematical Optimization together.
A version of this blog post has been published here: SAP Community Blogs
Updated 1 year ago
Version 3.0