Generative AI, Machine Learning, Large Language Models (LLMs),… are topics on everyone’s tongue. For people in the mathematical optimization community, questions arise as to whether these can replace certain existing optimization methods and how Optimization and LLMs might work together on an algorithmic level and in applications.
In this blog, we demonstrate experimentation with generative AI techniques. We use a practical example, you, the reader, can use as well. We will discuss potential risks when not using these techniques responsibly. This blog is not a recommendation about the dos and don'ts of generative AI best practice, and a lot more effort would be needed to turn this prototype into a useful application.
We explore how you can enrich a decision support tool, developed with Python and Xpress Insight, with a chatbot. In a decision support tool, we provide means for business analysts and decision makers to easily compare business strategies, and make a value-based comparison of what-if scenarios to identify the best strategy. We are considering complex decisions such as finding the best pricing strategy, a cost- and CO2-optimal supply chain network design, or a marketing strategy that improves customer satisfaction and increases market share. Such practical problems typically involve mathematical optimization models.
The experiments in this blog, enable a business user / strategy analyst to talk with a chatbot about the strategies they develop – i.e., analyze the data. We focus on a simple example – the input data. We assume they are using some of our solutions based on FICO Xpress Insight. Business users do not need (or want) to know the details of the Python model behind the scenes. There are two key tasks for a developer: (i) develop a user interface for chatting, and (ii) a backend that connects with some generative AI tool. For (i), we will see how the front end can be easily developed with drag-and-drop View Designer, and for (ii), we will leverage existing Python packages (langchain) and connect to OpenAI’s API for GPT.
Note: When it comes to the technical implementation details presented in this blog, we assume the reader is familiar with the development of Python-based Insight apps. The concept itself can be followed without that knowledge. To demonstrate a possible integration with generative AI, we are using the portfolio optimization example app that can be found in the XI5devkit in folder: <xpressdir>\examples\insight\basic_apps\python\portfolio_optimization.
Let’s get started! We will pick a portfolio optimization example in which a set of shares is given with an estimated return on investment (ROI) and other characteristics such as a risk level and origin (country issued). The target is to select a subset of shares with several practical constraints related to risk and operational limits. A dummy data set of ten shares is provided in csv format. See the pictures below for the app. We extend the given app by adding a drawer. We create a file genaidrawer.vdl that will hold the chat functionality with text fields and a button to execute the model. We embed the drawer into the existing view (portfolio.vdl).
The drawer (genaidrawer.vdl) holds a text field for user input, a button “Ask chatbot” to execute the Python model and to call the LLM, and a text area with the answer. To show the response in the UI, we have added the entity “UIResponse” (a string) to the model and linked it to the text area to make the response persistent.
To embed the drawer and allow for it to be opened with a button, you just need a few code snippets - easy enough to do with View Designer, still we are showing the code here to show how the view elements are linked to each other.
<vdl-action-group name="closeDrawer">
<vdl-action-drawer drawer-id="my-drawer" action="close"></vdl-action-drawer
</vdl-action-group>
<vdl-button vdl-event="click:actions.openDrawer" label="Open Messages"></vdl-button>
<vdl-page type="drawer" id="my-drawer">
<vdl-include src="genaidrawer.vdl"></vdl-include>
</vdl-page>
From the perspective of an Xpress Insight app developer, changing the UI is easy. The Python backend is more involved. When asking any chatbot, one gets recommended to use langchain – available as a Python package and a framework to integrate with large language models. Lots of blogs and online videos suggest how easy it is to use. So, let’s experiment with it to familiarize ourselves!
Based on the documentation and some blogs, you just need to import a few packages and add a few lines of code to our Python script. Sounds easy. Most blogs also suggest creating an OpenAI account and using one of the subscription-based LLMs to get responses of decent quality. Simple enough. So, here are a few lines of code that need to be added to a custom run mode – they are taken from langchain documentation and adjusted to fit the purposes of this example which reads the input file shares.csv:
# Python package imports
from langchain.agents.agent_types import AgentType
from langchain_experimental.agents.agent_toolkits import create_csv_agent
from langchain_openai import ChatOpenAI, OpenAI
# Entities for UI interactions
UIRequest : xi.types.Scalar(default="")
UIResponse : xi.types.Scalar(default="", update_after_execution=True)
# initiate LLM with data set
agent = create_csv_agent(
OpenAI(temperature=0),
"shares.csv",
verbose=True,
agent_type=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
)
# invoke LLM
response = agent.invoke(self.UIRequest)
self.UIResponse = response['output']
By using the langchain library, two Python statements suffice to get from a question (request) to an answer (response). The backend is surely much more involved. See their code and documentation links on GitHub to learn more: https://github.com/langchain-ai/langchain
By default, OpenAI’s chat-gpt-3.5-turbo is being used as the LLM. You will need an OpenAI account and supply the OPENAI_API_KEY to your system environment variables.
Well, all quickly set up. If you think the approach is limited, this is just for the sake of brevity here.
You may also use multiple data sources or directly supply Python dataframes. You can take control of the intermediate steps of this invocation.
When reading more about langchain and Python code generation and execution, there should be some caution! The underlying LLM generates Python code and executes it. There is no guarantee of its secure execution. Langchain provides documentation around security best practices – none of which are implemented here. Without guardrails, we should better be aware of risks such as data loss, corruption or leakage on the system it is running on.
To keep the scope of this blog narrow, let us turn back to our goal which is to allow a business expert to chat in a decision support system – in our portfolio optimization app. We have the UI and the model. Let’s get started. In the portfolio example, a user can now load data from an input file (shares.csv), open the drawer, write a question and then click the button to get a response from a chat bot. First try: Which share has the highest Return and what is the value? Answer: The share with the highest Return is Software with a value of 31.0.
Great, it works. Correct answer. Really promising
You may try a few more involved questions, e.g., what are the top three shares with low risk, or try to calculate the sum over a subset of the shares, it just works. Nice! The potential seems huge. Next, you might want to ask the chat bot to run a what-if scenario analysis and write a comparison about multiple scenarios. Finally, a user might make a decision based on that summary. Seems simple enough by just providing data. But caution! When I demo’ed the second time, it suddenly gave a different answer, no longer was “Software” the highest return, but “Electronics”. Maybe I edited the data, fine, great, difference explained!
So, what happened? You can inspect the log and see the differences between both questions:
Stepping back from the comparison for a moment. A chatbot with ‘thoughts’, ‘actions’, ‘action input’,… that all seems very structured and understandable. Human thoughts and actions can go wrong in a similar way – can’t they... We don't know why it's chain-of-thought went wrong, which demonstrates the inherent risks.
So, what have we learned? I’d like to start with: it is a great onboarding experience when connecting and starting to use LLMs as chat bots. It is a smooth developer experience and very easy to expose such functionality to a business user in Xpress Insight – custom for each app. This would be so awesome if and only if it could not get that easily wrong. With responses that flaky and unpredictable, we need to pause for caution. Would a decision support tool be perceived as better and easier to use if a chat bot is included? Trust in the data and the responses is key. When trusting the answer and thereby making a business decision, it can have a serious business impact. In this portfolio optimization example, a solution with about one-third of the return on investment would have been selected. More thorough testing would be needed to uncover such unexpected behavior and more research is needed to understand why this situation occurred and find counter measures that can be applied generically under such circumstances.
Further, we have ignored any security concerns in this blog so far. Python code is being generated and executed. Access was unrestricted, the code being executed may have corrupted other data sources or leaked it. There is a lot to consider before blindly bringing this approach into production – or running it on your company hardware.
If you want to learn more about FICO’s take on generative AI, and what responsible AI is all about, we recommend to follow Scott Zoldi on LinkedIn as well as follow our upcoming blogs. Or join us at FICO World 2024 from April 15-18 and such sessions:
In this blog, we demonstrate experimentation with generative AI techniques. We use a practical example, you, the reader, can use as well. We will discuss potential risks when not using these techniques responsibly. This blog is not a recommendation about the dos and don'ts of generative AI best practice, and a lot more effort would be needed to turn this prototype into a useful application.
We explore how you can enrich a decision support tool, developed with Python and Xpress Insight, with a chatbot. In a decision support tool, we provide means for business analysts and decision makers to easily compare business strategies, and make a value-based comparison of what-if scenarios to identify the best strategy. We are considering complex decisions such as finding the best pricing strategy, a cost- and CO2-optimal supply chain network design, or a marketing strategy that improves customer satisfaction and increases market share. Such practical problems typically involve mathematical optimization models.
The experiments in this blog, enable a business user / strategy analyst to talk with a chatbot about the strategies they develop – i.e., analyze the data. We focus on a simple example – the input data. We assume they are using some of our solutions based on FICO Xpress Insight. Business users do not need (or want) to know the details of the Python model behind the scenes. There are two key tasks for a developer: (i) develop a user interface for chatting, and (ii) a backend that connects with some generative AI tool. For (i), we will see how the front end can be easily developed with drag-and-drop View Designer, and for (ii), we will leverage existing Python packages (langchain) and connect to OpenAI’s API for GPT.
Note: When it comes to the technical implementation details presented in this blog, we assume the reader is familiar with the development of Python-based Insight apps. The concept itself can be followed without that knowledge. To demonstrate a possible integration with generative AI, we are using the portfolio optimization example app that can be found in the XI5devkit in folder: <xpressdir>\examples\insight\basic_apps\python\portfolio_optimization.
Let’s get started! We will pick a portfolio optimization example in which a set of shares is given with an estimated return on investment (ROI) and other characteristics such as a risk level and origin (country issued). The target is to select a subset of shares with several practical constraints related to risk and operational limits. A dummy data set of ten shares is provided in csv format. See the pictures below for the app. We extend the given app by adding a drawer. We create a file genaidrawer.vdl that will hold the chat functionality with text fields and a button to execute the model. We embed the drawer into the existing view (portfolio.vdl).
The drawer (genaidrawer.vdl) holds a text field for user input, a button “Ask chatbot” to execute the Python model and to call the LLM, and a text area with the answer. To show the response in the UI, we have added the entity “UIResponse” (a string) to the model and linked it to the text area to make the response persistent.
To embed the drawer and allow for it to be opened with a button, you just need a few code snippets - easy enough to do with View Designer, still we are showing the code here to show how the view elements are linked to each other.
<vdl-action-group name="closeDrawer">
<vdl-action-drawer drawer-id="my-drawer" action="close"></vdl-action-drawer
</vdl-action-group>
<vdl-button vdl-event="click:actions.openDrawer" label="Open Messages"></vdl-button>
<vdl-page type="drawer" id="my-drawer">
<vdl-include src="genaidrawer.vdl"></vdl-include>
</vdl-page>
From the perspective of an Xpress Insight app developer, changing the UI is easy. The Python backend is more involved. When asking any chatbot, one gets recommended to use langchain – available as a Python package and a framework to integrate with large language models. Lots of blogs and online videos suggest how easy it is to use. So, let’s experiment with it to familiarize ourselves!
Based on the documentation and some blogs, you just need to import a few packages and add a few lines of code to our Python script. Sounds easy. Most blogs also suggest creating an OpenAI account and using one of the subscription-based LLMs to get responses of decent quality. Simple enough. So, here are a few lines of code that need to be added to a custom run mode – they are taken from langchain documentation and adjusted to fit the purposes of this example which reads the input file shares.csv:
# Python package imports
from langchain.agents.agent_types import AgentType
from langchain_experimental.agents.agent_toolkits import create_csv_agent
from langchain_openai import ChatOpenAI, OpenAI
# Entities for UI interactions
UIRequest : xi.types.Scalar(default="")
UIResponse : xi.types.Scalar(default="", update_after_execution=True)
# initiate LLM with data set
agent = create_csv_agent(
OpenAI(temperature=0),
"shares.csv",
verbose=True,
agent_type=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
)
# invoke LLM
response = agent.invoke(self.UIRequest)
self.UIResponse = response['output']
By using the langchain library, two Python statements suffice to get from a question (request) to an answer (response). The backend is surely much more involved. See their code and documentation links on GitHub to learn more: https://github.com/langchain-ai/langchain
By default, OpenAI’s chat-gpt-3.5-turbo is being used as the LLM. You will need an OpenAI account and supply the OPENAI_API_KEY to your system environment variables.
Well, all quickly set up. If you think the approach is limited, this is just for the sake of brevity here.
You may also use multiple data sources or directly supply Python dataframes. You can take control of the intermediate steps of this invocation.
When reading more about langchain and Python code generation and execution, there should be some caution! The underlying LLM generates Python code and executes it. There is no guarantee of its secure execution. Langchain provides documentation around security best practices – none of which are implemented here. Without guardrails, we should better be aware of risks such as data loss, corruption or leakage on the system it is running on.
To keep the scope of this blog narrow, let us turn back to our goal which is to allow a business expert to chat in a decision support system – in our portfolio optimization app. We have the UI and the model. Let’s get started. In the portfolio example, a user can now load data from an input file (shares.csv), open the drawer, write a question and then click the button to get a response from a chat bot. First try: Which share has the highest Return and what is the value? Answer: The share with the highest Return is Software with a value of 31.0.
Great, it works. Correct answer. Really promising
You may try a few more involved questions, e.g., what are the top three shares with low risk, or try to calculate the sum over a subset of the shares, it just works. Nice! The potential seems huge. Next, you might want to ask the chat bot to run a what-if scenario analysis and write a comparison about multiple scenarios. Finally, a user might make a decision based on that summary. Seems simple enough by just providing data. But caution! When I demo’ed the second time, it suddenly gave a different answer, no longer was “Software” the highest return, but “Electronics”. Maybe I edited the data, fine, great, difference explained!
So, what happened? You can inspect the log and see the differences between both questions:
| Console log from the correct answer | Console log from the wrong answer |
| The user request: Which share has the highest Return and what is the value? > Entering new AgentExecutor chain... Thought: I need to find the maximum value in the Return column and then find the corresponding Share Ids value. Action: python_repl_ast Action Input: df['Return'].max() Observation: 31.0 Thought: Now I need to find the corresponding ShareIds value for this maximum Return value. Action: python_repl_ast Action Input: df.loc[df['Return'] == 31.0, 'ShareIds'] Observation: 6 Software Name: ShareIds, dtype: object Thought: I now know the final answer. Final Answer: The share with the highest Return is Software with a value of 31.0. | The user request: Which share has the highest Return? > Entering new AgentExecutor chain... Thought: I need to find the maximum value in the 'Return' column. Action: python_repl_ast Action Input: df['Return'].max() Observation: 31.0 Thought: I now know the final answer Final Answer: The share with the highest Return is Electronics with a value of 21. |
Stepping back from the comparison for a moment. A chatbot with ‘thoughts’, ‘actions’, ‘action input’,… that all seems very structured and understandable. Human thoughts and actions can go wrong in a similar way – can’t they... We don't know why it's chain-of-thought went wrong, which demonstrates the inherent risks.
So, what have we learned? I’d like to start with: it is a great onboarding experience when connecting and starting to use LLMs as chat bots. It is a smooth developer experience and very easy to expose such functionality to a business user in Xpress Insight – custom for each app. This would be so awesome if and only if it could not get that easily wrong. With responses that flaky and unpredictable, we need to pause for caution. Would a decision support tool be perceived as better and easier to use if a chat bot is included? Trust in the data and the responses is key. When trusting the answer and thereby making a business decision, it can have a serious business impact. In this portfolio optimization example, a solution with about one-third of the return on investment would have been selected. More thorough testing would be needed to uncover such unexpected behavior and more research is needed to understand why this situation occurred and find counter measures that can be applied generically under such circumstances.
Further, we have ignored any security concerns in this blog so far. Python code is being generated and executed. Access was unrestricted, the code being executed may have corrupted other data sources or leaked it. There is a lot to consider before blindly bringing this approach into production – or running it on your company hardware.
If you want to learn more about FICO’s take on generative AI, and what responsible AI is all about, we recommend to follow Scott Zoldi on LinkedIn as well as follow our upcoming blogs. Or join us at FICO World 2024 from April 15-18 and such sessions:
- Innovative, Responsible Use of Generative AI at FICO (Mike Gualtieri, Chris Werner, Jean-Luc Marce) April 17, 1:00 PM PT & 1:45 PM PT
- Sneak Peek: Collaborating with GenAI on FICO Platform (Chris Werner) April 17, 11:45 AM PT
- Sneak Peek: The Future of Digital Twins & Simulation with GenAI (Horia Tipi, Libin Varghese) April 17, 2:10 PM PT
- Sneak Peek: The Role of GenAI in Enterprise Optimization (Joc O'Connor) April 16, 3:30 PM PT
- Building enterprise ready Generative AI applications using Amazon Bedrock on AWS April 16, 1:15 PM PT
Updated 2 years ago
Version 3.0