QnA: Professor Powell is answering our questions
If you missed our webinar with Professor Powell last week, don’t worry. Here you can find the presentation used by Prof Powell during the webinar!
Also, a second session with him is being organised soon! We will follow up about it in the new year calendar!
Until then, you can read the Q&A below where the professor is responding to the questions that were imposed on him during the webinar! Enjoy!
How can we use digital twins together with sequential decision-making algorithms efficiently? What are the main considerations when selecting appropriate sequential decision-making algorithms to work in tandem with digital twins? Are there specific algorithms or approaches that tend to perform well in different application domains or scenarios?
> A digital twin in a simulator that simulates a real system. Real systems involve on-line learning, which means we want to optimize the cumulative costs/contributions. In a simulator, we can do iterative simulations to optimize policies or parameters. During these simulations, we do not care about how well we do, as long as we get a good design in the end. This is also a sequential decision problem but would use a final reward objective.
How can we assess the quality of the control decisions given by a sequential decision-making algorithm, especially when the training data/time is limited?
> Have to be clear whether you are learning in the field, so you are optimizing as you go along, or in a simulator. Even with a simulator, since these can be expensive you are still going to have a limited budget. See my chapter 7 for online learning in a derivative-free context. I give both types of objective functions, and both have finite objectives. WHen you are searching over, say, a continuous vector of parameters, you want the best value of these parameters given your budget of N simulations, and given your policy \pi (often called an algorithm), to give you the final estimate x^{N,\pi}. If N is comparatively small, then you are going to need \pi to be time dependent.
In optimizing complex systems with digital twins, how do you address the issue of model fidelity? What are some strategies or best practices for ensuring the digital twin accurately represents the physical system, especially in dynamic and uncertain environments?
> Ahh - calibration. We do this all the time at Opimal Dynamics, since our models have to mimic what a trucking company is doing. Typically you have performance metrics you are trying to match, which creates an objective function to minimize the deviations between simulated and target performance metrics. How you do the tuning depends very much on your model and the nature of the tuning parameters (discrete? continuous?). I suspect you are likely going to use derivative-free stochastic search. Chapter 7 of my book illustrates all four classes of policies for derivative-free stochastic search.
> Don't forget - your simulator is typically simulating a sequential decision problem. Tuning your simulator is *also* a sequential decision problem
When dealing with data-driven optimization and control, what strategies or techniques can be used to handle model uncertainties or changes in the underlying physical system? How does the digital twin adapt and inform the decision-making process in such situations?
> Uncertainties come in different flavors. You are typically simulating a stationary system in the presence of random inputs. But you are probably interested in optimizing a nonstationary system, which means you have to detect and adapt to changes in the underlying dynamics. We do this in our trucking problem by using a special adaptive stochastic modeling algorithm that estimates probably distributions of random activities (e.g. demands) at different levels of aggregation. We allow the baseline to move fairly quickly to pick up market changes.
Could you share insights into the computational and data requirements for implementing digital twins with sequential decision-making algorithms? How can organizations effectively manage the computational complexity while ensuring real-time or near-real-time decision-making?
> Our models of truckload fleets are updated every 5 minutes for the real-time dispatch module, less often for the load acceptance system that looks out 7-10 days to do load acceptance (not sure - we might run this every 15-30 minutes). We do a *lot* in parallel. WHen we are planning into an uncertain future, we will run 20 simulations in parallel.
> This issue of computational requirements is very problem dependent, and also very dependent on exactly how the simulator is coded and implemented.
Can you provide examples of real-world applications where digital twins have been effectively integrated with sequential decision-making algorithms? What were the key benefits and challenges encountered in these cases?
> Below is an animation of one of our fleets. This might be the fleet in a simulator (which is where we got this graphic), or it might mimic the real world as it unfolds. But it also could represent one of the 20 parallel simulations when we simulate into the future for our load acceptance module. This is the heart of what we are doing at Optimal Dynamics. My lab prepared simulators like this for other applications: Norfolk Southern Railways locomotive fleet, the intercity operations of UPS, the energy grid, (many others - my lab spent most of our time building these simulators).
