Marty Mukherjee

Marty Mukherjee

PhD Student at the University of Waterloo, Department of Applied Mathematics

Publications

Manifold-Guided Lyapunov Control with Diffusion Models

Preprint, 2024

The core objective of our proposed method is to develop stabilizing control functions by identifying the closest asymptotically stable vector field relative to a predetermined manifold and adjusting the control function based on this finding. To achieve this, we employ a diffusion model trained on pairs consisting of asymptotically stable vector fields and their corresponding Lyapunov functions. Our numerical results demonstrate that this pre-trained model can achieve stabilization over previously unseen systems efficiently and rapidly, showcasing the potential of our approach in fast zero-shot control and generalizability.

Download here

Physics-Informed Neural Network Policy Iteration: Algorithms, Convergence, and Verification

International Conference on Machine Learning (ICML), 2024

We approach the problem of optimal control for nonlinear systems by employing extreme learning machines (ELM) and physics-informed neural networks (PINN) to construct successive approximations to the Generalized Hamilton-Jacobi-Bellman (GHJB) equation. We show that these approximations converge uniformly to the viscosity solution of the Hamilton-Jacobi-Bellman (HJB) equation. Our proposed algorithms, ELM policy iteration (ELM-PI) and PINN policy iteration (PINN-PI) are trained to minimize the loss function given by the GHJB equation for three different control problems and outperform existing reinforcement learning and optimal control algorithms in the literature.

Recommended citation: Meng, Y., Zhou, R., Mukherjee, A., Fitzsimmons, M., Song, C., Liu, J. (2024). "Physics-Informed Neural Network Policy Iteration: Algorithms, Convergence, and Verification." International Conference on Machine Learning https://arxiv.org/pdf/2402.10119

Denoising Diffusion Restoration Tackles Forward and Inverse Problems for the Laplace Operator

Preprint, 2024

Diffusion models have emerged as a promising class of generative models that map noisy inputs to realistic images. More recently, they have been employed to generate solutions to partial differential equations (PDEs). However, they still struggle with inverse problems in the Laplacian operator, for instance, the Poisson equation, because the eigenvalues that are large in magnitude amplify the measurement noise. This paper presents a novel approach for the inverse and forward solution of PDEs through the use of denoising diffusion restoration models (DDRM).

Download here

Enhancing Reinforcement Learning in Vision-Based Environments with Optical Flow

Journal of Computational Vision and Imaging Systems, 2023

While convolutional neural networks have been effective in extracting meaningful features from frames, the representation of motion in reinforcement learning tasks remains a challenge. We propose an approach to improve the performance of RL models in Atari environments by concatenating OF with raw image frames as input.

Recommended citation: Mukherjee, A., Liu, J. (2024). "Enhancing Reinforcement Learning in Vision-Based Environments with Optical Flow." Journal of Computational Vision and Imaging Systems, 9(1), 1–3. DOI: https://doi.org/10.15353/jcvis.v9i1.10000 /files/OpticalFlow_RL.pdf

Harmonic Control Lyapunov Barrier Functions for Constrained Optimal Control with Reach-Avoid Specifications

arXiv Preprint, 2023

This paper introduces harmonic control Lyapunov barrier functions (harmonic CLBF) that aid in constrained control problems such as reach-avoid problems. Harmonic CLBFs exploit the maximum principle that harmonic functions satisfy to encode the properties of control Lyapunov barrier functions (CLBFs). As a result, they can be initiated at the start of an experiment rather than trained based on sample trajectories.

Recommended citation: Mukherjee, A., Zhou, R., Liu, J. (2023). "Harmonic Control Lyapunov Barrier Functions for Constrained Optimal Control with Reach-Avoid Specifications." arXiv preprint arXiv:2105.11617 https://arxiv.org/abs/2310.02869

Methods and Devices for Extracting Motion Vector Data from Compressed Video Data

United States Patent US11729395B2, 2023

At a video decoder, an encoded video data for a frame of video from an input buffer is obtained. The encoded video data is decoded to obtain decoded image data for a decoded frame, where the decoding includes extracting corresponding motion vector data for the decoded frame. The decoded image data is stored in a temporary storage indexed with a given index, and the corresponding motion vector data is stored in a same or different temporary storage indexed with the given index. An output buffer indexed with the given index is filled with the decoded image data and the corresponding motion vector data stored in the respective temporary storage indexed with the given index.

Recommended citation: Sheral Kumar, Amartya Mukherjee, Seel Patel, Rui Xiang Chai, Wentao Liu, Yuanhao Yu, Yang Wang, Jin Tang. "Methods and Devices for Extracting Motion Vector Data from Compressed Video Data." US Patent 11,729,395 B2, filed November 26, 2021, issued August 15, 2023. https://patents.google.com/patent/US11729395B2/

Actor-Critic Methods using Physics-Informed Neural Networks: Control of a 1D PDE Model for Fluid-Cooled Battery Packs

ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems, 2023

This paper proposes an actor-critic algorithm for controlling the temperature of a battery pack using a cooling fluid. This is modeled by a coupled 1D partial differential equation (PDE) with a controlled advection term that determines the speed of the cooling fluid. We propose an algorithm that treats the value network as a Physics-Informed Neural Network (PINN) to solve for the continuous-time HJB equation rather than a discrete-time Bellman optimality equation, and we derive an optimal controller for the environment that we exploit to achieve optimal control.

Recommended citation: Mukherjee, A., Liu, J. (2023). "Actor-Critic Methods using Physics-Informed Neural Networks: Control of a 1D PDE Model for Fluid-Cooled Battery Packs." ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems https://openreview.net/pdf?id=0mFwZHN2FN

Bridging Physics-Informed Neural Networks with Reinforcement Learning: Hamilton-Jacobi-Bellman Proximal Policy Optimization (HJBPPO)

ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems, 2023

Our work combines the HJB equation with reinforcement learning in continuous state and action spaces to improve the training of the value network. We treat the value network as a Physics-Informed Neural Network (PINN) to solve for the HJB equation by computing its derivatives with respect to its inputs exactly. The HJBPPO algorithm shows an improved performance compared to PPO on the MuJoCo environments.

Recommended citation: Mukherjee, A., Liu, J. (2023). "Bridging Physics-Informed Neural Networks with Reinforcement Learning: Hamilton-Jacobi-Bellman Proximal Policy Optimization (HJBPPO)." ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems https://openreview.net/pdf?id=TZMUQ6mkhs

Stochastic Parameterization Using Compressed Sensing: Application to the Lorenz-96 Atmospheric Model

Tellus A: Dynamic Meteorology and Oceanography, 2022

This paper deals with the parameterization of the Lorenz-96 model with two time-scale simplified ordinary differential equations describing advection, damping and forcing. We apply compressed sensing to parameterize the unresolved variables in terms of resolved variables.

Recommended citation: Mukherjee, A., Aydogdu, Y., Ravichandran, T. and Namachchivaya, N.S. (2022). "Stochastic Parameterization Using Compressed Sensing: Application to the Lorenz-96 Atmospheric Model." Tellus A: Dynamic Meteorology and Oceanography. 74(2022), pp.300–317. http://doi.org/10.16993/tellusa.42