CONSTRAINED DEEP REINFORCEMENT LEARNING FOR ROBOT-ASSISTED SURGICAL TRAINING WITH SOFT TISSUE DEFORMATION

Abstract

In this thesis, we address the challenges in laparoscopic surgery training by leveraging Deep Reinforcement Learning (DRL). We develop a simulation environment using the SOFA framework and SofaPython3 plugin, allowing seamless integration with Python-based machine learning algorithms. Our Gym environment interfaces with standard RL algorithms for training and evaluation in a simulated surgical environment. We present an uncertainty-aware Model-Based Greedy Search (MBGS) algorithm for constrained RL, promoting safety during DRL training. The laparoscopic surgery task is specified as a target-reaching task, guiding the robot to move the end effector to a given target point in/around the liver while stabilizing environment parameters. The ensemble of uncertainty-aware networks predicts parameters, evaluating multiple actions to select the optimal one. Experimental results show effective parameter stabilization and rapid learning. The algorithm's efficiency and compatibility with other off-policy algorithms make it a promising tool for high-safety scenarios.