AI-powered surgical training programme for medical students

As medical students conduct surgical exercises, the AI software scans a live video feed and provides immediate, personalised feedback for them.

The tool is being developed by associate professor Usman Roshan from Ying Wu College of Computing’s Department of Data Science at the New Jersey Institute of Technology, with colleagues from Robert Wood Johnson Medical School (RWJ) and Robust AI, a software company focused on AI-powered human activity recognition products.

Roshan, collaborating with transplant surgeon and director of surgical simulation at RWJ, Dr Advaith Bongu, and AI engineer Yunzhe Xue from Robust AI, have been developing the platform since 2023 and are refining it for student training at RWJ, with an expectation of having it embedded to the curriculum in 2025.

The researchers said that simulation has become an accepted part of the surgical educational curriculum. Surgical trainees develop laparoscopic skills over time and have to gain a Fundamentals of Laparoscopic Surgery (FLS) certification before graduation.

These simulations, while cost-effective and safe for patients, lack an automatic evaluation component and require constant supervision and manual feedback.

Laparoscopic surgery, also known as minimally invasive surgery, is performed through small incisions using specialised instruments and a tiny camera called a laparoscope.

In a statement, Dr Bongu said: “Our focus has been on the first task of FLS, which has residents using graspers to transfer six rings from one set of pegs to another and then back again to the original set of pegs without dropping the peg with time constraints.”

The software designed and implemented by Yunzhe Xue uses an underlying single-pass object detection computer vision model, ‘You Only Look Once’, to detect the ‘surgery’ and its components. It then determines whether a student has passed or failed the simulation and provides feedback on performance and specified training direction, all in real-time video that runs entirely on a laptop that doubles as the simulation monitor.

“We’re very excited about this project since it steps outside the bounds of the generative AI hype and into the domain of helping humans learn better. We expect broader usage of our software in surgical training programs nationwide and ultimately into other areas of human learning where physical activity is involved,” said Roshan.

The research has already been awarded a Rutgers Medical Educator Innovator award and is currently being presented at surgical meetings and conferences, including the recent IEEE International Conference on Bioinformatics and Biomedicine in Istanbul at the end of 2023.