Evaluation of procedural virtual reality simulation training: a randomised controlled trial
Nicholas Raison, Patrick Harrison, London, United Kingdom; Takashige Abe, Sapporo, Japan; Abdullatif Aydin, Senthil Nathan, Shamim Khan, Kamran Ahmed, Prokar Dasgupta, London, United Kingdom
Published in the JOURNAL OF UROLOGY
INTRODUCTION AND OBJECTIVES: Improvements in virtual reality (VR) technology have enabled the development of procedural simulation training which closely replicate surgical procedures. VR simulation training has been shown to be highly effective for robotic surgical training however to-date curricula are limited to generic basic skills training. This RCT aims to compare the transfer of learning following procedural VR or standard basic skills VR training.
METHODS: Initially 25 novice surgeons underwent basic robotic skills training, completing three FRS tasks. Participants were then block randomised to standard basic VR training or procedural VR training. All training was performed on the RobotiX Mentor (3D Systems (Airport City, Israel) VR robotic simulator. Standard basic skills training comprised further training following the FRS curriculum. The procedural simulation group underwent training on the guided bladder neck dissection and guided urethrovesical anastomosis tasks, parts of radical prostatectomy training module. Both groups completed a total of at least 5 hours of training. Following training both groups underwent transfer of skills assessment on fresh frozen cadavers using a Da Vinci Xi surgical robot in a simulated operating room environment. Their performances were compared to a control group of novice training without training. All performances were video recorded and were assessed blindly post hoc by a trained expert using GEARS.
RESULTS: Baseline FRS scores were equal between the two groups (p[0.5). Subjects in both arms completed an average of 5.6 0.3 hours of training. VR training (basic or procedural) resulted in a significantly higher GEARS score than no training, (mean GEARS score 11.3 0.6 vs 8.8 2.9 p[0.002). Procedural training resulted in significantly higher GEARS score than either basic training or control (p[0.03)(Figure 1).
CONCLUSIONS: This study provides the first evidence supporting the use of procedural-based VR simulation for training robotic skills even in novice participants. It also provides further validity evidence to support the use of VR simulation and the effective transfer of learned skills.
Head-to-Head Comparison of Three Virtual-Reality Robotic Surgery Simulators
Alexandria M. Hertz, MD, Evalyn I. George, BS, Christine M. Vaccaro, DO, Timothy C. Brand, MD
Published in JSLS : Journal of the Society of Laparoendoscopic Surgeons 22:e2017.00081
BACKGROUND AND OBJECTIVES: There are several different commercially available virtual-reality robotic simulators, but very little comparative data. We compared the face and content validity of 3 robotic surgery simulators and their pricing and availability.
METHODS: Fifteen participants completed one task on each of the following: dV-Trainer (dVT; Mimic Technologies, Inc., Seattle, Washington, USA), da Vinci Skills Simulator (dVSS; Intuitive Surgical Inc., Sunnyvale, California, USA), and RobotiX Mentor (RM; 3D Systems, Rock Hill, South Carolina, USA). Participants completed previously validated face and content validity questionnaires and a demographics questionnaire. Statistical analysis was then performed on the scores.
RESULTS: Participants had a mean age of 29.6 (range, 25– 41) years. Most were surgical trainees, having performed a mean of 8.6 robotic primary surgeries. For face validity, ANOVA showed a significant difference favoring the dVSS over the dVT (P .001), and no significant difference between the RM, dVSS, and dVT. Content validity revealed similar results, with a significant difference between the dVSS and dVT (P .021), a trend toward a difference between the RM and dVT (P .092), and no difference between the dVSS and RM (P .99).
CONCLUSION: All simulators demonstrated evidence of face and content validity, with significantly higher scores for the dVSS; it is also the least costly ($80,000 for the simulator), although it is frequently unavailable because of intra-operative use. The dVT and RM have similar face and content validity, are slightly more expensive, and are readily available.
Validity assessment of a simulation module for robot-assisted thoracic lobectomy
George Whittaker, Abdullatif Aydin, Sinthuri Raveendran, Faizan Dar, Prokar Dasgupta, Kamran Ahmed
Published in Asian Cardiovascular and Thoracic Annals, November 2018
BACKGROUND:Training for robot-assisted thoracic lobectomy remains an issue, prompting the development of virtual reality simulators. Our aim was to assess the construct and face validity of a new thoracic lobectomy module on the RobotiX Mentor, a robotic surgery simulator. We also aimed to determine the acceptability and feasibility of implementation into training.
METHODS:This prospective, observational, and comparative study recruited novice (n = 16), intermediate (n = 9), and expert (n = 5) participants from King’s College London, the 25th European Conference on General Thoracic Surgery, and the Society of Robotic Surgery conference 2018. Each participant completed two familiarization tasks followed by the Guided Robotic Lobectomy module and an evaluation questionnaire. Outcome measures were compared using Mann-Whitney U tests.
RESULTS:Construct validity was demonstrated in 12/21 performance evaluation metrics. Significant differences between groups were found in all metrics including: time taken to complete module, vascular injury, respect for tissue, number of stapler firings, time instruments out of view, number of instrument collisions, and number of movements. Participants deemed aspects of the simulator (mean 3/5) and module (3/5) as realistic and rated the simulator as both acceptable (3.8/5) and feasible (3.8/5) for robotic surgical training.
CONCLUSION: Face validity, acceptability, and feasibility were established for the thoracic lobectomy module of the RobotiX Mentor simulator. Moderate evidence of construct validity was also demonstrated. With further work, this simulation module could help to reduce the initial part of the learning curve for trainees and decrease the risk of errors during live training.
Establishing objective benchmarks in robotic virtual reality simulation at the level of a competent surgeon using the RobotiX Mentor simulator
William Watkinson, Nicholas Raison, Takashige Abe, Patrick Harrison, Shamim Khan, Henk Van der Poel, Prokar Dasgupta, Kamran Ahmed
Published: Postgraduate Medical Journal, Volume 94, Issue 1111
Published Online First 6 March 2018
BACKGROUND: To establish objective benchmarks at the level of a competent robotic surgeon across different exercises and metrics for the RobotiX Mentor virtual reality (VR) simulator suitable for use within a robotic surgical training curriculum.
METHODS: This retrospective observational study analysed results from multiple data sources, all of which used the RobotiX Mentor VR simulator. 123 participants with varying experience from novice to expert completed the exercises. Competency was established as the 25th centile of the mean advanced intermediate score. Three basic skill exercises and two advanced skill exercises were used.
SETTING: King’s College London.
PARTICIPANTS: 84 Novice, 26 beginner intermediates, 9 advanced intermediates and 4 experts were used in this retrospective observational study.
RESULTS: Objective benchmarks derived from the 25th centile of the mean scores of the advanced intermediates provided suitably challenging yet also achievable targets for training surgeons. The disparity in scores was greatest for the advanced exercises. Novice surgeons are able to achieve the benchmarks across all exercises in the majority of metrics.
CONCLUSION: We have successfully created this proof-of-concept study, which requires validation in a larger cohort. Objective benchmarks obtained from the 25th centile of the mean scores of advanced intermediates provide clinically relevant benchmarks at the standard of a competent robotic surgeon that are challenging yet also attainable. That can be used within a VR training curriculum allowing participants to track and monitor their progress in a structured and progressional manner through five exercises. Providing clearly defined targets, ensuring that a universal training standard has been achieved across training surgeons.
Validity evidence for procedural competency in virtual reality robotic simulation, establishing a credible pass/fail standard for the vaginal cuff closure procedure
Lisette Hvid Hovgaard, Steven Arild Wuyts Andersen, Lars Konge, Torur Dalsgaard, Christian Rifbjerg Larsen
Published in Surgical Endoscopy, March 2018
OBJECTIVE: The use of robotic surgery for minimally invasive procedures has increased considerably over the last decade. Robotic surgery has potential advantages compared to laparoscopic surgery but also requires new skills. Using virtual reality (VR) simulation to facilitate the acquisition of these new skills could potentially benefit training of robotic surgical skills and also be a crucial step in developing a robotic surgical training curriculum. The study’s objective was to establish validity evidence for a simulation-based test for procedural competency for the vaginal cuff closure procedure that can be used in a future simulation-based, mastery learning training curriculum.
METHODS: Eleven novice gynaecological surgeons without prior robotic experience and 11 experienced gynaecological robotic surgeons (> 30 robotic procedures) were recruited. After familiarization with the VR simulator, participants completed the module ‘Guided Vaginal Cuff Closure’ six times. Validity evidence was investigated for 18 preselected simulator metrics. The internal consistency was assessed using Cronbach’s alpha and a composite score was calculated based on metrics with significant discriminative ability between the two groups. Finally, a pass/fail standard was established using the contrasting groups’ method.
RESULTS: The experienced surgeons significantly outperformed the novice surgeons on 6 of the 18 metrics. The internal consistency was 0.58 (Cronbach’s alpha). The experienced surgeons’ mean composite score for all six repetitions were significantly better than the novice surgeons’ (76.1 vs. 63.0, respectively, p < 0.001). A pass/fail standard of 75/100 was established. Four novice surgeons passed this standard (false positives) and three experienced surgeons failed (false negatives.)
CONCLUSION: Our study has gathered validity evidence for a simulation-based test for procedural robotic surgical competency in the vaginal cuff closure procedure and established a credible pass/fail standard for future proficiency-based training.
The Validation of a Novel Robot-Assisted Radical Prostatectomy Virtual Reality Module
Patrick Harrison, BSc, Nicholas Raison, MRCS, Takashige Abe, PhD, MD, William Watkinson, BSc, Faizan Dar, MBBS, Ben Challacombe, MS, FRCS(Urol), Henk Van Der Poel, MD, PhD, Muhammad Shamim Khan, FRCS(Urol), Prokar Dasgupa, MD, FEBU, FRCS(Urol), Kamran Ahmed, FRCS(Urol), Ph
Published online:September 30, 2017
OBJECTIVE: To perform the first validation of a full procedural virtual reality robotic training module and analysis of novice surgeon’s learning curves.
DESIGN: Participants completed the bladder neck dissection task and urethrovesical anastomosis task (UVA) as part of the prostatectomy module. Surgeons completed feedback questionnaires assessing the realism, content, acceptability and feasibility of the module. Novice surgeons completed a 5.5-hour training programme using both tasks.
SETTING: King’s College London, London.
PARTICIPANTS: 13 novice, 24 intermediate and 8 expert surgeons completed the validation study.
RESULTS: Realism was scored highly for BDN (mean 3.4/5) and UVA (3.74/5), as was importance of BDN (4.32/5) and UVA (4.6/5) for training. It was rated as a feasible (3.95/5) and acceptable (4/5) tool for training. Experts performed significantly better than novice group in 6 metrics in the UVA including time (p = 0.0005), distance by camera (p = 0.0010) and instrument collisions (p = 0.0033), as well as task-specific metrics such as number of unnecessary needle piercing points (p = 0.0463). In novice surgeons, a significant improvement in performance after training was seen in many metrics for both tasks. For bladder neck dissection task, this included time (p < 0.0001), instrument collisions (p = 0.0013) and total time instruments are out of view (p = 0.0251). For UVA, this included time (p = 0.0135), instrument collisions (p = 0.0066) and task-specific metrics such as injury to the urethra (p = 0.0032) and bladder (p = 0.0189).
CONCLUSIONS: Surgeons found this full procedural VR training module to be a realistic, feasible and acceptable component for a robotic surgical training programme. Construct validity was proven between expert and novice surgeons. Novice surgeons have shown a significant learning curve over 5.5 hours of training, suggesting this module could be used in a surgical curriculum for acquisition of technical skills. Further implementation of this module into the curriculum and continued analysis would be beneficial to gauge how it can be fully utilised.
A comparative analysis and guide to virtual reality robotic surgical simulators
Julian D, Tanaka A, Mattingly P, Truong M, Perez M, Smith R
WILEY, The International Journal of Medical Robotics and Computer Assisted Surgery
Accepted: 9 October 2017; DOI: 10.1002/rcs.1874
BACKGROUND: Since the US Food and Drug Administration approved robotically assisted surgical devices for human surgery in 2000, the number of surgeries utilizing this innovative technology has risen. In 2015, approximately 650 000 robot-assisted procedures were performed worldwide. Surgeons must be properly trained to safely transition to using such innovative technology. Multiple virtual reality robotic simulators are now commercially available for educational and training purposes. There is a need for comparative evaluations of these simulators to aid users in selecting an appropriate device for their purposes.
METHODS: We conducted a comparison of the design and capabilities of all dedicated simulators of the da Vinci robot – the da Vinci Skills Simulator (dVSS), dV-Trainer (dVT), Robotic Skills Simulators (RoSS) and the RobotiX Mentor. This paper provides the base specifications of the hardware and software, with an emphasis on the training capabilities of each system.
RESULTS: Each simulator contains a large number of training exercises for skills development: dVSS n = 40, dVT n = 65, RoSS n = 52, RobotiX Mentor n = 31. All four offer 3D visual images but use different display technologies. The dVSS leverages the real robotic surgical console to provide visualization, hand controls and foot pedals. The dVT, RoSS and RobotiX Mentor created simulated versions of all of these control systems. Each includes systems management services that allow instructors to collect, export and analyze the scores of students using the simulators.
CONCLUSIONS: This study provides comparative information on the four simulators’ functional capabilities. Each device offers unique advantages and capabilities for training robotic surgeons. Each has been the subject of validation experiments, which have been published in the literature. But those do not provide specific details on the capabilities of the simulators, which are necessary for an understanding sufficient to select the one best suited for an organization’s needs. This article provides comparative information to assist with that type of selection.
The Robotix Simulator: Face And Content Validation Using The Fundamentals Of Robotic Surgery (FRS) Curriculum
Ismail Omar*, James Dilley, Philip Pucher, Philip Pratt, Torath Ameen,
Justin Vale, Ara Darzi, Erik Mayer, London, United Kingdom
The Journal of Urology
Volume 197, Issue 4, Pages e700-e701 (April 2017)
INTRODUCTION AND OBJECTIVES: Robotic surgical training is poorly delivered with limited skills-based training offered to trainees. To improve this, validated high quality robotic simulators need to be developed and made available. The RobotiX simulator is a new platform which this study sought to validate in the context of a recognised robotic surgery curriculum.
METHODs: Surgeons(n¼29) with ranging robotic experience and experience with other simulators(da Vinci Backpack and Mimic) were invited to complete all 6 FRS curriculum exercises. Participant performance was scored using the Global Evaluation Assessment of Robotic skills(GEARS). Participants completed a Likert scale based face and content validity questionnaire graded as negative(1-2/5), neutral(3), or positive(4-5).
RESULTS: Overall, analysis included 27 participants. There was good concurrent GEARS score reliability (Cronbach’s Alpha 0.801) between participants performing exercises on both the RobotiX and an alternative robotic simulator.
CONCLUSIONS: The RobotiX simulator demonstrated excellent Face and Construct validity evidence both in terms of general usage and in the specific context of the FRS curriculum.(Table 2 Q1,2)
The RobotiX performed at least as well as other simulators(da Vinci Backpack/Mimic) and was found to simulate a robotic platform such as the da Vinci Robot. (Table 2 Q4,5) Furthermore 81% of surgical trainees would recommend using the RobotiX(Table 2 Q3).
A methodological, task-based approach to Procedure-Specific Simulations training
Yaki Setty, Oren Salzman
International Journal of Computer Assisted Radiology and Surgery
December 2016, Volume 11, Issue 12, pp 2317–2324
PURPOSE: Procedure-Specific Simulations (PSS) are 3D realistic simulations that provide a platform to practice complete surgical procedures in a virtual-reality environment. While PSS have the potential to improve surgeons’ proficiency, there are no existing standards or guidelines for PSS development in a structured manner.
METHOD: We employ a unique platform inspired by game design to develop virtual reality simulations in three dimensions of urethrovesical anastomosis during radical prostatectomy. 3D visualization is supported by a stereo vision, providing a fully realistic view of the simulation. The software can be executed for any robotic surgery platform. Specifically, we tested the simulation under windows environment on the RobotiX Mentor.
RESULT: Using urethrovesical anastomosis during radical prostatectomy simulation as a representative example, we present a task-based methodological approach to PSS training. The methodology provides tasks in increasing levels of difficulty from a novice level of basic anatomy identification, to an expert level that permits testing new surgical approaches.
CONCLUSION: The modular methodology presented here can be easily extended to support more complex tasks. We foresee this methodology as a tool used to integrate PSS as a complementary training process for surgical procedures.
Validation of the RobotiX Mentor Robotic Surgery Simulator
Whittaker, A. Aydin, N. Raison, F. Kum, B. Challacombe, M.S. Khan,P. Dasgupta, K. Ahmed
MRC Centre for Transplantation, King’s College London; Department of Urology, Guy’s
and St. Thomas’ NHS Foundation Trust, London, United Kingdom
Proceedings of the Hamlyn Symposium on Medical Robotics
20-23 June 2015, Imperial College London London, UK
ISBN: 978-0-9563776-6-1 p 69-70
INTRODUCTION: With robotic-assisted surgery becoming more common practice in urology, effective training remains a challenge. There is a considerable learning curve associated with robotic training, though this has been reported as gentler than for laparoscopic training1.
Simulation has gained wide acceptance as a method of reducing the initial phase of the learning curve. The RobotiX MentorTM is a new virtual reality simulator which mimics the interface of the da Vinci® Surgical System, with integrated training modules including Fundamentals of Robotic Surgery (FRS) and Robotic Suturing. Face and content validity of the suturing module have been confirmed in a previous study2, though this was performed on a different platform. This study aims to assess face, content and construct validity of the RobotiX MentorTM. It also aims to assess its acceptability as a training tool and feasibility of its use in training.
MATERIALS AND METHODS: This prospective, observational and comparative study recruited novice (n=20), intermediate (n=15), and expert (n=11) robotic surgeons as participants from institutions
across the United Kingdom and at the 30th European Association of Urology Annual Meeting. Each participant completed nine surgical tasks across two modules on the simulator, followed by a questionnaire to evaluate subjective realism (face validity), task importance (content validity), feasibility, and acceptability. Outcome measures of novice, intermediate, and expert groups were compared using
Mann-Whitney U-tests to assess construct validity. Outcome measures of novice, intermediate, and expert groups were compared using Mann-Whitney U-tests to assess construct validity.
RESULTS: Construct validity was demonstrated in a total of 17/25 performance evaluation metrics (p<0.001). Experts performed better than intermediates in regard to time taken to complete the first (p=0.002) and second (p=0.043) module, number of instrument collisions (p=0.040), path length (p=0.049), number of cuts >2mm deep (p=0.033), average distance from suture target (p=0.015), and number of suture breakages (p=0.038). Participants determined both the simulator console and psychomotor tasks as highly realistic (mean: 3.7/5) and very important for surgical training (4.5/5), with system pedals (4.2/5) and knot tying task (4.6/5) scoring highest respectively. The simulator was also rated as an acceptable (4.3/5) tool for training and its use highly feasible (4.3/5).
CONCLUSIONS: Construct, face, and content validity were established for the RobotiX Mentor and feasibility and acceptability of incorporation into surgical training was ascertained. The RobotiX Mentor shows potential as a valuable tool for training and assessment of trainees in robotic skills and may reduce the initial learning curve if utilised as an adjunct to operating-room training. Investigation of
concurrent and predictive validity is necessary to complete validation and evaluation of learning curves
would provide insight into its value for training.
REFERENCES:  Goldstraw MA, Challacombe BJ, Patil K, Amoroso P, Dasgupta P, Kirby RS. Overcoming the challenges of robot-assisted radical prostatectomy. Prostate cancer and prostatic diseases. Mar 2012;15(1):1-7.
 Abboudi H, Khan MS, Aboumarzouk O, et al. Current status of validation for robotic surgery simulators-a systematic review. BJU international. Feb 2013;111(2):194-205. 69
From Design to Conception: An Assessment Device for Robotic Surgeons
Alyssa Tanaka, M.S.; Manuela Perez, M.D. Mireille Truong M.D., Khara Simpson M.D. Gareth Hearn, Roger Smith, Ph.D.
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2014
ABSTRACT: The daVinci Surgical System offers surgeons improved capabilities for performing complex minimally invasive procedures; however, there is no standardized assessment of robotic surgeons and a need exists to ensure that a minimal standard of care is provided to all patients. The Department of Defense and governing surgical societies convened consensus conferences to develop a national initiative, resulting in a curriculum called the Fundamentals of Robotic Surgery (FRS). FRS is comprised of an online curriculum and a psychomotor skills dome.This paper describes the production process used to create a psychomotor skills assessment device – the FRS Dome. The device was designed to measure the essential skills that are required of any robotic surgeon and to provide a basis upon which to grant or deny privileging with the robot. It was constructed to test seven tasks of manual dexterity: Docking, Ring Tower Transfer, Knot Tying, Suturing, 4th Arm Cutting, Puzzle Piece Dissection, and Energy Dissection.
The initial design of the device was created by a committee of experienced minimally invasive surgeons, with a background in testing protocols and materials. The design was rendered in computer animation, which kickstarted a prototyping effort with physical materials. These included platinum cure silicone approximating human tissue and a 3D polyjet printer for the structural framework. Usability testing was conducted and iterative modifications were made to improve ergonomics, standardization, and cost requirements. Final CAD diagrams and specifications were created and distributed to medical and simulation companies for both physical and digital manufacturing. This development process demonstrates the evolution of a simulation and a physical testing device based on international expert consensus. The specifications are open source, allowing competitive production and future iterations. The goal of this paper is to discuss how this device evolved from an idea to a manufactured product and a digital simulation.
Current status of robotic simulators in acquisition of robotic surgical skills.
Kumar A, Smith R, Patel VR.
Department of Urology, Global Robotics Institute, Florida Hospital-Celebration Health Chief Technology Officer, Florida Hospital-Nicholson Center, University of Central Florida College of Medicine, Orlando, Florida, USA.
Curr Opin Urol. 2015 Mar;25(2):168-74.
PURPOSE OF REVIEW: This article provides an overview of the current status of simulator systems in robotic surgery training curriculum, focusing on available simulators for training, their comparison, new technologies introduced in simulation focusing on concepts of training along with existing challenges and future perspectives of simulator training in robotic surgery.
RECENT FINDINGS: The different virtual reality simulators available in the market like dVSS, dVT, RoSS, ProMIS and SEP have shown face, content and construct validity in robotic skills training for novices outside the operating room. Recently, augmented reality simulators like HoST, Maestro AR and RobotiX Mentor have been introduced in robotic training providing a more realistic operating environment, emphasizing more on procedure-specific robotic training . Further, the Xperience Team Trainer, which provides training to console surgeon and bed-side assistant simultaneously, has been recently introduced to emphasize the importance of teamwork and proper coordination.
SUMMARY: Simulator training holds an important place in current robotic training curriculum of future robotic surgeons. There is a need for more procedure-specific augmented reality simulator training, utilizing advancements in computing and graphical capabilities for new innovations in simulator technology. Further studies are required to establish its cost-benefit ratio along with concurrent and predictive validity.
Further studies are required to establish its cost-benefit ratio along with concurrent and predictive validity.
Surgical suturing training with virtual reality simulation versus dry lab practice: an evaluation of performance improvement, content, and face validity
Michael J. Amirian, Samuel M. Lindner, Edouard J. Trabulsi, Costas D. Lallas
Department of Urology, Thomas Jefferson University Hospital, 1015 Walnut Street, Suite 1100, Philadelphia, PA 19107, USA
J Robotic Surg (2014) 8:329–335
The purpose of this study is to evaluate the effectiveness of virtual reality (VR) simulation versus dry lab suturing practice at improving suturing performance in robotic surgery. Nineteen novice participants with no prior robotic suturing experience were randomized to two groups, VR simulation and dry lab, which consisted of inanimate training on a da Vinci Si surgical system. Each group underwent baseline suturing evaluation, then trained on the Simbionix™ Suturing Module (SSM) or undertook suturing practice using the da Vinci Surgical System in a dry lab. Final suturing performance was evaluated using the objective suture scoring method. Participants in the VR simulation group were surveyed to assess the face and content validity of the SSM. Both groups experienced significant improvement after training (VR simulation group p = 0.0078; dry lab group p = 0.0039). There was no significant difference in improvement between the two groups after undergoing training with either SSM or in the dry lab. Improvements in composite timing scores were 123 and 172 in the VR simulation and dry lab test groups, respectively (p = 0.36). Face validation varied with respect to the category assessed, but participants confirmed content validity of the SSM in all categories. In this sample of novice operators, there was no significant advantage in training with VR simulation using the SSM over dry lab training in improving suturing performance. Users of the SSM found it useful and relevant as a training tool for improving suturing performance.