Augmented Reality Technology Applied to Surgical Planning in
Oncological Colorectal Surgery

Fernando Trebolle J; Perez Serrano N; Yanez Benitez C; Menal Munoz
P; Luesma Bartolome MJ; Cantarero Carmona I; Cros Montalban B; Cantalejo Díaz M; Blas Laina JL and Sanchez Margallo FM

doi:10.23880/crij-16000204

Clinical Radiology and Imaging Journal Research Article 10 min read

Augmented Reality Technology Applied to Surgical Planning in Oncological Colorectal Surgery

Fernando Trebolle J^*, Perez Serrano N, Yanez Benitez C, Menal Munoz P, Luesma Bartolome MJ, Cantarero Carmona I, Cros Montalban B, Cantalejo Díaz M, Blas Laina JL and Sanchez Margallo FM

^* Corresponding author

ISSN: 2640-2343 10.23880/crij-16000204 Received: May 31, 2023 Published: July 07, 2023

— views

32 references

4 figures

PDF

Keywords

3D Reconstruction Augmented Reality Colorectal Surgery Laparoscopic Surgery Smartphone Application

Abstract

This paper describes the application of augmented reality technology from three-dimensional colon models as a method of preoperative planning in oncological colorectal surgery. We have developed holograms of augmented reality from threedimensional anatomical models of the colon, obtained from computed tomography images (Siemens Somatom Perspective 64) with abdominal image cuts 3 mm thick. The recovery of the images was in DICOM format. The processing to achieve the threedimensional reconstruction was performed with the program OsiriX, which made a complete segmentation of the colon surface and modified the image density. 3D models were obtained of the isolated colon and in relationship with the bony structures. The smartphone Colon 3D AR application was designed (increased hyper experience- visualizer with SLAM technology) to apply augmented reality technology. An individualized hologram of augmented reality (scale 1:1) was created from each three-dimensional model. A projection with the smartphone on the patient's abdomen was made by modifying the position in the height of the reconstruction, using the pelvic bones as an anatomic reference point to calibrate the hologram's orientation. Three-dimensional reconstruction of the tumor in the preoperative plan of oncological colorectal surgery, combined with hyperreality technology, allows developing augmented reality models to improve colon anatomy knowledge and plan the surgical technique. The application is easy to use and may have advantages in preoperative colon surgery planning.

Introduction

The anatomy of the colon has been extensively studied. Seminal works have described the colon’s embryonic development [1] its different segments’ morphology, and its relationships with adjacent structures, including viscera and bones [2, 3, 4]. The colon’s radiological anatomy is well-known, assisting the study of the substantial anatomical variability of this organ [5]. Three-dimensional reconstruction technology based on radiological images, whether obtained via MDCT, nuclear magnetic resonance, or virtual colonoscopy, has been an essential development in this respect, allowing an enhanced anatomical description of the colon [6, 7, 8, 9].

Three-dimensional reconstruction improves the clinical applicability of results, helping to establish strategies for preoperative surgical planning that can facilitate the surgical approach and develop surgical techniques for this organ [7, 10, 11, 12]. It can also allow the development of virtual anatomical models and simulation models for teaching and training in specific medical techniques, such as through computer models [13], models integrated into Portable Document Format (PDF) files for virtual navigation through the reconstructed structures [14, 15], and physical models developed with current 3D printing technology [16]. Augmented reality technology plays an essential role in this field since it can allow the development of real- time simulation of the patient’s anatomy, creating virtual holograms that can be viewed on the operating table complementing the information available to the surgical team prior to the procedure [17, 18, 19].

This study aims to apply augmented reality technology from three-dimensional colon models as a preoperative planning method in oncological colorectal surgery.

Materials and Methods

The Siemens Somatom Perspective 64®-slice MDCT (Siemens Medical Solutions, Erlangen, Germany) scanner was used to obtain all images. CT sections of the abdomen were made at 3-mm thicknesses. The images were converted into DICOM format using the platforms Syngo Via® and Indra Alma 3D®. The processing and measurement software packages used were OsiriX® and Adobe Photoshop Elements 11 Editor®.

Results

The colon’s surface was manually identified in all sections, changing the image density to obtain the three- dimensional reconstruction (Figure 1).

Figure 1: Manual segmentation and image density change of the colon surface in CT slices.

Color was applied to this three-dimensional image with the software Adobe Photoshop Elements 11. 3D models were obtained of the isolated colon and in relationship with the bone structures. A second modification of the sections’ density allowed identifying the location of the tumor lesion location in the colon (Figure 2).

Figure 2: 3D reconstruction image of the colon and location of the tumor with color change.

Holograms of Augmented Reality

The smartphone Colon 3D AR application was designed (increased hyper experience- visualizer with SLAM technology) to apply augmented reality technology. A hologram of augmented reality to scale 1:1 from each three- dimensional model was obtained to make a projection of it on the abdomen of the patient by modifying the position in the height of the reconstruction, using the pelvis bone structure as an anatomic reference point to calibrate the orientation of the hologram (Figure 3).

Figure 3: Virtual projection of the hologram on the abdomen using the bone pelvis as an anatomical reference point.

A limitation of the mobile application is the inability to visualize various textures, so it is only possible to project the colon’s morphology in relation to the bone structure.

An example of a 3D reconstruction model is shown in an upper rectal tumor, with its corresponding augmented reality hologram obtained with the application Colon 3D AR and projected applied on the operating room table (Figure 4).

Figure 4: Real projection of the hologram on the patient`s abdomen placed in the operating room.

Discussion

For surgeons, the individualized evaluation of patients and the correct surgical planning improves efficiency in the operating room, reduces surgical time, and reduces the number of complications [15].

3-D technology has been an important advance for anatomical and radiological imaging. The processing of DICOM images, whether MRI or CT can obtain three- dimensional reconstructions, providing anatomical data challenging to get with conventional imaging technology. These new imaging techniques applied to the colon have superior quality and are currently considered essential for normal imaging anatomy, modified anatomy, and lesions on this organ [5, 20, 21].

There are three fundamental questions in the preoperative approach to colorectal surgery [22], the structural anatomical variability of the colon, its anatomical relationships with other organs and vascular elements, and the exact location of the lesions.

Regarding the enormous anatomical variability of the colon, its study has made it possible to determine the normal positioning and measurements of the different segments and see how factors such as age, sex, and body volume influence them [23]. 3-D analysis of colonic anatomy allows assessing its relationship with adjacent organs, with the arterial and venous elements, from any angle, being the relationship with the superior mesenteric vein being the most studied [24]. Accurately locating the lesion to be removed in the 3-D reconstructions is essential for the preoperative planning of the technique to be performed [25].

The new virtual reality or augmented reality technologies, which have been in use for years, represent a crucial advance in developing new strategies in surgical planning, training strategies, and high-fidelity simulation [17, 26, 27]. An essential requirement to develop this technology is obtaining 3-D reconstruction models and having adequate visualization and image projection systems [17].

These techniques have not flourished in abdominal surgery as much as in other surgical specialties. Neurosurgery and maxillofacial surgery use intraoperative navigation as an essential tool for planning complex surgical techniques [17, 28]. Additionally, hepatobiliary surgery and pancreatic surgery have achieved better technical refinements [17, 19, 29]. Augmented reality has made a significant contribution by improving the imaging of the anatomy, allowing surgeons to perform highly complex procedures using virtual projection on a screen of the 3-D reconstructions of the lesions [17].

Undoubtedly, the most significant advantage provided by this technology is the better planning of the surgical technique, the identification and visualization of adjacent structures in real-time, and the improvements in patient positioning and location of the ports of entry [17].

In colorectal surgery, there is a lack of experience in applying augmented reality technology, given the difficulty and complexity of obtaining 3-D reconstructions that are real and reproducible [17, 18]. There are published works concerning the identification of specific lesions proposed for trans-anal resections, however these advances have not been generalized for the rest of the colon [30, 31, 32].

Our work is based on the 3-D reconstruction models of the colon obtained from MDCT images. The standardization that we have achieved applying the reconstruction techniques designed by our group have allowed us to relate the colon with neighboring organs, the vascular structures, and above all to locate the specific site of the lesion, allowing us to improve the planning of the laparoscopic surgical technique [25].

Obtaining augmented reality holograms with the Colon 3D AR mobile application from the reconstructions obtained and bringing an immediate projection on the patient’s abdomen represents a tremendous improvement in the surgical planning as mentioned above, allowing for a safer procedure, and achieving better outcomes [17]. These elements can help decrease intraoperative complications and reduce the need for intraoperative improvisation [17].

Achieving specific improvements in the development of the application, such as the possibility of visualizing more than one texture in the hologram and specifying the method of calibration and projection on the patient can be definitive when including this practice in any colorectal surgical procedure.

As a future line of work, we propose the possibility of using this augmented reality mobile application combined with simulation systems. Both elements combined with other virtual reality technologies and developing training and learning programs can improve the training of surgeons in their daily practice in the operating room [32].

Conclusion

Three-dimensional reconstruction of the tumor in the preoperative planning of oncological colorectal surgery, combined with hyperreality technology, allows developing models of augmented reality to improve knowledge of colon anatomy knowledge and plan the surgical technique.

Institutional Review Board Statement: The study was reviewed and approved by the Clinical Research Ethics Committee of Aragon (CE PI17/O168).

References

Moore K, Persaud T, Torchia M (2008) Embriologia clinica. 8th (Edn.), Barcelona: Elsevier, Spain.
Testut L, Latarjet A (1958) Tratado de anatomia humana. 9th (Edn.), Barcelona: Salvat, Spain.
Rouviere H, Delmas A, Delmas V (2005) Anatomia humana: Descriptiva, topografica y funcional. 11th (Edn.), Barcelona: Masson, India.
Netter F (2013) Atlas of human anatomy. 5th (Edn.), Barcelona: Elsevier Masson, India.
Ramachandran I, Rodgers P, Elabassy M, Sinha R (2009) Multidetector computed tomography of the mesocolon: Review of anatomy and pathology. Curr Probl Diagn Radiol 38(2): 84-90.
Bourgouin S, Bege T, Lalonde N, Mancini J, Masson C, et al. (2012) Three-dimensional determination of variability in colon anatomy: applications for numerical modeling of the intestine. J Surg Res 178(1): 172-180.
Hirai K, Yoshinari D, Ogawa H, Nakazawa S, Takase Y, et al. (2013) Three-dimensional computed tomography for analyzing the vascular anatomy in laparoscopic surgery for right-sided colon cancer. Surg Laparosc Endosc Percutan Tech 23(6): 536-539.
Hong D, Tavanapong W, Wong J, Oh J, de Groen PC (2014) 3D Reconstruction of virtual colon structures from colonoscopy images. Computerized Medical Imaging and Graphics 38(1): 22-33.
Mark EB, Poulsen JL, Haase AM, Frokjaer JB, Schlageter V, et al. (2017) Assessment of colorectal length using the electromagnetic capsule tracking system: a comparative validation study in healthy subjects. Colorectal Dis 19(9): O350-O357.
Targarona EM, Balague C, Pernas JC, Martinez C, Berindoague R, et al. (2008) Can we predict immediate outcome after laparoscopic rectal surgery? Multivariate analysis of clinical, anatomic, and pathologic features after 3-dimensional reconstruction of the pelvic anatomy. Ann Surg 247(4): 642-649.
Killeen T, Banerjee S, Vijay V, Francis D, Warren S, et al. (2010) Magnetic resonance (MR) pelvimetry as a predictor of difficulty in laparoscopic operations for rectal cancer. Surg Endosc 24(12): 2974-2979.
Szura M, Pasternak A, Solecki R, Matyja M, Szczepanik A, et al. (2017) Accuracy of preoperative tumor localization in large bowel using 3D magnetic endoscopic imaging: Randomized clinical trial. Surg Endosc 31(5): 2089- 2095.
Trelease RB, Rosset A (2008) Transforming clinical imaging data for virtual reality learning objects. Anat Sci Educ 1(2): 50-55.
Mavar Haramija M, Prats Galino A, Mendez JA, Puigdelivoll Sanchez A, de Notaris M (2015) Interactive 3D-PDF presentations for the simulation and quantification of extended endoscopic endonasal surgical approaches. J Med Syst 39(10): 127.
Prats GA, Reina MA, Haramija MM, Puigdellivol SA, Mendez JAJ, et al. (2015) 3D interactive model of lumbar spinal structures of anesthetic interest. Clin Anat 28(2): 205-212.
Jones DB, Sung R, Weinberg C, Korelitz T, Andrews R (2016) Three-dimensional modelling may improve surgical education and clinical practice. Surg Innov 23(2): 189-195.
Vavra P, Roman J, Zonca P, Ihnat P, Nemec M, et al. (2017) Recent development of augmented reality in surgery: a review. J Healthc Eng 2017: 4574172.
Guerriero L, Quero G, Diana M, Soler L, Agnus V, et al. (2018). Virtual reality exploration and planning for precision colorectal surgery. Dis Colon Rectum 61(6): 719-723.
Quero G, Lapergola A, Soler L, Shahbaz M, Hostettler A, et al. (2019) Virtual and augmented reality in oncologic liver surgery. Surg Oncol Clin N Am 28(1): 31-44.
Kontovounisios C, Tekkis P, Bello F (2019) 3D imaging and printing in pelvic colorectal cancer: ‘The New Kid on the Block’. Techn Coloproctol 23(2): 171-173.
Madiba TE, Haffajee MR, Sikhosana MH (2008) Radiological anatomy of the sigmoid colon. Surg Radiol Anat 30(5): 409-415.
Emile SH, Wexner SD (2019) Systematic review of the applications of three‐dimensional printing in colorectal surgery. Colorectal Dis 21(3): 261-269.
Khashab MA, Pickhardt PJ, Kim DH, Rex DK (2009) Colorectal anatomy in adults at computed tomography colonography: normal distribution and the effect of age, sex, and body mass index. Endoscopy 41(8): 674-678.
Nesgaard JM, Stimec BV, Bakka AO, Edwin B, Ignjatovic D (2015) Navigating the mesentery. A comparative pre and per-operative visualization of the vascular anatomy. Colorectal Dis 17(9): 810-818.
Perez Serrano N, Fernando Trebolle J, Sanchez Margallo FM, Blanco Ramos JR, Garcia Tejero A, et al. (2020) Digital 3-dimensional virtual models in colorectal cancer and its application in surgical practice. Surg Innov 27(2): 246-247.
Izard SG, Juanes JA, Garcia Penalvo FJ, Estella JMG, Ledesma MJS, et al. (2018) Virtual reality as an educational and training tool for medicine. J Med Syst 42(3): 50.
Rizzetto F, Bernareggi A, Rantas S, Vanzulli A, Vertemati M (2020) Immersive virtual reality in surgery and medical education: diving into the future. Am J Surg 220(4): 856-857.
Ayoub A, Pulijala Y (2019) The application of virtual reality and augmented reality in oral & maxillofacial surgery. BMC Oral Health 19: 238.
Lang H, Huber T (2020) Virtual and augmented reality in liver surgery. Ann Surg 271(1): e8.
Atallah S, Larach SW, Monson JR (2016) Stereotactic navigation for TAMIS-TME. Minim Invasive Ther Allied Technol 25(5): 271-277.
Atallah S, Parra Davila E, Melani AGF, Romagnolo LG, Larach SW, et al (2019) Robotic-assisted stereotactic real-time navigation: initial clinical experience and feasibility for rectal cancer surgery. Tech Coloproctol 23(1): 53-63.
Keller DS, de Lacy FB, Hompes R (2021) Education and training in transanal endoscopic surgery and transanal total mesorectal excision. Clin Colon Rectal Surg 34(3): 163-171.

← Previous Article The Placenta: Anatomy, Pathophysiology, Pathological and Ultrasonograpic Findings Next Article → Tuberous Sclerosis with Typical Neuroimaging Findings-A Case Report