• Corpus ID: 254069545

Generalisable 3D Fabric Architecture for Streamlined Universal Multi-Dataset Medical Image Segmentation

@inproceedings{Liu2020Generalisable3F,
  title={Generalisable 3D Fabric Architecture for Streamlined Universal Multi-Dataset Medical Image Segmentation},
  author={Siyu Liu and Weiqun Dai and Craig B. Engstrom and Jurgen Fripp and Stuart Crozier and Jason A Dowling and Shekhar S. Chandra},
  year={2020}
}

• Siyu Liu, Weiqun Dai, +4 authors S. Chandra
• Published 28 June 2020
• Computer Science, Environmental Science

Data scarcity is common in deep learning models for medical image segmentation. Previous works proposed multi-dataset learning, either simultaneously or via transfer learning to expand training sets. However, medical image datasets have diverse-sized images and features, and developing a model simultaneously for multiple datasets is challenging. This work proposes Fabric Image Representation Encoding Network (FIRENet), a universal 3D architecture for simultaneous multidataset segmentation and transfer learning involving arbitrary numbers of dataset(s). To handle different-sized image and feature, a 3D fabric module is used to encapsulate many multiscale sub-architectures. An optimal combination of these subarchitectures can be implicitly learnt to best suit the target dataset(s). For diverse-scale feature extraction, a 3D extension of atrous spatial pyramid pooling (ASPP3D) is used in each fabric node for a fine-grained coverage of rich-scale image features. In the first experiment, FIRENet performed 3D universal bone segmentation of multiple musculoskeletal datasets of the human knee, shoulder and hip joints and exhibited excellent simultaneous multi-dataset segmentation performance. When tested for transfer learning, FIRENet further exhibited excellent single dataset performance (when pre-training on a prostate dataset), as well as significantly improved universal bone segmentation performance. The following experiment involves the simultaneous segmentation of the 10 Medical Segmentation Decathlon (MSD) challenge datasets. FIRENet demonstrated good multidataset segmentation results and inter-dataset adaptability of highly diverse image sizes. In both experiments, FIRENet’s streamlined multi-dataset learning with one unified network that requires no hyper-parameter tuning. Deep learning for medical image segmentation is a rapidly evolving field with the potential to enhance disease diagnosis [36] and treatment planning [43]. The effectiveness of deep learning can be largely attributed to its data-driven nature. However, this reliance on data is also a major limitation in medical image segmentation due to data scarcity. Unlike the abundance of large-scale public datasets [34], [18], [9] in 2D computer-vision tasks, (expert) labelled medical image datasets are much smaller in quantity due to several factors: • Data acquisition challenges: The acquisition of medical images can be highly specialised and resource-intensive. Voxel-wise manual annotation of 3D volumes (for segmentation) is time-intensive and subject to variable operator error. In addition, careful planning and expert contouring protocols are usually required to minimise intraand inter-rater segmentation variability. • Data fragmentation: Clinical studies involving medical imaging are typically highly focused, relatively small 3D Seg 3D CNN Decoder Brain Tumour Heart Liver Hippocampus Prostate Pancreas Lung Cancer Spleen Colon Hepatic Vessel Diverse-scale Fabric Representation Module (FRM) Shortcuts Multi-dataset Input Multi-dataset Segmentation Conv3D 3×3 3D CNN Encoder 3D Image Fig. 1: Simultaneous 3D segmentation of the 10 Medical Segmentation Decathlon (MSD) challenge datasets using only one instance of Fabric Image Representation Encoding Network (FIRENet). FIRENet uses a the latent fabric module to process diverse-sized features and images. investigations due to high imaging costs. Datasets from different studies can exhibit considerable inter-dataset variations (for example, different imaging fields of view, and dimensions) associated with different acquisition sequences and protocols. Most models lack versatility as they are highly optimised for only one dataset at a time. • Access to imaging datasets: Collecting large-scale medical image datasets with expert annotations is difficult as explicit consent, strict adherence to ethics and systematic coordination are required, as the publication of patient data, even when de-identified, is a highly sensitive matter. When facing data scarcity, multi-dataset applications including transfer learning [52], [7] and simultaneous multi-dataset ar X iv :2 00 6. 15 57 8v 4 [ ee ss .I V ] 2 9 N ov 2 02 2 learning [23]) have been shown to improve segmentation performance. However, most current deep learning methods for medical image analysis are designed for application on very few datasets. Even architectures that can adapt to different datasets datasets, like nnUNet [25], are optimised for one dataset at a time. Works that use one model to aggregate from and transfer features to many datasets remain limited. Currently, existing (and notable) works for simultaneous multi-dataset segmentation in the literature are 3D MDUNet and 3DUNet, which applied a U-Net-like architecture simultaneously to several MSD challenge datasets. However, they only demonstrated limited applicability on a subset of the available MSD datasets. To extend the coverage to more datasets with diverse features and sizes, a model with more focus on diverse-scale feature extraction and architecture-level versatility is desirable. Fabric-based architectures [57], [19], [42] are an ideal candidate for multi-dataset processing as they create a superposition of many multi-scale sub-architectures. With this inspiration, this work proposes FIRENet, a versatile 3D network that leverages a convolutional fabric for multi-dataset medical image segmentation. The central features of FIRENet are: 1) For improved adaptation to different medical image datasets, FIRENet’s uses a 3D fabric representation module (FRM) to emulate a superposition of many multi-scale sub-architectures. The major advantage of FRM is architecture-level generalisability, which is essential as different datasets contain different image and feature sizes. 2) To harness the benefits of multi-scale feature extraction for processing diverse features, FRM consists of multiscale branches of fabric nodes. Furthermore, a 3D extension of atrous spatial pyramid pooling (ASPP) [5] is employed in each FRM node to provide fine-grained coverage of different-sized features. 3) The nodes in FRM are connected via trainable feature sum (TFS) to further optimise FIRENet’s multi-dataset adaptability. These connections can be trained to automatically form an architecture from different medical image datasets. (Figure 1). 4) FIRENet’s generalisable architecture ensures no hyperparameter modification is required when applying or transferring it to multiple datasets. FIRENet’s demonstrated good performance and versatility in various transfer learning and multi-dataset segmentation tasks. The transfer learning and simultaneous multi-dataset segmentation experiments show FIRENet’s versatility in various medical image segmentation tasks, including I) standard single dataset segmentation (during pre-training on an MR prostate dataset) II) simultaneous universal (support for multi-anatomy) bone segmentation with and without transfer learning, and III) simultaneous multi-object segmentation performance for all the 10 image datasets in the MSD challenge. Notably, FIRENet also exhibits multi-dataset adaptability without tailored training procedures to reach convergence.