Deep learning based image analysis with enhanced reasoning capability

Abstract

To create high-quality hybrid models interactively based on Convolutional Neural Networks (CNNs) or by conducting cross-module learning from diverse feature representations usually suffers from the challenging task of combining multiple latency information from a sparse input, such as a semantic object, a medical image, and images with special topology. In this thesis, we use deep learning strategies to present novel algorithms for three problems: representing the basic object semantic information, creating a hybrid CNNs and Graph Neural Networks (GNNs) model for 3D nodule recognition, and learning special topological information for 3D medical vessel images.

At first, to represent the object semantic information, we proposed a novel Graph Feature Pyramid Network (GraphFPN). Feature pyramids have been proven powerful in image understanding tasks that require multi-scale features. State-of-the-art methods for multi-scale feature learning focus on performing feature interactions across space and scales using neural networks with a fixed topology. In this section, we propose graph feature pyramid networks capable of adapting their topological structures to varying intrinsic image structures and supporting simultaneous feature interactions across all scales. We first define an image-specific superpixel hierarchy for each input image to represent its intrinsic image structures. The graph feature pyramid network inherits its structure from this superpixel hierarchy. Contextual and hierarchical layers are designed to achieve feature interactions within the same scale and across different scales, respectively.

In clinical practice, doctors often use attributes, e.g. morphological and appearance characteristics of a lesion, to aid disease diagnosis. Effectively modeling all relationships among attributes could boost the accuracy of medical image diagnosis. In this section, we introduce a hybrid neural-probabilistic reasoning algorithm for interpretable attribute-based medical image diagnosis. There are two parallel branches in our hybrid algorithm, a Bayesian network branch performing probabilistic causal relationship reasoning and a graph convolutional network branch performing more generic relational modeling and reasoning using a feature representation. Tight coupling between these two branches is achieved via a cross-network attention mechanism and the fusion of their classification results.

Finally, vessel segmentation is widely used to help with vascular disease diagnosis. Vessels reconstructed using existing methods are often not sufficiently accurate to meet clinical use standards. This is because 3D vessel structures are highly complicated and exhibit unique characteristics, including sparsity and anisotropy. In this section, we propose a novel hybrid deep neural network for vessel segmentation. Our network consists of two cascaded subnetworks performing initial and refined segmentation respectively. The second subnetwork further has two tightly coupled components, a traditional CNN-based U-Net and a graph U-Net. Cross-network multi-scale feature fusion is performed between these two U-shaped networks to effectively support high-quality vessel segmentation. The entire cascaded network can be trained from end to end. The graph in the second subnetwork is constructed according to a vessel probability map as well as appearance and semantic similarities in the original CT volume.