An Effective Deep Learning Architecture Combination for Tissue Microarray Spots Classification of H&E Stained Colorectal Images

Tissue microarray (TMA) assessment of histomorphological biomarkers contributes to more accurate prediction of outcome of patients with colorectal cancer (CRC), a common disease. Unfortunately, a typical problem with the use of TMAs is that the material contained in each TMA spot changes as the TMA block is cut repeatedly. A re-classification of the content within each spot would be necessary for accurate biomarker evaluation. The major challenges to this end however lie in the high heterogeneity of TMA quality and of tissue characterization in structure, size, appearance and tissue type. In this work, we propose an end-to-end framework using deep learning for TMA spot classification into three classes: tumor, normal epithelium and other tissue types. It includes a detection of TMA spots in an image, an extraction of overlapping tiles from each TMA spot image and a classification integrated into two effective deep learning architectures: convolutional neural network (CNN) and Capsule Network with the prior information of intended tissue type. A set of digitized H&E stained images from 410 CRC patients with clinicopathological information is used for the validation of the proposed method. We show experimentally that our approach brings state-of-the-art performances for several relevant CRC H&E tissue classification and that these results are promising for use in clinical practice.