Class-Aware Image Search for Interpretable Cancer Identification

In recent times, the performance of computer-aided diagnosis systems in classification of malignancies has significantly improved. Search and retrieval methods are specifically important as they assist physicians in making the right diagnosis in medical imaging owing to their ability of obtaining similar cases for a query image. Supervised classification algorithms are generally more accurate than unsupervised search-based classifications; however, the latter may more easily provide insights into the decision-making process by providing a group of similar cases and their corresponding metadata (i.e., diagnostic reports) and not simply a class probability. In this study, we propose a class-aware search operating on deep image embeddings to increase the accuracy of content-based search. We validate our methodology using two different publicly available datasets, one containing endometrial cancer images and the other containing colorectal cancer images. The proposed class-aware scenarios can enhance the accuracy of the search-based classifier, thereby making them more feasible in practice. With search results providing access to the metadata of retrieved cases (i.e., pathology reports of evidently diagnosed cases), such a combination has clear benefits for assisting experts with explainable results.