Semantic based medical visual question answering with explainable artificial intelligence

Abstract

The medical visual question answering (MVQA) system takes the advantage of both computer vision (CV) and natural language processing (NLP) to accept the medical image and corresponding question as input and generates the respective answer as output. One step further, the MVQA system capable of generating the answer based on the semantics has a distinct place and hence semantic based medical visual question answering (SMVQA) system is proposed in this research. In SMVQA, the semantics for input image and question are generated using layerwise relevance propagation explainable artificial intelligence (LRP XAI) technique and the answer is derived using deductive reasoning method. For this, seven MVQA datasets are used for model creation, testing and validation. The training phase of the SMVQA system is implemented using VGGNet, long short-term memory (LSTM), LRP XAI, ResNet and bidirectional encoder representations from transformers (BERT) to generate a model file. Then the inference is derived in the testing phase based on the generated model file for the test set. Finally, the answer is derived from the inference using natural language toolkit (NLTK) library, term frequency-inverse document frequency (TF-IDF), cosine similarity, best match25 (BM25) techniques along with deductive reasoning. As a result, the proposed SMVQA system gives improved performance then the existing MVQA system especially for abnormality type samples.