Automatic medical image analysis (e.g., medical image classification) is widely used in the early diagnosis of various diseases. The computer-aided diagnosis (CAD) systems enable accurate disease detection and treatment. Nowadays, deep learning (DL)-based CAD systems have been able to achieve promising results in most of the healthcare applications. Also, uncertainty quantification in the existing DL methods have not gained enough attention in the field of medical research. To fill this gap, we propose a novel, simple and effective fusion model with uncertainty-aware module for medical image classification called Binary Residual Feature fusion (BARF). To deal with uncertainty, we applied the Monte Carlo (MC) dropout during inference to obtain the mean and standard deviation of the predictions. The proposed model has two main strategies: direct and cross validated using four different medical image datasets. Our experimental results demonstrate that the proposed model is efficient for medical image classification in real clinical settings.