Plants show visible symptoms of getting infected with a disease. Presently an experienced plant pathologist can diagnose the condition through visual inspection of disease-affected plants. However, manual visualization is time-consuming and depends on the plant pathologist’s expertise in identifying plant disease. Hence this problem can be solved by a computer-aided diagnostic system with artificial intelligence (CADS-AI). This system will aid in improving and protecting the yield of the plant, but it lacks trust as the existing system is not flawless. Hence, in this research work, a plant disease classification with an explainable AI pipeline is developed which ensures trust in the CADS solution. Furthermore, an expert recommendation system will act as an alternative to expert plant pathologists. Tomato leaf diseases data from the PlantVillage dataset is used in the proposed solution. Transfer learning technique was adopted in training deep neural network models with original and augmented data of 16,684 and 53,476 images respectively. The best model for the dataset was efficientNet B5 with best F1 score accuracy of 0.9842 and 0.9930. The predicted output of B5 was interpreted with explainable AI techniques and validated using YOLOv4. Inference of the proposed solution was a client-server interface where end-users can upload infected leaf images via mobile phones or web browsers. This entire system was tested in real-time with 250 volunteers with 4G mobile network or 100 MBPS wifi. The average throughput time of the system is around 4.3 seconds.