Deep learning in computer vision has shown remarkable success in the performance of detection systems for plant diseases. However, due to the complexity and deeply nested structure of these models, these are still considered as black-box and explanations are not intuitive for human users. Many researchers have developed deep neural architectures for plant disease detection but have not provided classification explanations. To be used in practical applications, our model needs to explain why the model classified a given image. Explainable Artificial Intelligence (XAI) provides algorithms that can generate human-understandable explanations of AI decisions. In this paper, we summarize recent developments in XAI techniques, develop a plant disease detection system, and most importantly an explainable AI method named Gradient-weighted Class Activation Mapping ++ (GradCAM++) is used to locate the disease and highlight the most important regions on the leaves contributing towards the classification.