Ocular disease is an eye disease that reduces the eye’s ability to work normally. Early ocular disease detection is important to avoid blindness caused by some of the diseases like cataracts, glaucoma, diabetes, age-related macular degeneration (AMD), etc. Artificial intelligence (AI) techniques have been used to build systems for the speedy diagnosis of such diseases. In recent years, the deep neural network (DNN) has shown remarkable success in this area. But the black box nature of such systems has created questions on the use of DNN in a high-risk system like health care. Explainable AI (XAI) is a suite of methods and techniques that provides explanations of predictions made by AI systems. This helps to achieve accountability, transparency, and debugging of the model in the healthcare domain. In this paper, we have proposed to develop an ocular disease classification model and an XAI method that can be used to explain the classification of eye diseases, from eye fundus images.