Many machine learning softwares are available which help the researchers to accomplish various tasks. These software packages have various conventional algorithms which perform well if the training and test data are independent and identically distributed. However, this might not be the case in the real world. The training data may not be available at one time. In the case of neural networks, the architecture has to be retrained with new data that are made available subsequently. In this paper, we present a novel training algorithm which can avoid complete retraining of any neural network architecture meant for visual pattern recognition. To show the utility of the algorithm, we have investigated the performance of convolutional neural network (CNN) architecture for a face recognition task under transfer learning. The proposed training algorithm may be used for enhancing the utility of machine learning software by providing researchers with an approach that can reduce the training time under transfer learning.