An electromagnetic field and artificial intelligence-based transmission line protection system for smart grid is proposed in this paper. Bayesian regularization neural network is used as an intelligence technique. Instead of conventional current transformers, the use of magnetic field sensors for monitoring the current in transmission lines is proposed. A mathematical model of the magnetic field due to the current in transmission line is developed by considering two positions of the magnetic field sensors. The developed model is simulated for extensive scenarios. Line to ground, line to line, double line to ground and three-phase faults are created. The fault current values are transformed into magnetic field values and given to the dyadic analysis filter bank for feature extraction process. The extracted features are given as an input to the neural network for the training. The Bayesian regularization algorithm is used as a training algorithm. The performance of the proposed system is compared with other algorithms. It shows that the proposed system with Bayesian regularization have 96.11% of accuracy with the combination of different faults. The results obtained show that the proposed system outperforms the existing approaches. © 2021 Taylor & Francis Group, LLC.