A rock failure criterion is very important for prediction of the ultimate strength in rock mechanics and geotechnics; it is determined for rock mechanics studies in mining, civil, and oil wellborn drilling operations. Also shales are among the most difficult to treat formations. Therefore, in this research work, using the artificial neural network (ANN), a model was built to predict the ultimate strength of shale, and comparison was made with support vector machine (SVM), multiple linear regression models, and the widely used conventional polyaxial failure criteria in the stability analysis of rock structures, Drucker-Prager, and Mogi-Coulomb. For building the model, the corresponding results of triaxial and polyaxial tests have been performed on shales by various researchers. They were collected from reliable published articles. The results obtained showed that a feed forward back propagation multi-layer perceptron (MLP) was used and trained using the Levenberg–Marquardt algorithm, and the 2-4-1 architecture with root-mean-square-error (RMSE) of 24.41 exhibits a better performance in predicting the ultimate strength of shale in comparison with the investigated models. Also for further validation, triaxial tests were performed on the deep shale specimens. They were prepared from the Ramshire oilfield in SW Iran. The results obtained were compared with ANN, SVM, multiple linear regression models, and the conventional failure criterion prediction. They showed that the ANN model predicted ultimate strength with a minimum error and RMSE being equal to 43.81. Then the model was used for prediction of the threshold broken pressure shale layer in the Gachsaran oilfield in Iran. For this, a vertical and horizontal stress was calculated based on a depth of shale layer. The threshold broken pressure was calculated for the beginning and ending of a shale layer to be 154.21 and 167.98 Mpa, respectively.