Determination of the borehole and fracture initiation positions is the main aim of a borehole stability analysis. A wellbore trajectory optimization with the help of the mud pressure may be unreasonable since the mud pressure can only reflect the degree of difficulty for the initial damage to occur at the wellbore rather than the extent of the wellbore damage. In this work, we investigate the failure extension in different arbitrary inclination boreholes under different in-situ stress regimes. Assuming the plane strain condition, the Mohr-Coulomb, Mogi-Coulomb, and Modified Lade rock failure criteria are utilized. We present an analytical equation to determine the optimum drilling trajectory of an Iranian oilfield. In order to predict the degree of wellbore damage, the initial shear failure location, failure width, and failure depth of arbitrary wellbores are determined. Then a new model is derived to calculate the initial failure area of a directional wellbore because it is more efficient in a wellbore stability analysis. The results obtained show that in the target oilfield, the vertical and low-deviated direction is the optimum drilling path. According to the results of this work, optimization of the wellbore trajectory based on the estimated failure zone is a reasonable method if a considerable failure zone takes place around the borehole wall.