In geotechnical engineering, stress parameter monitoring is constrained by complex environments, posing challenges to traditional single-point monitoring techniques. To enhance dynamic monitoring of geotechnical strain and shear deformation, this study proposes a sensor based on improved fiber Bragg grating. The research first models and demodulates the sensors, and optimizes the monitoring system. Subsequently, the study combines the dual-grating compensation method to eliminate temperature interference and optimizes the strain offset step size through an improved convolutional neural network. In the calibration experiment, the sensor demonstrated a fitting coefficient of 0.998 between wavelength shift and temperature, indicating excellent stability. During the loading experiment, the sensitivities of Sensor 3 and Sensor 4 were 12.472nm/mm and 12.468nm/mm, respectively, meeting the precision requirements. Additionally, in geotechnical dynamic strain monitoring, the research sensor achieved a monitoring error controlled within 7.8µε, outperforming traditional sensors. In shear deformation monitoring, the optimized sensor had a maximum monitoring error of 0.47mm, which was superior to traditional sensors. The improved fiber Bragg grating sensor has excellent application effects in the field of geotechnical detection. This study will provide technical support for structural deformation and environmental monitoring in geotechnical engineering.