Abstract: To improve the mechanical properties of steel fiber reinforced ultra-high performance concrete (UHPFRC), a new composite modification method of steel fiber surfaces was proposed using silane coupling agent KH550 and nano-SiO₂ particles. Four parameters, including the mass ratio of ethanol to water (W_e:W_d), the content of KH550 (wt%), the content of nano-SiO₂ (wt%), and the water bath temperature (T_wb), were considered for the orthogonal design of proportions (L₉(3⁴)). Firstly, four mixtures with good solution stability were selected for fiber surface modification. The FTIR and SEM were then used to analyze the composition and morphology of the coatings. Finally, the optimal modification process parameters were determined based on the bending and compressive strength of UHPFRC specimens at 28 days. The results show that: (1) the ratio of W_e:W_d is the main factor affecting the stability of the solution, and the optimal modification parameters are: W_e:W_d = 3, KH550 (wt%) = 10%, Nano-SiO₂ (wt%) = 0.5%, and T_wb = 80℃ ; (2) the FTIR results exhibit characteristic peaks of Fe-O-Si, indicating successful bonding of KH550 and Nano-SiO₂ on the steel fiber surfaces; (3) the SEM results indicate uniform distributions of composite coatings without aggregation of Nano-SiO₂ particles on the fiber surfaces modified by the optimal parameters; (4) for the fiber volume fractions of 1%, 1.5%, and 2%, the bending strength (28MPa, 30.5MPa and 37MPa) and compressive strength (133.3MPa, 151.7MPa and 163.9MPa) of UHPFRC specimens with optimally-modified fibers are increased by 40.4%, 28.5%, 32.7%, and 7.5%, 8.3%, 13%, respectively, compared to the specimens with untreated fibers; (5) the composite modification technique significantly improves the fiber-matrix interfaces, as demonstrated by more tortuous and complex crack morphologies in the specimens with fiber volume fractions of 1.5% and 2%.