Abstract: By designing curvilinear fiber trajectory, composite rotary shell with variable stiffness (VS) has more excellent buckling resistance than straight-fiber shell with constant stiffness (CS). This work studies the influence of ply patterns and geometrical parameters on buckling performance of composite rotary shell under combined loads. Firstly, directional arc of the cross section of rotary shell was used to describe the linear variation of fiber angle, and based on this, a parametric finite element model of VS rotary shell was established. Secondly, for the purpose of achieving the maximum buckling capacity, the fiber trajectory of the composite rotary shell was optimized by combining the sequence quadratic response surface method (SRSM) with the rotary shell model. Finally, effects of ply patterns and geometrical parameters on both the cylindrical shells with bending-torsion load and the elliptical cylindrical shells with axial compression-bending-torque load were studied. It is found that the buckling capacity of VS composite cylindrical shell increases with the increase of the proportion of bending moment, and this is better than the quasi-isotropic (QI) cylindrical shells. However, when the torsional moment is the dominant factor, the buckling performance of the optimized CS cylindrical shell shows more advantages. The buckling performance of the elliptical VS cylindrical shell with smaller ratio of radius is obviously better than that of the QI cylindrical shell under different loads. The closer the cross-section is to a circle, the weaker the improvement of the buckling performance of the elliptical cylindrical shell by the curvilinear fiber.