We investigate the stellar populations of passive spiral galaxies as a function of mass and environment, using integral field spectroscopy data from the Sydney-AAO Multi-object Integral field spectrograph Galaxy Survey. Our sample consists of 52 cluster passive spirals and 18 group/field passive spirals, as well as a set of S0s used as a control sample. The age and [Z/H] estimated by measuring Lick absorption line strength indices both at the center and within 1Re do not show a significant difference between the cluster and the field/group passive spirals. However, the field/group passive spirals with log(M_star/M_sun) > 10.5 show decreasing [α/Fe] along with stellar mass, which is ~0.1 dex smaller than that of the cluster passive spirals. We also compare the stellar populations of passive spirals with S0s. In the clusters, we find that passive spirals show slightly younger age and lower [α/Fe] than the S0s over the whole mass range. In the field/group, stellar populations show a similar trend between passive spirals and S0s. In particular, [α/Fe] of the field/group S0s tend to be flattening with increasing mass above log(M_star/M_sun) > 10.5, similar to the field/group passive spirals. We relate the age and [α/Fe] of passive spirals to their mean infall time in phase space; we find a positive correlation, in agreement with the prediction of numerical simulations. We discuss the environmental processes that can explain the observed trends. The results lead us to conclude that the formation of the passive spirals and their transformation into S0s may significantly depend on their environments.