Current climate change (e.g., temperature and precipitation variations) profoundly influences terrestrial vegetation growth and production, ecosystem respiration, and nutrient circulation. Grasslands are sensitive to climate change, and the carbon sequestration ability is closely related to water availability. However, how the terrestrial water budget influences regional carbon sequestration by the grassland ecosystem is still unclear. In this study, we modified a terrestrial biogeochemical model to investigate net ecosystem productivity (NEP) of Chinese grasslands under different aridity index (AI) levels from 1982 to 2008. The results showed that Chinese grasslands acted as a carbon sink of 33.7 TgC. yr− 1, with a clear decrease in the spatial distribution from the humid end (near-forest) to the arid end (near-desert). During these 27 years, gross primary productivity (GPP) and net primary productivity (NPP) significantly increased with regional warming over the entire range of the AI, but no significant tendency was found for NEP. Meanwhile, only NPP in the arid zone (AR) and the semiarid zone (SAR) were significantly correlated with mean annual precipitation (MAP), and no significant correlation was found between heterotrophic respiration (Rh) and MAP; NPP and Rh were both positively correlated with mean annual temperature (MAT) in all AI zones except for NPP in AR; no significant correlation between NEP and MAP or MAT was found. These results revealed that the grasslands with different AI levels keep different response patterns to temperature and precipitation variations. On the basis of these results, we predicted that the gap of carbon sequestration ability between humid and arid grassland will expand. The total carbon sink in Chinese grasslands will continue to fluctuate, but there is a danger that it might shrink in the future because of a combination of climatic and human factors, although CO2 fertilization and N deposition might partly mitigate this reduction.