Paddy soils in some areas of southern China are contaminated by arsenic (As) and cadmium (Cd), threatening human health via the consumption of As- and/or Cd-tainted rice. To date, a quantitative understanding of how soil characteristics control As and Cd accumulation in rice grains under field conditions is still deficient. Based on 31 paired soil-grain samples collected in southern China, we statistically explored which soil parameter or parameter combination from various soil analyses best estimates As and Cd in rice. We found that CaCl2 extraction of field-moist soil collected at rice harvest provided the best estimation (R2adj = 0.47–0.60) for grain Cd followed by dry soil CaCl2 extraction (R2adj = 0.38–0.49), where CaCl2 extractable Cd from moist or dry soil was the dominant soil parameter. Compared to soil totals, parameters from neither dry soil ascorbate-citrate extraction nor anoxic soil incubation improved model performance for grain As (R2adj ≤ 0.44), despite their closer relevance to soil redox conditions during plant As uptake. A key role of soil-available sulfur in controlling grain As was suggested by our models. Our approach and results may help develop potential soil amendment strategies for decreasing As and/or Cd accumulation from soils.