The primary method for inferring the stellar mass (M-*) of a galaxy is through spectral energy distribution (SED) modeling. However, the technique rests on assumptions such as the galaxy star formation history (SFH) and dust attenuation law that can severely impact the accuracy of derived physical properties from SED modeling. Here we examine the effect that the assumed SFH has on the stellar properties inferred from SED fitting by ground-truthing them against mock observations of high-resolution cosmological hydrodynamic galaxy formation simulations. Classically, SFHs are modeled with simplified parameterized functional forms, but these forms are unlikely to capture the true diversity of galaxy SFHs and may impose systematic biases with underreported uncertainties on results. We demonstrate that flexible nonparametric SFHs outperform traditional parametric forms in capturing variations in galaxy SFHs and, as a result, lead to significantly improved stellar masses in SED fitting. We find a decrease in the average bias of 0.4 dex with a delayed-tau model to a bias under 0.1 dex for the nonparametric model, though this is heavily dependent on the choice of prior for the nonparametric model. Similarly, using nonparametric SFHs in SED fitting results in increased accuracy in recovered galaxy star formation rates and stellar ages.