The Reynolds-averaged Navier-Stokes (RANS)-based method is a practical tool to provide rapid assessment of jet noise-reduction concepts. However, the RANS-based method requires modeling assumptions to represent noise generation and propagation, which often reduces the predictive accuracy due to the model-form uncertainties. In this work, the ensemble Kalman filter-based acoustic inversion method is introduced to reduce uncertainties in the turbulent kinetic energy and dissipation rate based on the far-field noise and the axial centerline velocity data. The results show that jet noise data are more effective from which to infer turbulent kinetic energy and dissipation rate compared to velocity data. Moreover, the inferred noise source is able to improve the estimation of the turbulent flowfield and the far-field noise at unobserved locations. Further, the noise model parameters are also considered uncertain quantities, demonstrating the ability of the proposed framework to reduce uncertainties in both the RANS and noise models. Finally, one realistic case with experimental data is investigated to show the practicality of the proposed framework. The method opens up the possibility for the inverse modeling of jet noise sources by incorporating far-field noise data that are relatively straightforward to be measured compared to the velocity field.