A phase-field model was revised to study the abnormal growth of Goss grains during the annealing process in Fe-3%Si steels, in which the interaction between the second-phase particles and Goss grain boundaries (GBs) was considered. The results indicate that the abnormal growth of Goss grains occurs due to the different dissolvability of the particles at Goss GBs compared with the other GBs. Moreover, the degree of abnormal growth increases first and then decreases with an increasing particle content. Meanwhile, the size advantage of Goss grain can further promote the degree of abnormal growth. Two types of island grains were found according to the simulated results, which is consistent with the experimental observations. A proper GB dissolvability of particles is the key factor for the formation of isolated island grains, and a higher local particle density at GBs is the main reason for the appearance of serial island grains. These findings can provide guidance for the desired texture control in silicon steels.