The optimal allocation of factors is of great significance to the realization of high-quality economic development. Based on the panel data of 28 manufacturing sub-sectors from 2006 to 2016, this paper discusses the heterogeneous impact mechanism of specialization agglomeration and diversification agglomeration on labor mismatch. The study found that there is heterogeneity in the impact mechanism of specialization and diversification on labor mismatch. There is a non-linear relationship between the degree of specialization agglomeration and labor mismatch. When the degree of specialization agglomeration is weak, agglomeration can improve labor mismatch. With the deepening of specialization, agglomeration began to deteriorate the allocation of labor force. The optimal intensity of diversification agglomeration on labor mismatch decreases with the deepening of agglomeration degree. Industrial agglomeration mainly affects the distribution of labor force by affecting the structure of labor force. The research conclusion has reference value for improving the mismatch of labor factors in China''s manufacturing industry.