Point of the matter is that there is more Islamic heritage in Hijaz itself than all other places to this day. Nothing of religious significance has even been destroyed. Of course you have some people who consider 200-300 year old mausoleums and elaborate shrines built for important Islamic figures as "sacred" despite them attracting people who were doing obvious non-Islamic rituals. Or that sand castle built 200 years ago which hindered any expansion work which is absolutely crucial.
Damn, if only certain people had thought like this when the Babri Masjid was destroyed. A shrine built as a tribute to an invader over one of the holiest spots in hinduism.