Although what you said may or may not be true. Prophet (SAW) asked us not to adopt any practices that idolaters are doing. As far as I see, yoga is directly mentioned in Hindu vedic texts and it was practiced as part of Hinduism. It is as Hindu as anything can get. And this is just my view hence I don't do yoga. I go to gym.