Logically, there are stones between you and mecca. And logically, you bow down before stone,though you might think of bowing only to your god. Btw, I'm not hindu, I'm irreligious. And I always look for logic.
The kaaba made of stone in mecca is not a stone? Also the stone embedded is also not a stone? Also, when muslims do namaz, there will be lakhs of stones between your head and the kaaba. So geographically and logically, you are bowing to the stone four times more than idolators...