It's none of anybody's business; neither your nor mine. Fact is, that people want a temple to be built in Ayodhya and people, hence, should get a temple in Ayodhya.
An act of grave injustice was committed by an army of looters and pillagers centuries ago.
It won't be entirely misplaced if that act is reversed now, peacefully, though, I must emphasize.
As per your assertion that people do not want the temple there. You are plain wrong.
Most people I have met in my life want a temple at the site.
It's only that their desire to have a temple in Ayodhya is very low down their priority list.
That doesn't imply that the desire doesn't exist.
BJP, once upon a time, claimed that it would build the temple there and people voted for it in huge numbers.
That's a testimony to the fact that the people do want the temple to be bult there.
It is another issue that BJP turned out to be a non-performer as far as temple issue is concerned.
However, this is also true that even today is people see a sincere effort, be it in the form of a political party, to build the temple in Ayodhya, they will who-heartedly support it.
The desire is there.
Only the instrument to execute that desire is missing.