Abraham first appears in Judaism then bible then in Quran. As per Judaism and biblical accounts Abraham did not travel to Arab peninsula or construct any Kaaba.
What is undisputed though that before Islam, mecca was a pagan temple.
For historical facts I would refer Judaism or Christianity as those predate Islam and will not give any twist to Kaaba related facts.
Prophet Ibrahim (PBUH) was the same person that is mentioned in Judaism, Bible and Holy Quran. It is called mentioned in the Abrahamaic faiths [Holy Scriptures], not appeared.
That's why most of the stories from Judaism, Bible to Holy Quran are persistent, hence acknowledged, and historically documented as well.
There was no civilization in the barren land with no water called Makka before Prophet Ibrahim and his sons Ishmail and Isaac (PBUTA). That is historically documented.
Prophet Ibrahim's son Ishmail (PBUH) emigrated to what is known as Makkah, the Arab civilization. His second son, Isaac (PBUH) emigrated to Egypt what is known as Hebrew generations who converted to Judaism. Without Prophet Ibrahim (PBUH), there was no civilization in Makkah which was barren land as historically documented attested, hence no point was conquering empty and barren land called Makkah.
Judaism and Bible have divided into the sects that had mentioned the names of Prophet Muhammad (PBUH) as Final Messenger for the whole mankind. Nonetheless, their knowledge on Prophet Ishmail (PBUH) and Makkah is pretty limited, hence Holy Quran and Authentic Hadith are the best option to gain more information on Prophet Ishmail (PBUH) and his family lineage, and from family lineage, Prophet Muhammad (PBUH) was born.
Even in neutral source, there was no civilization in the barren land with no water before Prophet Ibrahim (PBUH). This is well documented. Their sons emigrated, hence the new beginning of Judaism/Christianity and Islam. Everything is tied together links to Prophet Ibrahim and his sons (PBUTA). Again, this is historically documented in the neutral sources. You cannot argue against that.
Even Mohammed started with Jerusalem being qibla of Muslims but later made mecca as qibla. Mecca might just have been place of tribal pagan temples and to give it significance story of Abram etc might have been attached.
The direction of Qibla was Jerusalem by default, but later changed it to Makkah after the conquest, no doubt. But you are ignoring the origin of Makkah.
There was no civilization in the barren land with no water called Makkah. Prophet Ibrahim and his son Ishmail (PBUTA) happened which resulted the miracle of water stream called Zam Zam and the construction of Makkah with their bare hands.
The question is that were they polytheism or Monotheism? In neutral source as historically documented, they were Monotheism worshipers who erected Makkah in the barren land with no sign of civilization. Then the civilization happened. Then, that's how Makkah was hijacked with third-party religions like idols,...etc. Then, Prophet Muhammad (PBUH) from the family lineage of Prophet Ibrahim and his son Ishmail (PBUTA) through the Ishmailites tribes that endorsed Monotheism practices for years revived Makkah to the original Monotheism practice.
The bottom line is that Makkah was always family business, and always will be. Prophet Ibrahim and his son Ishmail (PBUTA) built Monotheism Makkah with their bare hands, and Prophet Muhammad (PBUH) from the family lineage of Prophet Ibrahim and his son (PBUTA) revived Makkah [filled with idols that came later] to Monotheism Makkah by cleansing the filth of polytheism.
You cannot ignore the origin of Makkah which is well-known and historically documented in neutral sources. And to be honest, Monotheism Makkah is family enterprise from Prophet Ibrahim, and his son Ishmail to Muhammad (PBUTA).