That thinking is error filled.
There was no oil in Arabia in those times - why would the Barbarians invade desolate places like the deserts?
No Invaders has ventured deep in Arabian desert mate. Mongols only invaded the large and rich cities like Baghdad. Most sensible invaders wouldn't waste their men and wealth on regions that carry no economic or strategic benefit to them.