if i remember correctly, FBR detected some 1 million persons who visit abroad once or twice a year and have assets more then 10 millions and 'don't pay taxes". but that data was destroyed by some fire i guess. anyone here with the knowledge can add more.
Point is, that's why we only have 1.2 million tax payers. because other's just turn things into ashes.
the fire was very convenient