There are 2 main theories regarding the Brahuis.
1) ancient Elamo-Dravidians of the IVC
2) migrants from central India
Over the centuries, due to their location, the Brahui have mixed with Iranians, Sindhis etc. Physically and culturally , they resemble their neighbors, rather than the Dravidians of South India.
Sinhalas were ex Tamils before they conversion to Buddhism, similar to , Keralites , who were ex Tamils 800 yrs ago. Both lost their Tamilness due to foreign religion infusion , Buddhism into Sinhala Lanka and Aryan Brahmanism into Kerala . This shows religion is the key factor in replacing a language as it happened in India after Aryanisation, with Sanskrit replacing the old Dravidian languages