We can even wear silk in small quantities, if it is embroidery on a garment, etc. Hazrat Umar RA used to do so.
People do not realize Islamic restrictions of dress are very specific, and not general. Even about the location of awrah in men differs depending on mazhab or scholars, whether knee is included or not.
Hazrat Abu Bakr RA used to wear long garments below his ankles, Rasullulah saws told him the rule of keeping garments above the ankle is to dissuade arrogance (which Arab nobility displayed,) and hence did not apply to him.